Does anyone have any experience for how the generalized LLM's like llama/gpt4 compare to much smaller, purpose built summarizers like some of the FLAN variants?
Thanks, my initial response to summarization as killer feature: "is it though?" but the "historic" nature of this post makes a bit more sense now. Unlike crypto, there are many diverse use cases for LLM. While summarization is useful I don't think it's killer.
Don’t forget classification. Even the zero shot classification can be quite decent, but put a solid prompt together, or fine tune a set of examples and it’s way easier with an LLM than previous methods in my experience.
I think it's one of a small set of killer related use cases:
- Summarization
- Data and fact extraction, which is strongly related to summarization. Find all factual assertions in this news article. Extract the names of every person and location mentioned.
Aside from these summarization-adjacent applications, the thing I find most exciting is LLMs incredible aptitude at generating code.
Code is especially interesting because it can be automatically executed (in a sandbox), which means many hallucinations can be automatically detected and fixed!
I never coded but I know what's needed and how things should be arranged to work.
Result: I've developed and deployed a fully fledged web service both for human and machine use. Backend, front-end, endpoints, db, integration with our company systems, authentication, testing and deploy.
In 5 sessions of 4 hours each. It was incredible while it was happening. I pay openai.
> This Connection is Untrusted
Go Back
The owner of costantini.pw has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.
I tried to see what it built for you, and this was the only url in your bio.
That's a personal profile page which has an expired certificate, sorry for that.
If you're (or anyone really) interested in what I did and would like to try it out for curiosity and feedback, please feel free to reach me at j@costantini.pw as I'd be pleased to share it. Thanks
The halting problem is one matter (and LLMs will often write terrible code performance wise, though you can often determine this automatically with a myriad of approaches and either fix it or discard it and start anew) but in terms of "will it run" this is more akin to the old "will it blend" youtube channel than any complicated mathematical problems which require Church or Turing to intervene on.
This is a simple smoke test and the biggest problem I've found is that it simply writes code which probably worked in the past in its training data (or is very close to what would work) but is just so ever so slightly off.
This was just a front page article today because flask3.0 broke a login module, so it's still going to be a problem no matter how recent your training data is.
You don't even need a python interpreter fired up for the first checks, in fact, you can generate the tree of valid libs, functions, their sigs, etc within your interpreter, save it as a json and and directly parse incoming python markup to evaluate if it is valid or not (at least, for your particular environment)
I'm actually a bit surprised that openAI themselves doesn't do this since they have a fixed sandbox and it would save them compute cycles to handle the "little fix" directly with either non-LLM code or a much smaller "LLM call" to figure out how to call a function for lib X version Y which is similar to this.
Or you could probably just do some non-insignificant double digit percentage of the time with Levenshtein distance (or similar fuzzy matching) and having an internal db of every libraries functions and function signatures for every version and automatically identifying where something changed and reducing your problem space to those.
The Halting problem is only relevant if you want to be completely correct. If you also reject code that looks weird, then you have bypassed the problem entirely.
Kind of .. depending on how sensitive you are to accuracy.
Also also, we've had powerful summarization techniques for decades now (deep-learning, or otherwise) - and they aren't susceptible to the hallucinations of present day Generative LLMs.
I found an 'old' (2016) reddit thread describing how auto tldr works, which doesn't suffer from hallucination (because it's only a statistical model that outputs original sentences from the text):
>we've had powerful summarization techniques for decades now (deep-learning, or otherwise) - and they aren't susceptible to the hallucinations of present day Generative LLMs.
do you mean BERT, Seq2Seq and similar?
My biggest takeaway from summarization tasks was that the benchmarks/evals seemed rather.. lacking.
Like many use cases for LLMs, this seems like it would benefit a lazy student trying to avoid work. Rolling in some statements that are related but not mentioned in the input isn't a bad thing when you're studying for a test.
However, for most use cases, falsely attributing statements is a terrible failure case for a summarizing tool.
3. Transform (translation, style transfer, switch modality between text/image/audio, etc.)
All of the major consumer use cases of LLMs at this point boil down to one of these three core operations. To the extent that LLMs allow you to perform "magic", that magic is comprised by a chain of {condense, expand, transform} operations, each of which could be performed in half an hour by a reasonably informed college graduate with access to Google.
Even if chain length maxes out at 3 or 4 for production-grade applications, that solves a frightening number of problems.
I don't understand why this got downvoted, but there's absolutely a huge unserved(or, rather, anti-served) and somewhat-looked-down-upon market here, and it's actually seemingly more women then men when i ran some numbers.
I'm an early engineer at a startup that has nothing to do with AI. I use LLMs for 2 and 3 all the time. A lot of times we'll release a new feature and I'll write my own fairly technical summary of it, then I'll ask an LLM to either target it for a specific channel (e.g. create a version of the post targeting execs vs creating one that's focused on users) or condense it down if we're posting a summary onto a social media channel. In all cases I check the LLM's work, but I rarely have to do much.
When I led teams at a big tech, I generally had a human whose job it was to do this kind of copy refinement/editing when my teams released new features. I find the LLM to be just as good, though obviously our reach is much smaller and the consequences of a mistake are much milder given our size. This is the kind of productivity boost I think of that LLMs offer; pre-LLM we would probably contract out a copy editor for this kind of work but now we do it all in-house and don't have to deal with working with a contractor's hours.
I would argue it's translation with summarization as a close second, but not for people who spend much time on HN since their command of English is almost always good enough.
Tools like DeepL already were cutting into the field of translation/localization, but now we can have something which can basically translate ANY language to ANY language (even things like fictional languages from e.g. LoTR, although that may be very hallucinogenic due to the paltry amount of training data) rather rapidly (gpt3) or rather slowly but more accurately (gpt4) including understanding of things like metaphors and idioms.
Not to mention you can probably fix some of the choice of vocab and style just by specifying (once converted to target language) "reword in the stylings of X" or countless other multi-invocation approaches to the previously very expensive and artistic task of converting human thought from one variation to another.
The biggest problem is probably over-training on english (and being dependent on openai), but even Microsoft made a LLaMA[!] finetune just for translation which iirc was significantly better than the non-translation versions from everyone else
„However, LLMs appear to be quite good at knowing what they do and don’t know“
Really? I think LLMs have not the slightest idea what they know and you have to go through great lengths to stop them from making stuff up.
I think the "killer use case" is slightly more general: operations in linguistic space, of which summarization is one. Some other useful uses include:
- the inverse of summarization; given a bulleted list of facts, write a sensible set of paragraphs presenting those facts
- semantically searching in large documents
- make some text sound more professional / casual
- translate some text between human languages (they're not perfect at this yet, but they're surprisingly good, and I think they will be the way to do translations in the future).
- expand abbreviations given some context
The key here is that none of these depend on the language model injecting factual information into its output. All of the facts are encoded in your input - the language model just does some operation in linguistic space. If you don't give it the opportunity to give you incorrect assertions, it won't give you incorrect assertions.
Another application is to use the LLM to build an embeddings database, and then do semantic search only within that database. Don't just let the LLM talk about whatever it wants to hallucinate up, force it to use a set of known good answers while accepting natural language input and summarizing documents in the known good dataset.
everything there sounds great (though i suspect for actual human-passing linguistic register-transfer we will need something different than the bog standard GPT models because everyone has this six sense for things being written by it and, particularly in English, register is one of oldest forms of 'discrimination' in the language/culture, and certainly persists to the current day. And there's a huge competitive advantage to being able to have an LLM write things for you but make it sound like you yourself wrote it.)
For translations considering how 'mature' the field is relative to everything else I'm a little surprised we haven't seen any rigorous benchmarks on round-trip translations, especially given that you aren't even limited to natural languages. You can take text, flip it into Klingon, and back, if you so chose.
>- the inverse of summarization; given a bulleted list of facts, write a sensible set of paragraphs presenting those facts
Except this one, this sounds unintentionally evil for some reasons i can't entirely put my finger on. Perhaps the inevitable conclusion of countless GPU farms spamming terabytes of text at each other predicated upon some short bullet point lists and the fact that "those are our marching orders"
I guess Fundamentally I would like to live in a future where LLMs can only decrease the length of text written by humans, never increase it or fabricate it.
> The key here is that none of these depend on the language model injecting factual information into its output.
This feels like an under-appreciated insight. There are lots of valuable (and in some cases previously difficult, expensive, or impossible) linguistic manipulations that fall under this lower-risk end of spectrum.
"LLMs: The Summarization Superheroes of the Modern Era
LLMs, like the one you're reading from, shine brightest when tasked with summarization. They may not craft unique insights, but they excel in distilling vast amounts of information into digestible bites. As information overload continues to plague our modern age, a tool that can swiftly and effectively condense info is invaluable. Bing Chat's emergence hints at a future where LLM-driven search engines index and summarize vast content troves, from books to blogs. But it's not just about search. Inspired by a recent Hacker News discussion likening the LLM movement to the early days of the web, it's clear that these models will reshape how we handle information, from helping CEOs extract corporate insights to tidying up the sprawling mess of unstructured data in enterprises. While LLMs have diverse potential applications, summarization stands out as their killer use case, at least for now."
Just because the model is unlikely to come up with novel insights doesn’t mean it’s not a useful conversationalist. I have all sorts of ideas I like to bounce off of ChatGPT and it’s good at coming up with reasonable sanity checks in the meantime.
I really think LLMs are great for translation between programming languages. I don't trust them to write complicated code, but when I am learning a new language or framework, the fastest way for me is to write a function in X and ask it to translate it to Y. It doesn't get fancy, it almost always compiles and if it makes a mistake it's not too bad.
Related thought - sometimes I ask for a "tldr" of long articles and people shame me for being lazy or something like that. I think this is really illogical. There's nothing wrong about reading a summary before or instead of reading the full article
I find using the word "regurgitate" when talking about LLMs pretty ironic since the word itself is almost never used except by people who "regurgitate" arguments from other people.
Also, I wish people would reduce the use of such derogatory terms in arguments because it distracts from the core reasoning.
I don't think "regurgitated" is a derogatory term at all, at least in the sense that I'm using the word — "[to] repeat (information) without analyzing or comprehending it." [0]
For me, the killer user case is having it as an AI assistant. A few months ago, I've started building a machine learning model for a keyboard iOS App as a solo dev, without python or deep learning knowledge.
The model I wanted to build, needed text correction, word completion and prediction in one pass. I'm pretty confident, it was an impossible task without the help of chatGPT, first GPT-3.5 and then GPT-4.
Oh my I really hope that you are building the app of my dreams.
I’m a an awful typer on my phone and on my desktop. Autocorrect is ok but I honestly don’t understand how we have all this amazing ML in the wild but we still seem to have the same shitty autocorrect.
My dream autocorrect (powered by ML) learns how you type and how you speak. It learns what mistakes you always make (due to muscle memory) and corrects based on your past mistakes. Furthermore my dream autocorrect actually understands what you’re trying to say and corrects words based on the context of the conversation rather than how close the mistake you made is to some other word.
For example, if someone texts me to say “I’m at the restaurant” and I type with a mistake “I’ll be right theyre” the autocorrect should understand the context of the conversation and rather than correct to “they’re” which is thinks is the correct spelling for this mistake, or autocorrects to “there” because it’s looking at the wider conversation. This is a poor example but I hope you get the point. I really don’t think what I’m asking for is hard. Apple should/could easily improve autocorrect
I would love to build that keyboard, but unfortunately there is a RAM limit for keyboard apps. My current model is not so smart as I would like. For fun, I just tested your example, using only 8 tokens (current limitation): "restaurant I’ll be right theyre" and it's corrected to "restaurant I’ll be right there", there is hope :) I'm pretty sure that I can expand it a few more tokens without much impact in memory and latency.
the killer use case for me is quickly finding specific commands involving parts of tools that i don't use often. how do i turn the last three commits into a patch using magit? how do i grep for this weird pattern and exclude this directory? give me an elisp snippet that runs my last "compile" command that i used without prompting me, and bind it to f5.
It's one of the things LLMs do really well. Summarization, translation, and guided blithering for generating ad copy work very well. Sometimes a bit of understanding seems to appear. Sometimes that's illusory.
Here's a video where someone tries to use GPT-4 to read a drawing and generate G-code for a milling machine.[1] He puts in a simple textbook problem drawing, and reasonable G-code comes out. The trouble is, GPT-4 plagiarized the answer from a Chegg homework assignment which has the same design of 4 holes in a square. This is clear because the problem didn't specify the material thickness, and the GPT-4 result used the material thickness from the Chegg homework assignment.
For harder, but still easy drawings, GPT-4 does worse. It's clear there's no underlying geometric model.
Is there any work being done on summarization based on loading content in and doing some sort of manipulation on the internal representation? It seems to me that this ought to be a better solution for summarization than simply asking the model to summarize and hoping its training set taught it how to do what you want. Seems like it would generally fix the potential confabulation issues as well w.r.t. summarization at least.
LLM summaries are in the sweet spot: more specific than typical website short glossaries, easier to read quickly than Wikipedia, and well-prioritized for industry professionals.
LLM summaries are providing excellent results (IMHO) for team training for technology groups, including for UI/UX, product management, innovation partnerships, software programming, and the like.
I'm not familiar with the summarization or NLP space really but I remember ~2011-2015 I signed up for a couple of daily email services that summarized a number of news articles and the summaries were fantastic. I don't even remember what they were called, they eventually sold out with ads and worse formatting/summaries to make money I guess. I often use them as an example of 1) why LLMs are a bit old news for the summary use case and 2) how various LLM use cases will probably also be ruined because for a lot of people tools like that seem novel and useful but all I can see is onboarding to more advertising.
So to someone who is actually knowledgeable in this space, are LLMs really that much better than whatever we had 10 years ago? Is this tech the key to some features we truly didn't have before?
Not an NLP expert but the biggest difference in my experience is guided focus, so to speak. When summarizing something huge like the US Code, for example, you can tell the LLM to focus on specific topics and anything adjacent to them so that it ignores irrelevant details (which is usually >99.9% of the text in my use case). The word relationships encoded in the LLM are really good at identifying important adjacent topics and entities.
LLMs are also really good at the harder NLP problems like coreference resolution, dependency parsing, and relations which makes a huge difference when using recursive summarization on complex documents where something like "the Commisioner" might be defined at the beginning and used throughout a 100,000 token document. When instructed, the LLM can track the definitions in memory itself and even modify it live by calling OpenAI functions.
I use OpenAI function calling most of the time I use the OpenAI API since that's the easiest way to get structured data and implement retry logic.
The simplest implementation is "retrieve_definition(word_to_lookup, word_to_replace)" with some number of tokens at the beginning of the prompt dedicated to definitions. You can use a separate LLM call with a long list of words (without their definitions) to do the actual selection since sometimes there might be ambiguity, which the LLM can usually figure out itself (it can also include both definitions when it's too uncertain if instructed).
A more complex variant does multiple passes: first pass identifies ambiguous words in each chunk, second pass identifies their definitions, third pass does actual summarization using the output of the previous passes to craft the prompt.
I recently went to a talk by Don Syme, the creator of F#. He was talking about the LLM tools he is working on at GitHub Next for summarising PR and code base content. He seemed pretty upbeat about the impact these tools will have.
66 comments
[ 3.8 ms ] story [ 124 ms ] thread(Relevant here because even five months is a long time in LLMs!)
Update: turns out there's a date on https://www.sebastianmellen.com/post/ - this post is from 18th March, so seven months ago.
But if the article is longer than the context window, you can try either RWKV or Claud-200k
Claude for up to 100,000 tokens.
I think it's one of a small set of killer related use cases:
- Summarization
- Data and fact extraction, which is strongly related to summarization. Find all factual assertions in this news article. Extract the names of every person and location mentioned.
- Question answering: given this context, answer this question. That's key to implementing Retrieval Augmented Generation, eg https://til.simonwillison.net/llms/embed-paragraphs#user-con...
Aside from these summarization-adjacent applications, the thing I find most exciting is LLMs incredible aptitude at generating code.
Code is especially interesting because it can be automatically executed (in a sandbox), which means many hallucinations can be automatically detected and fixed!
I tried to see what it built for you, and this was the only url in your bio.
how? you just set off my halting-problem neuroses with that claim, since how do you know what inputs to test with, in your idea?
It's quite something to watch. It writes code, then executes it, gets an error, adjusts the code to fix the error, runs it again...
I've seen it loop like that 4-5 times before it gets to code that works.
This is a simple smoke test and the biggest problem I've found is that it simply writes code which probably worked in the past in its training data (or is very close to what would work) but is just so ever so slightly off.
This was just a front page article today because flask3.0 broke a login module, so it's still going to be a problem no matter how recent your training data is.
You don't even need a python interpreter fired up for the first checks, in fact, you can generate the tree of valid libs, functions, their sigs, etc within your interpreter, save it as a json and and directly parse incoming python markup to evaluate if it is valid or not (at least, for your particular environment)
I'm actually a bit surprised that openAI themselves doesn't do this since they have a fixed sandbox and it would save them compute cycles to handle the "little fix" directly with either non-LLM code or a much smaller "LLM call" to figure out how to call a function for lib X version Y which is similar to this.
Or you could probably just do some non-insignificant double digit percentage of the time with Levenshtein distance (or similar fuzzy matching) and having an internal db of every libraries functions and function signatures for every version and automatically identifying where something changed and reducing your problem space to those.
But you have to nail down a definition of “looks weird”?
Also also, we've had powerful summarization techniques for decades now (deep-learning, or otherwise) - and they aren't susceptible to the hallucinations of present day Generative LLMs.
https://old.reddit.com/r/askscience/comments/4s5b5q/how_exac...
do you mean BERT, Seq2Seq and similar?
My biggest takeaway from summarization tasks was that the benchmarks/evals seemed rather.. lacking.
However, for most use cases, falsely attributing statements is a terrible failure case for a summarizing tool.
1. Condense (summarize, synthesize, extract, etc.)
2. Expand (explain, copilot, suggest, etc.)
3. Transform (translation, style transfer, switch modality between text/image/audio, etc.)
All of the major consumer use cases of LLMs at this point boil down to one of these three core operations. To the extent that LLMs allow you to perform "magic", that magic is comprised by a chain of {condense, expand, transform} operations, each of which could be performed in half an hour by a reasonably informed college graduate with access to Google.
Even if chain length maxes out at 3 or 4 for production-grade applications, that solves a frightening number of problems.
When I led teams at a big tech, I generally had a human whose job it was to do this kind of copy refinement/editing when my teams released new features. I find the LLM to be just as good, though obviously our reach is much smaller and the consequences of a mistake are much milder given our size. This is the kind of productivity boost I think of that LLMs offer; pre-LLM we would probably contract out a copy editor for this kind of work but now we do it all in-house and don't have to deal with working with a contractor's hours.
Tools like DeepL already were cutting into the field of translation/localization, but now we can have something which can basically translate ANY language to ANY language (even things like fictional languages from e.g. LoTR, although that may be very hallucinogenic due to the paltry amount of training data) rather rapidly (gpt3) or rather slowly but more accurately (gpt4) including understanding of things like metaphors and idioms.
Not to mention you can probably fix some of the choice of vocab and style just by specifying (once converted to target language) "reword in the stylings of X" or countless other multi-invocation approaches to the previously very expensive and artistic task of converting human thought from one variation to another.
The biggest problem is probably over-training on english (and being dependent on openai), but even Microsoft made a LLaMA[!] finetune just for translation which iirc was significantly better than the non-translation versions from everyone else
If you paste text into ChatGPT and then ask about specific information within the text, it's pretty good at answering.
- the inverse of summarization; given a bulleted list of facts, write a sensible set of paragraphs presenting those facts
- semantically searching in large documents
- make some text sound more professional / casual
- translate some text between human languages (they're not perfect at this yet, but they're surprisingly good, and I think they will be the way to do translations in the future).
- expand abbreviations given some context
The key here is that none of these depend on the language model injecting factual information into its output. All of the facts are encoded in your input - the language model just does some operation in linguistic space. If you don't give it the opportunity to give you incorrect assertions, it won't give you incorrect assertions.
For translations considering how 'mature' the field is relative to everything else I'm a little surprised we haven't seen any rigorous benchmarks on round-trip translations, especially given that you aren't even limited to natural languages. You can take text, flip it into Klingon, and back, if you so chose.
>- the inverse of summarization; given a bulleted list of facts, write a sensible set of paragraphs presenting those facts
Except this one, this sounds unintentionally evil for some reasons i can't entirely put my finger on. Perhaps the inevitable conclusion of countless GPU farms spamming terabytes of text at each other predicated upon some short bullet point lists and the fact that "those are our marching orders"
I guess Fundamentally I would like to live in a future where LLMs can only decrease the length of text written by humans, never increase it or fabricate it.
This feels like an under-appreciated insight. There are lots of valuable (and in some cases previously difficult, expensive, or impossible) linguistic manipulations that fall under this lower-risk end of spectrum.
"LLMs: The Summarization Superheroes of the Modern Era
LLMs, like the one you're reading from, shine brightest when tasked with summarization. They may not craft unique insights, but they excel in distilling vast amounts of information into digestible bites. As information overload continues to plague our modern age, a tool that can swiftly and effectively condense info is invaluable. Bing Chat's emergence hints at a future where LLM-driven search engines index and summarize vast content troves, from books to blogs. But it's not just about search. Inspired by a recent Hacker News discussion likening the LLM movement to the early days of the web, it's clear that these models will reshape how we handle information, from helping CEOs extract corporate insights to tidying up the sprawling mess of unstructured data in enterprises. While LLMs have diverse potential applications, summarization stands out as their killer use case, at least for now."
Also, I wish people would reduce the use of such derogatory terms in arguments because it distracts from the core reasoning.
[0]: https://www.google.com/search?hl=en&q=regurgitated%20definit...
The model I wanted to build, needed text correction, word completion and prediction in one pass. I'm pretty confident, it was an impossible task without the help of chatGPT, first GPT-3.5 and then GPT-4.
I’m a an awful typer on my phone and on my desktop. Autocorrect is ok but I honestly don’t understand how we have all this amazing ML in the wild but we still seem to have the same shitty autocorrect.
My dream autocorrect (powered by ML) learns how you type and how you speak. It learns what mistakes you always make (due to muscle memory) and corrects based on your past mistakes. Furthermore my dream autocorrect actually understands what you’re trying to say and corrects words based on the context of the conversation rather than how close the mistake you made is to some other word.
For example, if someone texts me to say “I’m at the restaurant” and I type with a mistake “I’ll be right theyre” the autocorrect should understand the context of the conversation and rather than correct to “they’re” which is thinks is the correct spelling for this mistake, or autocorrects to “there” because it’s looking at the wider conversation. This is a poor example but I hope you get the point. I really don’t think what I’m asking for is hard. Apple should/could easily improve autocorrect
The fact that it can output quite valid ffmpeg commands was the commandline cherry on the cake for me
Here's a video where someone tries to use GPT-4 to read a drawing and generate G-code for a milling machine.[1] He puts in a simple textbook problem drawing, and reasonable G-code comes out. The trouble is, GPT-4 plagiarized the answer from a Chegg homework assignment which has the same design of 4 holes in a square. This is clear because the problem didn't specify the material thickness, and the GPT-4 result used the material thickness from the Chegg homework assignment.
For harder, but still easy drawings, GPT-4 does worse. It's clear there's no underlying geometric model.
[1] https://scanalyst.fourmilab.ch/t/programming-a-computer-nume...
LLM summaries are providing excellent results (IMHO) for team training for technology groups, including for UI/UX, product management, innovation partnerships, software programming, and the like.
If you want to see some of these, my GitHub has a six of these LLM summary guides for tech people: https://github.com/joelparkerhenderson
So to someone who is actually knowledgeable in this space, are LLMs really that much better than whatever we had 10 years ago? Is this tech the key to some features we truly didn't have before?
LLMs are also really good at the harder NLP problems like coreference resolution, dependency parsing, and relations which makes a huge difference when using recursive summarization on complex documents where something like "the Commisioner" might be defined at the beginning and used throughout a 100,000 token document. When instructed, the LLM can track the definitions in memory itself and even modify it live by calling OpenAI functions.
The simplest implementation is "retrieve_definition(word_to_lookup, word_to_replace)" with some number of tokens at the beginning of the prompt dedicated to definitions. You can use a separate LLM call with a long list of words (without their definitions) to do the actual selection since sometimes there might be ambiguity, which the LLM can usually figure out itself (it can also include both definitions when it's too uncertain if instructed).
A more complex variant does multiple passes: first pass identifies ambiguous words in each chunk, second pass identifies their definitions, third pass does actual summarization using the output of the previous passes to craft the prompt.