This could use a bit more nuance around training from AI. While the naive approaches will experience worse replies, there are many documented cases where the quality improves instead. Star, self-reflection, groups of agents, and likely others I don't know about all improve the results using only the same model's output.
This can be replicated on a small scale by using two LLMs (which can be two instances of the same LLM). Start with a human prompt, then feed the answer of LLM-1 as the prompt to LLM-2, then feed that answer of LLM-2 to LLM-1 and so forth.
The answers soon converge to some boring, bland repetition that isn't even logical.
No intelligence here, which means that code emitting LLMs are just stealing human IP that they happened to have read.
I would reject the implicit materialistic worldview, but anyone is free to have their own opinions of course.
Aside from that, if a human creates a new work that builds on other people's work, it must have sufficient creativity and magnitude to qualify as new IP.
A human also does not connect to the Internet and distributes the IP that they have read to millions of users for subscription fee.
> A human also does not connect to the Internet and distributes the IP that they have read to millions of users for subscription fee.
Patreon, substack, reaction videos, the news?
"It's one of those irregular verbs, isn't it? I am inspired, you plagiarise, they are being prosecuted under Section 7 of the Copyright Act" - https://news.ycombinator.com/item?id=36830574
What do you think would happen if you did that with people who have no reason to talk, but are forced to give an answer each time? I'm sure at some point they'd just degrade to nonsense, but I don't think that says anything about intelligence. (Being stuck for a long time with a friend does occasionally degrade into nonsense and absurd injokes in my experience)
In a political discussion people are forced to give an answer each time and yet the discussion does not converge to utter nonsense, provided that both participants are intelligent. For the younger people here who have only watched recent political discussions, search on YouTube for discussions before 2000, preferably before 1980.
Same for a scientific discussion.
By the way, Bing Copilot absolutely aborts conversations it does not like. LLMs could also change the subject, like humans do if they detect repetition.
That's a completely different scenario than OP stated. In a political discussion you have an agenda, reasons to engage, plan for progress, constraints on topics/time, context and experience, and you know when to withdraw.
The scenario described is closer to "Do a political discussion." and being forced to continue forever.
> To train GPT-3, OpenAI needed over 650 billion English words of text – about 200x more than the entire English Wikipedia.
Since, I assume, humans don't need this much training (?) this area would seem to be ripe to explore — can you achieve similar training with a fraction of the data needed for GPT-3.
It’s an apples to oranges comparison- humans are getting massive amounts of data from their nervous system far richer than the token stream of Wikipedia. The model has to learn everything about the world solely from statistical patterns of text whereas humans are seeing, hearing and touching the world in full resolution. Humans also take an active role in seaking novel stimuli that they can learn from.
But that does I think shed light on why multi-modality is so promising; beyond additional use cases, it could make the models better at existing text tasks too when they train on all that video data.
And I think it also reinforces why everyone is so hyped for reinforcement-learning techniques like the self play that alpha zero used to come LLMs.
> humans are getting massive amounts of data from their nervous system far richer than the token stream of Wikipedia.
And their tokenizers are better - that's the primary reason why ChatGPT is so bad at math, its network literally doesn't (and can't!) comprehend numbers longer than the token size by definition.
Additionally, humans can cross-reference information with information they already know at learning time. For example, an AI "reads" some climate change denier politician's speech - how is it supposed to know that the politician is a climate change denier and downrank their opinion on climate change as a result, at least without a human having pre-classified the data?
That’s exactly what transformers do. They parse things in context. If it can parse C++ well enough to write C++ code, then it can parse thousand-separated numbers.
sure, you can get the model to understand with enough examples, but OP of this thread was saying LLMs need many many more steps than humans do. Mschuster was saying this could partially be an artifact of the tokenization - humans visually seeing 1,000,000 and 1000001 for the first time can see that they are similar based on how they look. A transformer must learn two token sequences, which may not have any tokens in common, are different by 1.* And it must do this based entirely on context and how likely one is to appear in the same place as another. That takes a massive amount of data in practice and could be one reason why humans appear to learn from much sparser data and faster.
* in gpt-2’s case, you get
1,000,000 -> 16, 11, 830, 11, 830, 198, 49
And
1000001 ->
388, 486
which means the model will see a sequence of noise embeddings entirely different when training. That means seeing 1,000,000 any number of times will not really help the model prepare for when it sees 1000001 for the first time.
You can play around with https://tiktokenizer.vercel.app/?model=gpt2 to see how tokenization of numbers has been improved by different models to try and mitigate this.
> And it must do this based entirely on context and how likely one is to appear in the same place as another. That takes a massive amount of data in practice and could be one reason why humans appear to learn from much sparser data and faster.
That, and another thing: A LLM fundamentally boils down to a statistical repetition engine. That means, if it has never seen the input - say, some random large numbers -, it will hallucinate garbage if you ask it a question ("multiply 245358943543548 by 438574589675486789654"), whereas a human would recognize where the boundaries of the numbers are and plug it into a calculator (or, some particularly big brained people would actually do it in their mind) to figure out the result.
Some AI or pseudo-AI (dunno what Wolfram Alpha uses under the hood) has to have a "front" AI to recognize such questions and redirect them to a calculator, but in my eyes that's just cheating for a very specific case and not enabling general intelligence.
While that could be an explanation I am not fully sold on that idea. For example we don't get continuous streams of information on something like abstract math but we still dona decent job with it without going through millions of math problems.
If multi modality gets us through this phase then you are right in your analysis. Let's see what come out in the next 5 years
I don’t mean to say that is _all_ that is separating us from the transformer architecture. I do think we’re more efficient than gradient descent, and sgd+attention is a function of how modern parallel processing works more than it is a function of any deep insight into the brain’s computation or learning methods.
I just think we’re really making a silly comparison when we compare a stream of text to the multi sensory experiences humans have directly, reducing what we are doing to just reading words off a screen or page.
I don't buy this explanation, yes we are getting "constant data", but most of that is completely useless like, traffic noise, looking at the wall etc, and most of it isn't novel at all.
I.e. of the GB of data I get every day, most of it is visual data of my apartment which I've already seen 10000 times and audio data of the same shit my girlfriend says every day. I'm not ingesting the whole of Wikipedia or every book written, which is far richer and more varied.
On the richness of the data- When you were a child every time you looked at any wall, how many wikipedia articles would it have taken to describe that entire experience?
The model needs to discover what is “noise” by rereading something as sparse as Wikipedia over and over. Think of it as understanding films purely from the script. Or learning what sand is and how it behaves physically from Wikipedia rather than picking it up and having it run through your fingers.
The experience of breathing and staring at a wall has an incredible amount of data that has to be learned before it is noise and is boring, and I’d argue it would take many Wikipedia articles to describe it fully.
And on the novelty function- SGD does update things based on a loss and how unexpected things were to the model. But that’s very crude compared to what humans are doing, where everything gets the same learning rate multiple, and we feed nearly random data to the model. There is a lot of evidence that if you pick your training set carefully you can train a lot faster, and I think humans are picking their own training set.
Humans also remember and update their model in real-time (with various caveats obviously). LLM’s forget when you close the window and need 1000 examples before they learn anything.
Humans no longer have the big picture. Our knowledge is being atomised because there's so much of it. You can see this in medicine: a cardiologist spends all of his time and effort on a single organ. In the 19th century it was possible for a doctor to keep up with the entire field.
That's depth-versus-breadth anxiety. Is the body of human knowledge, everything there is to know, losing deep meaning due to becoming too broad, making it impossible to grasp everything? No. Consolidation happens. Knowledge as a whole should appear deep, not broad, you're just looking at it from the wrong end.
In the model testing I've conducted, I've seen that LLMs from competing companies including GPT-4o, Gemini Flash 1.5, Llama 3.1 and Phi-3 all converge on the exact same joke. For a test of creativity this was alarming. They all tell slight variations of the same joke about ladders.
> For instance, researchers found a 16% drop in activity on the coding website StackOverflow one year after the release of ChatGPT. This suggests AI assistance may already be reducing person-to-person interactions in some online communities.
I don't get why this is a bad news. It meant that most likely these questions would have been duplicates and/or very easy, as ChatGPT could fix them.
My issue with SO is on the opposite side of the spectrum. By the time I am ready to post a question after having done my homeworks, I know it is either an unfixable bug or out of reach for most if not all users, and the probability of getting an answer is very low.
Most of the time I then answer my own questions a few months later.
Agreed - those who get the most out of SO seem to use it as a personal blogging tool - answering questions as a trigger for writing out what they know.
I have always assumed that a good marketing ruse for FOSS would be to pay an intern to basically Q&A your entire docs into SO - if you do it right it would even build a better roadmap for your project.
The article actually does explain this in the first few paragraphs.
It is an issue because communities like StackOverflow are actually one of the sources of training material. LLM's are not capable of innovation, they can only do variations based on what is in their training material. People asking questions to other people about things does allow for actual new and fresh answer, where people might come up with solutions other people have not yet.
And yes, you are right, some of these answers would be duplicates and easy to answer. But only to a certain point, there are a lot of answers on StackOverflow that are technically correct but severely outdated by this point. They have been superseeded by more modern ways of doing things or the APIs to do so have changed in newer versions of frameworks.
With a human community you will have people leaving comments on older answers pointing this out and answering with the current answers on new questions.
LLM's, again, are not capable of picking up this new information on their own. You can actually often see this in the answers you get from them, they are often ever so slightly dated already.
When the communities that feed them die they will have less learning material so at the very least they will stagnate.
> LLM's are not capable of innovation, they can only do variations based on what is in their training material
I've seen this claim a few times; what I have yet to see is a concrete definition of what constitutes "innovation" — the vagueness of what it means having been a problem I've seen even back when Dilbert occupied the cultural niche now filled by xkcd.
That example doesn't match my experience of either LLMs (I've given them my own private projects and libraries and the best LLMs are fine) or human developers (who, myself included, say or think "WTF" a lot on new codebases), and also isn't a definition.
> (I've given them my own private projects and libraries and the best LLMs are fine)
It all does depend on how much you have given them and how unique your libraries are. If you have given them a relatively simple library with a code base that uses all the API calls in that library in multiple ways, then yeah, an LLM will be able to work on that as you have given it plenty of examples to base answers on.
Give them just jsdoc/javadoc documentation, and they will give you very little in the way of useful implementations. Where a human in your own words might "WTF" a bunch but will be able to produce something.
In fact, the ability to say WTF might even be what is important here. As an LLM will not even stop to consider if things make sense.
> and also isn't a definition.
Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available.
Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".
> Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available.
A letter can part of some alphabet, but it is not itself an alphabet. A word can be part of some dictionary, but it is not itself a dictionary. An anecdote can be part of a dataset, but it is not itself a dataset.
I think the definition you now give seems to be reasonable, but I'd like to compare that with the example to illustrate the problem of vagueness here:
> Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation.
vs.
> When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available.
The example you give is the kind of thing that LLMs are really good at: translating one thing into another thing. The original use case of the transformer model was natural language translation, and they can do that well even when there's no explicit map from the input language to the output language, as they learn an intermediate representation for all.
The same applies when the languages are jargon and synthetic: "business plan", "code" and "documentation".
Is that innovation? It meets the example you gave, but does it meet the definition you gave? It depends exactly what you mean by "no *previous* examples": myself, I would say that "documentation" counts as "a previous example".
But the more I think about it, the less that actually feels like innovation. It's in-context learning, which is neat, but it doesn't seem to meet what I'd expect from real innovation — at most it is the corporate (and government) self-congratulation that gets called by the same word.
> Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".
This is a confusing point: that sounds like it's about following a new principle, whereas I would expect "innovation" to involve designing and specifying a new principle, determining its weaknesses through experiment. I believe LLMs are used as part of larger systems to do this, though I have no frame of reference for the quality, and I would accept that "LLM as component of bigger system" is different from "LLM" — a letter can part of some alphabet, but it is not itself an alphabet, etc.
I've performed a couple different experiments around this with niche programming languages.
I gave one of OAI's 128k models the entirety of official documentation on MOO[1], which is almost entirely absent in training because materials online for it are scarce. It was able to answer questions to a reasonable degree, and provide passable example code. The downside is prompting this cost $0.35 per prompt, but maybe a finetune could handle it.
I then attempted to have it walk me through writing a Greyscript[2] program, which is an interesting challenge since it carries many similarities to Lua, but is distinctly not. While I didn't have nearly the same amount of context as [1], relying on explaining it myself, it frequently hallucinated Lua functionality and had to be corrected every step of the way on this.
Would be interesting if someone found niche languages in different styles with similar levels of documentation to test with.
When people say "LLMs are not capable of innovation", what exactly do they consider as innovation? If LLMs are not capable of innovation on their own, what if we augment them with means to interact with the environment so they can obtain new training data?
e.g. The minecraft bot Voyager can explore the game environment and extend its skill library(stored as a vector database), is that considered as innovation? There are also systems like leandojo/alphaproof that discover new proofs and use LLM in non-trivial ways(not just naively predict the next token in one shot). Reinforcement learning algorithms like AlphaGo/AlphaZero use self play and use monte carlo tree search to learn to outperform humans. You can similarly use LLMs to generate actions and estimate state values(check the language agent tree search paper).
Most people use LLMs by prompting them with some additional context(chat history and data retrived from database) but there is nothing that stops us from continuously improving a LLM(either by modifying its weight or augmenting it with external database) by asking it to evaluate the task outcome/error message and feeding it back to the LLM. We can also ask it to just keep on generating new tasks to experiment with the environment/internet to get new knowledge.
> what if we augment them with means to interact with the environment so they can obtain new training data?
So far, the current generation of LLMs that are in widespread use do not have that ability as far as I am aware. To actually do it to a degree that would rival human learning, they would need access to a lot more environment than you might be thinking of.
Sure, for programming, the most basic environment would be the platform to run the code on and the output of the code. But choices in programming are made based on more things like performance, load impact, behavior in production environments, interaction with other applications, platform logging, adjacent application logging. Or even before that, using previous experience to judge specifications for an application which takes things in account like the expected user base, costs, etc.
The real world is a lot more complex than a minecraft world or a game of Go. Which is to say, I am not saying that it is impossible. I am sure research is ongoing to do exactly that.
But the LLMs that are currently already disrupting communities like StackOverflow are not doing any of that. Given how complex the task is to plug in all relevant stimuli and the fact that for now they can get by without doing all of this I think things are more likely to get worse before they potentially will get better.
> . By the time I am ready to post a question after having done my homeworks, I know it is either an unfixable bug or out of reach for most if not all users, and the probability of getting an answer is very low.
And even then it gets marked as duplicate of something unrelated.
I mean apart from wow ! 98% of the internet is shit (that’s higher than even I assumed) - how did they differentiate between good and shit? PageRank? Length? I can tell the difference between good writing and bad writing (and porn and erotica) but I have to read it using my brain based LLM - how did they do it ?
This leads to the whole open source, show us your training data thing
Surely the only way you overcome this is to write a system which can actually generate something new, rather than regurgitate an incredibly complicated statistical reworking of things it got trained on.
I did note how the author said that training models on other models doesn't have the ethical implications of training on stolen human data - except it does, because where did the first model get its training set from? This is why we make it illegal not just to steal but also to handle stolen goods.
How does doing real world experiments to test your hypothesis not help with the very common problem of being wrong?
> It's not the way science is done any way.
That's a very sweeping statement - especially since the scientific method is quite a catch-all phrase.
In the end science is driven forward by new data, gathered either intentionally or incidentally.
Self-driving cars can continue to learn because they are continually generating more data. Advertising algorithms can continue to learn and adapt as the campaigns they create are in effect experiments - and some are even well designed.
I suppose large language models can continue to learn by chatting to humans - but they are potentially only learning how to please humans, not necessarily underlying truths.
People flooding the 'net with LLM-generated crap are both eating their seed corn and poisoning the well.
Steel manufactured before 1944[0] can be incredibly valuable[1] because it hasn't been tainted by the nuclear tests/bombs fallout, and maybe fairly soon any archive internet material written pre-2010 will be considered equally valuable.
Not sure what point this is trying to make, but steel is a man-made alloy, not a naturally occurring substance (except possibly in tiny quantities by chance). The background radiation gets into it through the use of atmospheric air in the production process, so it seems rather a good analogy for background AI output entering the production process of internet "content" and rendering it less suitable for certain purposes.
Language is entirely human made , and it s affected by both communication and the medium. Every time a new mode of communication was invented, language was polluted with other people's language. AI is both a medium and communication which indirectly communicates using the "medial value content" of all other people on the internet. That content was itself already contaminated via too much communication over the past 30 years. So it's not really a fundamental shift , rather an acceleration of a process that s already happening
Not to mention pissing in the pool and tragedying the commons!
(By the way, the radioactivity in the atmosphere that was poisoning the steel has pretty much vanished by now, so don't go investing everything in battleship reclamation in 2024.)
I dunno - this seems like a problem ideally situated for adversarial networks to resolve. All you need is a smaller, dumber, validated model that can check and validate the output of the bigger, smarter model, and either use that to directly act as a filter between the user and the larger model, or use it to retrain the larger model until it isn’t dumb and wrong.
5 minutes later, someone will post about it on HN, explaining how we could all use this character to shield our private online writings from Big Tech's AI crawlers. 5 minutes after that, someone will change the defaults on nginx and Apache to output that character as the first character of every HTML web page.
Absolutely, except it will 100% go down the Do Not Track way. Nobody will care. I mean, at this very moment AI companies are training their models on data that's been obtained in a less-than-compliant way...
We don't need "more" data. We already have all the data we need in terms of quantity. We don't need more "synthetic" data for supervised learning. We need better training algorithms that go beyond minimizing token level loss.
We can speculate all day long how the current "AI" phenomenon will evolve but, alas, there isn't much solid on which to ground arguments.
In the oral phase of human communication "AI models" were residing within human brains: the origin of any stream of messages was literally in front of you, in flesh and blood.
In the print phase we have the first major decoupling. Whether in the form of a cuneiform tablet or a modern paperback, the provenance was no longer assured. We had to rely on "controlled" seals, trust the publishers and their distribution chains etc. Ultimately this worked because the relative difficulty of producing printed artifacts helped developed a legal/political apparatus to control the spread of the "fake" stuff.
Enter the digital communications era and we have the second major decoupling. The amount (and increasingly the apparent veracity) of generating human oriented messaging is no longer a limiting factor. This has no precedent. You can create now plausibly create a fake Wikipedia [1] by just running a model. The signal-to-noise of digitally exchanged messages can experience catastrophic collapse.
> A flood of synthetic content might not pose an existential threat to the progress of AI development, but it does threaten the digital public good of the (human) internet.
Indeed, this is the real risk. I don't care about idiot "AI" feeding on itself but I do care about destroying any basis of sane digital communication between humans.
Will we develop the legal/political apparatus to control the flood of algorithmic junk? Will it be remotely democratic? The best precedent of digital automation destroying communication channels and our apparent inability and/or unwillingness to do something about it is email spam [2]. Decades after it first appeared spam is still degrading our infosphere. The writing is on the wall.
64 comments
[ 2.6 ms ] story [ 108 ms ] threadThe answers soon converge to some boring, bland repetition that isn't even logical.
No intelligence here, which means that code emitting LLMs are just stealing human IP that they happened to have read.
Aside from that, if a human creates a new work that builds on other people's work, it must have sufficient creativity and magnitude to qualify as new IP.
A human also does not connect to the Internet and distributes the IP that they have read to millions of users for subscription fee.
Patreon, substack, reaction videos, the news?
"It's one of those irregular verbs, isn't it? I am inspired, you plagiarise, they are being prosecuted under Section 7 of the Copyright Act" - https://news.ycombinator.com/item?id=36830574
Same for a scientific discussion.
By the way, Bing Copilot absolutely aborts conversations it does not like. LLMs could also change the subject, like humans do if they detect repetition.
That's a completely different scenario than OP stated. In a political discussion you have an agenda, reasons to engage, plan for progress, constraints on topics/time, context and experience, and you know when to withdraw.
The scenario described is closer to "Do a political discussion." and being forced to continue forever.
Well, "stealing" is incorrect as they don't consume the original text - it's still there. "Plagiarising " would be more accurate.
Since, I assume, humans don't need this much training (?) this area would seem to be ripe to explore — can you achieve similar training with a fraction of the data needed for GPT-3.
But that does I think shed light on why multi-modality is so promising; beyond additional use cases, it could make the models better at existing text tasks too when they train on all that video data.
And I think it also reinforces why everyone is so hyped for reinforcement-learning techniques like the self play that alpha zero used to come LLMs.
And their tokenizers are better - that's the primary reason why ChatGPT is so bad at math, its network literally doesn't (and can't!) comprehend numbers longer than the token size by definition.
Additionally, humans can cross-reference information with information they already know at learning time. For example, an AI "reads" some climate change denier politician's speech - how is it supposed to know that the politician is a climate change denier and downrank their opinion on climate change as a result, at least without a human having pre-classified the data?
* in gpt-2’s case, you get 1,000,000 -> 16, 11, 830, 11, 830, 198, 49 And 1000001 -> 388, 486 which means the model will see a sequence of noise embeddings entirely different when training. That means seeing 1,000,000 any number of times will not really help the model prepare for when it sees 1000001 for the first time. You can play around with https://tiktokenizer.vercel.app/?model=gpt2 to see how tokenization of numbers has been improved by different models to try and mitigate this.
That, and another thing: A LLM fundamentally boils down to a statistical repetition engine. That means, if it has never seen the input - say, some random large numbers -, it will hallucinate garbage if you ask it a question ("multiply 245358943543548 by 438574589675486789654"), whereas a human would recognize where the boundaries of the numbers are and plug it into a calculator (or, some particularly big brained people would actually do it in their mind) to figure out the result.
Some AI or pseudo-AI (dunno what Wolfram Alpha uses under the hood) has to have a "front" AI to recognize such questions and redirect them to a calculator, but in my eyes that's just cheating for a very specific case and not enabling general intelligence.
If multi modality gets us through this phase then you are right in your analysis. Let's see what come out in the next 5 years
I just think we’re really making a silly comparison when we compare a stream of text to the multi sensory experiences humans have directly, reducing what we are doing to just reading words off a screen or page.
I.e. of the GB of data I get every day, most of it is visual data of my apartment which I've already seen 10000 times and audio data of the same shit my girlfriend says every day. I'm not ingesting the whole of Wikipedia or every book written, which is far richer and more varied.
Have you tried increasing the temperature?
The experience of breathing and staring at a wall has an incredible amount of data that has to be learned before it is noise and is boring, and I’d argue it would take many Wikipedia articles to describe it fully.
And on the novelty function- SGD does update things based on a loss and how unexpected things were to the model. But that’s very crude compared to what humans are doing, where everything gets the same learning rate multiple, and we feed nearly random data to the model. There is a lot of evidence that if you pick your training set carefully you can train a lot faster, and I think humans are picking their own training set.
I've posted about it here: https://news.ycombinator.com/item?id=41125309
I don't get why this is a bad news. It meant that most likely these questions would have been duplicates and/or very easy, as ChatGPT could fix them.
My issue with SO is on the opposite side of the spectrum. By the time I am ready to post a question after having done my homeworks, I know it is either an unfixable bug or out of reach for most if not all users, and the probability of getting an answer is very low.
Most of the time I then answer my own questions a few months later.
Thank you for doing that. This is very valuable.
I have always assumed that a good marketing ruse for FOSS would be to pay an intern to basically Q&A your entire docs into SO - if you do it right it would even build a better roadmap for your project.
It is an issue because communities like StackOverflow are actually one of the sources of training material. LLM's are not capable of innovation, they can only do variations based on what is in their training material. People asking questions to other people about things does allow for actual new and fresh answer, where people might come up with solutions other people have not yet.
And yes, you are right, some of these answers would be duplicates and easy to answer. But only to a certain point, there are a lot of answers on StackOverflow that are technically correct but severely outdated by this point. They have been superseeded by more modern ways of doing things or the APIs to do so have changed in newer versions of frameworks.
With a human community you will have people leaving comments on older answers pointing this out and answering with the current answers on new questions. LLM's, again, are not capable of picking up this new information on their own. You can actually often see this in the answers you get from them, they are often ever so slightly dated already.
When the communities that feed them die they will have less learning material so at the very least they will stagnate.
I've seen this claim a few times; what I have yet to see is a concrete definition of what constitutes "innovation" — the vagueness of what it means having been a problem I've seen even back when Dilbert occupied the cultural niche now filled by xkcd.
Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation.
It will struggle to come up with code that makes sense. Even if it does have training data on the language itself, similar frameworks, etc.
Ask the same of a human who does have the same "experience" and they will be able to come up with something reasonable.
It all does depend on how much you have given them and how unique your libraries are. If you have given them a relatively simple library with a code base that uses all the API calls in that library in multiple ways, then yeah, an LLM will be able to work on that as you have given it plenty of examples to base answers on.
Give them just jsdoc/javadoc documentation, and they will give you very little in the way of useful implementations. Where a human in your own words might "WTF" a bunch but will be able to produce something.
In fact, the ability to say WTF might even be what is important here. As an LLM will not even stop to consider if things make sense.
> and also isn't a definition.
Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available.
Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".
A letter can part of some alphabet, but it is not itself an alphabet. A word can be part of some dictionary, but it is not itself a dictionary. An anecdote can be part of a dataset, but it is not itself a dataset.
I think the definition you now give seems to be reasonable, but I'd like to compare that with the example to illustrate the problem of vagueness here:
> Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation.
vs.
> When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available.
The example you give is the kind of thing that LLMs are really good at: translating one thing into another thing. The original use case of the transformer model was natural language translation, and they can do that well even when there's no explicit map from the input language to the output language, as they learn an intermediate representation for all.
The same applies when the languages are jargon and synthetic: "business plan", "code" and "documentation".
Is that innovation? It meets the example you gave, but does it meet the definition you gave? It depends exactly what you mean by "no *previous* examples": myself, I would say that "documentation" counts as "a previous example".
But the more I think about it, the less that actually feels like innovation. It's in-context learning, which is neat, but it doesn't seem to meet what I'd expect from real innovation — at most it is the corporate (and government) self-congratulation that gets called by the same word.
> Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".
This is a confusing point: that sounds like it's about following a new principle, whereas I would expect "innovation" to involve designing and specifying a new principle, determining its weaknesses through experiment. I believe LLMs are used as part of larger systems to do this, though I have no frame of reference for the quality, and I would accept that "LLM as component of bigger system" is different from "LLM" — a letter can part of some alphabet, but it is not itself an alphabet, etc.
I gave one of OAI's 128k models the entirety of official documentation on MOO[1], which is almost entirely absent in training because materials online for it are scarce. It was able to answer questions to a reasonable degree, and provide passable example code. The downside is prompting this cost $0.35 per prompt, but maybe a finetune could handle it.
I then attempted to have it walk me through writing a Greyscript[2] program, which is an interesting challenge since it carries many similarities to Lua, but is distinctly not. While I didn't have nearly the same amount of context as [1], relying on explaining it myself, it frequently hallucinated Lua functionality and had to be corrected every step of the way on this.
Would be interesting if someone found niche languages in different styles with similar levels of documentation to test with.
[1] https://en.m.wikipedia.org/wiki/MOO
[2] https://greyhackgame.com/
e.g. The minecraft bot Voyager can explore the game environment and extend its skill library(stored as a vector database), is that considered as innovation? There are also systems like leandojo/alphaproof that discover new proofs and use LLM in non-trivial ways(not just naively predict the next token in one shot). Reinforcement learning algorithms like AlphaGo/AlphaZero use self play and use monte carlo tree search to learn to outperform humans. You can similarly use LLMs to generate actions and estimate state values(check the language agent tree search paper).
Most people use LLMs by prompting them with some additional context(chat history and data retrived from database) but there is nothing that stops us from continuously improving a LLM(either by modifying its weight or augmenting it with external database) by asking it to evaluate the task outcome/error message and feeding it back to the LLM. We can also ask it to just keep on generating new tasks to experiment with the environment/internet to get new knowledge.
So far, the current generation of LLMs that are in widespread use do not have that ability as far as I am aware. To actually do it to a degree that would rival human learning, they would need access to a lot more environment than you might be thinking of.
Sure, for programming, the most basic environment would be the platform to run the code on and the output of the code. But choices in programming are made based on more things like performance, load impact, behavior in production environments, interaction with other applications, platform logging, adjacent application logging. Or even before that, using previous experience to judge specifications for an application which takes things in account like the expected user base, costs, etc.
The real world is a lot more complex than a minecraft world or a game of Go. Which is to say, I am not saying that it is impossible. I am sure research is ongoing to do exactly that.
But the LLMs that are currently already disrupting communities like StackOverflow are not doing any of that. Given how complex the task is to plug in all relevant stimuli and the fact that for now they can get by without doing all of this I think things are more likely to get worse before they potentially will get better.
And even then it gets marked as duplicate of something unrelated.
I mean apart from wow ! 98% of the internet is shit (that’s higher than even I assumed) - how did they differentiate between good and shit? PageRank? Length? I can tell the difference between good writing and bad writing (and porn and erotica) but I have to read it using my brain based LLM - how did they do it ?
This leads to the whole open source, show us your training data thing
- https://time.com/6247678/openai-chatgpt-kenya-workers/
- https://www.theguardian.com/technology/2024/apr/16/techscape...
Another hidden aspect of the current LLM landscape.
> discard as much as 90% of the data they initially collect for training models.
I did note how the author said that training models on other models doesn't have the ethical implications of training on stolen human data - except it does, because where did the first model get its training set from? This is why we make it illegal not just to steal but also to handle stolen goods.
This is why the scientific method was developed.
If you want an AI that truly learns then it has to get up of the sofa, and test the ideas in the real world, rather than learning to parrot hearsay.
It's not the way science is done any way.
How does doing real world experiments to test your hypothesis not help with the very common problem of being wrong?
> It's not the way science is done any way.
That's a very sweeping statement - especially since the scientific method is quite a catch-all phrase.
In the end science is driven forward by new data, gathered either intentionally or incidentally.
Self-driving cars can continue to learn because they are continually generating more data. Advertising algorithms can continue to learn and adapt as the campaigns they create are in effect experiments - and some are even well designed.
I suppose large language models can continue to learn by chatting to humans - but they are potentially only learning how to please humans, not necessarily underlying truths.
Steel manufactured before 1944[0] can be incredibly valuable[1] because it hasn't been tainted by the nuclear tests/bombs fallout, and maybe fairly soon any archive internet material written pre-2010 will be considered equally valuable.
[0] https://en.wikipedia.org/wiki/Low-background_steel
[1] https://interestingengineering.com/science/what-is-pre-war-s...
(By the way, the radioactivity in the atmosphere that was poisoning the steel has pretty much vanished by now, so don't go investing everything in battleship reclamation in 2024.)
Then we can hopefully filter out the crap before training new models.
In the oral phase of human communication "AI models" were residing within human brains: the origin of any stream of messages was literally in front of you, in flesh and blood.
In the print phase we have the first major decoupling. Whether in the form of a cuneiform tablet or a modern paperback, the provenance was no longer assured. We had to rely on "controlled" seals, trust the publishers and their distribution chains etc. Ultimately this worked because the relative difficulty of producing printed artifacts helped developed a legal/political apparatus to control the spread of the "fake" stuff.
Enter the digital communications era and we have the second major decoupling. The amount (and increasingly the apparent veracity) of generating human oriented messaging is no longer a limiting factor. This has no precedent. You can create now plausibly create a fake Wikipedia [1] by just running a model. The signal-to-noise of digitally exchanged messages can experience catastrophic collapse.
> A flood of synthetic content might not pose an existential threat to the progress of AI development, but it does threaten the digital public good of the (human) internet.
Indeed, this is the real risk. I don't care about idiot "AI" feeding on itself but I do care about destroying any basis of sane digital communication between humans.
Will we develop the legal/political apparatus to control the flood of algorithmic junk? Will it be remotely democratic? The best precedent of digital automation destroying communication channels and our apparent inability and/or unwillingness to do something about it is email spam [2]. Decades after it first appeared spam is still degrading our infosphere. The writing is on the wall.
[1] https://en.wikipedia.org/wiki/Wikipedia:List_of_hoaxes_on_Wi...
[1] https://en.wikipedia.org/wiki/Email_spam