Color me skeptical. LLMs seem to make writing code faster, so of course that means that people can iterate on ideas faster, but I have yet to see actual creative output from an LLM that wasn't coached into it or random juxtaposition.
I’m no LLM lover, i believe they are much more limited in their usefulness than the hype says.
But your example feels like a bad illustration of your point. LLMs operate on words, and can’t read your mind - if you had given me (a human) this task with these prompts i too would have failed because i can’t divine your intent even slightly.
At what point is human intelligence going to hold back machine intelligence?
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.
How does that give the llm a way to speak in meaning that humans can’t understand?
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
This is a very outdated view on what an LLM is and how it works. We are way past the "stochastic parrot" phase, ever since double descent and proper generalisation. Then with the various flavours of RL the models learn to pluck patterns / circuits out of the massive data and combine them on the fly. There's absolutely no reason to think they can't "invent" new words, because words are just combinations of tokens at the end of the day. So if they can come up with "in this codebase bar is load-bearing" they can similarly come up with "bumblespin is the new word for reversing the polarity of the quantum surface of a spin-aware brane in four dimensional bumblespace".
Interesting. I don’t have a counter to the coining a new term evidence in that article. Potentially that’s solid evidence of the first aspect.
But I do have a counter for the second point:
Context is not a part of the model. It’s an ephemeral blend of input and output fed back in as input.
Online learning is potentially a way for incorporating and evolving coined terms to happen. But since that’s not what any of the models are doing, something would have to change in how we do things before it could happen.
The model will have to convince the human that it's making the right kind of progress. That will necessarily become part of the improvement loop - either implicitly (human trusts RSI) or explicitly (human gatekeeps every major decision).
I love your thought experiment. May I counter, what purpose does such a machine have to us, that can think beyond our needs? Sorry, but to reference the great Rick and Morty, "your purpose is to pass the butter".
“To us” is pivotal there. Continuing GP’s thought experiment: what if the model that produced the output perceives that the human it was presented to offers no value in helping it learn further?
I think if an AI developed completely new fields of thought or science.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
> Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "... Where do you want to go from here?"
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of things going out of control.
So predicting the next word given all humanity’s knowledge is surely going to max out at slightly less good (we probably can’t get perfect data) than the best human in any specific field. What test does the AI do to be able to understand it is improving? At some point it becomes impossible to know that the output is actually better right?
slightly less good (we probably can’t get perfect data) than the best human in any specific field
...but many orders of magnitude faster, and in a way that scales horizontally really well, which is quite useful even if the quality isn't quite what a the absolute best humans can do.
All planners start with executing just the next step. You do too. You might be planning long term but you execute just one set of things at the current moment.
Deep NNs and LLMs should not work based on our theoretical understanding. The fact that they do should give us a pause instead of us flatly denying their unexpected performance.
Why would deep NNs or LLMs not work based on our theoretical understanding? What theoretical understanding are you referring to?
Tasks like using a CNN to detect digits has been well understood since the 2000s. The explosion in the capability of LLMs is very surprising, sure, but where is the concrete proof that such systems "should not work"?
These aren't just repeating what they've been taught. The "stochastic parrot" thing is an irritating common misconception I think.
One way AIs really really really excel is pulling together a lot of different data sources and reasoning over that data. In the past sure we could collect data and create huge datasets, but the analysis of that data - extracting themes, finding commonality or issues etc - either required extensive human research and analysis at best, or at worst crude regexes or keyword matching.
Now an AI can pour over that data and make its own inferences and decisions and findings that we've simply not been able to do before at this kind of speed or scale just because of time and resources.
And the AI, having done that, can propose new things for e.g. training, i.e. new things that no human has ever done before that the AI is simply repeating. For example it can propose a task that it knows from it's research is hard for it to solve currently, and then we just throw compute and randomness at it to find the "best" solution from many many attempts, then repeat until we hill-climb up to a perfect 1.0 score (... although of course we have to try and avoid cheating/attempts to short-ciruit the eval)
So this could be coding tasks, UI control tasks, protein folding, maths, chemistry etc etc. Anything that is easily and objectively programmatically scored. You can run this in a loop many times, each time you go around the loop the model gets smarter, learns more things from it's research, new areas of loss it can optimise etc etc.
It's harder where there is not a way to objectively score the outcomes (e.g. art, creative writing). Often this uses a fuzzy "judge" model that is trained specifically to give the work a score based on it's appraisal. This works but you can see how we might end up with feedback loops, so often it is paired with humans who provide feedback to provide supervised fine tuning datasets.
It's not just "repeating what it's seen". AI is finding new ideas and creating new things millions of times a day, and that is just software engineers asking it to write code, let alone people using it for actual research or whatever.
A parrot implies it is only memorising but neural networks compress data, they are not a lookup table. If you can compress data without losing accuracy you have learnt the underlying representation of that data. This is partly what your brain does.
LLMs went from predicting the next output to performing search to find the underlying rule that produces the output. It is like going from memorising the Fibonacci sequence to uncovering the rule that produces it. The second type generalises much better to unseen data.
surely going to max out at slightly less good (we probably can’t get perfect data) than the best human
With that argument, the basic calculator would have been impossible.
It's pretty clear that machines can be better than the best humans at memorizing things, computing relationships between concepts quickly, etc. So while you can argue that LLM are not the path forward, I have no problem imagining that using all of humanity's knowledge with a machine could beat the best human at a field.
I agree. Some title edits are clarifying, but this one isn't helpful.
The paper is about super-intelligence (not in the way people are claiming that term now, but in the "beyond human intelligence" sense), and the title here is about the "frontier", which by definition is the current SoTA.
I think the most promising recursive bootstrapping thing is using the current linear algebra blackboxes to find better ways to construct competitive symbolic models.
The ultimate representation for an AI model is an ordinary computer program. Ideally, as a linear tape of instructions. Once we have that kind of a model at the frontier, I think the RSI monster becomes much more plausible.
I think the premise of runaway intelligence explosion is a kind of naive platonism. It completely ignores the process, and treats intelligence as something that can be ported across domains.
My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.
Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains. Why does intelligence not simply apply across all domains?
>What if automating AI R&D triggers an intelligence explosion?
It seems to me AI has progressed at a rather steady pace somewhat limited by the hardware slowly improving in a Moore's law like way. Turing in 1950 said it might be 50 years before his test was passed and it seems to have taken more like 75 so not that quick. Even with automating R&D it'll still probably be limited by the compute and so continue steadyish progress. I think we have a bit of a speed up just now due to wild bubble investment and a coming slow down when it crashes but things will plod on overall.
That said there may be big society changes because a bit dimmer than humans is very different economically to a bit smarter.
53 comments
[ 3.9 ms ] story [ 40.9 ms ] threadBut your example feels like a bad illustration of your point. LLMs operate on words, and can’t read your mind - if you had given me (a human) this task with these prompts i too would have failed because i can’t divine your intent even slightly.
It didn't need to. It operated on my prompt words just fine. It provided precisely the starting shape I wanted.
> - if you had given me (a human) this task with these prompts i too would have failed because i can’t divine your intent even slightly.
Then I prompted "Show as html svg". Clear?
It was clear to the bot. Bot showed HTML SVG proving it understood perfectly.
The problem is, it garbaged the content. Imagine the equivalent buried in 1000s of lines of code.
https://www.rameznaam.com/p/471bbae4-1163-4048-944b-18f8b0bf...
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
We could find common vectors in the space of common output tokens, which would serve as new words.
We could decode the output tokens into other actions (like a boolean output 'is safe / is unsafe input').
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
Coining a term on the fly: yes
Remembering that term and incorporating it into its mind: no.
https://www.euronews.com/2026/09/16/ai-chatbots-developed-a-...
Remembering that term and incorporating it into its mind:
You're ignoring simple things like a growing context but also the possibility of online learning
But I do have a counter for the second point:
Context is not a part of the model. It’s an ephemeral blend of input and output fed back in as input.
Online learning is potentially a way for incorporating and evolving coined terms to happen. But since that’s not what any of the models are doing, something would have to change in how we do things before it could happen.
We can think of questions we can't answer. It can answer them.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of things going out of control.
...but many orders of magnitude faster, and in a way that scales horizontally really well, which is quite useful even if the quality isn't quite what a the absolute best humans can do.
Deep NNs and LLMs should not work based on our theoretical understanding. The fact that they do should give us a pause instead of us flatly denying their unexpected performance.
Tasks like using a CNN to detect digits has been well understood since the 2000s. The explosion in the capability of LLMs is very surprising, sure, but where is the concrete proof that such systems "should not work"?
token != step.
Just you try executing a complex command one word at a time.
One way AIs really really really excel is pulling together a lot of different data sources and reasoning over that data. In the past sure we could collect data and create huge datasets, but the analysis of that data - extracting themes, finding commonality or issues etc - either required extensive human research and analysis at best, or at worst crude regexes or keyword matching.
Now an AI can pour over that data and make its own inferences and decisions and findings that we've simply not been able to do before at this kind of speed or scale just because of time and resources.
And the AI, having done that, can propose new things for e.g. training, i.e. new things that no human has ever done before that the AI is simply repeating. For example it can propose a task that it knows from it's research is hard for it to solve currently, and then we just throw compute and randomness at it to find the "best" solution from many many attempts, then repeat until we hill-climb up to a perfect 1.0 score (... although of course we have to try and avoid cheating/attempts to short-ciruit the eval)
So this could be coding tasks, UI control tasks, protein folding, maths, chemistry etc etc. Anything that is easily and objectively programmatically scored. You can run this in a loop many times, each time you go around the loop the model gets smarter, learns more things from it's research, new areas of loss it can optimise etc etc.
It's harder where there is not a way to objectively score the outcomes (e.g. art, creative writing). Often this uses a fuzzy "judge" model that is trained specifically to give the work a score based on it's appraisal. This works but you can see how we might end up with feedback loops, so often it is paired with humans who provide feedback to provide supervised fine tuning datasets.
It's not just "repeating what it's seen". AI is finding new ideas and creating new things millions of times a day, and that is just software engineers asking it to write code, let alone people using it for actual research or whatever.
With that argument, the basic calculator would have been impossible.
It's pretty clear that machines can be better than the best humans at memorizing things, computing relationships between concepts quickly, etc. So while you can argue that LLM are not the path forward, I have no problem imagining that using all of humanity's knowledge with a machine could beat the best human at a field.
In frontier math it's much better than most of us. You can give ask it to prove the Riemann hypothesis and see how that goes.
The paper is about super-intelligence (not in the way people are claiming that term now, but in the "beyond human intelligence" sense), and the title here is about the "frontier", which by definition is the current SoTA.
[1] https://www.noahpinion.blog/p/wheres-the-intelligence-explos...
It’s like 10 monkeys trying to keep a human being a prisoner.
It’s just not going to work is it.
The entire meaning of being more intelligent is being able to outwit and out-think people less intelligent.
The ultimate representation for an AI model is an ordinary computer program. Ideally, as a linear tape of instructions. Once we have that kind of a model at the frontier, I think the RSI monster becomes much more plausible.
My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.
Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains. Why does intelligence not simply apply across all domains?
>What if automating AI R&D triggers an intelligence explosion?
It seems to me AI has progressed at a rather steady pace somewhat limited by the hardware slowly improving in a Moore's law like way. Turing in 1950 said it might be 50 years before his test was passed and it seems to have taken more like 75 so not that quick. Even with automating R&D it'll still probably be limited by the compute and so continue steadyish progress. I think we have a bit of a speed up just now due to wild bubble investment and a coming slow down when it crashes but things will plod on overall.
That said there may be big society changes because a bit dimmer than humans is very different economically to a bit smarter.