- are capable of evaluating the LLM's output to the degree that they can identify truly unique insights
- are prompting the LLM in such a way that it could produce truly unique insights
I've prompted an LLM upwards of 1,000 times in the last month, but I doubt more than 10 of my prompts were sophisticated enough to even allow for a unique insight. (I spend a lot of time prompting it to improve React code.) And of those 10 prompts, even if all of the outputs were unique, I don't think I could have identified a single one.
I very much do like the idea of the day-dreaming loop, though! I actually feel like I've had the exact same idea at some point (ironic) - that a lot of great insight is really just combining two ideas that no one has ever thought to combine before.
Totally agree, most prompts (especially for code) aren’t designed to surface novel insights, and even when they are, it’s hard to recognize them. That’s why the daydreaming loop is so compelling: it offloads both the prompting and the novelty detection to the system itself. Projects like https://github.com/DivergentAI/dreamGPT are early steps in that direction, generating weird idea combos autonomously and scoring them for divergence, without user prompting at all.
If the breakthrough comes, most if not all links on HN will be to machine generated content. But so far it seems that the I in current AI is https://www.youtube.com/watch?v=uY4cVhXxW64 ..
Something I haven't seen explored, but I think could perhaps help is to somehow introduce feedback regarding the generation into the context, based on things that are easily computed w/ other tools (like perplexity). In "thinking" models we see a lot of emerging behaviour like "perhaps I should, but wait, this seems wrong", etc. Perhaps adding some signals at regular? intervals could help in surfacing the correct patterns when they are needed.
There's a podcast I listened to ~1.5 years ago, where a team used GPT2, further trained on a bunch of related papers, and used snippets + perplexity to highlight potential errors. I remember them having some good accuracy when analysed by humans. Perhaps this could work at a larger scale? (a sort of "surprise" factor)
Oh, in the middle of "AI is PhD-level" propaganda (just check Google News to see this is not a strawman argument), some people finally admit in passing "no LLM has ever made a breakthrough".
I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it?
First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for whether specific ideas came from the model or me. I tell people I code with LLMs, but I don’t commit a comment saying “wow, this clever bit came from the model!”
If people are getting actual bombshell breakthroughs from LLMs, maybe they are rationally deciding to use those ideas without mentioning the LLM came up with it first.
Anyway, I still think Gwern’s suggestion of a generic idea-lab trying to churn out insights is neat. Given the resources needed to fund such an effort, I could imagine that a trading shop would be a possible place to develop such a system. Instead of looking for insights generally, you’d be looking for profitable trades. Also, I think you’d do a lot better if you have relevant experts to evaluate the promising ideas, which means that more focused efforts would be more manageable. Not comparing everything to everything, but comparing everything to stuff in the expert’s domain.
If a system like that already exists at Jane Street or something, I doubt they are going to tell us about it.
This is bordering conspiracy theory. Thousands of people are getting novel breakthroughs generated purely by LLM an not a single person discloses such result? Not even one of the countless LLM corporation engineers who depend on the billion dollar IV injections from deluded bankers just to continue surviving, and not one has bragged about LLM doing that revolution? Hard to believe.
> but I don’t commit a comment saying “wow, this clever bit came from the model!”
The other day, Claude Code started adding a small signature to the commit messages it was preparing for me. It said something like “This commit was co-written with Claude Code” and a little robot emoji
I wonder if that just happened by accident or if Anthropic is trying to do something like Apple with the “sent from my iPhone”
It is hard to accept as a premise because the premise is questionable from the beginning.
Google already reported several breakthroughs as a direct result of AI, using processes that almost certainly include LLMs, including a new solution in math, improved chip designs, etc. DeepMind has AI that predicted millions of protein folds which are already being used in drugs among many other things they do, though yes, not an LLM per se. There is certainly the probability that companies won’t announce things given that the direct LLM output isn’t copyrightable/patentable, so a human-in-the-loop solves the issue by claiming the human made said breakthrough with AI/LLM assistance. There isn’t much benefit to announcing how much AI helped with a breakthrough unless you’re engaged in basically selling AI.
As for “why aren’t LLMs creating breakthroughs by themselves regularly”, that answer is pretty obvious… they just don’t really have that capacity in a meaningful way based on how they work. The closest example is Google’s algorithmic breakthrough absolutely was created by a coding LLM, which was effectively achieved through brute force in a well established domain, but that doesn’t mean it wasn’t a breakthrough. That alone casts doubt on the underlying premise of the post.
Most interesting novel ideas originate at the intersection of multiple disciplines. Profitable trades could be found in the biomedicine sector when the knowledge of biomedicine and finance are combined. That's where I see LLMs shining because they span disciplines way more than any human can. Once we figure out a way to have them combine ideas (similar to how Gwern is suggesting), there will be, I suspect, a flood of novel and interesting ideas, inconceivable with humans.
Almost certainly an LLM has, in response to a prompt and through sheer luck, spat out the kernel of an idea that a super-human centaur of the year 2125 would see as groundbreaking that hasn't been recognized as such.
We have a thin conception of genius that can be challenged by Edison's "1% inspiration, 99% perspiration" or the process of getting a PhD were you might spend 7 years getting to the point where you can start adding new knowledge and then take another 7 years to really hit your stride.
I have a friend who is 50-something and disabled with some mental illness, he thinks he has ADHD. We had a conversation recently where he repeatedly expressed his fantasy that he could show up somewhere with his unique perspective and sprinkle some pixie dust on their problems and be rewarded for it. I found it exhausting. When I would hear his ideas, or if I hear any idea, I immediately think "how would we turn this into a product and sell it?" or "write a paper about it?" or "convince people of it?" and he would have no part of it and think that operationalizing or advocating for that was uninteresting and that somebody else would do all that work and my answer is -- they might, but not without the advocacy.
And it comes down to that.
If an LLM were to come up with a groundbreaking idea and be recognized as having a groundbreaking idea it would have to do a sustained amount of work, say at least 2 person × years equivalent to win people over. And they aren't anywhere near equipped to do that, nobody is going to pay the power bill to do that, and if you were paying the power bill you'd probably have to pay the power bill for a million of them to go off in the wrong direction.
Broadly agree (I see lots of "ideas people" who have no interest in doing), the only thing I would say is that the occasional results from the big AI groups suggests it takes less than 1e6 machines — but probably more than 100 even for low-hanging fruit, which is already too much for a lot of people to stomach, so the point is still valid.
I have not yet seen AI doing a critical evaluation of data sources. AI willcontradict primary sources if the contradiction is more prevalent in the training data.
Something about the whole approach is bugged.
My pet peeve: "Unix System Resources" as explanation for the /usr directory is a term that did not exist until the turn of the millenium (rumor is that a c't journalist made it up in 1999), but AI will retcon it into the FHS (5 years earlier) or into Ritchie/Thompson/Kernigham (27 years earlier).
The bug is that LLMs are fundamentally designed for natural language processing and prediction, not logic or reasoning.
We may get to actual AI eventually, but an LLM architecture either won't be involved at all or it will act as a part of the system mimicking the language center of a brain.
I also hope we have something like this. But sadly, this is not going to work. The reason is this line from the article, which is so much harder that it looks:
> and a critic model filters the results for genuinely valuable ideas.
In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to satisfy the critic, and the critic itself is far from a good reasoner.
So the reason that we don't hear too much about this idea is not that nobody tried it. But that they tried, and it didn't work, and people are reluctant to publish about something which does not work.
The LLM doesn't have to know about the critic though. It can just output things and the critic is a second process that filters the output for the end user.
The models are currently trained on a static set of human “knowledge” — even if they “know” what novelty is, they aren’t necessarily incentivized to identify it.
In my experience, LLMs currently struggle with new ideas, doubly true for the reasoning models with search.
What makes novelty difficult, is that the ideas should be nonobvious (see: the patent system). For example, hallucinating a simpler API spec may be “novel” for a single convoluted codebase, but it isn’t novel in the scope of humanity’s information bubble.
I’m curious if we’ll have to train future models on novelty deltas from our own history, essentially creating synthetic time capsules, or if we’ll just have enough human novelty between training runs over the next few years for the model to develop an internal fitness function for future novelty identification.
My best guess? This may just come for free in a yet-to-be-discovered continually evolving model architecture.
In either case, a single discovery by a single model still needs consensus.
Google's effort with AlphaEvolve shows that the Daydream Factory approach might not be the big unlock we're expecting. They spent an obscene amount of compute to discover a marginal improvement over the state of the art in a very narrow field. Hours after Google published the paper, mathematicians pointed out that their SOTA algorithms underperformed compared to techniques published in the 50 years ago.
Intuitively, it doesn't feel like scaling up to "all things in all fields" is going to produce substantial breakthroughs, if the current best-in-class implementation of the technique by the worlds leading experts returned modest results.
I'd be happy to spend my Claude Max tokens during the night so it can "ultrathink" some Pareto improvements to my projects. So far, I've mostly seen lateral moves that rewrites code rather than rearchitecture/design the project.
Variations on increasing compute and filtering results aside, the only way out of this rut is another breakthrough as big, or bigger than transformers. A lot of money is being spent on rebranding practical use-cases as innovation because there's severe lack of innovation in this sphere.
AlphaEvolve and similar systems based on map-elites + DL/LLM + RL appears to be one of the promising paths.
Setting up the map-elites dimensions may still be problem-specific but this could be learnt unsupervisedly, at least partially.
The way I see LLMs is as a search-spqce within tokens that manipulate broad concepts within a complex and not so smooth manifold. These concepts can be refined within other spaces (pixel -space, physical spaces, ...)
In a recent talk [0] Francois Chollet made it sound like all the frontier models are doing Test-Time Adaptation, which I think is a similar concept to Dynamic evaluation that Gwern says is not being done. Apparently Test-Time Adaptation encompasses several techniques some of which modify model weights and some that don't, but they are all about on-the-fly learning.
Regardless of accusations of anthropomorphizing, continual thinking seems to be a precursor to any sense of agency, simply because agency requires something to be running.
Eventually LLM output degrades when most of the context is its own output. So should there also be an input stream of experience? The proverbial "staring out the window", fed into the model to keep it grounded and give hooks to go off?
Humans daydream about problems when they think a problem is interesting. Can an LLM know when a problem is interesting and thereby prune the daydream graph?
The feedback loop on novel/genuine breakthroughs is too long and the training data is too small.
Another reason is that there's plenty of incentive to go after the majority of the economy which relies on routine knowledge and maybe judgement, a narrow slice actually requires novel/genuine breakthroughs.
The question is: How do we get LLMs to have "Eureka!" moments, on their own, when their minds are "at rest," so to speak?
The OP's proposed solution is a constant "daydreaming loop" in which an LLM is does the following on its own, "unconsciously," as a background task, without human intervention:
1) The LLM retrieves random facts.
2) The LLM "thinks" (runs a chain-of-thought) on those retrieved facts to see if they are any interesting connections between them.
3) If the LLM finds interesting connections, it promotes them to "consciousness" (a permanent store) and possibly adds them to a dataset used for ongoing incremental training.
I’m once again begging people to read David Gelernter’s 1994 book “The Muse in the Machine”. I’m surprised to see no mention of it in Gwern’s post, it’s the exact book he should be reaching for on this topic.
In examining the possibility of genuinely creative computing, Gelernter discovers and defends a model of cognition that explains so much about the human experience of creativity, including daydreaming, dreaming, everyday “aha” moments, and the evolution of human approaches to spirituality.
55 comments
[ 3.5 ms ] story [ 59.8 ms ] threadI want to remember I heard about it in several podcasts
- are capable of evaluating the LLM's output to the degree that they can identify truly unique insights
- are prompting the LLM in such a way that it could produce truly unique insights
I've prompted an LLM upwards of 1,000 times in the last month, but I doubt more than 10 of my prompts were sophisticated enough to even allow for a unique insight. (I spend a lot of time prompting it to improve React code.) And of those 10 prompts, even if all of the outputs were unique, I don't think I could have identified a single one.
I very much do like the idea of the day-dreaming loop, though! I actually feel like I've had the exact same idea at some point (ironic) - that a lot of great insight is really just combining two ideas that no one has ever thought to combine before.
There's a podcast I listened to ~1.5 years ago, where a team used GPT2, further trained on a bunch of related papers, and used snippets + perplexity to highlight potential errors. I remember them having some good accuracy when analysed by humans. Perhaps this could work at a larger scale? (a sort of "surprise" factor)
(See original argument: https://nitter.net/dwarkesh_sp/status/1727004083113128327 )
First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for whether specific ideas came from the model or me. I tell people I code with LLMs, but I don’t commit a comment saying “wow, this clever bit came from the model!”
If people are getting actual bombshell breakthroughs from LLMs, maybe they are rationally deciding to use those ideas without mentioning the LLM came up with it first.
Anyway, I still think Gwern’s suggestion of a generic idea-lab trying to churn out insights is neat. Given the resources needed to fund such an effort, I could imagine that a trading shop would be a possible place to develop such a system. Instead of looking for insights generally, you’d be looking for profitable trades. Also, I think you’d do a lot better if you have relevant experts to evaluate the promising ideas, which means that more focused efforts would be more manageable. Not comparing everything to everything, but comparing everything to stuff in the expert’s domain.
If a system like that already exists at Jane Street or something, I doubt they are going to tell us about it.
The other day, Claude Code started adding a small signature to the commit messages it was preparing for me. It said something like “This commit was co-written with Claude Code” and a little robot emoji
I wonder if that just happened by accident or if Anthropic is trying to do something like Apple with the “sent from my iPhone”
Google already reported several breakthroughs as a direct result of AI, using processes that almost certainly include LLMs, including a new solution in math, improved chip designs, etc. DeepMind has AI that predicted millions of protein folds which are already being used in drugs among many other things they do, though yes, not an LLM per se. There is certainly the probability that companies won’t announce things given that the direct LLM output isn’t copyrightable/patentable, so a human-in-the-loop solves the issue by claiming the human made said breakthrough with AI/LLM assistance. There isn’t much benefit to announcing how much AI helped with a breakthrough unless you’re engaged in basically selling AI.
As for “why aren’t LLMs creating breakthroughs by themselves regularly”, that answer is pretty obvious… they just don’t really have that capacity in a meaningful way based on how they work. The closest example is Google’s algorithmic breakthrough absolutely was created by a coding LLM, which was effectively achieved through brute force in a well established domain, but that doesn’t mean it wasn’t a breakthrough. That alone casts doubt on the underlying premise of the post.
We have a thin conception of genius that can be challenged by Edison's "1% inspiration, 99% perspiration" or the process of getting a PhD were you might spend 7 years getting to the point where you can start adding new knowledge and then take another 7 years to really hit your stride.
I have a friend who is 50-something and disabled with some mental illness, he thinks he has ADHD. We had a conversation recently where he repeatedly expressed his fantasy that he could show up somewhere with his unique perspective and sprinkle some pixie dust on their problems and be rewarded for it. I found it exhausting. When I would hear his ideas, or if I hear any idea, I immediately think "how would we turn this into a product and sell it?" or "write a paper about it?" or "convince people of it?" and he would have no part of it and think that operationalizing or advocating for that was uninteresting and that somebody else would do all that work and my answer is -- they might, but not without the advocacy.
And it comes down to that.
If an LLM were to come up with a groundbreaking idea and be recognized as having a groundbreaking idea it would have to do a sustained amount of work, say at least 2 person × years equivalent to win people over. And they aren't anywhere near equipped to do that, nobody is going to pay the power bill to do that, and if you were paying the power bill you'd probably have to pay the power bill for a million of them to go off in the wrong direction.
Something about the whole approach is bugged.
My pet peeve: "Unix System Resources" as explanation for the /usr directory is a term that did not exist until the turn of the millenium (rumor is that a c't journalist made it up in 1999), but AI will retcon it into the FHS (5 years earlier) or into Ritchie/Thompson/Kernigham (27 years earlier).
The bug is that LLMs are fundamentally designed for natural language processing and prediction, not logic or reasoning.
We may get to actual AI eventually, but an LLM architecture either won't be involved at all or it will act as a part of the system mimicking the language center of a brain.
> and a critic model filters the results for genuinely valuable ideas.
In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to satisfy the critic, and the critic itself is far from a good reasoner.
So the reason that we don't hear too much about this idea is not that nobody tried it. But that they tried, and it didn't work, and people are reluctant to publish about something which does not work.
The LLM doesn't have to know about the critic though. It can just output things and the critic is a second process that filters the output for the end user.
The models are currently trained on a static set of human “knowledge” — even if they “know” what novelty is, they aren’t necessarily incentivized to identify it.
In my experience, LLMs currently struggle with new ideas, doubly true for the reasoning models with search.
What makes novelty difficult, is that the ideas should be nonobvious (see: the patent system). For example, hallucinating a simpler API spec may be “novel” for a single convoluted codebase, but it isn’t novel in the scope of humanity’s information bubble.
I’m curious if we’ll have to train future models on novelty deltas from our own history, essentially creating synthetic time capsules, or if we’ll just have enough human novelty between training runs over the next few years for the model to develop an internal fitness function for future novelty identification.
My best guess? This may just come for free in a yet-to-be-discovered continually evolving model architecture.
In either case, a single discovery by a single model still needs consensus.
Peer review?
Intuitively, it doesn't feel like scaling up to "all things in all fields" is going to produce substantial breakthroughs, if the current best-in-class implementation of the technique by the worlds leading experts returned modest results.
We're looking at our reflection and asking ourselves why it isn't moving when we don't
Setting up the map-elites dimensions may still be problem-specific but this could be learnt unsupervisedly, at least partially.
The way I see LLMs is as a search-spqce within tokens that manipulate broad concepts within a complex and not so smooth manifold. These concepts can be refined within other spaces (pixel -space, physical spaces, ...)
[0] https://www.youtube.com/watch?v=5QcCeSsNRks&t=1542s
Eventually LLM output degrades when most of the context is its own output. So should there also be an input stream of experience? The proverbial "staring out the window", fed into the model to keep it grounded and give hooks to go off?
The feedback loop on novel/genuine breakthroughs is too long and the training data is too small.
Another reason is that there's plenty of incentive to go after the majority of the economy which relies on routine knowledge and maybe judgement, a narrow slice actually requires novel/genuine breakthroughs.
The OP's proposed solution is a constant "daydreaming loop" in which an LLM is does the following on its own, "unconsciously," as a background task, without human intervention:
1) The LLM retrieves random facts.
2) The LLM "thinks" (runs a chain-of-thought) on those retrieved facts to see if they are any interesting connections between them.
3) If the LLM finds interesting connections, it promotes them to "consciousness" (a permanent store) and possibly adds them to a dataset used for ongoing incremental training.
It could work.
The breakthrough isn't in their datasets.
In examining the possibility of genuinely creative computing, Gelernter discovers and defends a model of cognition that explains so much about the human experience of creativity, including daydreaming, dreaming, everyday “aha” moments, and the evolution of human approaches to spirituality.
https://uranos.ch/research/references/Gelernter_1994/Muse%20...