OK as prototypist I can safely say "Yes!" and I've been repeating this for a while now. If you want something that is close to what everybody else does, or said, using statistical means make sense. If you are interested by genuine novelty, things on the fringe where the usual process breaks, then it still remains hard.
Anyway, going to read the actual piece but felt I needed this off my chest.
This is my biggest concern. Speaking as someone who recently had to read 60 AI generated reports (the whole issue of how much students are using AI is one discussion), it was genuinely soul-destroying reading the same phrases, seem sentence structures, same arguments over and over. Depressed me the whole of the next day.
Good potential for discussion here. I full agree with the underlying premise: This technology CANNOT be allowed to just give us more of the same, but lazily. It HAS TO be an empowering tool. It has to unlock NEW discoveries.
For the purpose of discussion though, this also undersells AIs:
1. They CAN be great tools for novelty + discovery! You just need to ask and explore and put in work. Its not "easy", but it does help.
2. Sometimes "the mean" is what you want. Sometimes I'm not after art. I'm after something efficient and recognizable and easy to maintain.
How this oversells AIs:
1. We are losing the muscle of forced creativity and problem solving. There is a certain kind of learned privilege that comes from facing a problem and having your instinctive reaction be to ask for and expect help from something else rather than to roll up your sleeves or sit back and have a think. If the system incentivizes loss of muscle en-masse, we're gonna lose something beautiful and powerful.
This touches on something I've (and many others) have felt throughout my life, not just since the advent of LLMs.
To take a simple example: I grew up with computer games in the '80s where there were no 'physics engines' or frameworks for building games. As a result, each game was an expression of the author's personality somehow. Fast forward to the noughties, games bored me as they mostly looked and felt the same, or maybe felt like 3-5 different games all packaged differently.
Another example: going abroad on holiday in Europe (I'm from London) used to be a relatively wild, vibrant experience, filled with unexpected differences and challenges (not all positive). There were no McDonalds or Starbucks and the shops were filled with unfamiliar products and foods. Now everywhere in Europe feels the same when I visit, especially with smartphone in hand.
And films went from wildly different to one another to what now feels like 'arty' and 'CGI' being the two choices.
This article continues that into the realm of ideas, or idea production. Everywhere you go looks and feels familiar.
Obviously not, since 20-somethings also dislike that and express the same things openly, which they didn't in the past.
Also obviously not, since one can just see it by comparing e.g. 10 pictures of NY or Chicago or LA vs 10 pictures of Berlin or London or Barcelona cafes from 1980s and from 2020s, and the latter all look alike. It's also been studied (even academically) and has several names: airport aesthetic, international style, placelessness, global monoculture, airspace aesthetic (air as in airbnb), etc.
I agree with the basic premise: that when using LLMs, they will tend towards some mean when resolving ambiguity.
There's a very interesting opportunity here for a deeper investigation.
- What does "regression to the mean" actually mean in practice when the LLM is conditioned on a possibly large amount of context?
- How does this perceived regression to the mean affect the result in different applications? When implementing code, it may show up as keeping it simple, hence easily understandable, "nonclever". When writing documentation, it may show up as simple language, short sentences, etc. supporting the intent of communicating with little friction to a broad audience. When brainstorming product ideas, it may show up as regurgitating old and boring ideas, but dressed in fancy language and affirmations that hide the shallowness of the content.
- What can be done to alter this behavior? Now that temperature doesn't seem to be a parameter anymore in new models, how can we steer creativity of the model?
- If the model's creativity is fundamentally limited, is there a way we can use it to support us in the expression of our creativity, leveraging the different strengths of humans and LLMs in a way that the result transcends the limits of either?
Unfortunately, I don't see the article doing that. And, while I know pointing out LLM-isms is often a cheap shot these days, I feel compelled to point out that this article is full of what I perceived as LLM-ism, quite ironic given the premise and the statement ("written off-distribution · on purpose").
E.g.
> Trained on the past, it answers in the past tense of thought. Not what is true. What is typical.
> We converge — not on what is right, but on what is average.
> Not the answer it was sure of — the one it would not stop correcting
> But ask it anything and it returns the most probable continuation — the center of mass of everything already written. Trained on the past, it answers in the past tense of thought. Not what is true. What is typical.
The problem is that this is a contradiction, and a pretty common misunderstanding. When we talk about something being probable, we're talking about what we don't know. When you extrapolate, you're saying something about the unknown, you're creating something new. While you can extrapolate into the past or the future, in the way this line is talking, it's answering in a future tense. I think even worse is that beneath this it says
> returned at the speed of certainty
At that point we're no longer talking about probabilities!
The problem isn't that these machines aren't capable of making new things. The whole of their mathematical grounding is in the creation of the unknown from the known. The problem is precisely that they are sold as miracle cures where they can produce great results for little effort. The law of "you get out of it what you put into it" still holds true. Undirected, uninspired usage of these statistical models gets you mediocre at-best results. Without an understanding of the underlying theory and mechanisms of how these models work (it's not just transformers, but any statistical model), driving the whole of the inference chain with maximal control as one might Max/MSP, as well as mastery over the target domain, you will effectively achieve nothing but "slop".
Of course, there's a whole other discussion here, which is that this site seems to be victim to the same grave ignorance that has caused the supposed "crisis of newness" within the arts (which has been talked about for much longer than these models have existed). That's a whole other can of worms, but essentially it's bunk. In modernity we can point to the last century of unending artistic innovation, and panic that this is slowing down, that this is the end of history. In truth, that century is anomalous. It's the most anomalous we've ever recorded, where real material changes were reacted to in real time. The innovations of modernism weren't born because of pretense to being original. It was wholly derived from the changes happening in reality irrespective of the arts, as a result of the industrial revolution, and later the information revolution. The norm in history is centuries of very slow refinement, barely perceptible on the timeline of a generation. Tiny little incremental changes stacked up over a long period of time. Bombastic, revolutionary artistic progress is the anomaly. An unending cacophony of that progress has happened exactly once in the entire history of humanity, as far as we can tell. There is a stupid expectation that the 20th century's breakneck pace was going to last forever. Obviously it wasn't. It was never a sustainable momentum, statistical models or not. People are still in the mindset it's the norm. The languishing over creative bankruptcy is simply the death of this delusional fantasy.
It's true that in a project, a novel idea undeclared as such will be shaved off quietly by an llm. You really need to be explicit about wanting to keep it.
You will get pushed into the mean.
However, I'd say 90% of making something (that is useful) is repeating the old thing. We stand on the shoulders of giants. Or at least we should. Getting there can be difficult for most of us.
I say this as someone who chronically re-invents things. I then later get stuck and find someone already thought through my problem and solved it better.
I don't believe being unique in all the ways is useful. You need to be unique in the important ways and not unique in the other places.
There's also a cultural coherence angle that (my) unique things often fail at. Stuff has to look like other stuff enough for people to understand intuitively what it is and how it works. Here the mean is your friend.
I am able to explore more unique spaces because I no longer deal with the minutia of getting the things that should be the same correct. So paradoxically, this has made my output more unique.
The output of a GPT is an interpolation (an estimation of new data points inside the range of known data) rather than extrapolation (estimations outside that range).
99% of the time we don't need a true intellectual breakthrough to get the job done, and often 'new ideas' are simply riffs on or blends of old ones, like fashion or music genres.
The worry to me, however, is that if society comes to rely on this form of 'AI' then eventually the model collapse bleeds into academia (e.g. grant proposals reviewed by AI?) causing a kind of incremental sociocognitive atrophy. Everything becomes a reaffirmation of the status quo.
That being said I think people said something similar about electronic calculators (that if you couldn't do long division by hand then you'd be too incompetent for higher-level calculus.)
Noticed that most of the comments here are despairing.
I think that perhaps there's a bit of hope, that by the forces of the market, the value of human distinctiveness will rise in comparison to whatever is the generated mean. This is what I am looking into.
It's the same problem that AI faces of Model Collapse: AIs that train on the internet ultimately just end up training on one another, stop moving forward, and end up as identical polished versions of one another
I now think of it as a Dr. Jekyll/ Mr. Hyde situation for software projects:
- Dr. Jekyll: For makers, the only limit is your imagination, architectural guidance, and token budget. Time to build!
- Mr. Hyde: For projects to get off the treadmill of having to copy others to maintain you position, you need to redefine how the project works and provides unique value. Features and quality are no longer the answer. Time to fight!
That is not true - a model trained in the internet can both build verifiers to remove false/poor quality data from the next training, and build synthetic datasets that will supplement its training.
Similar to a human that wants to learn something and invents exercises to practice.
1-2 years ago it was a theory, but new models are trained, successfully, on synthetic datasets.
This feels like a bit of a semantic debate, but maybe a few useful perspectives, as interpret current bespoke work as not so rosy wrt collapse:
- Synthetic datasets are typically human-steered today, which points to model collapse wrt learning from the internet. I don't think standard practice is (yet) AI looking at the internet and deciding to build its own gyms to go further. When it does, model collapse may happen again, and be even more expensive
- Distillation attacks are getting interesting here. There seem to be 2 kinds: intentionally querying other models, and maybe not so intentionally, learning from reasoning traces going through shared routers, esp. coding ones
- A lot of neolabs are trying to go where the big labs might not look as directly to avoid being squashed, which suggests they aren't ready to bet on being smarter, and that means the general AI is more about collapse / $
The recent breakthrough of llm’s solving major open problem’s in math is a direct contradiction to the article.
But, there is some truth to the article and perhaps it is more true in chat based interactions. The agentic, hands-off mode might tell a different story.
> The recent breakthrough of llm’s solving major open problem’s in math is a direct contradiction to the article.
I disagree. LLMs are essentially trained on all the math we know, as well as - via RL training - how we apply it to solve given types of problems.
It should be obvious that not every known mathematical technique, or sequence of techniques, have yet been tried by humans to solve every unproven conjecture, so necessarily there is some amount of low hanging fruit where an automaton tasked with throwing the kitchen sink at a problem will be successful. This is not the same as creativity.
If you are familiar with the mathematical notion of a closure, then what LLMs are capable of is generating the generative closure of what they were trained on. They can generate output that is novel in the sense of a sequence of chess moves, or mathematical moves, that was not - as a sequence - in the training set, but yet it WAS in the training set as part of the implicit closure of what an LLM would be capable of generating.
I expect that a lot of bad norms that crop up in various programming communities will get tossed out after years of dysfunction, often as a result of outsiders and younger people arriving, and stating the obvious: This doesn't make sense. It's actually nice to know this when dealing with all of the mess in the present. You can't change it now, but eventually someone will.
But yes, LLMs are likely to force permanent conformity.
One could also talk about how language in general shifts with the population, but LLMs are likely to prevent it. One would think anthropologists are already looking into this experimentally...
LLM training requires a massive amount of manual labor by human experts. This goes beyond just the obvious, scanning the public and sometimes private previous work of human experts. We know from news media exposés that LLM vendors hire experts such as precariously employed academics as glorified gig workers to provide feedback to the models and correct them. Facebook even tried to record the keystrokes of its own engineers for this purpose until they pushed back.
At the same time, LLMs undermine the production of new human experts by attacking expertise at its source: education. Students are becoming addicted to LLM usage, and as a consequence, they're failing to learn anything in school. Kids are dumbing themselves down; teachers are perplexed and demoralized. This may seem rational to each individual student, taking the easy way out, but collectively it's a disaster.
Together, these two phenomena inevitably result in arrested intellectual development throughout society. It's a recipe for idiocracy.
"Offer it something it has never seen, and it doesn't light up. It corrects you. To a system built to predict the expected, the genuinely new is indistinguishable from a mistake.
The pushback is soft, and constant:
Did you mean: the familiar thing, offered in place of yours."
Ouch, that's scary to think of.
Also why does it read like its written by ChatGPT?
This is only surprising if you see AI as a kind of independent intelligence rather than as a new way to access existing media. If it's the later then of course it rarely goes beyond that training. With that perspective it would be just as unreasonable to expect a book to be different when we re-read it. And whatever stultifying/amplifying effect that the LLM has on creativity, so does the written word.
43 comments
[ 3.0 ms ] story [ 40.9 ms ] threadAnyway, going to read the actual piece but felt I needed this off my chest.
For the purpose of discussion though, this also undersells AIs:
1. They CAN be great tools for novelty + discovery! You just need to ask and explore and put in work. Its not "easy", but it does help.
2. Sometimes "the mean" is what you want. Sometimes I'm not after art. I'm after something efficient and recognizable and easy to maintain.
How this oversells AIs:
1. We are losing the muscle of forced creativity and problem solving. There is a certain kind of learned privilege that comes from facing a problem and having your instinctive reaction be to ask for and expect help from something else rather than to roll up your sleeves or sit back and have a think. If the system incentivizes loss of muscle en-masse, we're gonna lose something beautiful and powerful.
To take a simple example: I grew up with computer games in the '80s where there were no 'physics engines' or frameworks for building games. As a result, each game was an expression of the author's personality somehow. Fast forward to the noughties, games bored me as they mostly looked and felt the same, or maybe felt like 3-5 different games all packaged differently.
Another example: going abroad on holiday in Europe (I'm from London) used to be a relatively wild, vibrant experience, filled with unexpected differences and challenges (not all positive). There were no McDonalds or Starbucks and the shops were filled with unfamiliar products and foods. Now everywhere in Europe feels the same when I visit, especially with smartphone in hand.
And films went from wildly different to one another to what now feels like 'arty' and 'CGI' being the two choices.
This article continues that into the realm of ideas, or idea production. Everywhere you go looks and feels familiar.
Or am I just getting old?
Obviously not, since 20-somethings also dislike that and express the same things openly, which they didn't in the past.
Also obviously not, since one can just see it by comparing e.g. 10 pictures of NY or Chicago or LA vs 10 pictures of Berlin or London or Barcelona cafes from 1980s and from 2020s, and the latter all look alike. It's also been studied (even academically) and has several names: airport aesthetic, international style, placelessness, global monoculture, airspace aesthetic (air as in airbnb), etc.
There's a very interesting opportunity here for a deeper investigation.
- What does "regression to the mean" actually mean in practice when the LLM is conditioned on a possibly large amount of context?
- How does this perceived regression to the mean affect the result in different applications? When implementing code, it may show up as keeping it simple, hence easily understandable, "nonclever". When writing documentation, it may show up as simple language, short sentences, etc. supporting the intent of communicating with little friction to a broad audience. When brainstorming product ideas, it may show up as regurgitating old and boring ideas, but dressed in fancy language and affirmations that hide the shallowness of the content.
- What can be done to alter this behavior? Now that temperature doesn't seem to be a parameter anymore in new models, how can we steer creativity of the model?
- If the model's creativity is fundamentally limited, is there a way we can use it to support us in the expression of our creativity, leveraging the different strengths of humans and LLMs in a way that the result transcends the limits of either?
Unfortunately, I don't see the article doing that. And, while I know pointing out LLM-isms is often a cheap shot these days, I feel compelled to point out that this article is full of what I perceived as LLM-ism, quite ironic given the premise and the statement ("written off-distribution · on purpose").
E.g.
> Trained on the past, it answers in the past tense of thought. Not what is true. What is typical.
> We converge — not on what is right, but on what is average.
> Not the answer it was sure of — the one it would not stop correcting
The problem is that this is a contradiction, and a pretty common misunderstanding. When we talk about something being probable, we're talking about what we don't know. When you extrapolate, you're saying something about the unknown, you're creating something new. While you can extrapolate into the past or the future, in the way this line is talking, it's answering in a future tense. I think even worse is that beneath this it says
> returned at the speed of certainty
At that point we're no longer talking about probabilities!
The problem isn't that these machines aren't capable of making new things. The whole of their mathematical grounding is in the creation of the unknown from the known. The problem is precisely that they are sold as miracle cures where they can produce great results for little effort. The law of "you get out of it what you put into it" still holds true. Undirected, uninspired usage of these statistical models gets you mediocre at-best results. Without an understanding of the underlying theory and mechanisms of how these models work (it's not just transformers, but any statistical model), driving the whole of the inference chain with maximal control as one might Max/MSP, as well as mastery over the target domain, you will effectively achieve nothing but "slop".
Of course, there's a whole other discussion here, which is that this site seems to be victim to the same grave ignorance that has caused the supposed "crisis of newness" within the arts (which has been talked about for much longer than these models have existed). That's a whole other can of worms, but essentially it's bunk. In modernity we can point to the last century of unending artistic innovation, and panic that this is slowing down, that this is the end of history. In truth, that century is anomalous. It's the most anomalous we've ever recorded, where real material changes were reacted to in real time. The innovations of modernism weren't born because of pretense to being original. It was wholly derived from the changes happening in reality irrespective of the arts, as a result of the industrial revolution, and later the information revolution. The norm in history is centuries of very slow refinement, barely perceptible on the timeline of a generation. Tiny little incremental changes stacked up over a long period of time. Bombastic, revolutionary artistic progress is the anomaly. An unending cacophony of that progress has happened exactly once in the entire history of humanity, as far as we can tell. There is a stupid expectation that the 20th century's breakneck pace was going to last forever. Obviously it wasn't. It was never a sustainable momentum, statistical models or not. People are still in the mindset it's the norm. The languishing over creative bankruptcy is simply the death of this delusional fantasy.
It's true that in a project, a novel idea undeclared as such will be shaved off quietly by an llm. You really need to be explicit about wanting to keep it.
You will get pushed into the mean.
However, I'd say 90% of making something (that is useful) is repeating the old thing. We stand on the shoulders of giants. Or at least we should. Getting there can be difficult for most of us.
I say this as someone who chronically re-invents things. I then later get stuck and find someone already thought through my problem and solved it better.
I don't believe being unique in all the ways is useful. You need to be unique in the important ways and not unique in the other places.
There's also a cultural coherence angle that (my) unique things often fail at. Stuff has to look like other stuff enough for people to understand intuitively what it is and how it works. Here the mean is your friend.
I am able to explore more unique spaces because I no longer deal with the minutia of getting the things that should be the same correct. So paradoxically, this has made my output more unique.
99% of the time we don't need a true intellectual breakthrough to get the job done, and often 'new ideas' are simply riffs on or blends of old ones, like fashion or music genres.
The worry to me, however, is that if society comes to rely on this form of 'AI' then eventually the model collapse bleeds into academia (e.g. grant proposals reviewed by AI?) causing a kind of incremental sociocognitive atrophy. Everything becomes a reaffirmation of the status quo.
That being said I think people said something similar about electronic calculators (that if you couldn't do long division by hand then you'd be too incompetent for higher-level calculus.)
I think that perhaps there's a bit of hope, that by the forces of the market, the value of human distinctiveness will rise in comparison to whatever is the generated mean. This is what I am looking into.
It's the same problem that AI faces of Model Collapse: AIs that train on the internet ultimately just end up training on one another, stop moving forward, and end up as identical polished versions of one another
I now think of it as a Dr. Jekyll/ Mr. Hyde situation for software projects:
- Dr. Jekyll: For makers, the only limit is your imagination, architectural guidance, and token budget. Time to build!
- Mr. Hyde: For projects to get off the treadmill of having to copy others to maintain you position, you need to redefine how the project works and provides unique value. Features and quality are no longer the answer. Time to fight!
Similar to a human that wants to learn something and invents exercises to practice.
1-2 years ago it was a theory, but new models are trained, successfully, on synthetic datasets.
- Synthetic datasets are typically human-steered today, which points to model collapse wrt learning from the internet. I don't think standard practice is (yet) AI looking at the internet and deciding to build its own gyms to go further. When it does, model collapse may happen again, and be even more expensive
- Distillation attacks are getting interesting here. There seem to be 2 kinds: intentionally querying other models, and maybe not so intentionally, learning from reasoning traces going through shared routers, esp. coding ones
- A lot of neolabs are trying to go where the big labs might not look as directly to avoid being squashed, which suggests they aren't ready to bet on being smarter, and that means the general AI is more about collapse / $
But, there is some truth to the article and perhaps it is more true in chat based interactions. The agentic, hands-off mode might tell a different story.
I disagree. LLMs are essentially trained on all the math we know, as well as - via RL training - how we apply it to solve given types of problems.
It should be obvious that not every known mathematical technique, or sequence of techniques, have yet been tried by humans to solve every unproven conjecture, so necessarily there is some amount of low hanging fruit where an automaton tasked with throwing the kitchen sink at a problem will be successful. This is not the same as creativity.
If you are familiar with the mathematical notion of a closure, then what LLMs are capable of is generating the generative closure of what they were trained on. They can generate output that is novel in the sense of a sequence of chess moves, or mathematical moves, that was not - as a sequence - in the training set, but yet it WAS in the training set as part of the implicit closure of what an LLM would be capable of generating.
But yes, LLMs are likely to force permanent conformity.
One could also talk about how language in general shifts with the population, but LLMs are likely to prevent it. One would think anthropologists are already looking into this experimentally...
At the same time, LLMs undermine the production of new human experts by attacking expertise at its source: education. Students are becoming addicted to LLM usage, and as a consequence, they're failing to learn anything in school. Kids are dumbing themselves down; teachers are perplexed and demoralized. This may seem rational to each individual student, taking the easy way out, but collectively it's a disaster.
Together, these two phenomena inevitably result in arrested intellectual development throughout society. It's a recipe for idiocracy.
The pushback is soft, and constant: Did you mean: the familiar thing, offered in place of yours."
Ouch, that's scary to think of.
Also why does it read like its written by ChatGPT?