I see the claims being levied against LLMs, but in the generative media world these models are nothing short of revolutionary.
In addition to being an engineer, I'm also a filmmaker. This tech has so many orders of magnitude changes to the production cycle:
- Films can be made 5,000x cheaper (a $100M Disney film will be matched by small studios on budgets of $20,000.)
- Films can be made 5x faster (end-to-end, not accounting for human labor hour savings. A 15 month production could feasibly be done in 3 months.)
- Films can be made with 100x fewer people. (Studios of the future will be 1-20 people.)
Disney and Netflix are going to be facing a ton of disruptive pressure. It'll be interesting to see how they navigate.
Advertising and marketing? We've already seen ads on TV that were made over a weekend [1] for a few thousand dollars. I've talked to customers that are bidding $30k for pharmaceutical ad spots they used to bid $300k for. And the cost reductions are just beginning.
"Yet, every time I tried to get LLMs to perform novel research, they fail because they don’t have access to existing literature on the topic. Whereas, humans, on the other hand, discovered everything humanity knows."
Just because the author was unable to wrangle LLM to do novel research doesn't mean that it's impossible. We already have examples of LLMs either doing or aiding significantly with novel research.
> While Apple is still a major player in the AI space, they typically lean towards a preference for traditional AI. They’re almost certainly still doing LLM research in the background, but they’ve decided against dropping everything to go full throttle on the hype train. Ironically, this has started to make their investors, who have bought into the hype, quite upset. People are starting to question “why isn’t apply doing things with AI”.
It may very well be the case that Apple too finds themselves pressured into going all out on LLM
Besides my stance that LLMs can serve specific tasks very well and are likely going to take a place similar to spreadsheets and databases over the coming years, hasn’t Apple already? Rarely has Apple tried to appear so unified on one goal across their product stack as they did with Apple Intelligence, the vast majority of which is heavily LLM focused.
The Author appears to fully skip over their attempt and subsequent failure, which made the entire point the piece is trying to further rather unsubstantiated and made me check whether this wasn’t posted in 2022, even more for someone like myself who also is very confident that there is a large chasm between LLMs and whatever AGI may end up being.
"If a machine is consuming and transforming incalculable amounts of training data produced by humans, discussed by humans, and explained by humans. Why would the output not look identical to human reasoning? If I were to photocopy this article, nobody would argue that my photocopier wrote it and therefore can think. But add enough convolutedness to the process, and it looks a lot like maybe it did and can."
But its not copying it. That is the entire point. Its using the training data to adjust floating point numbers. If you train on a single data piece over and over again, then yes it can replicate it, just like you can memorize lines of a school play, but its still not copied/compressed in the traditional, deterministic sense.
You can't argue "we don't know how they work, or our own brains work with any certainty" and then over-trivialize what they do on the next argument.
People suffer brain damage and come out the other side with radically different personalities. What happened to "qualia", or "sense of self", where is their "soul". Its just a mechanistic emergent property of their biological neural network.
Who is to say our brains aren't just very high parameterized biological floating point machines? That is the true Occam's Razor here, as uncomfortable as that might make people.
One mistake I think many make is to look at technology X and ascribe outcomes primarily to technology X.
AI is one in a long long long line of new technologies. It is generating a lot of investment, new corporate processes and directives, declarations like "new era" and "civilizational milestone," etc.
If someone thinks any of the above are wrong or misguided, it's a mistake to "blame" or look to AI as the primary cause.
The primary cause is our system: humans are actors in the US economic system and when a new technology is rolling out, usually the response is the same and differs only in magnitude.
I came into the comments to see how the discourse is shaping up in light of the author’s claims, and I’m seeing the same old boosterism trying to dismiss it wholesale without providing conclusive evidence.
So without further ado:
* If LLMs can indeed produce wholly novel research independently without any external sources, then prove it. Cite sources, unlike the chatbot that told you it can do that thing. Show us actual results from said research or products that were made from it. We keep hearing these things exponentially increase the speed of research and development but nobody seemingly has said proof of this that’s uniquely specific to LLMs and didn’t rely on older, proven ML techniques or concepts.
* If generative AI really can output Disney quality at a fraction of the cost, prove it with clips. Show me AI output that can animate on 2s, 4s, and 1s in a single video and knows when to use any of the above for specific effects. Show me output that’s as immaculate as old Disney animation, or heck, even modern ToonBoom-like animation. Show me the tweens.
* Prove your arguments. Stop regurgitating hypeslop from CEBros, actually cite sources, share examples, demonstrate its value relative to humanity.
All people like us (myself and the author) have been politely asking for since this hype bubble inflated was for boosters to show actual evidence of their claims. Instead, we just get carefully curated sizzle reels and dense research papers making claims instead of actual, tangible evidence that we can then attempt to recreate for ourselves to validate the claims in question.
Stop insulting us and show some f*king proof, or go back to playing with LLMs until you can make them do the things you claim they can do.
> If LLMs can indeed produce wholly novel research independently without any external sources, then prove it.
I was actually thinking about it, and there could be a simple test - Remove all knowledge of X from knowledge corpus and train LLM on such corpus. Under X one can imagine anything - differential calculus, logarithms, Reimann Hypothesis, Special Theory of relativity, Fermat's theorems, ... And now ask AI questions which actually has lead to discovery of X.
If AI is able to rediscover X while not knowing about X, we can say it is proof of intelligence.
It was quite convincing, and I could see lower-budget studios trying to make it work. (There is a truckload of garbage tier animation on all platforms.)
The person who submitted it is an experienced producer who used something like 600 prompts to generate the end result, so it's not exactly few-shot prompting from novices with no film experience. But it happened
Then again, the astroturfing done (presumably) by big LLM is off the charts, so who knows if this was actually what happened.
Completely nailed it. The comparison to Adderall and other drugs was exactly how I've been feeling about it all--i watch my boss excitedly post another LLM documentation PR, raving about how he never used to have time for these things, yet the actual output is unclear, ambiguous, and almost entirely useless. But it made him feel good to "get it done," and that alone has largely blinded him to how bad the output actually is, and how long it took him to get.
This post is perfect. It summarizes my position on today's variant of 'AI' exactly. I'm sending this article to everyone that asks me how I feel about it. Thanks for writing it.
16 comments
[ 2.6 ms ] story [ 34.9 ms ] threadI see the claims being levied against LLMs, but in the generative media world these models are nothing short of revolutionary.
In addition to being an engineer, I'm also a filmmaker. This tech has so many orders of magnitude changes to the production cycle:
- Films can be made 5,000x cheaper (a $100M Disney film will be matched by small studios on budgets of $20,000.)
- Films can be made 5x faster (end-to-end, not accounting for human labor hour savings. A 15 month production could feasibly be done in 3 months.)
- Films can be made with 100x fewer people. (Studios of the future will be 1-20 people.)
Disney and Netflix are going to be facing a ton of disruptive pressure. It'll be interesting to see how they navigate.
Advertising and marketing? We've already seen ads on TV that were made over a weekend [1] for a few thousand dollars. I've talked to customers that are bidding $30k for pharmaceutical ad spots they used to bid $300k for. And the cost reductions are just beginning.
[1] https://www.npr.org/2025/06/23/nx-s1-5432712/ai-video-ad-kal...
Just because the author was unable to wrangle LLM to do novel research doesn't mean that it's impossible. We already have examples of LLMs either doing or aiding significantly with novel research.
Because AI can inadvertently say nasty things. This could potentially damage the company image.
It may very well be the case that Apple too finds themselves pressured into going all out on LLM
Besides my stance that LLMs can serve specific tasks very well and are likely going to take a place similar to spreadsheets and databases over the coming years, hasn’t Apple already? Rarely has Apple tried to appear so unified on one goal across their product stack as they did with Apple Intelligence, the vast majority of which is heavily LLM focused.
The Author appears to fully skip over their attempt and subsequent failure, which made the entire point the piece is trying to further rather unsubstantiated and made me check whether this wasn’t posted in 2022, even more for someone like myself who also is very confident that there is a large chasm between LLMs and whatever AGI may end up being.
But its not copying it. That is the entire point. Its using the training data to adjust floating point numbers. If you train on a single data piece over and over again, then yes it can replicate it, just like you can memorize lines of a school play, but its still not copied/compressed in the traditional, deterministic sense.
You can't argue "we don't know how they work, or our own brains work with any certainty" and then over-trivialize what they do on the next argument.
People suffer brain damage and come out the other side with radically different personalities. What happened to "qualia", or "sense of self", where is their "soul". Its just a mechanistic emergent property of their biological neural network.
Who is to say our brains aren't just very high parameterized biological floating point machines? That is the true Occam's Razor here, as uncomfortable as that might make people.
AI is one in a long long long line of new technologies. It is generating a lot of investment, new corporate processes and directives, declarations like "new era" and "civilizational milestone," etc.
If someone thinks any of the above are wrong or misguided, it's a mistake to "blame" or look to AI as the primary cause.
The primary cause is our system: humans are actors in the US economic system and when a new technology is rolling out, usually the response is the same and differs only in magnitude.
Don't hate the player, hate the game.
So without further ado:
* If LLMs can indeed produce wholly novel research independently without any external sources, then prove it. Cite sources, unlike the chatbot that told you it can do that thing. Show us actual results from said research or products that were made from it. We keep hearing these things exponentially increase the speed of research and development but nobody seemingly has said proof of this that’s uniquely specific to LLMs and didn’t rely on older, proven ML techniques or concepts.
* If generative AI really can output Disney quality at a fraction of the cost, prove it with clips. Show me AI output that can animate on 2s, 4s, and 1s in a single video and knows when to use any of the above for specific effects. Show me output that’s as immaculate as old Disney animation, or heck, even modern ToonBoom-like animation. Show me the tweens.
* Prove your arguments. Stop regurgitating hypeslop from CEBros, actually cite sources, share examples, demonstrate its value relative to humanity.
All people like us (myself and the author) have been politely asking for since this hype bubble inflated was for boosters to show actual evidence of their claims. Instead, we just get carefully curated sizzle reels and dense research papers making claims instead of actual, tangible evidence that we can then attempt to recreate for ourselves to validate the claims in question.
Stop insulting us and show some f*king proof, or go back to playing with LLMs until you can make them do the things you claim they can do.
I was actually thinking about it, and there could be a simple test - Remove all knowledge of X from knowledge corpus and train LLM on such corpus. Under X one can imagine anything - differential calculus, logarithms, Reimann Hypothesis, Special Theory of relativity, Fermat's theorems, ... And now ask AI questions which actually has lead to discovery of X.
If AI is able to rediscover X while not knowing about X, we can say it is proof of intelligence.
I don't think this makes any sense and suspect it requires a deep misunderstanding of “research” to even consider this an issue.
Research is inherently something done in response to, and grounded in, the external.
It was quite convincing, and I could see lower-budget studios trying to make it work. (There is a truckload of garbage tier animation on all platforms.)
The person who submitted it is an experienced producer who used something like 600 prompts to generate the end result, so it's not exactly few-shot prompting from novices with no film experience. But it happened
Then again, the astroturfing done (presumably) by big LLM is off the charts, so who knows if this was actually what happened.