The prediction wasn't the existence of the product, but what was feeding into the content of the product (WebMD doesn't exist spontaneously).
Also relevant:
> Six patients 65 years or older (2 women and 4 men) were included in the analysis. The accuracy of the primary diagnoses made by GPT-4, clinicians, and Isabel DDx Companion was 4 of 6 patients (66.7%), 2 of 6 patients (33.3%), and 0 patients, respectively. If including differential diagnoses, the accuracy was 5 of 6 (83.3%) for GPT-4, 3 of 6 (50.0%) for clinicians, and 2 of 6 (33.3%) for Isabel DDx Companion (Table).
I was genuinely asking, and not getting any answers I decided to research it for myself.
I didn't find anything particularly compelling, but I did dig up this gem of an article about how deep learning was hitting a wall and "we are still a long way from machines that can genuinely understand human language" exactly four days before GPT-4 released:
> Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns. Whatever doubling there might have been, has perhaps come to an end.
Maybe I’m misunderstanding how OpenAI names their models, but was there supposed to be a huge jump from GPT-4 to GPT-4 Turbo? Just from the names it sounds like it’s expected to be in the same realm of model performance, just faster.
Your understanding is correct, Gary Marcus is intentionally misrepresenting what GPT-4 turbo is to make a story. It’s clearly the cost optimized version of gpt-4. That’s why it’s much cheaper per token.
I think the point made in the article is that they wanted to build gpt 5 but they renamed it to 4 turbo because the improvements over 4 weren't what they were expecting
The author suggests that GPT4-Turbo should have been GPT-5, but turned out too weak. Not sure if I buy that personally, I'm quite confident that OpenAI can deliver a lot more than GPT4 does right now, just not in a way that is economically feasible. GPT4-Turbo may be an intermediate release to have something to show while more work is being done behind the scenes.
If OpenAI had GPT-5 they would’ve released that instead of 4-Turbo.
OpenAI was allegedly working on their Arrakis model last year that didn’t end up panning out. It’s clear they’re hitting some sort of limit in the scaling laws, or cost efficiency, or both.
Why should the release GPT-5 if GPT-4 is still at the top of all benchmarks? 95% of the population who are not techies will probably use ChatGPT anyway for everything. I think they are pretty much the leading LLM company in most ways.
It is all financially-driven. All of the AI companies are working on models constantly are are purposely putting out releases so they can be ahead of the game or have some nice news for investors in order to make sure their stocks grow.
OpenAI probably know they are ahead with GPT-5 as well and they are planning to drop it Q3/Q4 after Llama3 (which will maybe trump GPT-4), so they can be ahead again for the EOY earnings.
> And here’s the thing – we all know that GPT-3 was vastly better than GPT-2. And we all know that GPT-4 (released thirteen months ago) was vastly better than GPT-3. But what has happened since?
Other models have gradually caught up to GPT-4 while OpenAI has declined to discuss or release anything regarding their internal research and work on GPT-5. Given how far ahead they were with GPT-4, it seems odd to assume no further progress has been made and just not published yet.
The author’s assumption is based on the claim that GPT-4 Turbo “was a failed attempt at GPT-5” but I don’t see any good reason to believe this unless you take the claim that GPT-5 is impossible as a given and reason backwards.
Also, part of the issue with the argument is that smaller firms like Anthropic now have models that exceed OpenAI's offering.
If we were actually at a point of diminishing returns, they'd only be catching up with similar levels of investment, not bypassing at lower levels.
And when digging into the details of the differences between something like Opus and GPT-4, it becomes more readily apparent why OpenAI's product line up is struggling.
Their current issues (to the extent they exist in practice) are related to OpenAI specific policies in how they are approaching developing their models in the fine tuning stages, not to inherent limitations in the underlying tech.
Oh wow so the game Connections is apparently a robust indicator of LLM capabilities?
Give me a break. The thesis here may be correct but Gary Marcus’ hasn’t effectively argued that it is. Is there even a mention of the scaling law’s paper?
Is it just me or does this look like an advanced form of blogspam? The author seems to take one idea and just throw out a dozen variations of it over a few days to see if anything sticks. A few titles from the past few days:
- Evidence that LLMs are reaching a point of diminishing returns — and what that might mean
- Superhuman AGI is not nigh
- $10 million says we won’t see human-superior AGI by the end of 2025
Oroborus issues were seen from 100k miles away.
LLMs were bound to run into issues immediately based on training data. If I weren’t substantially poor I’d put all my savings into puts of LL
‘S like ChatGPT, and others. I cannot risk burning substantials of my own time to offset the burnage those themselves are creating.
Good luck y’all. I’d say I’m hope I’m right but you’re on a ship sunk by glacial ice that is hoping for a lifeboat off. Good fucking luck, you have no time to pivot.
That article sure was a lot of words for "the rest of the market is catching up to OpenAI".
The rest is just a negative critical light on some of the woes of early adopters who likely aren't using the tech effectively. That's kind of problematic because (a) it doesn't emphasize how the tech takes skill and iteration to use right, and (b) it doesn't balance that out with examples of organizations who have seen immediate value from this tech.
I thought this was to be expected. If you invest 1 billion USD in training a new model using existing tools and techniques it probably won't be as big of a leap in capabilities as previous generations. In particular, aren't we running out of content with which to train these models? There's obviously some key details that are still missing in the AI space precluding us from achieving things like "can learn how to drive in a couple hours".
I don't want to pile on the hate, but this creator has a fairly negative reputation on Twitter. He comes off as a bit of an asshole on social media sometimes. Making it difficult to disentangle the author from the topics he brings up when the author is so inflammatory.
32 comments
[ 2.6 ms ] story [ 121 ms ] thread- https://moores.samaltman.com/ (March 2021)
Also relevant:
> Six patients 65 years or older (2 women and 4 men) were included in the analysis. The accuracy of the primary diagnoses made by GPT-4, clinicians, and Isabel DDx Companion was 4 of 6 patients (66.7%), 2 of 6 patients (33.3%), and 0 patients, respectively. If including differential diagnoses, the accuracy was 5 of 6 (83.3%) for GPT-4, 3 of 6 (50.0%) for clinicians, and 2 of 6 (33.3%) for Isabel DDx Companion (Table).
- Use of GPT-4 to Analyze Medical Records of Patients With Extensive Investigations and Delayed Diagnosis (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10425828/)
I didn't find anything particularly compelling, but I did dig up this gem of an article about how deep learning was hitting a wall and "we are still a long way from machines that can genuinely understand human language" exactly four days before GPT-4 released:
https://nautil.us/deep-learning-is-hitting-a-wall-238440/
Maybe I’m misunderstanding how OpenAI names their models, but was there supposed to be a huge jump from GPT-4 to GPT-4 Turbo? Just from the names it sounds like it’s expected to be in the same realm of model performance, just faster.
OpenAI was allegedly working on their Arrakis model last year that didn’t end up panning out. It’s clear they’re hitting some sort of limit in the scaling laws, or cost efficiency, or both.
It is all financially-driven. All of the AI companies are working on models constantly are are purposely putting out releases so they can be ahead of the game or have some nice news for investors in order to make sure their stocks grow.
OpenAI probably know they are ahead with GPT-5 as well and they are planning to drop it Q3/Q4 after Llama3 (which will maybe trump GPT-4), so they can be ahead again for the EOY earnings.
Other models have gradually caught up to GPT-4 while OpenAI has declined to discuss or release anything regarding their internal research and work on GPT-5. Given how far ahead they were with GPT-4, it seems odd to assume no further progress has been made and just not published yet.
The author’s assumption is based on the claim that GPT-4 Turbo “was a failed attempt at GPT-5” but I don’t see any good reason to believe this unless you take the claim that GPT-5 is impossible as a given and reason backwards.
If we were actually at a point of diminishing returns, they'd only be catching up with similar levels of investment, not bypassing at lower levels.
And when digging into the details of the differences between something like Opus and GPT-4, it becomes more readily apparent why OpenAI's product line up is struggling.
Their current issues (to the extent they exist in practice) are related to OpenAI specific policies in how they are approaching developing their models in the fine tuning stages, not to inherent limitations in the underlying tech.
Give me a break. The thesis here may be correct but Gary Marcus’ hasn’t effectively argued that it is. Is there even a mention of the scaling law’s paper?
- Evidence that LLMs are reaching a point of diminishing returns — and what that might mean
- Superhuman AGI is not nigh
- $10 million says we won’t see human-superior AGI by the end of 2025
Etc, you get the idea.
Good luck y’all. I’d say I’m hope I’m right but you’re on a ship sunk by glacial ice that is hoping for a lifeboat off. Good fucking luck, you have no time to pivot.
The rest is just a negative critical light on some of the woes of early adopters who likely aren't using the tech effectively. That's kind of problematic because (a) it doesn't emphasize how the tech takes skill and iteration to use right, and (b) it doesn't balance that out with examples of organizations who have seen immediate value from this tech.
I don't want to pile on the hate, but this creator has a fairly negative reputation on Twitter. He comes off as a bit of an asshole on social media sometimes. Making it difficult to disentangle the author from the topics he brings up when the author is so inflammatory.