I’m not sure I understand what you mean by “the button”. If you’re comparing this to DeepSeek’s copying, it’s not really the same thing right? DeepSeek essentially stole intellectual property by violating OpenAI’s terms of service. As I understand it, this is a copy of Google’s Deep Research
Deepseek proved that there is no moat. Thus no path to profitability for openai, anthropic & co.
Stealing from thieves is fine by me. Sama was the one claiming that all information could be used to train LLMs, without permisdion of the copyright holders.
Now the same is being done to openai. Well, too bad.
> Stealing from thieves is fine by me. Sama was the one claiming that all information could be used to train LLMs, without permisdion of the copyright holders.
OpenAI and other LLMs scraping the internet is probably covered under fair use. DeepSeek’s violation of OpenAI’s terms is pretty clearly a violation of their terms and not legal.
Yes those cases will be interesting. By default a lot of copyrighted content may be legal to use for training (in the US but also many other places) under what’s called fair use. The cases you’re referring to will likely reinforce this, but it isn’t known yet. Note that it’s not just OpenAI on that side of the argument but also other (non tech) organizations that believe protecting fair use here is current law and essential.
Care to explain how something that cannot be copyrighted and was not generated by a human is “intellectual property“? Or are you just parroting a narrative?
Explain the trade secrets contained in non-copyrightable AI outputs and the reasonable efforts OpenAI takes to keep its AI output “secret”. Or are you confused about what a “trade secret” actually is?
What is the current state of DSPy optimizers? When I originally checked it out it appeared to just be optimizing the set of examples used for n-shot prompting.
It appears that OpenAI is in panic mode after the release of DeepSeek. Before they were confident in competing against Google on any AI model they release.
Now they are scrambling against open-source after their disastrous operator demonstration and using this deep research demo as cover. Nothing that Google or Perplexity could not already do themselves.
By the end of them month, this feature is going be added by a bunch of other open-source projects and this feature won't be as interesting very quickly.
I don’t think you’re comparing the right things here. This feature is more like Google’s Deep Research, which basically goes off and does a whole lot of search and compute to produce something more like a full research report. This has nothing to do with open weight models like DeepSeek (note: DeepSeek, Llama, etc are NOT open source). This feature doesn’t just require the research on the model but also enormous compute. Plus anyone using such a feature for real work is not going to be using DeepSeek or whatever, but a product with trustworthy practices and guarantees.
> This feature is more like Google’s Deep Research, which basically goes off and does a whole lot of search and compute to produce something more like a full research report.
Of course. It is in response to their disastrous operator demo which did not justify the $200 per month ChatGPT Pro subscription on top of the release of DeepSeek to make matters worse for them.
> This has nothing to do with open weight models like DeepSeek (note: DeepSeek, Llama, etc are NOT open source).
It obviously does. Even before they rushed this presentation, they made o3-mini available for ChatGPT free users so it in direct response to DeepSeek.
> This feature doesn’t just require the research on the model but also enormous compute. Plus anyone using such a feature for real work is not going to be using DeepSeek or whatever, but a product with trustworthy practices and guarantees.
Nothing that Perplexity + DeepSeek-R1 can already do.
I really don't like the snarky tone of the parent comment.
Nonetheless, I don't think this is even something that can easily be benchmarked. I'd recommend you take a look at aider [1], and consider how I drew similarities between it and what's presented here.
Has ClosedAI presented any benchmarks / evaluation protocols?
I don’t think you actually read it. The benchmarks are in reference to the model that’s underlying deep-research, and not deep-research itself. For the latter, they have anecdata from scientists.
Not sure if people picked up on it, but this is being powered by the unreleased o3 model. Which might explain why it leaps ahead in benchmarks considerably and aligns with the claims o3 is too expensive to release publicly. Seems to be quite an impressive model and the leading out of Google, DeepSeek and Perplexity.
They’ve only released o3-mini, which is a powerful model but not the full o3 that is being claimed as too expensive to release. That being said, DeepSeek for sure forced their hand to release o3-mini to the public.
I guess the question is, did DeepSeek force them to rethink pricing? It's crazy how much cheaper it (v3 and R1) is, but considering they (Deepseek) can't keep up with demand, the price is kind of moot right now. I really do hope they get the hardware to support the API again. The v3 and R1 models that are hosted by others are still cheap compared to the incumbents, but nothing can compete with DeepSeek on price and performance.
Interesting, thanks for highlighting! Did not pick up on that. Re:"leading", tho:
Effectiveness in this task environment is well beyond the specific model involved, no? Plus they'd be fools (IMHO) to only use one size of model for each step in a research task -- sure, o3 might be an advantage when synthesizing a final answer or choosing between conflicting sources, but there are many, many steps required to get to that point.
I don't believe we have any indication that the big offerings (claude.ai, Gemini, operator, tasks, canvas, chatgpt) use multiple models in one call (other than for different modalities like having Gemini create an image). It seems to actually be very difficult technically and I'm curious as to why.
I wonder how much of an impact our being still so early in the productization phase of this all is. Like it takes a ton of work and training and coordination to get multiple models synced up into an offering and I think the companies are still optimizing for getting new ideas out there rather truly optimizing them.
OpenAI is very much in an existential crisis and their poor execution is not helping their cause. Operator or “deep research” should be able to assume the role of a Pro user, run a quick test, and reliably report on whether this is working before the press release right?
I’m not sure if you’re implying this subtly in your comment or not, as it’s early here, but it does of course need to be a generation ahead of what 10 months of their competitors moving forward have done too. Nobody is standing still
> Which might explain why it leaps ahead in benchmarks considerably and aligns with the claims o3 is too expensive to release publicly
It's the only tool/system (I won't call it an LLM) in their released benchmarks that has access to tools and the web. So, I'd wager the performance gains are strictly due to that.
If an LLM (o3) is too expensive to be released to the public, why would you use it in a tool that has to make hundreds of inference calls to it to answer a single question? You'd use a much cheaper model. Most likely o3-mini or o1-mini combined with o4-mini for some tasks.
>why would you use it in a tool that has to make hundreds of inference calls to it to answer a single question? You'd use a much cheaper model.
The same reason a lot of people switched to GPT-4 when it came out even though it was much more expensive than 3 - doesn't matter how cheap it is if it isn't good enough/much worse.
> Powered by a version of the upcoming OpenAI o3 model that’s optimized for web browsing and data analysis, it leverages reasoning to search, interpret, and analyze massive amounts of text, images, and PDFs on the internet, pivoting as needed in reaction to information it encounters.
If that's what you're referring to, then it doesn't seem that "explicit" to me. For example, how do we know that it doesn't use less thinking than o3-mini? Google's version of deep research uses their "not cutting edge version" 1.5 model, after all. Are you referring to something else?
o3-mini is not really "a version of the o3 model", it is a different model (less parameters). So their language strongly suggests, imo, that Deep Research is powered by a model with the same number of parameters as o3.
Eh, not really. Google failed to launch first out of internal political dysfunction and then made a crash effort to launch something to counter the first ChatGPT release.
I highly doubt that the concerns of internal political commissars were holding up this particular openai release.
Yea but guy paying closedai to get "insights" that basically copy-pasted content from my blog is definitely violating my blogs copyright, and in the end no coin comes to me either. What about that?
Could you provide an example where OpenAI outputting verbatim quotes actually constitutes the copyright violation? Because mechanically retrieving relevant quotes seems analogous to grep/search - the copyright status would depend on how downstream users transform and use that content. Like how quoting your blog in a technical analysis or critique is fair use, but wholesale republishing isn't. This suggests the violation occurs at usage time, not retrieval time.
I see many are offended, but I am genuinely asking a question.
I want to understand does this mean it's ethical for anyone to create a research AI tool that will go through arXiv and related GitHub repo and use it to solve problems, implement ideas like cursor.
What does that even mean? Treating each iterative model as a new product is not any different than Google changing its search or youtube recommendation algorithm.
Different pre-cooked prompts and filters don’t really amount to new products either, despite them being marketed as such. It’s like adobe treating each tool in photoshop as its own product.
If I understood the graphs correctly, it only achieves 20% pass rate on their internal tests. So I have to wait 30min and pay a lot of money just to sift through walls of most likely incorrect text?
Unless the possibility of hallucinations is negligible, this is just way too much content to review at once. The process probably needs to be a lot more iterative.
LLMs often don't do well on tasks that require composition into smaller subtasks. In this case there is a chain of relations that depend on the previous result.
it tests syllogistic reasoning: Jason's mother was Tyro, whose father was Poesidon, whose father was Kronos. it also tests whether it "eagerly" rather than comprehensively considers something: a maternal great-grandfather could be the father of either one's maternal grandmother or maternal grandfather. so the answer could also be king Aeolus of the Etruscans.
ideally a model would be able to answer this accurately and completely.
I think there are more possible answers? Jason's mother differs depending on the author...
For example, Jason's mother was Philonis, daughter of Mestra, daughter of Daedalion, son of Hesporos. So Jason's maternal great-grandfather was Hesporos.
No it is not an actual question on this exam. From the paper: “To ensure question quality and integrity, we enforce strict submission criteria. Questions should be precise, unambiguous, solvable, and non-searchable, ensuring models cannot rely on memorization or simple retrieval methods. All submissions must be original work or non-trivial syntheses of published information, though contributions from unpublished research are acceptable. Questions typically require graduate-level expertise or test knowledge of highly specific topics (e.g., precise historical details, trivia, local customs) and have specific, unambiguous answers…”. (Emphasis mine)
Did you intentionally flip through all the questions to find the one that seemed the easiest? If so, why? That's question #7, and all other 7 questions in the sample set seem ridiculously difficult to me.
Maybe. Not enough data to say. Say it does a days worth of work in a query. It is sensible to use if it takes less than a day to review ~5 days worth of work. I don't know if we're near that threshold yet but conceptually this would work well for actual research where the amount of preparation is large compared to the amount of output written.
And eyeballing the benchmarks, it'll probably reach a >50% rate per query by the end of the year. Seems to double every model or two.
Here's an example of the type of question it is acheiving 20% on;
The set of natural transformations between two functors F,G :C→DF,G:C→D can be expressed as the end
Nat(F,G)≅∫AHomD(F(A),G(A)).
Nat(F,G)≅∫A HomD (F(A),G(A)).
Define set of natural cotransformations from FF to GG to be the coend
CoNat(F,G)≅∫AHomD(F(A),G(A)).
CoNat(F,G)≅∫AHomD (F(A),G(A)).
Let:
- F=B∙(Σ4)∗/F=B∙ (Σ4 )∗/ be the under ∞∞-category of the nerve of the delooping of the symmetric group Σ4Σ4 on 4 letters under the unique 00-simplex ∗∗ of B∙Σ4B∙ Σ4 .
- G=B∙(Σ7)∗/G=B∙ (Σ7 )∗/ be the under ∞∞-category nerve of the delooping of the symmetric group Σ7Σ7 on 7 letters under the unique 00-simplex ∗∗ of B∙Σ7B∙ Σ7 .
How many natural cotransformations are there between FF and GG?
btw isn't this question at least really badly worded (and maybe incorrect?) the definitions they give for F and G are categories not functors... (and both categories are in fact one object with contractible space of morphisms...)
It's very interesting to think about what kind of "mental model" might it have, if it's capable of "understanding" all this (to me) gibberish, but is then unable to actually work the problem.
As someone who doesn't understand anything beyond the word 'set' in that question, can anyone give an indication of how hard of a problem that actually is (within that domain)?
Also I'm curious as to what percentage of the questions in this benchmark are of this type / difficulty, vs the seemingly much easier example of "In Greek mythology, who was Jason's maternal great-grandfather?".
I'd imagine the latter is much easier for an LLM, and almost trivial for any LLM with access to external sources (such as deep research).
The difference is that it takes few minutes to an hour at most so it can be run multiple times a day, using the results of previous runs to further refine the search and reasoning process to get better outcomes. Pretty much how any human research works but much faster and with potentially vastly more world-knowledge and reasoning capability than average humans. And these capabilities will rapidly improve with further RL.
Yeah it can be more iterative. Just use individual queries and build on it yourself. This is all this is doing. It's a trick, and OpenAI is a PR hype company at this stage.
I have never believed a conspiracy theory more instantly. Deep Search vs. DeepSeek is way more than enough to confuse the average layman! Especially when you're googling something you heard about at work a few hours ago, or on Bloomberg TV
You might as well say that DeepSeek wanted to cause confusion with DeepMind. Deep isn't such a distinguishing name, deep learning has been a buzzword since 2012.
Oh God, this is such an astute observation. I think it worked so well on me that I didn't even think about the "deep" portion initially. Goes to show how effective these things are psychologically.
It absolutely can replace the research done by one person, for my use case at least. It’s also available on their $20/month subscription, unlike OpenAI’s $200/month.
Feels like only a matter of time before these crawlers are blocked from large swathes of the internet. I understand that they’re already prohibited from Reddit and YouTube. If that spreads, this approach might be in trouble.
I suppose there is an equilibrium, where sites that penalize these types of crawlers will also get less traffic from people reading ai citations, so for many sites the upsides of allowing it will be greater than the downsides.
TBF OpenAI in particular bought access to Reddit. Otherwise yeah this is my main confusion with all of these products, Perplexity being the biggest -- how do you get around the status-quo of refusing access to bots? Just to start off with, there is no Google Search API, and they work hard to make sure headless browsers can't access the normal service.
They do say "Currently, deep research can access the open web...", so maybe "open" there implies something significant. Like, "websites that have agreements with OpenAI and/or do not enforce norobot policies".
> Amazon listings are blocked from google shopping
I see Amazon results there all the time. 3 of the visible 8 sponsored results are Amazon, in the non-sponsored results an Amazon listing is either first or second in every category.
This is trivially bypassed by OpenAI asking the user to take control of their computer (or a sandboxed browser within it,) then for all intents and purposes it’s the user themselves accessing your site (with some productivity/accessibility aid from OAI.)
While people might attempt that, it's going to be an arms race, just like ads vs adblocks. There's already multiple crawlers that present fake user-agent when their original one is blocked. Temptation of more data is just to irresistible to them
Especially this is not a breakthrough justifying a 340B USD valuation, but rather the work that junior developers can do; implement a loop of Bing Searches connected to an LLM.
If they "knew the questions in advance," why'd they need Internet access at all? The ability to use the same data sources humans would use is not the insult you seem to think it is.
Again: the assertion was yours, so let us know the results of your own work.
I haven’t tried the OpenAI version yet, as I’m on their peasant-level $20 plan, but the Google equivalent is way superior to Perplexity (I use both extensively). The web search Perplexity carries out is superficial compared to the Google product; it misses a large percentage of what Gemini Deep Research finds, and for a particular task in my business this makes a huge difference.
Sure if you're viewing this as some kind of spectator thing, or entertainment, maybe it's less interesting. But it doesn't really matter whether "people care". What matters is whether it's useful and has impact. It's enough if the small number of people use it for whom it is useful. It doesn't matter if the average Joe on the street is excited by it.
Few people care or even know about various advances in various specialized fields. It's enough if AI simply seeps into various applications in boring and non-flashy ways for it to have significant effects that will affect a wider range of people, whether they get hyped by the news announcements or not. Jobs etc.
An analogy: the Internet as such is not very exciting nowadays, certainly not in the way it was exciting in the 90s with all the news segments about surfing the information superhighway or whatever. There was a lot of buzz around the web, but then it got normalized. It didn't disappear, it just got taken for granted. No average person got excited around HTML5 or IPv6. It just chugs along in the background. AI will similarly simply build into the fabric of how things get done. Sometimes visibly to the average person, sometimes just behind the scenes.
Not sure if it's just me, but it looks like all SOTA companies are doubling down to chase the new benchmark, which beyond hype, doesn't seem to translate into real world uses. Why don't these companies just plug it into a popular git repo and say, hey our AI fixed these 100 issues! Or something real? The only people who seem to be doing something real is DeepMind.
To anyone who's tried it: how does it handle captchas? I can't imagine that OpenAI's IP addresses are anyone's favorites for unfettered access to web properties these days.
You can buy residential proxies to pretend you're a regular person IIRC, some of the browser automation companies do that to bypass rate limiting, captchas, etc.
"will find, analyze, and synthesize hundreds of online sources"
Synthesize? Seems like the wrong word -- I think they would want to say something like, "analyze, and synthesize useful outputs from hundreds of online sources"..
> combine (a number of things) into a coherent whole: pupils should synthesize the data they have gathered | Darwinian theory has been synthesized with modern genetics.
I'm sorry but what the fuck is this product pitch?
Anyone who's done any kind of substantial document research knows that it's a NIGHTMARE of chasing loose ends & citogenesis.
Trusting an LLM to critically evaluate every source and to be deeply suspect of any unproven claim is a ridiculous thing to do. These are not hard reasoning systems, they are probabilistic language models.
o1 and o3 are definitely not your run of the mill LLM. I've had o1 correct my logic, and it had correct math to back up why I was wrong. I'm very skeptical, but I do think at some point AI is going to be able to do this sort of thing.
444 comments
[ 4.6 ms ] story [ 328 ms ] threadStealing from thieves is fine by me. Sama was the one claiming that all information could be used to train LLMs, without permisdion of the copyright holders.
Now the same is being done to openai. Well, too bad.
OpenAI and other LLMs scraping the internet is probably covered under fair use. DeepSeek’s violation of OpenAI’s terms is pretty clearly a violation of their terms and not legal.
Here is a new thing you learn today, ToS are not laws, you can ignore any ToS and at worst the company might close your account.
Meanwhile, their entire training corpus was the result of scraping the intellectual property and copyrighted materials of THE ENTIRE PUBLIC INTERNET.
Woe is them to be sure.
Now they are scrambling against open-source after their disastrous operator demonstration and using this deep research demo as cover. Nothing that Google or Perplexity could not already do themselves.
By the end of them month, this feature is going be added by a bunch of other open-source projects and this feature won't be as interesting very quickly.
Of course. It is in response to their disastrous operator demo which did not justify the $200 per month ChatGPT Pro subscription on top of the release of DeepSeek to make matters worse for them.
> This has nothing to do with open weight models like DeepSeek (note: DeepSeek, Llama, etc are NOT open source).
It obviously does. Even before they rushed this presentation, they made o3-mini available for ChatGPT free users so it in direct response to DeepSeek.
> This feature doesn’t just require the research on the model but also enormous compute. Plus anyone using such a feature for real work is not going to be using DeepSeek or whatever, but a product with trustworthy practices and guarantees.
Nothing that Perplexity + DeepSeek-R1 can already do.
So what is your point?
Nonetheless, I don't think this is even something that can easily be benchmarked. I'd recommend you take a look at aider [1], and consider how I drew similarities between it and what's presented here.
Has ClosedAI presented any benchmarks / evaluation protocols?
[1] https://aider.chat/
Effectiveness in this task environment is well beyond the specific model involved, no? Plus they'd be fools (IMHO) to only use one size of model for each step in a research task -- sure, o3 might be an advantage when synthesizing a final answer or choosing between conflicting sources, but there are many, many steps required to get to that point.
I wonder how much of an impact our being still so early in the productization phase of this all is. Like it takes a ton of work and training and coordination to get multiple models synced up into an offering and I think the companies are still optimizing for getting new ideas out there rather truly optimizing them.
https://news.ycombinator.com/item?id=42913575
OpenAI is very much in an existential crisis and their poor execution is not helping their cause. Operator or “deep research” should be able to assume the role of a Pro user, run a quick test, and reliably report on whether this is working before the press release right?
It's the only tool/system (I won't call it an LLM) in their released benchmarks that has access to tools and the web. So, I'd wager the performance gains are strictly due to that.
If an LLM (o3) is too expensive to be released to the public, why would you use it in a tool that has to make hundreds of inference calls to it to answer a single question? You'd use a much cheaper model. Most likely o3-mini or o1-mini combined with o4-mini for some tasks.
The same reason a lot of people switched to GPT-4 when it came out even though it was much more expensive than 3 - doesn't matter how cheap it is if it isn't good enough/much worse.
What makes you believe that?
> Powered by a version of the upcoming OpenAI o3 model that’s optimized for web browsing and data analysis, it leverages reasoning to search, interpret, and analyze massive amounts of text, images, and PDFs on the internet, pivoting as needed in reaction to information it encounters.
If that's what you're referring to, then it doesn't seem that "explicit" to me. For example, how do we know that it doesn't use less thinking than o3-mini? Google's version of deep research uses their "not cutting edge version" 1.5 model, after all. Are you referring to something else?
I highly doubt that the concerns of internal political commissars were holding up this particular openai release.
> when Google released Gemini to have a product in the space.
Bard preceded Gemini.
I want to understand does this mean it's ethical for anyone to create a research AI tool that will go through arXiv and related GitHub repo and use it to solve problems, implement ideas like cursor.
Though, the jump for Gaia relative to SOTA is relatively not that high. Especially given that this is o3
Different pre-cooked prompts and filters don’t really amount to new products either, despite them being marketed as such. It’s like adobe treating each tool in photoshop as its own product.
> In Greek mythology, who was Jason's maternal great-grandfather?
https://www.google.com/search?q=In+Greek+mythology%2C+who+wa...
ideally a model would be able to answer this accurately and completely.
For example, Jason's mother was Philonis, daughter of Mestra, daughter of Daedalion, son of Hesporos. So Jason's maternal great-grandfather was Hesporos.
pass rate really only matters in context of the difficulty of the tasks
I mean I too can complain that my iPhone doesn’t automatically screen out spammers and send my mom flowers on Mother’s Day.
Pixel phone launched in 2016.
And eyeballing the benchmarks, it'll probably reach a >50% rate per query by the end of the year. Seems to double every model or two.
The set of natural transformations between two functors F,G :C→DF,G:C→D can be expressed as the end Nat(F,G)≅∫AHomD(F(A),G(A)). Nat(F,G)≅∫A HomD (F(A),G(A)).
Define set of natural cotransformations from FF to GG to be the coend CoNat(F,G)≅∫AHomD(F(A),G(A)). CoNat(F,G)≅∫AHomD (F(A),G(A)).
Let: - F=B∙(Σ4)∗/F=B∙ (Σ4 )∗/ be the under ∞∞-category of the nerve of the delooping of the symmetric group Σ4Σ4 on 4 letters under the unique 00-simplex ∗∗ of B∙Σ4B∙ Σ4 . - G=B∙(Σ7)∗/G=B∙ (Σ7 )∗/ be the under ∞∞-category nerve of the delooping of the symmetric group Σ7Σ7 on 7 letters under the unique 00-simplex ∗∗ of B∙Σ7B∙ Σ7 .
How many natural cotransformations are there between FF and GG?
Also I'm curious as to what percentage of the questions in this benchmark are of this type / difficulty, vs the seemingly much easier example of "In Greek mythology, who was Jason's maternal great-grandfather?".
I'd imagine the latter is much easier for an LLM, and almost trivial for any LLM with access to external sources (such as deep research).
https://blog.google/products/gemini/google-gemini-deep-resea...
They do say "Currently, deep research can access the open web...", so maybe "open" there implies something significant. Like, "websites that have agreements with OpenAI and/or do not enforce norobot policies".
I wouldn’t even be surprised if a law is passed requiring sites to provide equal access to humans whether accessed directly or via these models.
It’s too important an innovation to stall, especially considering the US’s competitors (China) won’t respect robots.txt either.
Amazon listings are blocked from google shopping and other price comparison sites.
I see Amazon results there all the time. 3 of the visible 8 sponsored results are Amazon, in the non-sponsored results an Amazon listing is either first or second in every category.
How would you know its a crawler?
Agents that can search the internet exist for a while now and have been essentially solved and happily used in platforms like Perplexity.
It's really "meh", very far from revolutionary.
Keep in mind this company is trying to convince everybody they need 500B USD now (through the Stargate project).
To say this is trivial is like saying the one shot ai prompted twitter clone is the same thing as twitter.
Peak HN indeed.
Without internet: 10%
With internet: 23%
In addition:
> We found that the ground-truth answers for one dataset were widely leaked online
in very small letters, and they blocked these URLs at runtime but not training time.
It's not bad, but not revolutionary at all compared to the leap that was GPT-2 from GPT-3, or GPT-4o to DeepSeek-R1
Again: the assertion was yours, so let us know the results of your own work.
Few people care or even know about various advances in various specialized fields. It's enough if AI simply seeps into various applications in boring and non-flashy ways for it to have significant effects that will affect a wider range of people, whether they get hyped by the news announcements or not. Jobs etc.
An analogy: the Internet as such is not very exciting nowadays, certainly not in the way it was exciting in the 90s with all the news segments about surfing the information superhighway or whatever. There was a lot of buzz around the web, but then it got normalized. It didn't disappear, it just got taken for granted. No average person got excited around HTML5 or IPv6. It just chugs along in the background. AI will similarly simply build into the fabric of how things get done. Sometimes visibly to the average person, sometimes just behind the scenes.
Synthesize? Seems like the wrong word -- I think they would want to say something like, "analyze, and synthesize useful outputs from hundreds of online sources"..
**with browsing + python tools
Maybe we have different definitions of scaling?
Anyone who's done any kind of substantial document research knows that it's a NIGHTMARE of chasing loose ends & citogenesis.
Trusting an LLM to critically evaluate every source and to be deeply suspect of any unproven claim is a ridiculous thing to do. These are not hard reasoning systems, they are probabilistic language models.
This is like arguing an Airbus cannot possibly fly because it is 165 tonnes of aluminum, steel and plastic.
The proof is in the fact that it flies, not what it is constructed from.
And LLMs do not.
> "But it looks like reasoning to me"
My condolences. You should go see a doctor about your inability to count the number of 'R's in a word.
CoT reasoning is reasoning, whether you like it or not. If you don't understand that, it means the models are already smarter than you.