Its interesting because I didn't think it was, but then reading the NVIDIA approach, this kind of loop plus generating a program to explain things. Maybe that is AGI? I don't know, but it seems like an additional layer that maybe is a fundamental shift in capabilities (kind of like reinforcement learning and COT was).
The capability for a man-made machine machine to solve problems drawn from arbitrary problem domains without domain specific pre-training: Artificial. General. Intelligence. AGI. It is what the term of art means.
You could probably score 100% on ARC 3 if you were motivated enough. I find some of the current problems to be kind of like Chess - mechanically simple, and ~solvable, but it's difficult to force myself to think at length about a monotonous and meaningless problem. The machines do have an advantage on the "energy" front; they've become almost psychotically persistent (and don't get tired after too many prompts).
Anyway yes I think we've had AGI for a while now, even if the GI doesn't quite match up with what we expect from a human.
I thought ARC-AGI-3 was explicitly a test of raw model performance excluding the harness? Adding the harness back in doesn't tell us anything new. We've known that agents are capable of long horizon reasoning with sufficient harnesses. GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?
I don't think GPT-4 was ever used for beating pokemon with or without a harness. Successful attempts include Gemini 2.5 and Opus 4.7 both using relatively advanced custom harnesses that give access to game memory, notes systems, and one-off hacks to get around parts of the game the model gets stuck on. More recently, Fable 5 beat FireRed with a _very_ minimal harness (screenshots and button inputs). That's the only example I know of but that is a very sophisticated and very expensive model compared to GPT-4.
Most of this doesn't discredit your overall point, though.
GPT-o3 was the first OpenAI one to beat Red. The harness used by GPT Plays Pokemon is the most featureful one of the main competitors (GPT, Claude, Gemini), IIRC.
> GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?
GPT-4 was decidedly not capable of beating Pokemon 18 months ago. I doubt it would be able to complete a single level. I don't think people realize how large the advances in model capabilities have been. GPT-4 in a modern harness is absolutely horrendous compared to modern models.
Yes. Presumably you're referring to my use of the word "level". I mean here basically every "level" as denoted by the order of locations and places on the town map that you get (which is usually +- some other locations how game runners refer to different sections of the game).
None of the tweets, nor the press release, seems to mention how long time it actually took E2E to complete the evaluation, but they do mention it took "12% fewer actions" compared to just Opus 5 without AVO. Feels a bit suspicious they don't break down the timing involved, looking at the diagram from the press release, it gives the impression there is a lot of machinery here, and given they claim fewer actions, each action must be more carefully considered, doesn't it?
Curious to read more about it though, seems the paper for it is here: https://arxiv.org/pdf/2603.24517, I'm not sure I understand if it's better than just Codex with a /goal, as they talk about "can discover performance-critical micro-architectural optimizations" but leave Codex alone for a day or two and you'll get the same results without doing "additional autonomous adaptation" at all.
I wonder if these benchmarks swap words, meaning and more because you might as well be benchmaxxing for specific words. I notice a lot of recurring just structural sentences coming back in smaller LLM models where they're fit for a specific task which is fine because most of the work we do is repetitive and there are patterns to learn but they should be word agnostic which I wonder if LLM can really fix.
Can we please prioritize links to the papers, github repos, press releases, or blog articles for these types of posts? I don't use Twitter and I don't think anyone else should either.
There you will find the extremely important qualifier it's the public set, not the private set (with the risk of overfitting, ie the results not repeating when submitted to be run in competition), and the detail that this is essentially a harness added to Opus 5, not Nvidia's own models.
> would we be convinced that we have achieved AGI?
No.
AGI is impossible without a biological pineal gland. The pineal gland is the seat of consciousness and without one, any AI is merely a pattern matcher, not intelligent.
The approach feels asymmetric though now, since by the time AI reaches a point where humans cannot come up with any task that's easy for humans but hard for machines, it (AI) may be able to do some tasks that an average human can't, or do some much better than an average human.
AVO’s paper author (ex-NVIDIAN) is here. This work was done half a year ago for GPU kernels, and the same approach has now been applied to ARC-AGI-3. I think people are still underestimating the evolution progress; e.g., recently we made a self-improving evolution harness that generated an entire inference stack and is better than SGLang/vLLM on various tasks: https://int21.ai/insights/addressing-the-inference-bottlenec...
43 comments
[ 1.8 ms ] story [ 42.6 ms ] threadAnyway yes I think we've had AGI for a while now, even if the GI doesn't quite match up with what we expect from a human.
Yey, AGI is finally solved.
(0) https://arcprize.org/arc-agi/3
100% is some "RHAE" metric: its performance of median human first time seeing those problem.
The next year is going to be wild folks
https://arxiv.org/html/2603.24517v1
Most of this doesn't discredit your overall point, though.
Community maintained spreadsheet of the runs: https://docs.google.com/spreadsheets/d/e/2PACX-1vQDvsy5Dt_-P...
GPT-4 was decidedly not capable of beating Pokemon 18 months ago. I doubt it would be able to complete a single level. I don't think people realize how large the advances in model capabilities have been. GPT-4 in a modern harness is absolutely horrendous compared to modern models.
Have you ever played pokemon?
Curious to read more about it though, seems the paper for it is here: https://arxiv.org/pdf/2603.24517, I'm not sure I understand if it's better than just Codex with a /goal, as they talk about "can discover performance-critical micro-architectural optimizations" but leave Codex alone for a day or two and you'll get the same results without doing "additional autonomous adaptation" at all.
https://developer.nvidia.com/blog/nvidia-avo-reaches-100-on-...
There you will find the extremely important qualifier it's the public set, not the private set (with the risk of overfitting, ie the results not repeating when submitted to be run in competition), and the detail that this is essentially a harness added to Opus 5, not Nvidia's own models.
Obviously still impressive, you would think.
If not, is the benchmark just incorrectly named? (I personally think so.)
PS: I follow Wikipedia's definition for AGI (https://en.wikipedia.org/wiki/Artificial_general_intelligenc...), which also talks about some tests. However, I distinguish it from Strong AI.
No.
AGI is impossible without a biological pineal gland. The pineal gland is the seat of consciousness and without one, any AI is merely a pattern matcher, not intelligent.
(1) Consciousness is mandatory for intelligence.
(2) 'AGI' is no different than just 'intelligence'.
(3) Consciousness resides in pineal gland.
(4) Biology is mandatory for consciousness/AGI.
I cannot claim these to be wrong, however, have no reason to believe in any.
We do seem to agree though that the benchmark is incorrectly named.
no? once 3 is solved, we would come up with 4. then 5, 6...
it will be AGI when we cannot come up with a task easy for human but hard for machines. thet's the whole point.
The approach feels asymmetric though now, since by the time AI reaches a point where humans cannot come up with any task that's easy for humans but hard for machines, it (AI) may be able to do some tasks that an average human can't, or do some much better than an average human.
we are past that point for a while now (chess, go)!
the important part is the G in AGI: General.