So this year SotA models have gotten gold at IMO, IoI, ICPC and beat 9/10 humans in that atcoder thing that tested optimisation problems. Yet the most reposted headlines and rethoric is "wall this", "stangation that", "model regression", "winter", "bubble", doom etc.
People pattern match with a very low-resolution view of the world (web3/crypt/nfts were a bubble because there was hype, so there must be a bubble since AI is hyped! I am very smart) and fail to reckon with the very real ways in which AI is fundamentally different.
Also I think people do understand just how big of a deal AI is but don't want to accept it or at least publicly admit it because they are scared for a number of reasons, least of all being human irrelevance.
My response simply is that performance in coding competitions such as ICPC is a very different skillset than what is required in a regular software engineering job. GPT-5 still cannot make sense of my company's legacy codebase even if asked to do the most basic tasks that a new grad out of college can figure out in a day or two. I recently asked it to fix a broken test (I had messed with it by changing one single assertion) and it declared "success" by deleting the entire test suite.
People are having a tough time coping with what the near future holds for them. It is quite hard for a typical person to imagine how disruptive and exponential coming world events are like Covid showed.
Two days ago I talked to someone in water management about data centers. One of the big players wanted to build a center that consumed as much water as a medium town in semi arid bushland. A week before that it was a substation which would take a decade to source the transformers for. Before that it was buying closed down coal power plants.
I don't know if we're in a bubble for model capabilities, but we are definitely hitting the wall in terms of what the rest of the physical economy can provide.
You can't undo 50 years of deffered maintenance in three months.
the wall is how we need to throw trillions of hardware to do "breakthroughs", LLM uses the same algorthm from last few years. We need a new algorthm breakthrough otherwise buying hardware to increase intelligence isn't scalable.
Where these competitions differ from real life is that evaluating a solution is much easier than generating a solution. We're at the point where AI can do a pretty good job of evaluating solutions, which is definitely an impressive step. We're also at the point where AI can generate candidate solutions to problems like these, which is also impressive. But the degree to which that translates to practical utility is questionable.
The sibling commenter compared this to go, but we could go back to comparing it with chess. Deepblue didn't play chess the way a human did. It deployed massive amounts of compute, to look at as many future board states as possible, in order to see which move would work out. People who said that a computer that could play chess as well as a human would be as smart as a human ended up eating crow. These modern AIs are also not playing these competitions the way a human does. Comparing their intelligence to that of a humans is similarly fallacious.
Historically there has been a gap between the performance of AI in test environments vs the impact in the real world, and that makes people who have been through the cycle a few times cautious extrapolating.
In 2016 Geoffrey Hinton said vision models would put radiologists out of business within 5-10 years. 10 years on there is a shortage of Radiologists in the US and AI hasn't disrupted the industry.
The DARPA grand challenge for autonomous vehicles was won in 2006, 20 years on self driving cars still have limited deployment.
The real world is more complex than computer scientists apprecate.
"We used a custom AI that requires a small nuclear plant to be trained and function to beat three humans consuming 400 watts per day" isn't as impressive as it sounds
They apparently managed gold in the IOI as well. A result that was extremely surprising for me and causes me to rethink a lot of assumptions I have about current LLMs. Unfortunately there was very little transparency on how they managed those results and the only source was a Twitter post. I want to know if there was any third party oversight, what kind of compute they used, how much power what kind of models and how they were set up? In this case I see that DeepMind at least has a blog post, but as far as I can see it does not answer any of my questions.
I think this is huge news, and I cannot imagine anything other than models with this capability having a massive impact all over the world. It causes me to be more worried than excited, it is very hard to tell what this will lead which is probably what makes it scary for me.
However with so little transparency from these companies and extreme financial pressure to perform well in these contests, I have to be quite sceptical of how truthful these results are. If true I think it is really remarkable, but I really want some more solid proof before I change my worldview.
I went to ICPC's web pages, downloaded the first problem (problem A) and gave it to GPT-5, asking it for code to solve it (stating it was a problem from a recent competitive programming contest).
I think it's becoming clear that these mega AI corps are juggling with their models at inference time to produce unrealistically good results. By that it seems that they're just cranking up the compute beyond reasonable levels in order to gain PR points against each other.
The fact is most ordinary mortals never get access to a fraction of that kind of power, which explains the commonly reported issues with AI models failing to complete even rudimentary tasks. It's now turned into a whole marketing circus (maybe to justify these ludicrous billion-dollar valuations?).
The bleeding edge behind closed doors token burning monsters of 2023 are bad compared to the free LLMs we have now.
I believe it was Sundar in an interview with Lex who said that the reason they haven't developed another Ultra model is because by the time it is ready to launch, the flash and pro versions will have already made it redundant.
"It's now turned into a whole marketing circus (maybe to justify these ludicrous billion-dollar valuations?)."
Yes theres an entire ecosystem being built up around language models that has to stay afloat for another 5 years at least, to hope for a significant breakthrough.
More information on OpenAI's result (which seems better than DeepMind's) from the X thread:
> our OpenAI reasoning system got a perfect score of 12/12
> For 11 of the 12 problems, the system’s first answer was correct. For the hardest problem, it succeeded on the 9th submission. Notably, the best human team achieved 11/12.
> We had both GPT-5 and an experimental reasoning model generating solutions, and the experimental reasoning model selecting which solutions to submit. GPT-5 answered 11 correctly, and the last (and most difficult problem) was solved by the experimental reasoning model.
I'm assuming that "GPT-5" here is a version with the same model weights but higher compute limits than even GPT-5 Pro, with many instances working in parallel, and some specific scaffolding and prompts. Still, extremely impressive to outperform the best human team. The stat I'd really like to see is how much money it would cost to get this result using their API (with a realistic cost for the "experimental reasoning model").
I think in the future information will be more walled -- because AI companies are not paying anyone for that piece of information, and I encourage everyone to put their knowledge on their own website, and for each page, put up a few urls that humans won't be able to find (but can still click if he knows where to find), but can be crawled by AI, which link to pages containing falsified information (such as, oh the information on url blah is actually incorrect, here you can find the correct version, with all those explanations, blah blah -- but of course page blah is the only correct version).
Essentially, we need to poison AI in all possible ways, without impacting human reading. They either have to hire more humans to filter the information, or hire more humans to improve the crawlers.
Or we can simply stop sharing knowledge. I'm fine with it, TBF.
A database is good at leetcode, who would have thought. Give humans a database and they'll outperform your "AI" (which probably uses an extraordinary amount of graphics cards and electricity).
It is an idiotic benchmark, in line with the rest of the "AI" propaganda.
The best thing of the ICPC is the first C, which stands for "collegiate". It means that you get to solve a set of problems with three persons, but with only one computer.
This means that you have to be smart about who is going to spend time coding, thinking, or debugging. The time pressure is intense, and it really is a team sport.
It's also extra fun if one of the team members prefers a Dvorak keyboard layout and vi, and the others do not.
I wonder how three different AI vendors would cooperate. It would probably lift reinforcement learning to the next level.
ICPC = The International Collegiate Programming Contest. These are college level programmers, not elite competitive programmers.
Apparently Gemini solved one problem (running on who knows what kind of cluster) by burning 30 min of "thinking" time on it, and at a cost that Google have declined to provide.
According to one prior competition paricipant, writing in the comments section of this ArsClasica coverage, each year they include one "time sink" problem that smart humans will avoid until they have tackled everything else.
This would all seem to put a rather different spin on this. It's not a case of Google outwitting the worlds best programmers, but rather that by searching for solutions for 30 min on god knows what kind of cloud hardware, they were able to get something done that the college kids did not have time to complete, or deem worthwhile starting.
I've contemplated this a bit, and I think I have a bit of an unconventional take:
First, this is really impressive.
Second, with that out of the way, these models are not playing the same game as the human contestants, in at least two major regards. First, and quite obviously, they have massive amounts of compute power, which is kind of like giving a human team a week instead of five hours. But the models that are competing have absolutely massive memorization capacity, whereas the teams are allowed to bring a 25-page PDF with them and they need to manually transcribe anything from that PDF that they actually want to use in a submission.
I think that, if you gave me the ability to search the pre-contest Internet and a week to prepare my submissions, I would be kind of embarrassed if I didn't get gold, and I'd find the contest to be rather less interesting than I would find the real thing.
Whats the point? These models are still unreliable in every day work. And they're getting fat! For a moment, they were getting cheaper, but now they are only getting bigger and this is not going to be cheap in the future. The point is, what are we investing a trillion dollars in?
My understanding is that the way they do this is have some number of model instances generating solution proposals, and then another model which chooses which candidates to submit.
I haven't been able to find information on how many proposals were generated before a solution was chosen to submit. I'm curious to know whether this is "you can get ICPC gold medal performance with a handful of GPT-5 instances" or "you will drown yourself in API credit debt if you try this".
Here is the published 2025 ICPC World Finals problemset. The "Time limit: X seconds" printed on each ICPC World Finals problem is the maximum runtime your program is allowed. If any judged run of your program takes longer than that, the submission fails, even if other runs finish in time.
I wonder whether they allowed humans input for the AI besides the initial generic prompt? Could they provide guidance for the AI?
We all know that by this kind of problems, intuition/guiding principles to transform the problem is all you need. The human may not be fast enough or error-free to sample correctly the already restricted solution space, but machine can. And for them, it’s a huge advantage. So did they allow human input (as part of a centaur team!) input or not?
These AI teams often have one of the best (ex-) competitive programmers.
While very cool, this feels like another instance of the kind of thing that we already know they are good at: self-contained, perfectly-specified problems that can be done by humans in a short timespan (especially when a team of highly skilled engineers behind the model is wielding it). Yes, it's amazing that a computer can do this, consider what they could do 10 years ago to today, so on and so on - but I don't see this and go "holy shit", I see this and go "yep".
I wish they went into more detail about how exactly the interaction with the LLM works - I'm pretty sure there's significantly more to it than "drop the paper with the problems into a text box and hit go".
I briefly looked at a few of Gemini's solutions https://github.com/google-deepmind/gemini_icpc2025. What struck me was how Gemini finds clean ways to express an idea - perhaps because it knows a large set of tricks for each kind of sub-algorithm (within the larger algorithm). I am a former competitor and managed to reach world finals at Google CodeJam and Topcoder Open. It took me a lot of work to get there but I will gladly concede that Gemini is way better than my peak. I haven't competed in 15 years and have forgotten a lot of tricks but Gemini's code reminded me how quickly algorithms can get complicated sometimes without a bag of tricks.
There are parallels to tactics in chess - humans might miss them but a machine will not. And that can be a huge difference in a game or even in a software project.
46 comments
[ 2.7 ms ] story [ 55.8 ms ] threadAlso I think people do understand just how big of a deal AI is but don't want to accept it or at least publicly admit it because they are scared for a number of reasons, least of all being human irrelevance.
I don't know if we're in a bubble for model capabilities, but we are definitely hitting the wall in terms of what the rest of the physical economy can provide.
You can't undo 50 years of deffered maintenance in three months.
The sibling commenter compared this to go, but we could go back to comparing it with chess. Deepblue didn't play chess the way a human did. It deployed massive amounts of compute, to look at as many future board states as possible, in order to see which move would work out. People who said that a computer that could play chess as well as a human would be as smart as a human ended up eating crow. These modern AIs are also not playing these competitions the way a human does. Comparing their intelligence to that of a humans is similarly fallacious.
In 2016 Geoffrey Hinton said vision models would put radiologists out of business within 5-10 years. 10 years on there is a shortage of Radiologists in the US and AI hasn't disrupted the industry.
The DARPA grand challenge for autonomous vehicles was won in 2006, 20 years on self driving cars still have limited deployment.
The real world is more complex than computer scientists apprecate.
I think this is huge news, and I cannot imagine anything other than models with this capability having a massive impact all over the world. It causes me to be more worried than excited, it is very hard to tell what this will lead which is probably what makes it scary for me.
However with so little transparency from these companies and extreme financial pressure to perform well in these contests, I have to be quite sceptical of how truthful these results are. If true I think it is really remarkable, but I really want some more solid proof before I change my worldview.
It thought for 7m 53s and gave as reply
The fact is most ordinary mortals never get access to a fraction of that kind of power, which explains the commonly reported issues with AI models failing to complete even rudimentary tasks. It's now turned into a whole marketing circus (maybe to justify these ludicrous billion-dollar valuations?).
I believe it was Sundar in an interview with Lex who said that the reason they haven't developed another Ultra model is because by the time it is ready to launch, the flash and pro versions will have already made it redundant.
Yes theres an entire ecosystem being built up around language models that has to stay afloat for another 5 years at least, to hope for a significant breakthrough.
> our OpenAI reasoning system got a perfect score of 12/12
> For 11 of the 12 problems, the system’s first answer was correct. For the hardest problem, it succeeded on the 9th submission. Notably, the best human team achieved 11/12.
> We had both GPT-5 and an experimental reasoning model generating solutions, and the experimental reasoning model selecting which solutions to submit. GPT-5 answered 11 correctly, and the last (and most difficult problem) was solved by the experimental reasoning model.
I'm assuming that "GPT-5" here is a version with the same model weights but higher compute limits than even GPT-5 Pro, with many instances working in parallel, and some specific scaffolding and prompts. Still, extremely impressive to outperform the best human team. The stat I'd really like to see is how much money it would cost to get this result using their API (with a realistic cost for the "experimental reasoning model").
What's the judgement here? Was it within the allotted time, or just a "try as often as you need to"?
Google source post: https://deepmind.google/discover/blog/gemini-achieves-gold-l... (https://news.ycombinator.com/item?id=45278480)
OpenAI tweet: https://x.com/OpenAI/status/1968368133024231902 (https://news.ycombinator.com/item?id=45279514)
Nonetheless, I'm still questioning what's the cost and how long it would take for us to be able to access these models.
Still great work, but it's less useful if the cost is actually higher than hiring someone with the same level.
Essentially, we need to poison AI in all possible ways, without impacting human reading. They either have to hire more humans to filter the information, or hire more humans to improve the crawlers.
Or we can simply stop sharing knowledge. I'm fine with it, TBF.
It is an idiotic benchmark, in line with the rest of the "AI" propaganda.
This means that you have to be smart about who is going to spend time coding, thinking, or debugging. The time pressure is intense, and it really is a team sport.
It's also extra fun if one of the team members prefers a Dvorak keyboard layout and vi, and the others do not.
I wonder how three different AI vendors would cooperate. It would probably lift reinforcement learning to the next level.
Apparently Gemini solved one problem (running on who knows what kind of cluster) by burning 30 min of "thinking" time on it, and at a cost that Google have declined to provide.
According to one prior competition paricipant, writing in the comments section of this ArsClasica coverage, each year they include one "time sink" problem that smart humans will avoid until they have tackled everything else.
https://arstechnica.com/google/2025/09/google-gemini-earns-g...
This would all seem to put a rather different spin on this. It's not a case of Google outwitting the worlds best programmers, but rather that by searching for solutions for 30 min on god knows what kind of cloud hardware, they were able to get something done that the college kids did not have time to complete, or deem worthwhile starting.
ICPC finalists are very much in the world elite of competitive programmers.
First, this is really impressive.
Second, with that out of the way, these models are not playing the same game as the human contestants, in at least two major regards. First, and quite obviously, they have massive amounts of compute power, which is kind of like giving a human team a week instead of five hours. But the models that are competing have absolutely massive memorization capacity, whereas the teams are allowed to bring a 25-page PDF with them and they need to manually transcribe anything from that PDF that they actually want to use in a submission.
I think that, if you gave me the ability to search the pre-contest Internet and a week to prepare my submissions, I would be kind of embarrassed if I didn't get gold, and I'd find the contest to be rather less interesting than I would find the real thing.
I haven't been able to find information on how many proposals were generated before a solution was chosen to submit. I'm curious to know whether this is "you can get ICPC gold medal performance with a handful of GPT-5 instances" or "you will drown yourself in API credit debt if you try this".
Still extremely impressive either way.
- It's not a fair match, these models have more compute and memory than humans
- Contestants weren't really elite, they're just college level programmers, not the world's best
- This doesn't matter for the real world, competitive programming is very different from regular software engineering
- It's marketing, they're just cranking up the compute to unrealistic levels to gain PR points
- It's brute force, not intelligence
Here is the published 2025 ICPC World Finals problemset. The "Time limit: X seconds" printed on each ICPC World Finals problem is the maximum runtime your program is allowed. If any judged run of your program takes longer than that, the submission fails, even if other runs finish in time.
https://worldfinals.icpc.global/problems/2025/finals/problem...
We all know that by this kind of problems, intuition/guiding principles to transform the problem is all you need. The human may not be fast enough or error-free to sample correctly the already restricted solution space, but machine can. And for them, it’s a huge advantage. So did they allow human input (as part of a centaur team!) input or not?
These AI teams often have one of the best (ex-) competitive programmers.
I wish they went into more detail about how exactly the interaction with the LLM works - I'm pretty sure there's significantly more to it than "drop the paper with the problems into a text box and hit go".
There are parallels to tactics in chess - humans might miss them but a machine will not. And that can be a huge difference in a game or even in a software project.
EDIT: minor correction.