Ten hours is a decent amount of time, so I'm not too surprised the human won. LLMs don't really tend to improve the longer they get to chew on a problem (often the opposite in fact).
The LLM was probably getting nowhere trying to improve after the first few minutes.
I despise the company that competed, but I feel obligated to acknowledge that headline buries the lede that their bot got SECOND place, and their 2nd place was closer to first than 3rd was to 2nd.
Are the submissions available online without needing to become a member of AtCoder?
I want to see what these 'heuristic' solutions look like.
Is it just that the ai precomputed more states and shoved their solutions in as the 'heuristic' or did it come up with novel, more broad, heuristics? Did the human and ai solutions have overlapping heuristics?
Remember this is the worst AI will ever be from here on out. Models are only going to get better, faster, cheaper, more accessible and more easily deployable.
I think people need to realize that just because an AI model fails at one point, or some certain architecture has common failure modes, that billions of dollars are poured into correcting those failures and improving in every economically viable domain. Two years ago AI video looked like a garbled 140p nightmare, now it's higher quality video than all but professional production studios could make.
AI agents don't get tired. They don't need to sleep. They don't require sick days, parental leave, or PTO. They don't file lawsuits, they don't share company secrets, they don't disparage, deliberately sandbag to get extra free time, whine, burn out or go AWOL. The best AI model/employee is infinitely replicatable, and can share its knowledge with other agents perfectly and clone itself arbitrarily many times, and it doesn't have a clash of egos working with copies of itself, it just optimizes and is refit to accomplish whatever task its given.
All this means is that gradually the relative advantage of humans in any economically viable domain will predictably trend towards zero. We have to figure out now what that will mean for general human welfare, freedom and happiness, because barring extremely restrictive measures on AI development or voluntary cessation by all AI companies, AGI will arrive.
I'm bullish on specific areas improving (I'm sure you could selectively train an LLM on the latest Angular version to replace the majority of front-end devs given enough time and money, it's a limited problem space and a strongly opinionated framework after all), but for the most part enshittification is already starting to happen with the general models.
Nowadays even ChatGPT doesn't bother to even refer to the original question posed after a few responses, so you're left summarising a conversation and starting a new context to get anywhere.
So, yeah, I think we're very much into finding the equilibrium now. Cost vs scale. Exponential improvements won't be in the general LLMs.
I'm guessing he didn't have access to any LLMs while competing, but I think a "centaur" approach probably would have outperformed both "only human" and "only LLM" competitors.
Reading through the challenge, there's a lot of data modelling and test harness writing and ideating that an LLM could knock out fairly quickly, but would take even a competitive coder some time to write (even if just limited by typing speed).
That'd give the human more time to experiment with different approaches and test incremental improvements.
> All competitors, including OpenAI, were limited to identical hardware provided by AtCoder, ensuring a level playing field between human and AI contestants.
The power of this chosen hardware will very much determine how well the AI performs. Everyone receiving the same computer does not make the competition inherently fair.
It’s likely that human competitors would outperform the AI on hardware that is even a few years old.
18 comments
[ 4.8 ms ] story [ 40.2 ms ] threadThe LLM was probably getting nowhere trying to improve after the first few minutes.
Are the submissions available online without needing to become a member of AtCoder?
I want to see what these 'heuristic' solutions look like.
Is it just that the ai precomputed more states and shoved their solutions in as the 'heuristic' or did it come up with novel, more broad, heuristics? Did the human and ai solutions have overlapping heuristics?
https://en.wikipedia.org/wiki/John_Henry_(folklore)
First, there's a world coding championship?! Of course there is. There's a competition for anything these days.
Why is he exhausted?
> The 10-hour marathon left him "completely exhausted."
> ... noting he had little sleep while competing in several competitions across three days. "I'm completely exhausted. ... I'm barely alive."
oh! That's a lot.
> beating an advanced AI model from OpenAI ...
> On Wednesday, programmer Przemysław Dębiak (known as "Psyho"), a former OpenAI employee,
Interesting that he used to work there.
> Dębiak won 500,000 yen
JPY 500,000 -> USD 3367.20 -> EUR 2889.35
I'm guessing it's more about the clout than it is about the payment, because that's not a lot of money for the effort spent
I think people need to realize that just because an AI model fails at one point, or some certain architecture has common failure modes, that billions of dollars are poured into correcting those failures and improving in every economically viable domain. Two years ago AI video looked like a garbled 140p nightmare, now it's higher quality video than all but professional production studios could make.
AI agents don't get tired. They don't need to sleep. They don't require sick days, parental leave, or PTO. They don't file lawsuits, they don't share company secrets, they don't disparage, deliberately sandbag to get extra free time, whine, burn out or go AWOL. The best AI model/employee is infinitely replicatable, and can share its knowledge with other agents perfectly and clone itself arbitrarily many times, and it doesn't have a clash of egos working with copies of itself, it just optimizes and is refit to accomplish whatever task its given.
All this means is that gradually the relative advantage of humans in any economically viable domain will predictably trend towards zero. We have to figure out now what that will mean for general human welfare, freedom and happiness, because barring extremely restrictive measures on AI development or voluntary cessation by all AI companies, AGI will arrive.
This does not follow. Your argument, set in the 1950s, would be that cars keep getting faster, therefore they will reach light speed.
I'm bullish on specific areas improving (I'm sure you could selectively train an LLM on the latest Angular version to replace the majority of front-end devs given enough time and money, it's a limited problem space and a strongly opinionated framework after all), but for the most part enshittification is already starting to happen with the general models.
Nowadays even ChatGPT doesn't bother to even refer to the original question posed after a few responses, so you're left summarising a conversation and starting a new context to get anywhere.
So, yeah, I think we're very much into finding the equilibrium now. Cost vs scale. Exponential improvements won't be in the general LLMs.
Happy to be wrong on this one..
Reading through the challenge, there's a lot of data modelling and test harness writing and ideating that an LLM could knock out fairly quickly, but would take even a competitive coder some time to write (even if just limited by typing speed).
That'd give the human more time to experiment with different approaches and test incremental improvements.
The power of this chosen hardware will very much determine how well the AI performs. Everyone receiving the same computer does not make the competition inherently fair.
It’s likely that human competitors would outperform the AI on hardware that is even a few years old.