I use GPT about daily now and have noticed a funny thing, which is to be expected really.
I can ask it to help me code for example a physics engine, so we're talking really hard and intricate code and it'll come up with some amazing optimizations, we're talking (recent) research paper level implementations.
Then I ask it to work on something that's relatively trivial, let's say we need a flowfield. It'll think and reason about it just as well as in the first example, but then it'll start spitting out a lot of subpar code. Its error rate will increase 10x while the global cohesiveness of the produced code will be substantially worse.
As to why that's happening, maybe its being trained on a lot more as well as worse examples of the second, whereas the first is relatively "pure".
These programming competitions are pretty much the same thing in my opinion. For us humans its a hard challenge, but in general they're asking the same-ish questions, just in different formats. They should add some questions where the participant has to invent something new, or alternatively use two or more existing concepts in a totally novel fashion.
3 comments
[ 5.1 ms ] story [ 21.6 ms ] thread- gold at IMO
- gold at IoI
- beat 9/10 humans in atcode heuristics
- longer context, better models, routing calls to cheaper models, 4-6x cheaper inference for 90% of the top models capabilities
- longer agentic sessions while being coherent/solving tasks (30-90min)
Yet every other post here and there are about "bubble this", "winter that", "plateauing this", "wall that"...
Are we in the denial stage, or bargaining stage? Can't quite tell...
I can ask it to help me code for example a physics engine, so we're talking really hard and intricate code and it'll come up with some amazing optimizations, we're talking (recent) research paper level implementations.
Then I ask it to work on something that's relatively trivial, let's say we need a flowfield. It'll think and reason about it just as well as in the first example, but then it'll start spitting out a lot of subpar code. Its error rate will increase 10x while the global cohesiveness of the produced code will be substantially worse.
As to why that's happening, maybe its being trained on a lot more as well as worse examples of the second, whereas the first is relatively "pure".
These programming competitions are pretty much the same thing in my opinion. For us humans its a hard challenge, but in general they're asking the same-ish questions, just in different formats. They should add some questions where the participant has to invent something new, or alternatively use two or more existing concepts in a totally novel fashion.