So far in this benchmark we based the tasks on a couple of open-source projects (like curl, jq, GNU Coreutils).
Even on those "simple" projects we managed to make the tasks difficult - Claude Opus 4.1 was the only one to correctly cross-compile curl for arm64 (+ make it statically-linked) [1].
In the future we'd like to test it with projects like FFmpeg or chromium - those should be much more difficult.
LGTM! I'm sure it comes with a correctness proof, too!
The newer blog posts appear to scan forums like this one for objections ("AI" does not work for legacy code bases) and then create custom "benchmarks" for their sales people to point to if they encounter these objections.
> Our toughest challenges include cross-compiling to Windows or ARM64 and resurrecting 22-year-old source code from 2003 on modern systems. Some agents needed 135 commands and 15 minutes just to produce a single working binary.
I found that "just" there to be so funny in terms of how far the goal posts moved over these last few years (as TFA does mention). I personally am certain that it would have taken me significantly longer than that to do it myself.
I’ve been doing this a lot! AI seems to really excel at setting up compiler boilerplate/minor modifications for new arch. I made a simple cpu information utility work on HP PA-RISC and Sparc64 :)
the libs in the bench don’t really have an external deps. will be much more interesting to see the results with ffmpeg, Qt, etc. The original source releases from any repo here would also be great candidates: https://github.com/id-software
I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook.
Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was the best investment of non-time I could do... It definitely felt magical. Claude would tweak headers, `make` it, try to run it, and apply more fixes based on the errors it got back.
Now that I think of it, I regret not opening an issue/PR with its findings...!
(((I then went on to make more vibe-changes to the Doom code and made a video out of those which went semi-viral, which I will now unashamedly plug [1])))
I mean this in the nicest possible way because you were just messing around on a fun thing, but...
I feel like there's a real metaphor here. 86+ people did work over two decades to maintain a cross-platform codebase and that "definitely deserves to be commended", but what "definitely felt magical" was Claude bumbling through header tweaks from compilation errors until the project compiled. And in the end what has AI wrought? A viral video but not anything to give back to the original project. Really there are multiple layers here :)
For C projects, the task should be passing the full test suite with at least address-sanitizer enabled. Amusing how some would discourage fellow human from using a programming language because of its unsafeness or undefined behavior, yet AI doing unaudited source modification on the same language is encouraged.
Though this is more "LLM uses a variety of open source tools and compilers to compile source," I do wonder about whether there will eventually be a role for transformers in compiling code.
I've mentioned this before, but "sufficiently smart compiler" would be the dream here. Start with high level code or pseudo code, end up with something optimized.
I might start to accept this LLM stuff when it can directly compile programs, i.e. not spit out a compiler command but take in source code and output linked object code in an executable format via token inference. And have it be correct.
Then, I'd start to trust in its ability to manage context and reliably work through complex tasks.
"I'll try this newfangled steel constructions if they actually forge each rebar on site".
You are saying that you'd trust the new and unproven technology more if it didn't rely on old and proven technology and instead reinvented everything from scratch. That's a somewhat illogical take.
25 comments
[ 5.1 ms ] story [ 88.2 ms ] threadSo far in this benchmark we based the tasks on a couple of open-source projects (like curl, jq, GNU Coreutils).
Even on those "simple" projects we managed to make the tasks difficult - Claude Opus 4.1 was the only one to correctly cross-compile curl for arm64 (+ make it statically-linked) [1].
In the future we'd like to test it with projects like FFmpeg or chromium - those should be much more difficult.
[1] https://www.compilebench.com/curl-ssl-arm64-static/
Now if it could fix React Native builds after package upgrades I'd be impressed...
The newer blog posts appear to scan forums like this one for objections ("AI" does not work for legacy code bases) and then create custom "benchmarks" for their sales people to point to if they encounter these objections.
I found that "just" there to be so funny in terms of how far the goal posts moved over these last few years (as TFA does mention). I personally am certain that it would have taken me significantly longer than that to do it myself.
Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was the best investment of non-time I could do... It definitely felt magical. Claude would tweak headers, `make` it, try to run it, and apply more fixes based on the errors it got back.
Now that I think of it, I regret not opening an issue/PR with its findings...!
(((I then went on to make more vibe-changes to the Doom code and made a video out of those which went semi-viral, which I will now unashamedly plug [1])))
[0] https://github.com/chocolate-doom/chocolate-doom
[1] https://www.youtube.com/watch?v=LcnBXtttF28
I feel like there's a real metaphor here. 86+ people did work over two decades to maintain a cross-platform codebase and that "definitely deserves to be commended", but what "definitely felt magical" was Claude bumbling through header tweaks from compilation errors until the project compiled. And in the end what has AI wrought? A viral video but not anything to give back to the original project. Really there are multiple layers here :)
The point was to get it running, not solve world peace. Without AI, the problem might not have been tackled at all.
https://buildd.debian.org/status/package.php?p=chocolate-doo...
$ git clone --depth=1 https://github.com/chocolate-doom/chocolate-doom
$ cd c*doom; ls
Ok, there is a CMakeFile.txt, so it's probably a cmake project, so:
$ cmake .
Ok, that seems to work, but three libraries are missing, SDL2_Mixer, SDL2_Net and FluidSynth, so lets install them:
$ sudo apt install libsdl2-mixer-dev libsdl2-net-dev libfluidsynth-dev
Let's try again:
$ cmake .
Works, so now for compiling:
$ cmake --build . -j $(nproc)
Build completed in a few seconds first try.
I've mentioned this before, but "sufficiently smart compiler" would be the dream here. Start with high level code or pseudo code, end up with something optimized.
Then, I'd start to trust in its ability to manage context and reliably work through complex tasks.
You are saying that you'd trust the new and unproven technology more if it didn't rely on old and proven technology and instead reinvented everything from scratch. That's a somewhat illogical take.