I have breach parser that i had written to parse through over 3 billion rows of compressed data (by parsing i simply mean searching for a particular substring), I’ve tried multiple LLMs to make it faster (currently it does so in <45 seconds on an M3 pro mac) none have been able to do that yet.
I'm totally not surprised by this. It would be strange if, at this point, we couldn't find anything that a specialized tool could do better.
But rest assured that the LLM folks are watching, and learning from this, so the issue will probably be resolved in the next version. Of course without thanking/crediting the author of the article.
hopefully i am not sounding too pedantic in mentioning this. But LLMS are still deterministic if you're using the same prompt and seed , temp , (sometimes requires the same hardware even) etc.
Are they? AFAIK, the “etc” includes using hardware that produces the same results for a given input every time. Once you start to multi-thread/multi-process in combination with floating point math, that can be hard to accomplish.
For example, the result of summing a stream of floats depends on the order the floats arrive in, and that order can change depending on what’s in your CPU cache when you start a computation, on whether something else running on your system such as a timer interrupt evicts something from cache during a computation, etc.
If you’re running on your GPU, even if the behavior of your GPU is 100% predictable (I wouldn’t know of that’s true on modern hardware, but my guess is it isn’t) anything that also uses the GPU can change things.
Big takeaway for me: the win isn’t better prompts, it’s semantic guarantees. By proving at the bytecode level that the pixel loop is side-effect-free, you can safely split it into long-lived workers and use an order-preserving queue. It's an aggressive transform copilots won’t attempt because they can’t verify invariants. That difference in guarantees (deterministic analysis vs. probabilistic suggestion) explains the 2× gap more than anything else.
Anything built to purpose (by a competent dev) will usually beat out a general purpose tool. I remember burntsushi being surprised that my purpose-built unicode segmentation code so dramatically outperformed the unicode segmentation he had in bytestring which was based on regular expressions, but personally I would be surprised if it were any different.
Wouldn't the real test be to run all the code through the bytecode optimizer and see who's faster then?
I mean, this is sort of the same as testing the LLM output against the -O3 compiler optimization flag while compiling their programs with no optimizations. Actually, if I read TFA correctly, this is exactly what they're doing, am I wrong?
Or maybe I am wrong and they're testing their VM against compiled code, dunno?
It's unsurprising to me that the author got this outcome. However, instead of just prompting to optimize the code, I suspect they would have gotten much stronger results from the models if they'd prompted them to write an optimizer.
It is very likely that LLM will be able to plagiarize https://ispc.github.io/example.html and steal ready to use optimal code for Mandelbrot, while specialized optimizers are locked within a domain. Not even speaking of the fact, that author is producing graphics: the task should be solved on the GPU in the first place.
I certainly expect a human to do better here but if you wanna show it, giving a one line prompt to 2nd best LLMs to one-shot it isn't really the way to do it. Use Opus and o3, and give it to an agent that can measure things and try more than once.
This is cool. I wonder if your VM could work in conjunction with an LLM? Have you tried making this optimizer available as an MCP, or maybe some of the calculated invariants could be exposed as well?
15 comments
[ 4.3 ms ] story [ 39.8 ms ] threadhttps://github.com/44za12/breach-parse-rs
Feel free to drop ideas if any.
But rest assured that the LLM folks are watching, and learning from this, so the issue will probably be resolved in the next version. Of course without thanking/crediting the author of the article.
For example, the result of summing a stream of floats depends on the order the floats arrive in, and that order can change depending on what’s in your CPU cache when you start a computation, on whether something else running on your system such as a timer interrupt evicts something from cache during a computation, etc.
If you’re running on your GPU, even if the behavior of your GPU is 100% predictable (I wouldn’t know of that’s true on modern hardware, but my guess is it isn’t) anything that also uses the GPU can change things.
I suspect theses bench were run on the default model, ChatGPT 4o, which is now more than a year old.
I mean, this is sort of the same as testing the LLM output against the -O3 compiler optimization flag while compiling their programs with no optimizations. Actually, if I read TFA correctly, this is exactly what they're doing, am I wrong?
Or maybe I am wrong and they're testing their VM against compiled code, dunno?