Rust has its place but python is default for this kind of thing. By doing it in rust, they have now changed two things: the implementation of the transformer, and this new model.
Rust matters because it means you can actually deploy it without going insane. I wonder if the anti-Rust zealots have actually ever used pip, especially on Windows.
He could have said "... not written in Python" - would that have been acceptable?
I would say the ability to actually use software is kind of relevant to said software. There's definitely software I wanted to use but gave up on because it was just impossible to actually install. Hell I just spent half of today coaxing Claude to fix some Kotlin software so that it worked with the version of Java I have. Let me tell you that would have taken a good few weeks without AI, even if I was unfortunate enough to be an expert in Maven and Gradle. "Written in Rust" would have saved me a huge amount of (pre-AI) time for that software.
OP, you should not have written this in Rust. It should be in PyTorch, which is by far the most popular. We can’t tell if this architecture is good or whether there is a problem in your implementation.
You can test the whole thing for free on a GPU with Google Colab. Test both the transformer and your new architecture on a larger dataset. Something that maxes out the GPU for an hour each run.
Also, the readme mentions keeping the same optimizer schedule which sounds nice at first but they are completely different architectures. The loss is high on the transformer, did you try raising the learning rate on it?
In general I’m interested in parameter efficient architectures. I don’t think transformers are optimal, and indeed many improvements have been made to vanilla transformers. But if you have an idea for something better you need to show it.
One criticism is that you have to install the same package, torch, but from different Python indexes in order to install the cpu version or gpu version, on Linux. On windows, `pip install torch` gets you the cpu version. On linux, that gets you a ton of Nvidia extras that take a lot of space.
GPU support should really be a optional extra eg `torch[gpu]` or `torch[nvidia]`.
I would argue that we need a `torch[stubs]` package (or similar) that doesn't install any form of libtorch. The CPU version of libtorch is 160MB compressed and 400MB extracted.
I don't think it's caused by PyTorch on its own, but every AI-related Python project I try out locally manages to depend on a version of PyTorch that isn't in my disk cache yet. Having to download a gigabyte of dependencies for every project gets tiresome.
The Rust compile cycle will probably generate a gigabyte of files locally as well, but at least they can be `rm`'d out of `target/` once it's done.
The entire Python ecosystem is horrible and shouldn’t have been as falsely boosted by institutions as it was in the 2010s.
Yes, it got less bad with 3.8 or whatever version added type annotations. But making so much of ML depend on Python has made it distasteful to a lot of devs who would otherwise have contributed more to it.
We really need to move all AI/ML research off PyTorch to Candle or… just anything that isn’t Python or another old-gen, broken language like it.
Well, to be a bit pedantic, it's called Machine Learning, not Machine Programming. But also learning is just the other side of the skills transfer coin (that teaching, or programming, is on), so it's more a matter of perspective. And then, actually there has always been - traditional - programming in ML: the data has to be prepped, architecture created, etc.
Edit: the claimed reason of "updates" also seems to not exist in code. If the author is going to use LLM for this, the very least they can do is to ask it to check properly before publishing.
I have read somewhere that Transformer architecture has a quadratic cost (which explains the high costs associated with LLMs and the difficulty for constant improvement without state size pockets).
For what I understand PSSA belongs to a line of research for LLMs with scalable architecture because you don't need the full KV in memory to generate a single token:
I’m so tired of the “X written in Rust” posts on HN. Seriously, who cares what language you used? Especially nowadays, when LLMs are writing much of the code anyway.
32 comments
[ 3.9 ms ] story [ 22.6 ms ] threadWhy did you choose Rust? Why does that matter?
Nothing wrong with Rust. Lots wrong with the bandwagon that “in Rust” somehow adds value.
He could have said "... not written in Python" - would that have been acceptable?
A non transformer language model written from scratch is more interesting than
“I did X in Python/Rust/Flavour-of-the-month-thing”
There are a few layers of irony here, but see uv.
- https://docs.astral.sh/uv/
Edit: plausibly there’s no human in loop here at all, in which case fair play, I guess.
You can test the whole thing for free on a GPU with Google Colab. Test both the transformer and your new architecture on a larger dataset. Something that maxes out the GPU for an hour each run.
Also, the readme mentions keeping the same optimizer schedule which sounds nice at first but they are completely different architectures. The loss is high on the transformer, did you try raising the learning rate on it?
In general I’m interested in parameter efficient architectures. I don’t think transformers are optimal, and indeed many improvements have been made to vanilla transformers. But if you have an idea for something better you need to show it.
That said I don't know what's wrong with using a Rust AI framework like Candle.
GPU support should really be a optional extra eg `torch[gpu]` or `torch[nvidia]`.
The Rust compile cycle will probably generate a gigabyte of files locally as well, but at least they can be `rm`'d out of `target/` once it's done.
We really need to move all AI/ML research off PyTorch to Candle or… just anything that isn’t Python or another old-gen, broken language like it.
Edit: the claimed reason of "updates" also seems to not exist in code. If the author is going to use LLM for this, the very least they can do is to ask it to check properly before publishing.
> The implementation language is a detail, and a Python port is welcome.
I guess the post title could have dropped "in rust"
Weird turn of phrase very typical of AI.
For what I understand PSSA belongs to a line of research for LLMs with scalable architecture because you don't need the full KV in memory to generate a single token:
https://arxiv.org/abs/2503.00392
https://papers.nips.cc/paper_files/paper/2023/hash/6ceefa7b1...