This doesn't mean anything. A project can implement things from scratch inefficiently but there might be other libraries the project can use instead of reimplementing.
I’m curious where you got your training data? I will look myself, but saw this and thought I’d ask. I have a CPU-first, no-backprop architecture that works very well on classification datasets. It can do single‑example incremental updates which might be useful for continuous learning. I made a toy demo to train on tiny.txt and it can predict next characters, but I’ve never tried to make an LLM before. I think my architecture might work well as an on-device assistant or for on-premises needs, but I want to work with it more before I embarrass myself. Any open-source LLM training datasets you would recommend?
The training data is contained inside main.rs; it looks like about 50 statements about general facts. Probably to keep training time down. If you go off-script things fall apart pretty quickly:
Enter prompt: hello
Model output: What are eclipses ? Assistant : Eclipses occur when one celestial body moves into the shadow of another </s>
Enter prompt: what are facts
Model output: eclipses Assistant : Eclipses Eclipses What What was when What through you ? through you meet through using of What was What tall ? of What thank help you explain using , and do you a patterns you you a patterns through air in tall you help metal metal </s>
Enter prompt: how are mountains formed?
Model output: I ? ' I ' : Of : in happy Hi wind in yeast altering it it </s>
This looks rather similar to when I asked an AI to implement a basic xor problem solver I guess fundementally there's really only a very limited amount of ways to implement this.
As someone who has spent days wrestling with Python dependency hell just to get a model running, a simple cargo run feels like a dream. But I'm wondering, what was the most painful part of NOT having a framework? I'm betting my coffee money it was debugging the backpropagation logic.
I have heard of similar experiences on HN a few times. Haven't seen any such conflicts on real projects in the last five years or so, since I started using Poetry and then UV. I deal with data science code and the people writing it have a tendency to create dependency spaghetti, for example including the Scikit package in a mainly Pytorch code, just because they need a tried-and-tested accuracy() function.
I do remember banging my head against failed dependency resolution in my Early days of Python, circa 2014, with Pip and Conda, etc.
The dependency issues I have faced were mostly due to data science folks pinning exact package versions for the sake of replicability in requirements.txt for example
This is great ! Congratulations. I really like your project, especially I like how easily it is to peak at.
Do you plan on moving forward with this project ? I seem to understand that all the training is done on the CPU, and that you have next steps regarding optimizing that. Do you consider GPU accelerations ?
Also, do you have any benchmarks on known hardware ? Eg, how long would it take to train on a macbook latest gen or your own computer ?
Congrats - there is a very small problem with the LLM - its reusing transformer blocks and you want to use different instances of them.
Its a very cool excercise, I did the same with Zig and MLX a while back, so I can get a nice foundation, but since then as I got hooked and kept adding stuff to it, switched to Pytorch/Transformers.
"Python" is perfectly performant for AI and this demonstrates a deep lack of understanding. Virtually every library in python used for AI delegates to lower-level code written in C++.
The memory safety guarantees in Rust are probably useful here given how easy it is to have buffer overflows in transformer implementations. CUDA kernels are still going to dominate performance though.
Curious about the tokenization approach - are you implementing BPE from scratch too or using an existing library?
I've not written a single line of Rust ever, but I have occasionally looked under the hood of Tensorflow, Pytorch etc. and have been a machine learning practitioner for several years. The succinctness of the interfaces surprised me!
34 comments
[ 2.4 ms ] story [ 51.2 ms ] threadLooking good!
Enter prompt: hello Model output: What are eclipses ? Assistant : Eclipses occur when one celestial body moves into the shadow of another </s>
Enter prompt: what are facts Model output: eclipses Assistant : Eclipses Eclipses What What was when What through you ? through you meet through using of What was What tall ? of What thank help you explain using , and do you a patterns you you a patterns through air in tall you help metal metal </s>
Enter prompt: how are mountains formed? Model output: I ? ' I ' : Of : in happy Hi wind in yeast altering it it </s>
I do remember banging my head against failed dependency resolution in my Early days of Python, circa 2014, with Pip and Conda, etc.
The dependency issues I have faced were mostly due to data science folks pinning exact package versions for the sake of replicability in requirements.txt for example
- exports being broken if code is executed from a different directory
- packaging being more complicated than it should be
and I don't even have too much experience in the area of packaging, besides occasionally publishing to a private repo.
[0]: https://github.com/enricozb/picogpt-rust [1]: https://jaykmody.com/blog/gpt-from-scratch/
Do you plan on moving forward with this project ? I seem to understand that all the training is done on the CPU, and that you have next steps regarding optimizing that. Do you consider GPU accelerations ?
Also, do you have any benchmarks on known hardware ? Eg, how long would it take to train on a macbook latest gen or your own computer ?
Honestly, I don't know.
This was purely a toy project/thought experiment to challenge myself to learn exactly how these LLMs worked.
It was super cool to see the loss go down and it actually "train".
This is SUPER far from a the real deal. Maybe it could be cool to see how far a fully in memory LLM running on CPU can go.
// Increased for better learning
this doesn't tell me anything
// Use the constants from lib.rs
const MAX_SEQ_LEN: usize = 80;
const EMBEDDING_DIM: usize = 128;
const HIDDEN_DIM: usize = 256;
these are already defined in lib.rs, why not use them (as the comment suggests)
Its a very cool excercise, I did the same with Zig and MLX a while back, so I can get a nice foundation, but since then as I got hooked and kept adding stuff to it, switched to Pytorch/Transformers.
Very interesting to already see rust based inference frameworks as well.
In-browser demo: https://galqiwi.github.io/aqlm-rs
Source code: https://github.com/galqiwi/demo-aqlm-rs