Is there a sort order? Would be so nice to understand the threads of evolutions and revolution in the progression. A bit of a family tree and influence layout? It would also be nice to have a scaled view so you can sense the difference in sizes over time.
This is amazing, such a nice presentation. It reminds me of the Neural Network Zoo [1], which was also a nice visualization of different architectures.
Thank you for this! I help teach a "CS enrichment course", and I'm having students play with Keras (with my own written scaffolding of course.) I'm struggling to find a resource to help me plan beyond "this is a perceptron/FFNN", and with my lack of experience (I'm a statistician) this is going to be extremely helpful.
Interesting collection. The architecture differences show up in surprising ways when you actually look at prompt patterns across models. Longer context windows don't just let you write more, they change what kind of input structure works best.
Thank you so much! As a (bio)statistician, I've always wanted a "modular" way to go from "neural networks approximate functions" to a high-level understanding about how machine learning practitioners have engineered real-life models.
This is great - always worth reading anything from Sebastian. I would also highly recommend his Build an LLM From Scratch book. I feel like I didn’t really understand the transformer mechanism until I worked through that book.
On the LLM Architecture Gallery, it’s interesting to see the variations between models, but I think the 30,000ft view of this is that in the last seven years since GPT-2 there have been a lot of improvements to LLM architecture but no fundamental innovations in that area. The best open weight models today still look a lot like GPT-2 if you zoom out: it’s a bunch of attention layers and feed forward layers stacked up.
Another way of putting this is that astonishing improvements in capabilities of LLMs that we’ve seen over the last 7 years have come mostly from scaling up and, critically, from new training methods like RLVR, which is responsible for coding agents going from barely working to amazing in the last year.
That’s not to say that architectures aren’t interesting or important or that the improvements aren’t useful, but it is a little bit of a surprise, even though it shouldn’t be at this point because it’s probably just a version of the Bitter Lesson.
We’re literally seeing digital evolution in real-time. These are basically primitive life forms such as bacteria evolving just with tiniest differences.
Right now we’re engineering every bit of it to make it better but in the long run this is unsustainable. It’s going to be so complex that even these digital life forms won’t be able to understand their own digital DNAs, like us.
We know we have DNA, we can measure every letter but it doesn’t mean we understand what’s going on our 14 trillion cells and how each and every one of them is regulated.
I think this analogy not only explains us, or digital beings we see today. It explains everything, quite literally. Still it would be amazing to think about these systems from the perspective of biology, and try to understand the parts analogous to existing frame that we already have. Then we might figure out what to optimize better. For instance if we figure out a certain part of a layer corresponds to “genes” then we might find out there is alternative splicing within it. Wild but worth a shot.
It is perhaps my eyes, but when I zoom in enough to make it readable, it gets blurry. A higher-res image would be much appreciated. Great idea otherwise.
32 comments
[ 4.3 ms ] story [ 1490 ms ] threadIs there a sort order? Would be so nice to understand the threads of evolutions and revolution in the progression. A bit of a family tree and influence layout? It would also be nice to have a scaled view so you can sense the difference in sizes over time.
[1] https://www.asimovinstitute.org/neural-network-zoo/
I even brought my popcorn :(
On the LLM Architecture Gallery, it’s interesting to see the variations between models, but I think the 30,000ft view of this is that in the last seven years since GPT-2 there have been a lot of improvements to LLM architecture but no fundamental innovations in that area. The best open weight models today still look a lot like GPT-2 if you zoom out: it’s a bunch of attention layers and feed forward layers stacked up.
Another way of putting this is that astonishing improvements in capabilities of LLMs that we’ve seen over the last 7 years have come mostly from scaling up and, critically, from new training methods like RLVR, which is responsible for coding agents going from barely working to amazing in the last year.
That’s not to say that architectures aren’t interesting or important or that the improvements aren’t useful, but it is a little bit of a surprise, even though it shouldn’t be at this point because it’s probably just a version of the Bitter Lesson.
Right now we’re engineering every bit of it to make it better but in the long run this is unsustainable. It’s going to be so complex that even these digital life forms won’t be able to understand their own digital DNAs, like us.
We know we have DNA, we can measure every letter but it doesn’t mean we understand what’s going on our 14 trillion cells and how each and every one of them is regulated.
I think this analogy not only explains us, or digital beings we see today. It explains everything, quite literally. Still it would be amazing to think about these systems from the perspective of biology, and try to understand the parts analogous to existing frame that we already have. Then we might figure out what to optimize better. For instance if we figure out a certain part of a layer corresponds to “genes” then we might find out there is alternative splicing within it. Wild but worth a shot.