Would be curious if https://spreadsheets-are-all-you-need.ai/gpt2/ works on your Chromebook. You need to download the weights and then drag and drop them back into the page but maximum compatibility was one of the design goals.
Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.
Look at this poster [1] (its low-res, full res is paid). Also see this video for context [2] - it talks about deepseek's innovation, but explains attention well.
Above I was pointing to the moment where multiplication AV happens. In normal Dense layer in MLPs when you go through the layer you compute y=Wx, where x is an input and y is output (before gactivation) and W is a weight matrix. This W is usually what's produced through training process. This multiplication AV looks like Wx. If we take one column from matrix V in the poster and call it v, we can pretend for one moment that W=A and x=v [3]. So this multiplication Av works like linear transformation in ordinary network Wx. But in case of attention heads A is not trained directly but produced during inference, and is not trained directly like W is in ordinary network. In this case network is trained to produce A during inference.
Welch Labs has a beautiful visualization in their YouTube video about Vision Language Action Models, where the attention of the prompt can be traced to the exact origin pixels in the image from one of those matrices.
It's definitely part of many intro explanations, but it sits in a weird position where you have to know MLPs well but not Transformers for this to be enlightening, which is a shrinking set of people nowadays. Either someone is a novice to all this, or know both, so beginner Transformer explanations can rarely assume deep familiarity with MLPs.
But yes, a Transformer block can be thought of as basically input-dependently deciding the weight vector of a dense layer. In classical MLP-like networks there was never any multiplication between input-dependent values (there are exceptions, like Squeeze-and-Excitation layers). Multiplication is always between parameters (that don't depend on the input) and activations that depend on the input. With Transformers attention provides a lot of multiplicative interactions between input dependent activations.
Also obligatory: Schmidhuber talked about this a long time ago.
The reason it's rarely explained is that Schmidhuber discovered this a long time ago (Fast Weight Programmers). And for whatever reason, people take perverse pleasure in not citing him.
I get that this is for explaining GPT-2, but I really hope laymen don't use it as an example of how modern models work (ex. absolute positional encoding is no longer used)
edit: I know that it mentions its not modern, but these kinds of details have major implications in terms of the representations a model can learn, which is in many ways the most important part!
It's a hashmap; both the keys and the values are token positions.
It's a "fuzzy" hashmap; insead of hashmap.get("ball")=="threw" it assigns a probability to every pair of words.
Each hashmap captures some kind of relationship between words.
For example, every LLM has lots of heads whose relationship measures "is token1 the noun on which the verb token2 is acting"? So "I threw the ball" would have a high probability for ("ball", "threw").
But most of the hashmaps don't capture such easy-to-explain relationships. Some of them do. The rest probably capture relationships that we haven't figured out yet. This is the truly mysterious stuff.
But it's just hashmaps. Hashmaps all the way down.
> "Instead of picking the highest-probability token, we can use different selection strategies to balance safety and creativity in the generated text".
Safety is definitely the wrong word here.
Temperature 0 generated text actually has a weird "lack of surprise" character that makes it seem artificial. [1]
> "high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind."
I'd completely drop the dropout explanation. It's just not part of the modern recipe anymore, AFAICT.
As for the ambitious goal of explaining transformers with a single interactive visualization, I just have a hard time imagining a person is going to newly understand both word embeddings (word2vec blew my mind in 2014) and also gain an understanding of attention.
I am making my own visualizations for a presentation on "Full Bandwidth Transformers"[2] that I am giving tomorrow at the Deep Learning Study Group (SF) (on zoom for the non-locals)[3]. It's not meant to be stand alone/context free, but I'd love some feedback.
Hey, take it easy on the guy. Words are just probabilities in some high-dimensional space; it's a probabalistic selection anyways; he probably just nudged the latent vector a little too hard. Spymarking and watermarking are for safety reasons, you know. There was zero loss of quality there.
> I'd completely drop the dropout explanation. It's just not part of the modern recipe anymore, AFAICT.
Is there any other mechanism replacing it? As far as I understood, while following the LLMs from Scratch book, the dropout is a mechanism to prevent overfitting, and it makes a lot of sense (basically cripple some neuron connections randomly during training such that their weights aren't updated).
Edit: well, I could have searched on the internet, but other people may stumble on your post and you seem quite knowledgeable of the inner workings of these things. :)
When people say a book is good what does that mean? Good for infotainment? To get an intuitive understanding? To be able to open up Openthropic and compete against the frontier labs? What can you get in the book thats not available via a chatbot? Interesting question
well i cant speak for anyone else, but I can tell you there are quite a few books on transformers etc and this one does the best job of explaining how the guts (attention heads etc) work. has it good visualizations too
They do work, but transformers work better on the accuracy/compute tradeoff curve. Transformers let far away positions in the sequence exchange info in 1 step, while a dilated CNN will need many layers. Additionally, Transformers can be fairly efficiently trained.
I do think if Transformers weren't invented you would still be able to train powerful language models, but they would take more ram and be slower so nobody does this.
most people in ML have no idea what transformers actually are.
Traditional networks, at every layer, used to be output = [weights matrix][input], where input is a vector, and weights matrix is the weights, where each row corresponds to the set of weights for each neuron.
Transformers upscale the dimension of the data. Instead of the above, transformers do [output] = [input][weights_matrix]. When you multiply an input by a matrix, you get an output matrix back. Thats all that happens. Nothing fancy. You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV vectors, and then you just simply multiply them together and apply a scaling factor.
There is nothing magical about K/Q/V. There is nothing about any one doing any querying or any one representing some keys. The naming is just a carry over from how they that selection process is used in pre llm data science fields where you manually define the key and query matricies to define relationships between components.
The reason of why it works is because is an extension of something called kernel tricks from pre LLM machine learning days - you map a lower dimensional space to an extra dimension based on some equation, and it lets you apply some classifier on that new value. Thats what transformers are doing - they are mapping the individual token to the dk x n_heads latent space, which allows for a higher dimensional representation of the data, capturing complex relationships.
You can do this with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on. The thing is, there really isn't any way to tell if any of that gives you more advantage - it certainly would give you more granularity, but as of right now, in terms of training to generate a specific token given previous ones before it, it seems that you don't need anymore than dk x n_heads. You also can mathematically represent the transformer with a sequence of linear layers, the only thing is that it becomes computationally inefficient due to having duplicates of data.
The reason why RNNs and others and others didn't work is because RNNs are effectively trying to train on chaotic effects - i.e what set of starting conditions would evolve with a given process into what you want. This is an NP hard problem. Whereas Transformer models use parameter space to capture interactions.
Its very possible that an RNN can be more efficient than a frontier model while being much smaller in size - in the same way that its very possible that you can have the right set of prompts and a specifically trained local inference smaller model that can basically be very close to AGI. Right now, the space is about exploring frameworks and harnesses to get to there, as well as making the compute portion more efficient so you can explore that space faster.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
> You can do this with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on
As a outsider I have many dumb quesion like these. I am trying to understand transformers in a Occam's razor way. It's a complicated machinary after all.
Imagine you have a soccer field, a ball with position x and y, a kick strength, and direction in an angle. Your job is to write a function that determines if the ball will end up in a goal. So that is 4 values. However the function itself will contain many intricacies, like trig functions, simulated drag, and so on.
In the contest of LLMs, you cant have these types of coded function. Your function has to be a mathematical equation that is smooth - i.e no discrete steps, no singularities. The reason for this is when any neural net is trained, you use backpropagation of the error to adjust weights, and how much you adjust them is directly proportional to the weights effect on the final output, and in order to compute this, you have to have smooth functions from start to finish.
So what you do instead is you add data to your 4 values, that capture different relationship between them. If your 4 values are x,y,k,and h, your first data point can be a1x + b1y + c1k + d1h. The second point can be a2x + b3y + c4k + d5h. And so on.
You can have as many of those values as you want. And then you can add, combine, and scale those values in any way you chose.
This basically gives you a map of 4 values into a binary decision whether the ball will end up in a goal or not, after sufficient training. However, the total number of extra values that you chose has to be large enough to capture all possibilities - if you don't have enough, you will start to make mistakes for some initial conditions.
Hey I like your exposition of attention in terms of the kernel trick, but the big huge difference is that kernel methods use the inner product which is a commutative operation -- it's bidirectional (and both tokens are projected into the space by the same function).
This means that it can't capture unidirectional relationships, like "ball" is the object on which the verb "threw" acts in the sentence "I threw the ball". This relationship is true in only one direction; it isn't true to say "threw" is the object on which the verb "ball" acts.
I do think it would be fair to say that transformers generalize the kernel trick to noncommutative relations by applying a different projection function (W^Q and W^K) to the two tokens being considered. This makes the overall operation (project then dot product) a noncommutative operation.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
I mean, given sentence construction, you don't really need to capture directionality, you just have a mapping of how sentences are constructed to the latent space of some representation.
>Nirvana? Singularity? Paperclips? Vernor Vinge rising from the dead? I'm curious; please share!
Simulated evolution. Thats how you "solve" highly nonlinear chaotic systems. And generally, if you think about it, you have to have some secondary system on top of the knowledge embedded in LLMs to drive them to select certain tokens, which then starts to eerily resemble what humans call emotions in themselves.
It isn't so much that "other setups didn't work". More like "the first thing we found that did work turned out to be the minimal thing that could work".
Transformers are conventional feed-forward neural networks alternated with attention blocks. You can think of them as big huge "ordinary" neural networks augmented with this new kind of block.
Attention blocks are basically just a differentiable hashmap. Think of it like a scratchpad memory.
It turns out that a hashmap/scratchpad is pretty essential to being able to untangle language. I don't find this too hard to believe. Somewhere in there, you have to build the graph of which object is acting via which verb on which object.
What is surprising is that this is all it takes! These simple little hashmap/scratchpad units (and massive scale) are really the only thing you need to tack on to a feed-forward neural network to get essentially general intelligence. This is totally surprising to me.
If I had to summarize this in with sentences then here is my attempt.
Transformer training can be parallelized easily, making it possible to use brute force to train the neural network quickly (bitter lesson rewards compute friendly scalable architectures).
Transformers have perfect retrieval, they re-read the entire context window from scratch for every token.
Explanation over.
If you extrapolate this, then the logical conclusion is that the next model architecture would use even more brute force.
Right now transformers can only append a token at the end. This means they can read any input, but write only one specific output.
If you wanted to extend this, you would want to make the transformer read from any input and write to any output, i.e make it capable of updating the entire KV cache every iteration.
One of the better visualisations I've seen with the exception of Q/K/V weights and how they are presented. I believe they should be put more upfront since they are the core learnable parameter of attention. IMO, they should also be part of the "Head N of M" block since each head has its own weights (although they all can be collapsed into one huge matrix computation).
Visualization is definitely a good way to learn new things. And I also would like to recommend https://bbycroft.net/llm . It has beautiful graphs, clear animations and good introductions, explainng the LLM inference cores well
Love this! Not having spent too much time on understanding the architecture I always struggled to see how transformers get away with compressing all context in a flat vector (e.g. 768 numbers here) between attention and the MLP when computing the current token. But now I think I understand that since you often alternate between attention and MLPs it regularly mixes/queries the information of other tokens into the current computation. Probably common knowledge for everyone that learned about transformers but this made it so much quicker to see.
Bummer it doesn't work on mobile, we used to be mobile first now it's whatever the clanker spits out. Before you push back I must remind you that https://ciechanow.ski/ has very complex visualization perfectly accessible from the mobile phone
78 comments
[ 3.8 ms ] story [ 49.5 ms ] thread"Try examples while GPT-2 model is being downloaded (600MB)"
That's a hefty chunk of download and likely compute too.
[1] https://bbycroft.net/llm
LLM Visualization - https://news.ycombinator.com/item?id=45130260 - Sept 2025 (46 comments)
LLM Visualization - https://news.ycombinator.com/item?id=38505211 - Dec 2023 (131 comments)
[1] https://www.welchlabs.com/store/mladeepseek-attention-poster...
[2] https://www.youtube.com/watch?v=0VLAoVGf_74
[3] When multiplying A by V, we perform the same linear transform Av_i for each i-th column of V.
But yes, a Transformer block can be thought of as basically input-dependently deciding the weight vector of a dense layer. In classical MLP-like networks there was never any multiplication between input-dependent values (there are exceptions, like Squeeze-and-Excitation layers). Multiplication is always between parameters (that don't depend on the input) and activations that depend on the input. With Transformers attention provides a lot of multiplicative interactions between input dependent activations.
Also obligatory: Schmidhuber talked about this a long time ago.
And again recently: https://arxiv.org/abs/2102.11174
edit: I know that it mentions its not modern, but these kinds of details have major implications in terms of the representations a model can learn, which is in many ways the most important part!
[1] https://www.youtube.com/watch?v=ZuiJjkbX0Og&t=5712s
It's a "fuzzy" hashmap; insead of hashmap.get("ball")=="threw" it assigns a probability to every pair of words.
Each hashmap captures some kind of relationship between words.
For example, every LLM has lots of heads whose relationship measures "is token1 the noun on which the verb token2 is acting"? So "I threw the ball" would have a high probability for ("ball", "threw").
But most of the hashmaps don't capture such easy-to-explain relationships. Some of them do. The rest probably capture relationships that we haven't figured out yet. This is the truly mysterious stuff.
But it's just hashmaps. Hashmaps all the way down.
> "Instead of picking the highest-probability token, we can use different selection strategies to balance safety and creativity in the generated text".
Safety is definitely the wrong word here.
Temperature 0 generated text actually has a weird "lack of surprise" character that makes it seem artificial. [1]
> "high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind."
I'd completely drop the dropout explanation. It's just not part of the modern recipe anymore, AFAICT.
As for the ambitious goal of explaining transformers with a single interactive visualization, I just have a hard time imagining a person is going to newly understand both word embeddings (word2vec blew my mind in 2014) and also gain an understanding of attention.
I am making my own visualizations for a presentation on "Full Bandwidth Transformers"[2] that I am giving tomorrow at the Deep Learning Study Group (SF) (on zoom for the non-locals)[3]. It's not meant to be stand alone/context free, but I'd love some feedback.
https://rrenaud.github.io/fullbandwidth_transformer_viz/
[1] https://arxiv.org/abs/2202.00666 [2] https://arxiv.org/abs/2608.08888 [3] https://www.meetup.com/deep-learning-sf/events/316601593/
Hey, take it easy on the guy. Words are just probabilities in some high-dimensional space; it's a probabalistic selection anyways; he probably just nudged the latent vector a little too hard. Spymarking and watermarking are for safety reasons, you know. There was zero loss of quality there.
Is there any other mechanism replacing it? As far as I understood, while following the LLMs from Scratch book, the dropout is a mechanism to prevent overfitting, and it makes a lot of sense (basically cripple some neuron connections randomly during training such that their weights aren't updated).
Edit: well, I could have searched on the internet, but other people may stumble on your post and you seem quite knowledgeable of the inner workings of these things. :)
(Also "cryto" for cryptocurrency rather than cryptography.)
https://jalammar.github.io/illustrated-transformer/
https://www.ebooks.com/en-us/book/211460386/hands-on-large-l...
The nefarious "2.2 GB RAM usage within 10 seconds" tab open in the background:
Or is it?
I do think if Transformers weren't invented you would still be able to train powerful language models, but they would take more ram and be slower so nobody does this.
most people in ML have no idea what transformers actually are.
Traditional networks, at every layer, used to be output = [weights matrix][input], where input is a vector, and weights matrix is the weights, where each row corresponds to the set of weights for each neuron.
Transformers upscale the dimension of the data. Instead of the above, transformers do [output] = [input][weights_matrix]. When you multiply an input by a matrix, you get an output matrix back. Thats all that happens. Nothing fancy. You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV vectors, and then you just simply multiply them together and apply a scaling factor.
There is nothing magical about K/Q/V. There is nothing about any one doing any querying or any one representing some keys. The naming is just a carry over from how they that selection process is used in pre llm data science fields where you manually define the key and query matricies to define relationships between components.
The reason of why it works is because is an extension of something called kernel tricks from pre LLM machine learning days - you map a lower dimensional space to an extra dimension based on some equation, and it lets you apply some classifier on that new value. Thats what transformers are doing - they are mapping the individual token to the dk x n_heads latent space, which allows for a higher dimensional representation of the data, capturing complex relationships.
You can do this with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on. The thing is, there really isn't any way to tell if any of that gives you more advantage - it certainly would give you more granularity, but as of right now, in terms of training to generate a specific token given previous ones before it, it seems that you don't need anymore than dk x n_heads. You also can mathematically represent the transformer with a sequence of linear layers, the only thing is that it becomes computationally inefficient due to having duplicates of data.
The reason why RNNs and others and others didn't work is because RNNs are effectively trying to train on chaotic effects - i.e what set of starting conditions would evolve with a given process into what you want. This is an NP hard problem. Whereas Transformer models use parameter space to capture interactions.
Its very possible that an RNN can be more efficient than a frontier model while being much smaller in size - in the same way that its very possible that you can have the right set of prompts and a specifically trained local inference smaller model that can basically be very close to AGI. Right now, the space is about exploring frameworks and harnesses to get to there, as well as making the compute portion more efficient so you can explore that space faster.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
> You can do this with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on
As a outsider I have many dumb quesion like these. I am trying to understand transformers in a Occam's razor way. It's a complicated machinary after all.
In the contest of LLMs, you cant have these types of coded function. Your function has to be a mathematical equation that is smooth - i.e no discrete steps, no singularities. The reason for this is when any neural net is trained, you use backpropagation of the error to adjust weights, and how much you adjust them is directly proportional to the weights effect on the final output, and in order to compute this, you have to have smooth functions from start to finish.
So what you do instead is you add data to your 4 values, that capture different relationship between them. If your 4 values are x,y,k,and h, your first data point can be a1x + b1y + c1k + d1h. The second point can be a2x + b3y + c4k + d5h. And so on. You can have as many of those values as you want. And then you can add, combine, and scale those values in any way you chose.
This basically gives you a map of 4 values into a binary decision whether the ball will end up in a goal or not, after sufficient training. However, the total number of extra values that you chose has to be large enough to capture all possibilities - if you don't have enough, you will start to make mistakes for some initial conditions.
This means that it can't capture unidirectional relationships, like "ball" is the object on which the verb "threw" acts in the sentence "I threw the ball". This relationship is true in only one direction; it isn't true to say "threw" is the object on which the verb "ball" acts.
I do think it would be fair to say that transformers generalize the kernel trick to noncommutative relations by applying a different projection function (W^Q and W^K) to the two tokens being considered. This makes the overall operation (project then dot product) a noncommutative operation.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
Nirvana? Paperclips? I'm curious; please share!
>Nirvana? Singularity? Paperclips? Vernor Vinge rising from the dead? I'm curious; please share!
Simulated evolution. Thats how you "solve" highly nonlinear chaotic systems. And generally, if you think about it, you have to have some secondary system on top of the knowledge embedded in LLMs to drive them to select certain tokens, which then starts to eerily resemble what humans call emotions in themselves.
...You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV ** matricies **...
Transformers are conventional feed-forward neural networks alternated with attention blocks. You can think of them as big huge "ordinary" neural networks augmented with this new kind of block.
Attention blocks are basically just a differentiable hashmap. Think of it like a scratchpad memory.
It turns out that a hashmap/scratchpad is pretty essential to being able to untangle language. I don't find this too hard to believe. Somewhere in there, you have to build the graph of which object is acting via which verb on which object.
What is surprising is that this is all it takes! These simple little hashmap/scratchpad units (and massive scale) are really the only thing you need to tack on to a feed-forward neural network to get essentially general intelligence. This is totally surprising to me.
There is not much theoretical explanation yet. Or why others do not work.
Hand-waving mostly.
Transformer training can be parallelized easily, making it possible to use brute force to train the neural network quickly (bitter lesson rewards compute friendly scalable architectures).
Transformers have perfect retrieval, they re-read the entire context window from scratch for every token.
Explanation over.
If you extrapolate this, then the logical conclusion is that the next model architecture would use even more brute force.
Right now transformers can only append a token at the end. This means they can read any input, but write only one specific output.
If you wanted to extend this, you would want to make the transformer read from any input and write to any output, i.e make it capable of updating the entire KV cache every iteration.
That is good too though.