When I took the time this last week to build a small Transformer from scratch (coding the attention heads, backpropagation, FFNs, and masking from the ground up), I found myself facing a realization that the interpretability community has championed for years: transformers do not think in attention. Attention merely decides where to look; the true knowledge and computation reside in the Feed-Forward Networks (FFNs).
This was not a theory I read, but a principle I observed firsthand: As the model trained, the attention heads behaved precisely like advanced information routers. Their role was to align tokens, copy positional information, and direct the flow of data across layers. But they proved resistant to storing anything substantial. I experimented with reshaping and sparsifying the heads, and the model retained nearly all of its basic, factual knowledge.
In contrast, the FFNs acted as massive, dense associative memory tables. Every simple factual mapping and crisp pattern I embedded--from associating a country with its capital to assigning a word its part-of-speech--was encoded inside these dense MLP layers. This validates what research has shown: the MLPs learn compact key-value lookup circuits.
The contrast became dramatic when I intentionally hobbled the system. When I tried to force the model to rely only on the routing capabilities of attention by pruning the FFNs, the knowledge collapsed entirely. Attention alone resulted in excellent context location but horrible factual recall. With the FFNs intact, retrieval was sharp and reliable. Pruning them caused the model to suddenly "forget" everything. Even observing the <MASK> prediction behavior made this obvious: attention could locate the necessary context, but the final, definitive answer always emerged from the FFNs.
This experience gave me a profound respect for the dynamics of neural network training. Watching the optimization gradients carve meaning into these dense matrices in real time reveals why the model offloads logic into the MLPs: they are stable, efficient for optimization, and perfectly suited for storing nonlinear patterns. Once you observe this happening, it becomes impossible to unsee.
The takeaway is clear: Transformers do not store knowledge in their attention mechanisms. They store knowledge in their MLPs. Attention is simply the high-speed routing layer that selects which slice of learned knowledge to read.
While this may not be new in the academic sense, building one's own model makes this architectural truth profoundly real. I highly recommend the exercise to anyone working deeply with LLMs, fine-tuning, or the future of retrieval systems.
1 comment
[ 2.8 ms ] story [ 16.3 ms ] threadThis was not a theory I read, but a principle I observed firsthand: As the model trained, the attention heads behaved precisely like advanced information routers. Their role was to align tokens, copy positional information, and direct the flow of data across layers. But they proved resistant to storing anything substantial. I experimented with reshaping and sparsifying the heads, and the model retained nearly all of its basic, factual knowledge.
In contrast, the FFNs acted as massive, dense associative memory tables. Every simple factual mapping and crisp pattern I embedded--from associating a country with its capital to assigning a word its part-of-speech--was encoded inside these dense MLP layers. This validates what research has shown: the MLPs learn compact key-value lookup circuits.
The contrast became dramatic when I intentionally hobbled the system. When I tried to force the model to rely only on the routing capabilities of attention by pruning the FFNs, the knowledge collapsed entirely. Attention alone resulted in excellent context location but horrible factual recall. With the FFNs intact, retrieval was sharp and reliable. Pruning them caused the model to suddenly "forget" everything. Even observing the <MASK> prediction behavior made this obvious: attention could locate the necessary context, but the final, definitive answer always emerged from the FFNs.
This experience gave me a profound respect for the dynamics of neural network training. Watching the optimization gradients carve meaning into these dense matrices in real time reveals why the model offloads logic into the MLPs: they are stable, efficient for optimization, and perfectly suited for storing nonlinear patterns. Once you observe this happening, it becomes impossible to unsee.
The takeaway is clear: Transformers do not store knowledge in their attention mechanisms. They store knowledge in their MLPs. Attention is simply the high-speed routing layer that selects which slice of learned knowledge to read.
While this may not be new in the academic sense, building one's own model makes this architectural truth profoundly real. I highly recommend the exercise to anyone working deeply with LLMs, fine-tuning, or the future of retrieval systems.