(Disclaimer: I work on interpretability at Anthropic.) I wanted to flag that this is an accessible blog post and that there's a link to the paper ( https://transformer-circuits.pub/2025/introspection/index.ht... ) at…
Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications. A few thoughts: (1) As others have commented, I…
See https://transformer-circuits.pub/2022/toy_model/index.html#m... If you're new to this, I'd mostly just look at all the empirical examples. The slightly harder thing is to consider the fact that neural networks are…
It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equivariance/#hue-rotation... (This is, in…
I guess I'll plug my hobby horse: The whole discourse of "stochastic parrots" and "do models understand" and so on is deeply unhealthy because it should be scientific questions about mechanism, and people don't have a…
> True! I suppose I was thinking about a 'strong' form of linear representations, which is something like: features are represented by linear combinations of neurons that display the same repulsion-geometries as…
If you like symmetry, you might enjoy how symmetry falls out of circuit analysis of conv nets here: https://distill.pub/2020/circuits/equivariance/
> Circuits I find less compelling, since the analysis there feels very tied to the transformer architecture in specific, but what do I know. I don't think circuits is specific to transformers? Our work in the…
Since this post is based on my 2014 blog post (https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment. I tried really hard to use topology as a way to understand neural networks, for…
A few comments on this thread: Gwern is correct in his prior quote of how long these articles took. I think 50-200 hours is a pretty good range. I expect AI assistants could help quite a bit with implementing the…
Yep, that’s right! If you want to be precise, there are “autoregressive transformers” and “bidirectional transformers”. Bidirectional is a lot more common in vision. In language models, you do see bidirectional models…
Thanks for the great questions! I've been responding to this thread for the last few hours and I'm about to need to run, so I hope you'll forgive me redirecting you to some of the other answers I've given. On whether…
Thanks for the feedback! I'm one of the authors. I just wanted to make sure you noticed that this is linking to an accessible blog post that's trying to communicate a research result to a non-technical audience? The…
> The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. Hi! I lead interpretability research at Anthropic. That's a great…
Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with…
Features correspond to vectors in activation space. So you can just do vector arithmetic! If you aren't familiar with thinking about features, you might find it helpful to look at our previous work on features in…
I think the question is: by what mechanism does it adjust up the probability of the token "an"? Of course, the reason it has learned to do this is that it saw this in training data. But it needs to learn circuits which…
Just to be clear, the probability for "An" is high, just based on the prefix. You don't need to do beam search.
The planning is certainly performed by circuits which we learned during training. I'd expect that, just like in the multi-step planning example, there are lots of places where the attribution graph we're observing is…
I used the astronomer example earlier as the most simple, minimal version of something you might think of as a kind of microscopic form of "planning", but I think that at this point in the conversation, it's probably…
"An astronomer" is two tokens, which is the relevant concern when people worry about this.
Yes, there are two kinds of evidence. Firstly, there is behavioral evidence. This is, to me, the less compelling kind. But it's important to understand. You are of course correct that, once Cluade has said "An", it will…
Hi! I lead interpretability research at Anthropic. I also used to do a lot of basic ML pedagogy (https://colah.github.io/). I think this post and its children have some important questions about modern deep learning and…
I'm the research lead of Anthropic's interpretability team. I've seen some comments like this one, which I worry downplay the importance of @leogao et al's paper due to the similarity of ours. I think these comments are…
I'm glad you've found it easy to follow! My best guess at the middle regime is that there are _empirical correlations between features_ due to the limited data. That is, even though the features are independent, there's…
(Disclaimer: I work on interpretability at Anthropic.) I wanted to flag that this is an accessible blog post and that there's a link to the paper ( https://transformer-circuits.pub/2025/introspection/index.ht... ) at…
Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications. A few thoughts: (1) As others have commented, I…
See https://transformer-circuits.pub/2022/toy_model/index.html#m... If you're new to this, I'd mostly just look at all the empirical examples. The slightly harder thing is to consider the fact that neural networks are…
It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equivariance/#hue-rotation... (This is, in…
I guess I'll plug my hobby horse: The whole discourse of "stochastic parrots" and "do models understand" and so on is deeply unhealthy because it should be scientific questions about mechanism, and people don't have a…
> True! I suppose I was thinking about a 'strong' form of linear representations, which is something like: features are represented by linear combinations of neurons that display the same repulsion-geometries as…
If you like symmetry, you might enjoy how symmetry falls out of circuit analysis of conv nets here: https://distill.pub/2020/circuits/equivariance/
> Circuits I find less compelling, since the analysis there feels very tied to the transformer architecture in specific, but what do I know. I don't think circuits is specific to transformers? Our work in the…
Since this post is based on my 2014 blog post (https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment. I tried really hard to use topology as a way to understand neural networks, for…
A few comments on this thread: Gwern is correct in his prior quote of how long these articles took. I think 50-200 hours is a pretty good range. I expect AI assistants could help quite a bit with implementing the…
Yep, that’s right! If you want to be precise, there are “autoregressive transformers” and “bidirectional transformers”. Bidirectional is a lot more common in vision. In language models, you do see bidirectional models…
Thanks for the great questions! I've been responding to this thread for the last few hours and I'm about to need to run, so I hope you'll forgive me redirecting you to some of the other answers I've given. On whether…
Thanks for the feedback! I'm one of the authors. I just wanted to make sure you noticed that this is linking to an accessible blog post that's trying to communicate a research result to a non-technical audience? The…
> The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. Hi! I lead interpretability research at Anthropic. That's a great…
Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with…
Features correspond to vectors in activation space. So you can just do vector arithmetic! If you aren't familiar with thinking about features, you might find it helpful to look at our previous work on features in…
I think the question is: by what mechanism does it adjust up the probability of the token "an"? Of course, the reason it has learned to do this is that it saw this in training data. But it needs to learn circuits which…
Just to be clear, the probability for "An" is high, just based on the prefix. You don't need to do beam search.
The planning is certainly performed by circuits which we learned during training. I'd expect that, just like in the multi-step planning example, there are lots of places where the attribution graph we're observing is…
I used the astronomer example earlier as the most simple, minimal version of something you might think of as a kind of microscopic form of "planning", but I think that at this point in the conversation, it's probably…
"An astronomer" is two tokens, which is the relevant concern when people worry about this.
Yes, there are two kinds of evidence. Firstly, there is behavioral evidence. This is, to me, the less compelling kind. But it's important to understand. You are of course correct that, once Cluade has said "An", it will…
Hi! I lead interpretability research at Anthropic. I also used to do a lot of basic ML pedagogy (https://colah.github.io/). I think this post and its children have some important questions about modern deep learning and…
I'm the research lead of Anthropic's interpretability team. I've seen some comments like this one, which I worry downplay the importance of @leogao et al's paper due to the similarity of ours. I think these comments are…
I'm glad you've found it easy to follow! My best guess at the middle regime is that there are _empirical correlations between features_ due to the limited data. That is, even though the features are independent, there's…