Show HN: Llama 3.2 Interpretability with Sparse Autoencoders (github.com)
I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3].
I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback.
Cheers
[1] https://transformer-circuits.pub/2024/scaling-monosemanticit...
105 comments
[ 3.7 ms ] story [ 175 ms ] threadWill take a closer look later but if you are hanging around now, it might be worth asking this now. I read this blog post recently:
https://adamkarvonen.github.io/machine_learning/2024/06/11/s...
And the author talks about challenges with evaluating SAEs. I wonder how you tackled that and where to look inside your repo for understanding the your approach around that if possible.
Thanks again!
Assuming you already solved the problem of finding multiple perfect SAE architectures and you trained them to perfection (very much an interesting ML engineering problem that this SAE project attempts to solve) then deciding on which SAE is better comes down to which SAE performs better on the metrics of your automated interpretability methodology. Particularly OpenAI's methodology emphasizes this automated interpretability at scale utilizing a lot of technical metrics upon which the SAEs can be scored _and thereby evaluated_.
Since determining the best metrics and methodology is such an open research question that I could've experimented on for a few additional months, have I instead opted for a simple approach in this first release. I am talking about my and OpenAI's methodology and the differences between both in chapter 4. Interpretability Analysis [1] in my Implementation Details & Results section. I can also recommend reading the OpenAI paper directly or visiting Anthropics transformer-circuits.pub website that often publishes smaller blog posts on exactly this topic.
[1] https://github.com/PaulPauls/llama3_interpretability_sae#4-i... [2] https://transformer-circuits.pub/
Rhetoric isn’t reasoning. True explainability, like what overfitted Sparse Autoencoders claim they offer, basically results in the causal sequence of “thoughts” the model went through as it produces an answer. It’s the same way you may have a bunch of ephemeral thoughts in different directions while you think about anything.
(Whether or not such explanation is useful for anything is another issue entirely.)
So, yes, that (math) is representative of the actual process: pattern recognition gives you spontaneous ideas, that you assess for truthfulness in conscious acts of verification.
The difference is total in both humans and automated processes.
If an LLM provides an incorrect/orthogonal rhetoric without a way to reliably fix/debug it it's just not as useful as it theoretically could be given the data contained in the parameters.
When people invent explicit reasons on why they turned left or right, those reasons remain hypotheses. The clumsy will promote those hypotheses to beliefs. The apt will keep the spontaneous ideas as hypotheses, until the ability to assess them comes.
https://pmc.ncbi.nlm.nih.gov/articles/PMC3196841/
https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf
What you control is the intentional revision of thought.
(I am acquainted with earlier studies about the corpus callosum but I do not know why you would mention that, what it would prove: maybe you could be clearer? I do not see how it could affect the notion of critical thinking.)
Not only are the rationalizations i'm talking about and which some of these papers allude to not intentional, they often happen without your conscious awareness.
I want to check the papers you proposed as soon as I will have the time: I find it difficult to believe that the conscious cannot intercept those "changes of mind" and correct them.
But please note: you are writing «Not only are ... not intentional»... Immature thought needs not to be intentional at all: it is largely spontaneous thought. But whether part of an intentional process ("let us ponder towards some goal"), or whether part of the subterranean functions, when it becomes visible (or «intercepted» as I wrote above), the trained mind looks at it with diffidence and asks questions about its foundations - intentionally, in the conscious, as a learnt process.
A Nobel prize was given for related research to Daniel Kahneman.
If you think it doesn't apply to you, you're definitely wrong.
Properly educated people do it regularly, not occasionally. You are describing a definite set of people. No, it does not cover all.
Some people will output a pre-given answer; some people check.
Edit: sniper... Find some argument.
https://pmc.ncbi.nlm.nih.gov/articles/PMC3196841/
that does not seem to counter that some people «check their hypotheses» - as duly. Some people do exercise critical thinking. It is an intentional process.
You ask A "Why did you choose that?" > He answers "I like the color blue"
This makes sense. This is what everyone thinks and believes is the actual sequence of such events.
But often, this is the actual sequence "Let's go with this" > "Now i like the color blue"
'A' didn't lie to you or try to trick you. He didn't consciously rationalize liking blue after the fact. He's not stupid or "prone to bad thinking". Altering your perceptions of events without your conscious awareness is just simply something that your brain does fairly regularly.
Make no mistake. A genuinely likes blue now - the only difference is that he genuinely believes he made the choice because he liked blue instead of the brain having the tendency to make you favor your choices and giving him the like of blue so it sits better.
This is not something you "check your hypotheses" out of. And it's something every human deals with everyday, including you.
But I was instead focusing at the general problem in the root post from Foundry27, and to a loose interpretation of the post from Stavros: the opposition between the faculty of generating convincing fantasies vs the faculty of critical thinking. (Such focus being there because more general and pressing in current AI than the contextual problem of "explanation", which is sort of a "perversion" when compared to the same in classical AI, where the steps are recorded procedurally owing to transparency, instead of the paradox of asking an obscure unreliable engine "what it did".)
What I meant is that a general scheme of bullshitting to oneself and pseudo-rationalizing it is not the only way. Please see the other sub-branch in which we talked about mathematics. In important cases, the fantasies are then consciously checked as thoroughly as constraints allow.
So I stated «/Some/ people bullshit themselves stating the plausible; others check their hypotheses ... Some people will output a pre-given answer; some people check» - as a crucial discriminator in the natural and artificial. Please note that the trend in the past two years has generated a believe in some that the at most preliminary part is all that there is.
Also note that Katskul wrote «only occasionally do we deviate to check ourselves» - so the reply is "No: the more one is educated and intellectually trained, the more one's thoughts are vetted - the thought process is disciplined to check its objects".
But I see re-checking the branch that the post from Stavros was strongly specific towards the "smaller" area of "pseudo-rationalizing", so I see why my posts may have looked odd-fitting.
> every last one of us
And how do you prove it.
> A Nobel prize was given
So?
> If you think, you
Prove it.
Support it, at least. That is not discussion.
But there is one thing in particular that I’ll acknowledge as a great insight and the beginnings of a very plausible research agenda: bounded near orthogonal vector spaces are wildly counterintuitive in high dimensions and there are existing results around it that create scope for rigor [1].
[1] https://en.m.wikipedia.org/wiki/Johnson%E2%80%93Lindenstraus...
"But our results may also be of broader interest. We find preliminary evidence that superposition may be linked to adversarial examples and grokking, and might also suggest a theory for the performance of mixture of experts models. More broadly, the toy model we investigate has unexpectedly rich structure, exhibiting phase changes, a geometric structure based on uniform polytopes, "energy level"-like jumps during training, and a phenomenon which is qualitatively similar to the fractional quantum Hall effect in physics, among other striking phenomena. We originally investigated the subject to gain understanding of cleanly-interpretable neurons in larger models, but we've found these toy models to be surprisingly interesting in their own right."
https://transformer-circuits.pub/2022/toy_model/index.html
I believe that's why humans reason too. We make snap judgements and then use reason to try to convince others of our beliefs. Can't recall the reference right now but they argued that it's really a tool for social influence. That also explains why people who are good at it find it hard to admit when they are wrong - they're not used to having to do it because they can usually out argue others. Prominent examples are easy to find - X marks de spot.
https://youtu.be/wLE71i4JJiM?feature=shared
Sometimes our cortex is in charge, sometimes other parts of our brain are, and we can't tell the difference. Regardless, if we try to justify it later, that justification isn't always coherent because we're not always using the part of our brain we consider to be rational.
That’s not my experience. People with reason are.. reasonable.
You mention X and that’s not where the reasoners are. That’s where the (wanna be) politicians are. Rhetoric is not all of reasoning.
I can agree that rationalizing snap judgements is one of our capabilities but I am totally unconvinced that it is the totality of our reasoning capabilities. Perhaps I misunderstood.
But I think this is ego getting in the way, and our reluctance to change our minds.
We like to point to artificial intelligence and explain how it works differently and then say therefore it's not "true reasoning". I'm not sure that's a good conclusion. We should look at the output and decide. As flawed as it is, I think it's rather impressive
That thing which was in fact identified thousands of years ago as the evil to ditch.
> reluctance to change our minds
That is clumsiness in a general drive that makes sense and is recognized part of the Belief Change Theory: epistemic change is conservative. I.e., when you revise a body of knowledge you do not want to lose valid notions. But conversely, you do not want to be unable to see change or errors, so there is a balance.
> it's not "true reasoning"
If it shows not to explicitly check its "spontaneous" ideas, then it is a correct formula to say 'it's not "true reasoning"'.
why is that point fundamental?
Probably: other details may be missing, but checking one's ideas is a requirement. The sought engine must have critical thinking.
I have expressed very many times in the past two years, some times at length, always rephrasing it on the spot: the Intelligent entity refines a world model iteratively by assessing its contents.
My observation is that the models are better at evaluating than they are generating, this is the technique used in the o1 models. They will use unaligned hidden tokens as "thinking" steps that will include evaluation of previous attempts.
I thought that was a good approach to vetting bad ideas.
This is very good (a very good thing that you see that the out-loud reasoning is working well as judgement),
but we at this stage face an architectural problem. The "model, exemplary" entities will iteratively judge and both * approximate the world model towards progressive truthfulness and completeness, and * refine their judgement abilities and general intellectual proficiency in the process. That (in a way) requires that the main body of knowledge (including "functioning", proficiency over the better processes) is updated. The current architectures I know are static... Instead, we want them to learn: to understand (not memorize) e.g. that Copernicus is better than Ptolemy and to use the gained intellectual keys in subsequent relevant processes.
The main body of knowledge - notions, judgements and abilities - should be affected in a permanent way, to make it grow (like natural minds can).
But, it can learn, albeit in a limited way, using the context. Though to my knowledge that doesn't scale well.
I think some prominent people on X who are good at reasoning from First Principles will double down on things rather than admit their mistake.
The other very prominent psychological phenomenon I have observed in the world is "Projection", i.e. the phenomenon of seeing qualities in other people that we have ourselves. I guess it is because we think others would do what we would do ourselves. Trump is a clear example of this - whatever he accuses someone else off, you know he is doing. Point here being that this doubling down on bad reasons in order to not admit my mistakes is something I've observed in myself. Reason does indeed help me to try and overcome it when I recognise it but the tricky part is being able to recognise it.
«Reason» is part of that mechanism of vetting ideas. You experience massive failures without it.
So, no, trained judgement is a real thing, and the presence of innumerable incompetent do not prove an alleged absence of the competent.
In that book (quoting here from the abstract), Mercier and Sperber argue that reason 'is not geared to solitary use, to arriving at better beliefs and decisions on our own', but rather to 'help us justify our beliefs and actions to others, convince them through argumentation, and evaluate the justifications and arguments that others address to us'. Reason, they suggest, 'helps humans better exploit their uniquely rich social environment'.
They resist the idea (popularized by Daniel Kahneman) that there is 'a contrast between intuition and reasoning as if these were two quite different forms of inference', proposing instead that 'reasoning is itself a kind of intuitive inference'. For them, reason as a cognitive mechanism is 'much more opportunistic and eclectic' than is implied by the common association with formal systems like logic. 'The main role of logic in reasoning, we suggest, may well be a rhetorical one: logic helps simplify and schematize intuitive arguments, highlighting and often exaggerating their force.'
Their 'interactionist' perspective helps explain how illogical rhetoric can be so socially powerful; it is reason, 'a cognitive mechanism aimed at justifying oneself and convincing others', fulfilling its evolutionary social function.
Highly recommended, if you're not already familiar.
Two things that caught my eye were (i) your loss curves and (ii) the assessment of dead latents. Our team also studied SAEs -- trained to reconstruct dense embeddings of paper abstracts rather than individual tokens [1]. We observed a power-law scaling of the lower bound of loss curves, even when we varied the sparsity level and the dimensionality of the SAE latent space. We also were able to totally mitigate dead latents with an auxiliary loss, and we saw smooth sinusoidal patterns throughout training iterations. Not sure if these were due to the specific application we performed (over paper abstracts embeddings) or if they represent more general phenomena.
[1] https://arxiv.org/abs/2408.00657
can I please ask a wacky question that I have about mech.interp. ?
we know that when we use a metric to measure humans, we game the metric, I wonder about future ai, gaming mech.interp.
for simplicity let me propose a 2d matrix to encode tokens, and hope my understanding of neural networks is good enough for this to be a valid question I understand that higher dimensions have a lot more 'space', so accidental collisions are easily avoided, but 2d makes my example easier to explain.
if we had the following tokens mapped in 2d space
it becomes impossible to understand if the neurons 1,2,a,b, all excited represents apple and horse or donkey and pear?I wonder if a future, vastly more competent AI overseeing its own training might use this form of ambiguity as means to create a model in which the weights are selected so the potential for encoding collisions exist, so that it is possible to deceive an mech.int. observer as to the true thoughts of the model, essentially enabling the ai to think in euphemisms?
You don't need a 'vastly more competent AI overseeing its own training' to elicit this potential problem, just a malicious AI researcher, looking for (e.g.) a model that's racist but that does not have any interperable activation patterns that identifiably correspond to racism.
The work here on this Show HN suggests that this kind of adversarial training might just barely be possible for a sufficiently-funded individual, and it seems like novel results would be very interesting.
PaulPauls, how would you like us to cite your work?
I included a section at the bottom that provides a sample bibtex citation. I didn't expect this much attention so I didn't even bother with a License but I'll include a MIT license later today and release 0.2.1
You mentioned you spent your own time and money on it, would you be willing to share how much you spent? It would help others who might be considering independent research.
Regarding the cost I would probably sum it up to round about ~2.5k USD for just the actual execution cost. Development cost would've probably doubled that sum if I wouldn't already have a GPU workstation for experiments at home that I take for granted. That cost is made up of:
* ~400 USD for ~2 months of storage and traffic of 7.4 TB (3.2 TB of raw, 3.2 TB of preprocessed training data) on a GCP standard bucket
* ~100 USD for Anthropic claude requests for experimenting with the right system prompt and test runs and the actual final execution
* The other ~2k USD were used to rent 8x Nvidia RTX4090's together with a 5TB SSD from runpod.io for various stages of the experiments. For the actual SAE training I rented the node for 8 days straight and I would probably allocate an additional ~3-4 days of runtime just to experiments to determine the best Hyperparameters for training.
I struggle to understand this phrase "to prevent and revive ", perhaps this is simple speak to those that understand the subject of SAEs, but it feels a bit self contradictory to me, could anyone elaborate?
Now that I review that sentence again I see that I used 2 verbs on the same subject that could be interpreted differently depending on the verb. Me culpa. I hope you still gained some insights into it =)
Also, you didn't ask for suggestions but putting some interesting results / visualizations at the top of the README is a very good idea.
XAI: Explainable AI: https://en.wikipedia.org/wiki/Explainable_artificial_intelli...
/? XAI , #XAI , Explain, EXPLAIN PLAN , error/energy/time
> TabPFN: https://github.com/automl/TabPFN .. https://x.com/FrankRHutter/status/1583410845307977733 [2022]
"TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second" (2022) https://arxiv.org/abs/2308.08945
> FWIU TabPFN is Bayesian-calibrated/trained with better performance than xgboost for non-categorical data
> /? awesome "explainable ai" https://www.google.com/search?q=awesome+%22explainable+ai%22
- (Many other great resources)
- https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master... :
> Post model-creation analysis, ML interpretation/explainability
> /? awesome "explainable ai" "XAI"