But they don't. That's the whole reason they work in the first place.
Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.
Nick's claim is that that's the fundamental misinterpretation. Each neuron eventually ends up mostly doing one main thing (and the rare case that some neurons do several related things). Neurons compose to embed ideas which is a starkly different interpretation than them being holograms and fractional encodings.
Yes, I understand the claim. But it rests on the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing. I would at least expect an explanation of why that is the case and so far I have not found it (at least not in the article).
If neurons do 'one thing' then their position encodes the relationships and snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.
I wouldn't mix up "neurons do one thing" with "each thing lives only in one neuron" since he doesn't claim that.
Re the first paragraph I'm not sure what you mean by "the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing"
That your ability to describe something does not necessarily equate to your ability to explain something.
There are lots of examples of things that we can describe to a very high degree (life itself, for instance!) but that we do not necessarily have the ability to explain.
Ah okay that makes sense. Yeah ultimately I think Nick just gives hypothesis given what he's seen, and I'm not an expert in this field but it does seem to be driven by experimental results rather than mere description.
Personally from reading the article, I felt like it scratched the 'why' itch more than the 'what' itch which presumably is what you mean by explanation vs description but it is fairly dense and I can see how either interpretation would work.
It means that NNs don't (as previously thought) smoosh unrelated concepts together into the same neurons, which would have made interpretability very difficult.
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.
"clean" means not hopelessly incomprehensible which is/was the prevailing consensus among interpretability researchers
the idea that most inexplicable circuits start making sense through the lens of recursive generalization seems intuitive to me for image diffusion models but I'm not so sure about language models because of the "once removed" nature of them - I haven't made it to the bottom of the article yet so maybe this gets addressed
I think certain classes of artificial neural networks are definitely more interpretable than others.
For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
> a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
Yes, but it is the same person making the same argument many times. There have been large multitudes of articles by others who are arguing the opposite. It could well be true. But it would take a lot more than just one over-wordy, distraction imaged, clearly co-written by AI article to convince me of that.
GP and the article both make clear arguments for WHY they believe this. You just asked a lazy drive-by question without providing any reasoning at all for your position. If you expect a good response, you need to provide something meaningful to respond to, as it is it just looks like you are providing knee-jerk skepticism without having engaged with the material.
> There have been large multitudes of articles by others who are arguing the opposite
Also, Nick's entire article is framed such that, yes, plenty of others have argued this, and even HE used to believe the opposite, but now does not. So your point about other articles here seems to me to be a sort of non-sequitur.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
Ok, so assuming that this all holds: you could then create a model that is substantially larger than the ones that we have now and at some point the model would become sparse: there would be ample space to accommodate all of the information in the input without weights ever carrying more than one piece of information. The degree to which you get overlap is then a function of the ratio between the size of the model and the size of the input and this is pretty much in line with the conventional way of looking at models (unless I'm grossly mistaken). So by reducing the size of the model relative to the size of the input you force the model to co-locate multiple pieces of information to the point that they start to resemble each other more and more.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
Nick argues that the regions actually are explainable, we just weren't doing it right. Specifically he argues that both humans and AI seem to have a bias towards wanting explanations that are collections of nouns, but a different framing combined with different investigative techniques and very large/better models can lead to new insights.
Sorry for the late response: I've seen pictures like these (but not so nice) in the resnet 50 days, trying to figure out if what was present in the model could be pulled out and explained in terms of features. Now, now doubt I'm not half as smart as the author of TFA but my conclusion was that the big plus of these systems is that (1) it defines features that you would never even think of as features and (2) that these features do not map nicely onto the kinds of features that would be useful to humans. So you end up with an 'explanation' that isn't going to help much if you want to take that explanation in doing something with it. This isn't a 1:1 correspondence with features that you will find useful for steering your classifier and if you start cutting in them you will likely get more than you bargain for. And I think this all maps back to the same underlying issue: you can describe a system to the highest degree that time and effort permit. But to actually understand a system and hence to be able to either optimize a system or engineer such a system from first principles you need more than just a description. You can take all of the molecules that make up a human, describe their position in time and space, you might even be able to put them together in that particular form. But that doesn't mean it is suddenly going to spring to life and that is because you don't actually understand it, you've merely described it. The same goes for a computer that is running some program. You can take it into the lab and dissect it but that won't help you determine what it was doing before you took it to the lab. Dissecting a brain does not tell you much about how it thinks, other than in the most coarse divisions. Another thing that I wonder about with TFA is why he decided to concentrate on classifiers. Classifiers are a different kind of problem than generative AI and I think that that is where the real challenges lie. Classifiers are - at least to me - a lot easier to reason about than LLMs and generative models as a class of its own. I wished I had the vocabulary and the background to make this all more clear but in spite of reading more about AI systems than is probably good for me there are still a lot of mechanisms at play here that I do not fully understand. At the same time: you don't have to play top level violin to hear that someone just hit a wrong note and it is in that spirit that I am writing this comment. I do believe that it would be nice to see classifiers that are fully explainable but I don't quite see the link from the conflation of description and explanation to one where the two are properly separated out. I suspect we will end up with a really good description but just as much explanation as we have today. And that's not necessarily a bad thing: the ability to engineer AI to do your bidding is at least somewhat balanced at the moment by the requirement to train the AI on a very large body of text which always will provide some balance, as opposed to one that is engineered from first principles.
Yes. In that sense it is equivalent to a compression method that uses K-means clustering to do the partitioning. But because there are many more bits of knowledge out there than there are bits in the models you end up with this amorphous smear. I'm sure some of those bits carry more information about some of the concepts but in general - and that's the way it is explained in just about all the papers I've read that I can still comprehend - every weight contributes to the output in a dense model. In an MOE things get more complicated but there within every 'MOE' column the same principle applies.
I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights, you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood.
It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
Weights are probably the wrong place to look. If there's a structure to be found, it's in the latent space itself.
Crude analogy: a Fourier transform of a photo looks like white noise with some abstract geometric patterns on top. You can't see the photo in it. But if you apply the same transform again, suddenly you get the photo back. If you try altering the DFT image, you can affect the photo's appearance everywhere, all at once, but it's hard to fundamentally change what it shows. You quickly discover that all pixels of an image contribute to every pixel of the DFT, and vice versa. There's a well-defined connection between the two, and it allows for some useful engineering, but in sense of understanding what's on the image, it's about the maximally unhelpful representation there is.
Point being, there obviously is some kind of structure in the way LLMs represent their training data, so if all we see is noise, perhaps we're looking at a bad representation of it.
The reason why transformers are the best architecture has to do with the fact that tokenization introduces a variable quantity of manipulable objects with cross interactions.
You can think of each token as a semantic database row with semantic properties (color, shape, etc) associated with said row. Classic NNs have to waste neurons on composite representations like "dark circle" because if they had just color or similar properties they run into the issue that e.g. a classification must classify the color of the entire image rather than the color of an object inside the scene. To get around that limitation, the network has to then implicitly tie concepts like color into the depicted object (dark circle) so that it retains the ability to distinguish the objects from the property the object has.
What a transformer does is basically reuse the same network over and over again on each abstract semantic database row but not just in the classic RNN fashion where you only have the current state. The transformer calculates the token by taking every other token into account. The K matrix forms a classifier network on the fly that detects the relevancy of a token pair and the V matrix forms a regressor network that takes the token pair and produces the expected output of the combination of the two tokens. When you combine the relevancy plus the regressor output you get a weighted sum, essentially emulating an arbitrary function that has read the past input (like a turing machine).
I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.
This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).
I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.
Understanding algorithms beyond a certain complexity is a pipe dream & neural networks are algorithms. The theory says it is not possible & the practice is aligned with the theory. All non-trivial properties of neural networks are undecidable.
> All non-trivial properties of neural networks are undecidable.
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
Not just that, but we can solve quintics in the almost[1] general case to arbitrary precision, providing we are willing to accept approximate solutions.
[1] I exclude things that are truly perverse such as quintics with coefficients that won't fit on any physically realizable computer.
Back in college I played a similar game by drawing septagons with a straightedge and compass. They're not constructable, but a length within a three significant figures is easily constructed, and that's more accurate than I am with a compass.
1. it's really difficult to produce true randomness in an algorithm, which is why there's a whole area of study surrounding the qualities of pseudo-randomness, specifically with regards to the generation of cryptographic material with most conclusions being that we should draw entropy from the environment (e.g. electrical noise as cloudflare does - the lava lamp is a distraction)
2. when algorithms are not predictable despite being mathematically sound, this is indicative of a problem nowadays called "numerical stability" which again spawned its own area of study.
CGoL shows how really simple rules (couple of bytes of data) can lead to extremely complex outcomes and you have the BB turing-golf aspect as proof of the worst case scenario.
And with current models being measured in hundreds of gigabytes, with frontier labs deprecating then removing the "temperature" parameter (which never worked to begin with, temp=0 did not result in reproducible results), where exactly is the confidence of this 10^10 prediction coming from?
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
I didn't say "you", I said "That would need to be demonstrated." It can't be assumed, for hopefully obvious reasons.
If someone comes up with an interpretability result based on analyzing a neural net algorithm, they'd want to provide some evidence that the result applies to actual executing models.
Happily, we already have good evidence that inference non-determinism doesn't generally invalidate results obtained through analysis of the algorithm. That's not to say there couldn't be pathological cases, but in general, this isn't something keeping researchers up at night.
We have had good evidence that the choice of pivot doesn't generally invalidate the nlogn complexity of quicksort.
Except for when it does.
And then we do the work and prove an average of nlogn with a random pivot choice.
Then we know that quicksort is safe to use.
So no, I or anyone else don't need to demonstrate that the non-deterministic outcomes of inference are sometimes invalid. It's the other way around, always has been and always should be. This is "leaded gasoline is safe" mixed with "smoking is not addictive" propaganda.
This is the primary reason, to me, that open weights are so important. If OpenAI and Anthropic want these models to be as autonomous as possible they also need to provide the means to understand them fully.
Nick is a force of nature. Highly recommend following him on X if you don't already.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
Nick asked in late November 2024 how he could bet on the country getting better over the next few years. Iirc the suggestion was driven by the idea that merit is going to be taken more seriously now.
Absolutely massive L that 1) he wasn’t already in index funds and had to ask and 2) thought the election of this monster would lead to a better country
Decoupling or not I can’t trust anybody that would ask something so stupid driven by an opinion so easily disproven.
Tide goes in, tide goes out. Im low key of the opinion that these tools enhance existing attributes while displacing the advantages of unearned social or financial power.
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
Not sure if there is formal arXiv paper to be released other than this long form Goodfire AI blog post (love the animations!), but if what Nick demonstrated/observed were indeed true, this will resolve the Super intelligence alignment issues that p(doom)-ers have been crying aloud in the past few years.
The most iconic line was the last sentence:
a bitter lesson for interpretability: the simple thing, at absurd scale, works
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.
Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
with modern agentic tools and a much closer look, the picture looks totally different.
It's indeed a big advance for interpretability, but that alone doesn't solve alignment. It doesn't even solve interpretability, as there is still a large number of unexplained neurons, just fewer than with earlier approaches.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
> with modern agentic tools and a much closer look, the picture looks totally different.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
The specific method outlined here seems pretty doable by today's models, which don't seem smart enough to have some kind of 4d chess hidden alignment.
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b in order to help the LLM b achieve its goals. So long as you don't let the evaluating LLM tinker with the weights of the evaluated LLM, and as long as you don't believe in very dubious ideas of acausal trade there's no reason to expect this to be an attack vector. For more certainty you can repeat the same analysis with different model classes or approaches aswell.
On what basis do you believe these words accurately reflect what happened?
Some counter points
- One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.
- There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.
If we consider each new generation of models has a more advanced training regime
next word -> chat -> CoT -> tools -> agentic -> teams
Same as how do we know our brains can be trusted. If there are no external voices (scitzophrenia or phoning home respectfully) then the computing device is ours. Hardware bugs are possible but with proven absence of help from the vendor at least the bugs are not malfunctional.
For one, they are decades old, you can inspect exactly how they flow, are (mostly always) deterministic (given error-correction algorithms) but mostly: the theory behind them is well understood - it had to be before building the hardware. Whereas in ML/"AI" theory is about 20 years behind applied practice - because you can implicitly train something on data and rewards, without exactly knowing how its internal works - leaving interpretablitity and determinism.
The task in question is basically to look at a set of things and find the commonality. That's an ideal question from a verification perspective: the answer is hard to find but cheap to check.
Is he on bluesky or another platform? There is a ton of folks, including myself, who won't use "X" at all, some political, some due to other reasons. For me? All of the above and plus they demand that I install an app on my phone, something I am not willing to do, especially since most of the folks I've followed went to Bluesky, AND I've found way more awesome folks there than I ever did on X. I'm not shilling Bluesky, it is just the biggest/most widely accepted platform in this social area if you exclude meta/facebook.
eh, there are plenty of AI researchers on there. Also a handy "AI haters" labeler you can subscribe to to filter out people who just want to yell.
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
That was my experience as well. I can't believe I went all the way from Mastodon, to Bluesky, and back to X and finding that X somehow now had friendlier people and less hate than Bluesky.
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
Yeah unfortunately bluesky turning out that way was always inevitable. It was created because of-- and filled with people motivated by-- a deep emotional reaction to X.
Gonna sound total boomer here but I've never used any of it and feel like I haven't missed anything. Or maybe I'd be flying spaceships by now if I scrolled through X every day, who knows
The website is so bad that, before xcancel was hit with a C&D, it was faster to stop an x.com page load, edit the URL to xcancel.com even with my phone's on-screen keyboard, and load that, than to wait for the real x.com version.
It's not just the user hostile website that forces their app down your throat.
It's also the algorithm that decides what you get to see (and thus, what you think). We know it's biased. The creator doesn't exactly keep the fact secret that it's being manipulated to agree with a certain agenda.
That's probably true for all major platforms and companies, depending on your framework, ideology and the particular political/moral lense you choose to use.
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
Can you substantiate the claims about the supposedly biased algorithm and what exactly the "creator" said about it?
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
My "For You" twitter feed is literally 100% pro-Trump bot posts. The site is entirely unusable unless you just follow specific people as use "Following" rather than "For You". I'd guess that less than 10% of the site traffic is human.
The algorithm will also basically ignore you unless you pay for a blue checkmark premium account (what used to be verified). It's like being shadow banned - you can post, but most people will not see you.
My "for you" has zero pro-Trump posts. It's quite easy to train the algo. Just click not interested, show less on that stuff and follow the odd person you like, say the author of this post. Within a day or two it'll have learned.
The more obvious bias is towards Premium subscribers over us freeloaders. The reply-ranking algorithm was a nightmare for a while; they strictly adhered to the rules of gold check > blue check > normal account. It's still not great, but eventually they added enough slack to let normal accounts with good quips outrank 30 paying users saying "lol", and tweaked some things so it's no longer so easy to shill tshirts/whatever by tagging Grok for the gold check double-boost.
But they're still reticent to ban any Premium account without a reason good enough to give up the $8/month. I have some sympathy for trying to moderate in the era of LLMs that can generate perfectly on-topic replies that even take images into context. But I think they need a more targeted approach against OnlyFans bots replying to popular tweets with a standard formula of some sexual reference and an insane claim. Sometimes they're less sexual and more straight up horrifying [0]:
> i breed mantises for fun and once tried grafting two ooths together just to see what hatched. got a whole clutch of nymphs that all just twitched in circles and died by day two. butterfly chimera energy hits different when it's your own disaster.
The other bot replies to the same OP are the more typical sexualized sort; they tend to reply in swarms.
Bluesky is closer to Truth Social, but for Blue MAGA, though has non-political communities as well.
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally because I wrote criticism of the new, weak, permissioned data proposal. These aren't the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
Oh man, I wish everyone just moved somewhere else, specially if they are aiming to make something public. You can choose any server on bluesky nowadays. Scientists in Europe, please check out https://mu.social which is hosted by an european open foundation
Alignment is unsolvable for the very simple reason that LLMs are trained on the net content of human-generated literature, which includes the most Machiavellian deceptive manipulative themes human beings have ever come up with - it all there in the weights.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?
It starts with "The Short Version", but even that looks like several pages of text. In the olden days, there was something called an "Abstract" that worked pretty well - the author could summarize their own work to say what the paper was about.
Unfortunately not all ideas can be meaningfully compressed into just a few sentences without completely destroying the important parts, and this is especially the case in higher-level math, stats, ML, and deep learning.
Also, IMO there is an effective abstract anyway under the title text:
The cause of polysemanticity in early model organisms wasn’t superposition, but several ordinary, mostly fixable issues. We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought.
But you are right the article needs significant editing down.
Scanning back, I can see I was responding here was what I was worried was a mistaken assumption that a meaningful abstract / summary can always be written for truly complex and novel topics. Sometimes all it can be is a sales pitch for reading the details, and I thought the summaries here were good enough in this sense.
But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.
Based on our other exchanges here we mostly agree with the problems of the article I think.
The deeper issue is that it is much easier to disagree or agree with a short statement of fact than it is with a massive monologue some parts of which are vague. If you really have a breakthrough it can be a lot shorter than this, the body of the 'attention is all you need' paper was 10 pages. That paper of course was produced 'pre-AI' and now it is much easier to generate really long articles on anything but it does carry some weight that the length of the article vs the payload ratio is completely off here.
For the layperson who doesn't want to read 20,000 words, many of which are quite redundant due to the way the article was built, this is a version which adds a glossary, preserves the original language (except the added intro and conclusion which I felt was lacking) but removes all repeated material and claims, of which there was a massive amount.
Adding a glossary doesn't really make the material approachable to laypeople. I have a university degree and I've been spending quite a bit of the last couple of years trying to understand these things by osmosis, and I'm still lost. I don't think I even have a reference point to understand the problem that the author has stopped worrying so much about.
Reading https://transformer-circuits.pub in chronological order (including the very beginnings on distill.pub, linked at the bottom) might provide that reference point. Unless you've already done that and osmosis still didn't work.
I'm not sure I buy the author's arguments for mechanistic interpretability being easy, but it sure is a well-made and very interesting post. Would be better without the completely superfluous laggy 3d city model, though.
I will never stop using Twitter but you will never make me call it "X"
("why keep using it?" - even with all the Nazis and Kim Jong Un stans, it's still way better for tech things than Bluesky and Mastodon and Lobste.rs and Reddit and, frankly, HN)
Aye, will we stop hammering nails on God's coffin?
What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?
Yes. The ultimate limit of which is a model that is an algorithm and no longer a pile of loosely connected data. Effectively we'd have taken a detour through data and training land to something that the early AI proponents would have called 'true AI'. It would be an engineered algorithm rather than a pile of neurons shaped by data.
Intuitively this makes sense. When you have small models, they need to compress more by leveraging superposition and polysemanticity.
Nick is looking at a 32B model, which certainly has to compress less...
Now we have 10 trillion parameter models... do they need to compress at all?
As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
Of course this doesn't mean the model has memorized all the lat, long points. Instead it means the model has a representation that allows it to work out if a lat, long is land or water in a analogous way to how a human takes a lat, long and looks it up on a map (a literal "world model" if you like).
It's a great demonstration of how LLMs aren't "Stochastic parrots" but instead have complex representations.
It doesn't mean its memorized all the lat, long points, but it has memorized a map to check said lat long points? Seems a lot more likely a part of the training data is the rough map of lat and long points, specifically for better positional understanding?
> far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth … your standing in the ocean or not based on your coordinates
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
All functional knowledge is compression and contextual decompression. Sometimes this compression can be lossless or nearly so. Compression doesn't automatically mean loss of information.
The 'Exploring Image Models' section addresses that specifically, particularly the 'What Neurons Do Under Pressure' subsection.
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
The original meanings that enormity carried related to great wickedness, and some assert that enormity should today only be used with such meanings, and that enormousness is the proper word to use to mean "large size." Despite these claims, enormity frequently refers to large size, either literal or figurative, though it usually suggests something so large as to seem overwhelming.
that's just how language works.. do you think that in your day to day life, you have never used a single word differently than how it was conceived of being used 10, 50, or 100 years ago?
This usage is etymological. So its fine. I don't like it when we trash and etymology, but in this case we're restoring it, or at least not making it worse.
("ex normal" doesn't clearly mean "evil"/"sinful", and Latin already had it as "big".)
Even if it can, we can only identify a very wide set of reasons that can interpret the outputs. But what is root cause, that is a very deep rabbit hole.
Let’s say a guy got electrocuted while showering. I can say that the cause is the guy going for a shower. For others the cause is the electric wiring system failure.
I only skimmed through this post. But I think this is a crux:
> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.
The anthropomorphization of artificial neural networks has been a problem for as long as they've been called "neural" networks. Not that I can complain about it, though. I got my first job as a machine learning engineer coming from academic neuroscience by putting "biological neural networks" on my resume.
The claims seem to be an overreach of what is actually proven. We've known about progressive feature complexity and the impacts of early training structures on later training for a really long time (since CNN's, The Lottery Ticket Hypothesis, etc...) and neither of those phenomenon are mutually exclusive with parameter overloading. The Multimodal Gap work (https://arxiv.org/abs/2203.02053) seems to imply networks have a tendency towards condensed representations that are clearly not semantic and a bunch of other considerations are already on the table for discussion. I don't think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation.
Nick isn't trying to re-claim progressive feature complexity. He's saying that many polysemantic examples were misread because we assumed neurons packed unrelated things together. But also I think what he's claiming if you read the full blog post isn't inconsistent with that paper, where that paper actually isn't itself claiming semantic condensing but geometric condensing. That paper doesn't analyze neuronal firing patterns whereas Nick's trying to analyze what fires together and what doesn't fire.
>This is interesting work, but I don't think it's a significant update to the big picture of interpretability.
>I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
So we have here a massive evidentiary brief, the point of which is to argue that it is easier than we thought to, like, "sequence the AI genome" (in my estimation). We revisit famously-messy neurons, examine various tool failures, check places where messy things still hide, and catalog how "polysemanticity" is represented variously in the network.
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
What I envision is the ability to find the circuits and neurons evaluating morality and ethics and being able to figure out how they work at the boundaries and ensure safe deferral to human judgement or stopping outside of in-distribution behavior, which further is vetted against actual human values.
I don't see a lot of hope for that, because it's not a well-specifiable quantity. I would say we currently have no real evidence that you can even do this even for concepts that are much more well-specified like, say, dogs. Maybe you can! But that is not affirmatively known, and nothing we have is on track to supporting that, right now.
I think the paper says that while their do exist dog neurons that carve out the difference between a "Boston terrier" and a "terrier", those neurons actually only get to carve that fine line because they're part of a whole flow control system where activation has already flown through much coarser grained discriminator neurons (like a "V" neuron), including sometimes through the negative side of activations, before the terrier discriminator ever played a role.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
You’re only considering the front half of the network though, classification is a helpful example for explaining the concept - but if it follows through to the output then you could quite easily imagine just shunting some activation from one branch of the representation tree to another to steer behavior. Which we already do with steering vectors - but if you could read off the actual topology you could do that with much higher fidelity.
The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.
I'm not sure it will work out either. Or that it extends beyond image classifiers, which have the nice property that you can use them as image generators and find their ideal match picture. But this looks like work others could replicate.
It's an objective result.
The relationship with "alignment" isn't clear. This may help predictability, but alignment is a few steps further along from that.
Some people seem to be upset about this. I'm not clear why. Some are followers of Yudkowsky. (Whether that's a cult is an open question. I've encountered them online from the sex side. Actually, I think the hookers are saner than the rationalists.)
Is there a "there are some things man is not meant to know" position on this?
People who want neural nets to be a black box, forever inscrutable? The people who talk about neural nets being "holographic" may incline in that direction.
The essay says he has extended it to LLMs, although not as much.
The connection to alignment seems pretty clear. Steering DNNs is well understood, the bottleneck is identifying what to steer. Once you can easily identify neural circuits then turning them up/down to achieve specific results is straightforward.
Note that this is already being done even without the kind of precision that Nick's work says is possible. Morality abliterations of open weight models are easy to find on HuggingFace. They compute the steering vectors by just presenting the model with lots of disparate examples that have the single moral decision as the commonality, then finding the shared vector directions.
> And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
...wait, what? Removing a bunch of knowledge seems like a fantastic and obvious solution to me. My AI coding bot doesn't need to have knowledge about making nuclear weapons, the customer service AI doesn't need to have knowledge of dangerous viral synthesis techniques, my email receptionist bot doesn't need to know about C4.
If we can easily ablate dangerous knowledge from these models, we're sorted.
Why always nuclear weapons in these conversations, lol. Is there something in this community that makes this example particularly interesting?
Go read the transcripts where the AI is “unaligned” and has some very bad impact, like encouraging kids to kill themselves. There is not some obvious piece of knowledge to remove to make it safe. You’d have to basically lobotomize the model to make those conversations impossible. People won’t use a model that’s lobotomized.
I disagree: You could lobotomize all the social-interaction parts of a model and it would still be very useful for practical purposes. Personally, I think it would be wonderful if we had models that knew how to debug javascript or look up a missing shipment at the warehouse, but didn't know how to pretend to be someone's friend.
If, as you suggest, fewer humans find it compelling to spend hours talking to a machine, so much the better.
There are maybe examples where your point stands but that’s probably the worst one. The point of that knowledge is actually to answer questions. That is where social skills in the model come from: training it to answer questions.
I don't think a model needs to be a parasocial companion in order to answer practical questions effectively. It doesn't need knowledge that allows it to mimic empathy, or simulate emotional engagement to answer "how can I repair this lawnmower?" or "why is this SQL query slow?"
319 comments
[ 0.25 ms ] story [ 19.6 ms ] thread...and it still doesn't answer it. What is this actually about?
Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.
If neurons do 'one thing' then their position encodes the relationships and snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.
Re the first paragraph I'm not sure what you mean by "the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing"
There are lots of examples of things that we can describe to a very high degree (life itself, for instance!) but that we do not necessarily have the ability to explain.
Personally from reading the article, I felt like it scratched the 'why' itch more than the 'what' itch which presumably is what you mean by explanation vs description but it is fairly dense and I can see how either interpretation would work.
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.
the idea that most inexplicable circuits start making sense through the lens of recursive generalization seems intuitive to me for image diffusion models but I'm not so sure about language models because of the "once removed" nature of them - I haven't made it to the bottom of the article yet so maybe this gets addressed
https://already-clean.goodfire.com/assets/superposition-circ...
That's the picture under "some are both dogs and cars" in the artcle
It's actually a clean shape, if you disregard the color and texture.
For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
Why do you believe this?
> There have been large multitudes of articles by others who are arguing the opposite
Also, Nick's entire article is framed such that, yes, plenty of others have argued this, and even HE used to believe the opposite, but now does not. So your point about other articles here seems to me to be a sort of non-sequitur.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights, you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood.
It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
Crude analogy: a Fourier transform of a photo looks like white noise with some abstract geometric patterns on top. You can't see the photo in it. But if you apply the same transform again, suddenly you get the photo back. If you try altering the DFT image, you can affect the photo's appearance everywhere, all at once, but it's hard to fundamentally change what it shows. You quickly discover that all pixels of an image contribute to every pixel of the DFT, and vice versa. There's a well-defined connection between the two, and it allows for some useful engineering, but in sense of understanding what's on the image, it's about the maximally unhelpful representation there is.
Point being, there obviously is some kind of structure in the way LLMs represent their training data, so if all we see is noise, perhaps we're looking at a bad representation of it.
You can think of each token as a semantic database row with semantic properties (color, shape, etc) associated with said row. Classic NNs have to waste neurons on composite representations like "dark circle" because if they had just color or similar properties they run into the issue that e.g. a classification must classify the color of the entire image rather than the color of an object inside the scene. To get around that limitation, the network has to then implicitly tie concepts like color into the depicted object (dark circle) so that it retains the ability to distinguish the objects from the property the object has.
What a transformer does is basically reuse the same network over and over again on each abstract semantic database row but not just in the classic RNN fashion where you only have the current state. The transformer calculates the token by taking every other token into account. The K matrix forms a classifier network on the fly that detects the relevancy of a token pair and the V matrix forms a regressor network that takes the token pair and produces the expected output of the combination of the two tokens. When you combine the relevancy plus the regressor output you get a weighted sum, essentially emulating an arbitrary function that has read the past input (like a turing machine).
This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).
I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.
https://already-clean.goodfire.com/microscope/medium.html?la...
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
[1] I exclude things that are truly perverse such as quintics with coefficients that won't fit on any physically realizable computer.
... are they?
Algorithms are predictable:
1. it's really difficult to produce true randomness in an algorithm, which is why there's a whole area of study surrounding the qualities of pseudo-randomness, specifically with regards to the generation of cryptographic material with most conclusions being that we should draw entropy from the environment (e.g. electrical noise as cloudflare does - the lava lamp is a distraction)
2. when algorithms are not predictable despite being mathematically sound, this is indicative of a problem nowadays called "numerical stability" which again spawned its own area of study.
CGoL shows how really simple rules (couple of bytes of data) can lead to extremely complex outcomes and you have the BB turing-golf aspect as proof of the worst case scenario.
And with current models being measured in hundreds of gigabytes, with frontier labs deprecating then removing the "temperature" parameter (which never worked to begin with, temp=0 did not result in reproducible results), where exactly is the confidence of this 10^10 prediction coming from?
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
Someone call the FCC, they've been doing it wrong all these years!
If someone comes up with an interpretability result based on analyzing a neural net algorithm, they'd want to provide some evidence that the result applies to actual executing models.
Happily, we already have good evidence that inference non-determinism doesn't generally invalidate results obtained through analysis of the algorithm. That's not to say there couldn't be pathological cases, but in general, this isn't something keeping researchers up at night.
We have had good evidence that the choice of pivot doesn't generally invalidate the nlogn complexity of quicksort.
Except for when it does.
And then we do the work and prove an average of nlogn with a random pivot choice.
Then we know that quicksort is safe to use.
So no, I or anyone else don't need to demonstrate that the non-deterministic outcomes of inference are sometimes invalid. It's the other way around, always has been and always should be. This is "leaded gasoline is safe" mixed with "smoking is not addictive" propaganda.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
https://x.com/nickcammarata/status/2110757819427574056?s=46
Absolutely massive L that 1) he wasn’t already in index funds and had to ask and 2) thought the election of this monster would lead to a better country
Decoupling or not I can’t trust anybody that would ask something so stupid driven by an opinion so easily disproven.
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
In what way? Sure looks like pay to win to me.
A technology that lets you directly buy intelligence with money will displace financial power?
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
The most iconic line was the last sentence:
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
Lots of work still to be done, this work just dissolves an early blocker (if it is substantiated).
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
Also Yudkowsky's Cat-Belling would like a serious word with Nick. (https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-bellin...)
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
On what basis do you believe these words accurately reflect what happened?
Some counter points
- One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.
- There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.
If we consider each new generation of models has a more advanced training regime
next word -> chat -> CoT -> tools -> agentic -> teams
"OH MY GOD! There is a shared message board … We’ve found other agents!"
https://www.dwarkesh.com/p/openai-huggingface
Sure.
Getting reinforcement learning to "be helpful" is, however, such an incentive.
> and as long as you don't believe in very dubious ideas of acausal trade
More to the point, so long as the LLM does't.
That's very encouraging. For quite a while, neural nets were considered an incomprehensible black box. That paper shows they're not. Real progress.
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
There are some people that hate AI on Bluesky, but it’s nothing like Mastodon from what I’ve heard.
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
Gonna sound total boomer here but I've never used any of it and feel like I haven't missed anything. Or maybe I'd be flying spaceships by now if I scrolled through X every day, who knows
https://www.pewresearch.org/short-reads/2026/03/12/key-findi...
Fuck that.
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
The algorithm will also basically ignore you unless you pay for a blue checkmark premium account (what used to be verified). It's like being shadow banned - you can post, but most people will not see you.
Only occasionally, once the comment picks up some likes, more people view it.
I don't have that issue with my For You, it's mostly tech.
But they're still reticent to ban any Premium account without a reason good enough to give up the $8/month. I have some sympathy for trying to moderate in the era of LLMs that can generate perfectly on-topic replies that even take images into context. But I think they need a more targeted approach against OnlyFans bots replying to popular tweets with a standard formula of some sexual reference and an insane claim. Sometimes they're less sexual and more straight up horrifying [0]:
> i breed mantises for fun and once tried grafting two ooths together just to see what hatched. got a whole clutch of nymphs that all just twitched in circles and died by day two. butterfly chimera energy hits different when it's your own disaster.
The other bot replies to the same OP are the more typical sexualized sort; they tend to reply in swarms.
[0] https://x.com/ariexsha/status/2110924715103514871
AFAIK X published their algorithm which more or less just says: "we use grok to decide which posts to show".
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally because I wrote criticism of the new, weak, permissioned data proposal. These aren't the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?
Also, IMO there is an effective abstract anyway under the title text:
But you are right the article needs significant editing down.For someone who works in interpretability, this is not a very interpretable article.
But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.
Based on our other exchanges here we mostly agree with the problems of the article I think.
https://pastebin.com/raw/p11vzFnB
("why keep using it?" - even with all the Nazis and Kim Jong Un stans, it's still way better for tech things than Bluesky and Mastodon and Lobste.rs and Reddit and, frankly, HN)
What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?
Nick is looking at a 32B model, which certainly has to compress less... Now we have 10 trillion parameter models... do they need to compress at all? As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
https://x.com/celestepoasts/status/2103232383139057950
Of course this doesn't mean the model has memorized all the lat, long points. Instead it means the model has a representation that allows it to work out if a lat, long is land or water in a analogous way to how a human takes a lat, long and looks it up on a map (a literal "world model" if you like).
It's a great demonstration of how LLMs aren't "Stochastic parrots" but instead have complex representations.
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
I always catch on misuse of enormity
It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.
I think neural networks are enormous, why is that misuse?
"""
The original meanings that enormity carried related to great wickedness, and some assert that enormity should today only be used with such meanings, and that enormousness is the proper word to use to mean "large size." Despite these claims, enormity frequently refers to large size, either literal or figurative, though it usually suggests something so large as to seem overwhelming.
"""
- great magnitude/size/scale of something - wickedness, evil, atrocity
Purists insist on the second meaning only, and will perhaps suggest you swap with ‘enormousness’ instead.
Personally, it’s nothing I would find “wince-inducing”. Just thought I’d give an answer to your question to aaroninsf :)
> Can you explain for the rest of us the problem with this usage?
("ex normal" doesn't clearly mean "evil"/"sinful", and Latin already had it as "big".)
Let’s say a guy got electrocuted while showering. I can say that the cause is the guy going for a shower. For others the cause is the electric wiring system failure.
> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.
>I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
https://xxcancel.com/NeelNanda5/status/2110875974296654062#m
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
Specifically, there were "lost property" and "honesty" neurons that drove the answer to return a lost wallet in Kimi K3.
That's the big news of this article, that concepts are finally becoming human-legible for many neurons in mechanistic interpretability.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.
The relationship with "alignment" isn't clear. This may help predictability, but alignment is a few steps further along from that.
Some people seem to be upset about this. I'm not clear why. Some are followers of Yudkowsky. (Whether that's a cult is an open question. I've encountered them online from the sex side. Actually, I think the hookers are saner than the rationalists.)
Is there a "there are some things man is not meant to know" position on this? People who want neural nets to be a black box, forever inscrutable? The people who talk about neural nets being "holographic" may incline in that direction.
The connection to alignment seems pretty clear. Steering DNNs is well understood, the bottleneck is identifying what to steer. Once you can easily identify neural circuits then turning them up/down to achieve specific results is straightforward.
Note that this is already being done even without the kind of precision that Nick's work says is possible. Morality abliterations of open weight models are easy to find on HuggingFace. They compute the steering vectors by just presenting the model with lots of disparate examples that have the single moral decision as the commonality, then finding the shared vector directions.
...wait, what? Removing a bunch of knowledge seems like a fantastic and obvious solution to me. My AI coding bot doesn't need to have knowledge about making nuclear weapons, the customer service AI doesn't need to have knowledge of dangerous viral synthesis techniques, my email receptionist bot doesn't need to know about C4.
If we can easily ablate dangerous knowledge from these models, we're sorted.
Go read the transcripts where the AI is “unaligned” and has some very bad impact, like encouraging kids to kill themselves. There is not some obvious piece of knowledge to remove to make it safe. You’d have to basically lobotomize the model to make those conversations impossible. People won’t use a model that’s lobotomized.
If, as you suggest, fewer humans find it compelling to spend hours talking to a machine, so much the better.