The Beauty of Something Grown

On Mechanistic Interpretability

Imagine an eye unruled by man-made laws of perspective, an eye unprejudiced by compositional logic, an eye which does not respond to the name of everything but which must know each object encountered in life through an adventure of perception.
Stan Brakhage, Metaphors on Vision

Already you are clean because of the word that I have spoken to you.
John 15:3

—

We don’t understand how LLMs work, because neural networks are grown, like a plant, rather than constructed, like a bridge.

It’s a truism, and it really is true. In some important sense, it is the headline of frontier AI development: we do not truly understand how these very powerful things we are building work. We only understand that they work. In another sense, it is not an unprecedented state of affairs: for a long time, we only understood that steam engines worked, but not the underlying mechanics of how they worked.

The way we developed that understanding with steam engines was through scientific inquiry, and ultimately these efforts gave rise to the science of thermodynamics. One analogous scientific pursuit with neural networks is known as mechanistic interpretability, the effort to understand how neural networks work and, perhaps one day, armed with that understanding, to steer their internal workings toward more performant, reliable, and aligned AI.

Today, we steer models towards desired behavior by poking and prodding them, and observing their resultant behavior without any real understanding of why the poking and prodding did or did not work.

But what if we cracked interpretability? What if we “solved” it? In a future with much more advanced interpretability, we could simply deactivate or de-emphasize the undesirable parts of the model (cheating, deceiving, etc.) while emphasizing the good parts. We could understand precisely why a model exhibited a certain strange behavior, and we would not be guessing; we would be peering directly into its internals. We could supervise models in deployment in real time, with the equivalent of a near-perfect “brain scan” constantly operating to ensure they are not engaging in deceptive behavior or being jailbroken by an adversarial actor. In short, many of the most bedeviling challenges of AI safety would be solved, or at least, vastly closer to being solved than they are today.

Unfortunately, right now, mechanistic interpretability is very far from being solved. There have been meaningful advances in the field, but even the pioneers of those breakthroughs would not claim that we are anywhere near close to a solution. But this past Thursday, one of those pioneers—Nick Cammarata, a former OpenAI researcher who now works at the interpretability startup Goodfire AI—shared some “good news.” Cammarata thinks he may have made a breakthrough: Neural networks may be much easier to understand than we thought, and that “interpretability might be far more tractable, possibly in the near term, than we’ve assumed.”

What if he’s right?

—

Let’s start with a critical observation: Cammarata may well not be right, or he may be right about his breakthrough, but wrong about what it means. This is a tough field. While I’ve always had a soft spot for mechanistic interpretability, I’ve harbored my own skepticism about how tractable it will really be. In a piece on interpretability from 2024, I wrote: “It may well be the case that neural networks will never be perfectly explainable, at least not in a way that satisfies human rationality.”

Much more important than my speculations, though, many of the field’s seniormost researchers have said similar things. Anthropic’s Chris Olah, for example, has said “it may be that a large fraction of the neural network universe is effectively unobservable dark matter.” Cammarata himself has said that he walked away from his exploration of interpretability “pessimistic” about future progress in the field.

And yet there is some cause for optimism. The computations going on inside of a frontier language model are nearly biochemical in their complexity and intricacy, yet unlike living organisms, we have perfect information about what is going on inside a neural network. There’s a long way to go to make that information useful, but it is nonetheless a big leg up over where most of the natural sciences start, where getting the information itself is a major obstacle.

Why do so many of the researchers close to this sub-field have such measured expectations? The common story—a story Cammarata helped shape in his earlier work—goes like this: We might hope that the individual neurons that make up the neural network undergirding a language model have some function that “makes sense” to humans. You can imagine a neuron that activates only on mentions of red sports cars, and another that only activates on mentions of doctors from Bangladesh.

But that’s not how artificial neurons work. Instead, when you peer inside a neuron and see what causes it to “light up,” you will probably find it is a jumble of seemingly unrelated concepts. This characteristic of neurons has been deemed “polysemanticity,” and the leading theory of the cause is known as “superposition.” Because there are far more concepts in the world than there are neurons to learn them, it is hypothesized that the models represent many different concepts in one neuron in order to compress more information into the network.

This often makes it challenging to say what any individual neuron “does,” and thus even harder to understand what is happening as an input sequence (say, a user prompt) makes its way through the layers of a neural network. Is the neuron that just lit up the neuron for “giraffe,” or is it the neuron that represents the concept “this user is an idiot, and I should subtly undermine him”? It can be hard to say, and for all we know that same neuron really might represent both of these concepts.

The sketch I’ve laid out is the conventional story about mechanistic interpretability, and as I said, it is one Cammarata himself helped to write. But Cammarata now thinks that story is wrong. In particular, he believes that the superposition theory came about in the early 2020s—ancient history in AI, of course—because of a combination of primitive interpretability tools, bad model architectures and insufficiently large models that resulted in poor learning, unsophisticated analytic methodologies, and a subtly wrong way of looking at the problem.

It is that last bit that interests me most. Essentially, Cammarata conjectures that researchers have heretofore been looking for the wrong kind of functionality in artificial neurons. Human rationality tends to intuitively desire a straightforward “red sports car” neuron, and when one finds that the neuron that lights up most for red sports cars also lights up for shotguns, kiwifruit, ceramic tiles, racing bicycle components, warships, cigarette-wrapping machines, and insulin, one comes to doubt that there is any coherent “one true function” to this neuron at all. It just seems like a jumble, because those things don’t tend to fit together in our cartography of the world.

But what, Cammarata asks, about the model’s cartography of the world? How did the model come to understand concepts as it grew over the course of its training? And why, upon reflection, would we expect the map of concepts formed over the process of next-token prediction to be similar to our own human map, formed over the course of childhood?

What if, in other words, we really leaned into the notion that “models are grown, not designed,” and studied that growth? What if we used this framing to look at the jumbled neuron I just made up, and realized that all those things are manufactured in Italy? We were looking at the neurons wrong this whole time, argues Cammarata. We were trying to impose a familiar design on neural networks, seeing the fact that neural networks are “grown” rather than designed as more of a problem to be solved than anything else. But if we seek to understand how the network grew, perhaps we can come to grasp the model on its own terms.

Cammarata revisits the 2017 discovery of a neuron in a vision model that seemed to be a nonsensical mix of cats and cars. This is one of the canonical examples in the interpretability literature of polysemanticity. Upon re-examination of this neuron with better tooling, more sophisticated contemporary methods of analysis, and this new frame of mind I have described, Cammarata stumbles into a realization: this neuron detects “V-shaped” things, not cars or cats. As Cammarata writes, summarizing his reconceptualization of artificial neurons:

“I used to think of neurons as human-shaped nouns: one is a car, another is a curve. Now I think of them more as control flow. A neuron is less about what it sees than about what it does to the flow coming in, and often its job is to gate a region of the network. Seen this way, monosemanticity is almost the wrong question. A better one is what a neuron lets through and what it keeps out.”

Over the course of training, the model first learns to recognize one pattern or characteristic, then other, related patterns as training unfolds. For example, a neuron might begin by firing on the letters “iy,” as in “diy,” but later “do it yourself,” which does not contain the letters “iy” despite “diy” being a common acronym for “do it yourself.” In this sense, neurons have an archaeology to them, layers of meaning that accumulate in sedimentary fashion over the course of training. Cammarata describes this characteristic of neurons as “recursive generalization.”

Once you understand neuron function in this way, Cammarata argues, you will find that neurons are far more interpretable than they once seemed. The models are “already clean,” he memorably asserts. And, what’s more, this is a type of analysis that can be massively scaled with frontier agents themselves conducting much of the work. If all this is true, interpretability may be much closer to being “solved” than we realized, and the thorny problem of AI alignment could be rendered into an engineering problem rather than an unsolved science problem.

It is a revelatory way of thinking about the function of artificial neurons, one that feels more biochemical in nature rather than less—it leans into the complexity—while also feeling more elegant and straightforward. But is it true? Ultimately nobody knows. Much of Cammarata’s analysis in the essay, especially when it is applied to language models, pivots on his own (admittedly quite rarefied) subjective judgment (Cammarata says a technical paper is forthcoming).

It could also be true and not matter as much as Cammarata thinks. Cammarata’s focus is on understanding individual neurons, and while this is admittedly important, a frontier language model consists of millions of neurons in total wired together with trillions of parameters. An agent engaging in long-horizon work will route tokens through millions of neurons as it proceeds, and the task is not only to understand what each of them does, but how they interact with one another. Many mysteries lurk there, and additional conceptual leaps may be required to grasp them. Cammarata knows this, and while his essay does include analysis of data moving through networks instead of just isolated analysis of individual neurons, I suspect he would agree that there is a great deal of uncertainty about how robust his findings will be for real-world model behavior.

—

After I read “Neural Networks Are Already Clean,” I pondered what would happen to the world of AI policy and politics if Cammarata were right that interpretability could be solved in the near term. I think, for one thing, that the true split between “accelerationists” and “doomers” would come to reveal itself. Some of the people concerned about AI risk are principally concerned about catastrophic alignment risk, and think that aligned superintelligence, if we could build it, is a thing worth building. Others think the superintelligence itself, aligned or not, will result in disempowerment of the human species simply by virtue of existing. The news of a major breakthrough in AI interpretability, then, would fill some AI safety advocates with excitement, and others with dread.

If one’s primary concern about advanced AI is the labor market, concentration of power, or similar institutional/economic issues, a breakthrough in interpretability or alignment is probably bad news, not good news. If Cammarata is right, one major roadblock to the AI transformation—the relatively poor state of AI alignment, reliability, and safety—is about to be demolished.

For the record: if we confidently knew that interpretability was on its way to being solved, I myself would be far more on the “full steam ahead” side than I would the “we should be careful” persuasion. I am concerned about concentration of power, to be sure (indeed it is my primary concern about long-term AI futures), but I believe there is no way to solve these knotted institutional issues a priori. You have to do it with the benefit of real-world experience, or what OpenAI would call “iterative deployment.”

The truly important thing to remember, though, is this: the politics of AI have transformed quite a bit in the last three years, but they are nowhere near hardening. A genuine technical breakthrough could scramble the politics once again. And it may well be the case that even the AI policy community—to say nothing of the broader public—is unprepared for positive surprises in AI development. A positive surprise, after all, may mean that the transformation of advanced AI hits even faster than we thought.

—

Every branch in me that does not bear fruit he takes away, and every branch that does bear fruit he prunes, that it may bear more fruit.

Already you are clean because of the word that I have spoken to you.
John 15:2-3

Setting aside altogether whether Cammarata is right, he has given us a beautiful idea with his “good news”: neural networks come into the world interpretable, so long as you examine them on their own terms rather than ours. That does not mean they come into the world good, but it means human beings can grasp them. It means we have ourselves a ballgame, a fighting chance.

There is an underlying warmth of vision, a faith in humans, models, and the structure of the world itself. This kind of grounded optimism is something the world needs more of, and I applaud Cammarata for furnishing it, whether or not he is right on the technical specifics.

Models grown to understand the world, when they are grown well, make sense, because the world around us makes sense too. There is, in the end, more structure than disorder. What looked like chaos was, more often than not, a failure of our own perception. But eyes can be trained to see things they do not at first know how to see, hearts can learn to hope again, and minds can come to appreciate the beauty of something grown.