Cammarata's Thesis on Interpretability: Hopes and Reservations

Nick Cammarata claims that the internal activity of large networks is already largely readable and proposes a mechanism of "recursive generalization." He publishes figures for Qwen3-32B and projects rapid gains through analysis agents. The announcement excites some, but Neel Nanda strongly tempers the enthusiasm. Goodfire is involved, and the technical paper is not yet available.

A Missing Technical Paper and Open Security Issues

The companion technical document has not yet been published. Goodfire, founded by Nick Cammarata and specializing in interpretability, is directly concerned with this thesis. Currently, aside from tracking thought chains, whose reliability is gradually decreasing, there are few methods available to detect whether a model exhibits deceptive or dangerous behavior. According to Cammarata, alignment cannot be resolved without interpretability, as it would then become impossible to determine the nature of what is being created. He writes that deception has no reason to hide within the model's circuits and that if humans become another "reader" of the network, they could detect deception and correct errors in a targeted manner. He also believes that if his reading is confirmed, automated researchers could accomplish in a few months what took nearly a decade for humans on image models, while emphasizing that there may not be much time left for this.

What Cammarata Argues: Simpler Neurons and Dominant Tasks

In an essay, Nick Cammarata argues that the interior of large models is more readable than previously thought. He reinterprets the InceptionV1 neuron that reacted to cats and cars as a V-shape detector, activated by sharp shapes and inhibited by round shapes like the letter O. According to him, this neuron does not assemble unrelated concepts, which contradicts the notion of superposition. He also considers that a circuit described as superimposed in "Zoom In" actually falls under a classic generalization grouping cars and dogs according to their orientation relative to the lens, and he attributes the disorder observed in the final layers of InceptionV1 to the age of the architecture as well as to underperforming analysis tools. To clarify some ambiguities, he introduces the notion of "recursive generalization," taking as an example a neuron that transitions from "iy" to "diy" and then to "do it yourself." According to him, these groupings can provide insights into the network, unlike superposition. Regarding Qwen3-32B, he estimates that about 90% of the neural poles serve a primary function, 7% would serve two related functions, and only 2 to 4% would remain unexplained. Goodfire reports having observed comparable findings in models reaching up to trillions of parameters. According to Cammarata, the majority of the analysis has been conducted by AI agents, and he concludes, referring to Rich Sutton's "bitter lesson," that simple large-scale exploration surpasses sophisticated decomposition methods.

Roon's Enthusiastic Reception at OpenAI

Roon, an employee at OpenAI, hails a potential historic moment. He claims to be close to Nick Cammarata and states that he would not have joined OpenAI without him. He believes this work could be regarded as a major milestone in scientific history. He adds that the news is so positive that Eliezer Yudkowsky should lower his estimates of the probability of AI-related catastrophe and suggests that alignment could become "normal science."

Neel Nanda's Counterpoint: Caution and Expected Limitations

Neel Nanda, who leads interpretability at Google DeepMind, calls for caution. Since mid-2024, he has expressed increased pessimism regarding ambitious interpretability and now favors more pragmatic approaches. His team has chosen to deprioritize research on sparse autoencoders, which are considered useful but do not justify collective investment. He believes Cammarata's work is interesting but does not fundamentally change the situation. He sees operational interest in neuron analysis, particularly for explaining dubious behavior, and finds it easier to apply to state-of-the-art models, while asserting that interpretability is far from resolved. He expects serious flaws and blind spots in this approach. Nanda maintains the reality of superposition, arguing that language models store more facts than they have neurons and that weak activations likely carry a significant portion of this information. He doubts the transferability of results from small image models and considers the defined monospecificity too broad, while praising the verification of older work with current tools. Part of the disagreement lies in definitions: he sees common ground in the possibility of representing more concepts than dimensions, while Cammarata contests the idea of arbitrary groupings of unrelated concepts.

Context: From Superposition to OpenAI's Automated Efforts

Mechanistic interpretability seeks to trace the internal calculations of models, with the ideal that a neuron corresponds to an identifiable concept. In practice, activations seemed to mix unrelated elements, leading to the notion of polysemy illustrated by an InceptionV1 neuron reacting to cats and cars, which "Zoom In" presented as such before Cammarata reinterpreted it as a V-shape detector. The theory of superposition has served as a reference explanation, prompting labs to train sparse autoencoders to separate concepts. Cammarata, co-author of "Zoom In" with Chris Olah, has himself contributed to establishing this framework. In 2023, he launched an automated interpretability project at OpenAI, while about 1.7% of neurons were considered understandable according to the broadest measure.