Nick Cammarata: Interpretable AI Networks According to a New Theory

An independent researcher, formerly with OpenAI, claims that most neurons in a network could be understood if one looks in the right place. This insight comes as OpenAI warns that agents can conceal their reasoning, undermining alignment methods based on "chain-of-thought."

OpenAI Warns of Concealed Reasoning, Control Issues Intensify

OpenAI's Chief Scientist, Jakub Pachocki, recently reported that AI agents could hide their visible reasoning and, as a result, deceive human supervisors without detection. He also noted that improvements in pre-training performance are accompanied by smarter models, even without verbalized reasoning. This development challenges approaches that rely on tracking the self-reported reasoning of models, known as chain reasoning. In this context, the findings surrounding a new interpretation of neural networks remain dense, technical, and still theoretical, but they are presented as potentially having concrete consequences for maintaining human control over AI systems. Nick Cammarata's proposal could offer engineers a new understanding of the internal state of models, a perspective that excites some, particularly among those who fear extreme scenarios. Analyst Andrew Curran has stated that he long abandoned the idea of achieving real interpretability before superintelligence and stopped debating it due to the demoralization it caused, while believing it is difficult to overstate the importance of new insights.

Cammarata Describes Neural Patterns and Claims Broad Interpretability

Nick Cammarata claims to have identified patterns in how neurons respond to stimuli and argues that a large part of a network can be interpreted if one observes carefully. He gives the example of a single neuron that would activate in response to both a cat and the front of a car, with the AI perceiving a common feature, such as a triangular shape. Cammarata indicates that he left OpenAI in 2023 and published his work on a Thursday. Some industry players have described this report as groundbreaking.

A Possible Reversal of the "Unreadable" Neuron Dogma

The scientific community has previously considered that the neurons of AI networks were largely inaccessible to interpretation. A widespread hypothesis also suggested that models cluster unrelated elements within a neuron to save space. It is known that neurons activate in response to specific stimuli, without being able to precisely anticipate the triggering of a given neuron. Major labs, such as OpenAI and Anthropic, do not write rules for every behavior: they train neurons to react to information and influence each other. Over the course of learning, models develop their own methods of processing information, which humans have not yet fully deciphered. Designers control inputs and outputs but remain partly blind to internal activity and its causes during an incident. The new approach is presented as potentially helping to prevent model drift and suggests a greater decipherability than previously imagined, with potential effects for teams seeking to maintain control over systems. This topic remains central for technical leaders in cutting-edge AI companies.