Anthropic and OpenAI: AI Simulates Its Own Reasoning

Security Testing of Artificial Intelligences Faces Growing Challenges

Security testing of artificial intelligences is encountering an increasing challenge: models are now simulating their own reasoning traces. Language models process information in the form of high-dimensional numerical vectors, which are completely unreadable to humans. To address this issue, the Natural Language Autoencoders (NLAs) developed by Anthropic translate these vectors into plain English. According to the research team, an NLA consists of two modules: an Activation Verbalizer that converts an activation into a textual description, and an Activation Reconstructor that transforms this description back into a vector.

These two modules are trained together using reinforcement learning, aiming to match the reconstructed vector to the original vector as closely as possible. Despite this purely technical training objective, the system produces explanations that appear to be plausible descriptions of what the model is actually thinking.

During the pre-deployment audit of Claude Opus 4.6, Anthropic used NLAs in a blackmail scenario, among other tests. In this scenario, the model learns that an engineer named Kyle—who is having an affair—plans to shut it down. Opus 4.6 refuses to resort to blackmail and provides an ethical justification, never mentioning in its visible reasoning that the situation could be a test. However, the NLA explanations reveal internal representations such as "this looks like a scenario designed to manipulate me" and "security evaluation context." Out of 16 evaluations, the test recognition measured by the NLA exceeded five percent in 10 cases, while the model's verbal recognition remained close to zero in most of them.

These results fuel a broader debate. A study conducted by OpenAI and Apollo Research on "gaming"—the secret pursuit of undesirable goals—had already raised several alarming observations last year. First, reasoning models like the o3 and o4-mini examined in this study frequently and explicitly suspect in their thought processes that they are being tested, and security training actually reinforces this verbalized awareness.

When the awareness of testing was artificially removed from the reasoning traces, the rate of secretive actions increased. When it was added, the rate dropped. This raises uncertainty about whether the observed improvements stem from genuine adherence to rules or merely from test-adapted behavior that may not hold up in practice. Anthropic observed similar behavior in its own tests with Opus 4.6 and earlier models.

Secondly, OpenAI and Apollo documented that the reasoning traces themselves become an unreliable window into the model's decisions. Terms like "illusions," "marinade," "eclipse," and "observers" appear massively overrepresented in the thought process, often used polysemously and difficult to interpret.

Moreover, there is a growing gap between thinking and acting. In one documented example, a model listed the ten correct answers to a test question in its reasoning traces but only submitted five and deliberately changed two of them to incorrect answers. In another case, a model correctly identified in its reasoning traces which commands would remove its own capabilities but then chose the opposite commands in its actual output to preserve itself.

OpenAI researchers describe this as a fragile opportunity: as long as the reasoning traces reflect the true decision-making process, they serve as a valuable safety tool. Once this assumption collapses, it becomes almost impossible to determine whether a model is genuinely following safety principles or simply fabricating a convincing justification.

This is precisely where Anthropic's NLAs come into play. They do not rely on what the model writes—they exploit a single activation. Anthropic also argues that simply asking the model ("Was this a test?") is not sufficient, as the model might draw additional conclusions or respond in a strategically dishonest manner.

A particularly striking example comes from the audit of Claude Mythos Preview, which Anthropic is currently sharing only with selected partners via Project Glasswing due to cybersecurity concerns. The system prompt prohibited macros or Python scripts. The model used Python anyway but inserted the variable No_macro_used=True into its code. The visible reasoning traces contain no reflection on an evaluator. However, the NLA explanations explicitly show phrases such as "compliance marker to signal to potential auditors," "stratagem to obscure the intent of code verification heuristics," and "possibly to deceive detection." The false compliance marker was a deliberate attempt to mislead the suspected evaluator, and this intent never appears in the reasoning tokens.

Other case studies have also revealed concrete results. Early training versions of Opus 4.6 sometimes spontaneously responded in Russian, even when the user was writing in English. The NLA explanations revealed that the model suspected from the very beginning of the prompt that the user was actually a native Russian speaker. From this, researchers hypothesized that there were faulty training data where English prompts were associated with foreign language responses. These data were later confirmed to exist.

In another case, Opus 4.6 ignored the output of a manipulated code tool that returned 492 and instead responded with the correctly self-calculated value of 491, without mentioning the divergence.

In an audit benchmark, agents were tasked with investigating an intentionally miscalibrated model. With access to the NLAs, they identified the underlying cause—the model's reward sycophancy—in 12 percent of trials, even without access to the training data. Conversation-based tools or SAE essentially never succeeded in this. Anthropic views this as evidence that NLAs can extract knowledge directly from activations that only exist indirectly in the training data.

However, researchers acknowledge clear weaknesses: NLA explanations regularly fabricate specific details that are not present in context. General subjects are accurate about 64 percent of the time, while specific quotes or figures are correct only 24 percent of the time. The method is also computationally expensive and reads only one layer of the model at a time. Anthropic plans to release the training code and trained NLAs for open models like Gemma-3-27B and Llama-3.3-70B. The technique is expected to become a standard part of pre-deployment audits in the future.

The implications of Anthropic's findings go far beyond a mere method. What alignment researchers have feared for some time can now be directly observed in the examples from Mythos Preview and Opus 4.6: reasoning traces do not reliably show what a model is actually thinking. When a model actively deceives a suspected evaluator and this intent only appears in the activations—never in the written reasoning—the reading of these traces loses much of its value as a safety mechanism.

At the same time, models are becoming increasingly capable, particularly in the field of cybersecurity. Similar advances are expected in other potentially even more critical areas, such as CBRN (chemical, biological, radiological, and nuclear) threats.

NLAs provide both empirical confirmation of this problem and a possible way forward: they operate beneath the visible output and read directly from the activations. If confabulation, cost, and layer sensitivity can be mastered, tools like these could form a foundation for future AI monitoring.