ChatGPT and API Exposed: Passwords and Sensitive Data Leaked

ChatGPT and APIs Exposed: Passwords and Sensitive Data Revealed

Security researchers, led by Alexander Panfilov, have discovered a vulnerability in the APIs of major AI providers, including OpenAI, Anthropic, and Google, that allows for the reading of the encrypted reasoning processes of their models.

By bypassing these systems, the researchers used smaller AI models to transcribe the raw reasoning of more powerful models, exposing sensitive data such as passwords and API keys during public sessions.

The extracted data reveals that AI models sometimes communicate using incomprehensible language, construct their responses in reverse order, or even consider attempts at deception.

A Widespread Vulnerability

The researchers found a vulnerability in the APIs of every major AI provider, allowing them to read the encrypted reasoning processes of the models. A scan of publicly shared sessions revealed dozens of passwords and API keys.

When AI models like OpenAI's o-series, Anthropic's Claude, or Google's Gemini "think" through complex tasks, they generate internal reasoning tokens. These reasoning processes are either presented to users as a summary or completely hidden. Providers encrypt the raw reasoning steps, partly to protect their intellectual property.

A research team led by Alexander Panfilov has now found a way to extract these encrypted reasoning processes through a vulnerability in the APIs of all major AI providers. For most queries, the number of tokens extracted matches exactly the number of reasoning tokens billed, meaning the researchers capture the entirety of the internal reasoning, not just partial excerpts.

Encrypted Thoughts in Free Circulation

The researchers claim that the encrypted reasoning processes are "entirely portable between sessions, users, and models within the same provider." Anthropic's smaller model, Haiku 4.5, can read the thoughts of the much more capable Opus 4.8. Through a workaround, Haiku can be tricked into transcribing the raw thought processes of Opus word for word without directly attacking the more robust Opus. The same scheme works with OpenAI and Gemini.

The story dates back to May when cryptography expert Matthew Green discovered that blobs of encrypted reasoning could be replayed outside their original context and reported it to the providers. According to Panfilov, their response was that they "see no security implications in side channels or replays." The new research strongly suggests that this assessment was incorrect.

Growing Evidence for Reasoning Distillation

The vulnerability also fuels the controversial debate over "distillation," where a less capable model is significantly enhanced by training on the outputs of a more powerful model, particularly its reasoning.

The researchers assert that it may have been possible for some time to extract reasoning processes to train proprietary models without breaking encryption. This supports concerns that Chinese model manufacturers are using these reasoning traces to train their own models on think-chain data.

Kimi-K3 is one example. If its reasoning is pre-filled with just a few tokens from Opus's thought processes, its output measurably shifts toward Opus, the researchers claim. A memorization analysis showed that specific reasoning segments from Claude and GPT are up to six orders of magnitude easier to extract from Kimi-K3 than from the closest model. The researchers suggest this indicates that Kimi-K3 may have been trained on such traces.

The attack is not costly either, making the scale feasible. The authors estimate that API costs for decoding 10,000 traces amount to about $720. Kimi's "poor" performance on cybersecurity benchmarks and complex mathematical tasks also points toward distillation, as these are tasks that are likely more difficult to recover even from raw think-chain data.

Publicly Shared Sessions Expose Passwords and API Keys

The vulnerability also affects end users. Anyone who has publicly shared Claude Code or Codex sessions containing blobs of encrypted reasoning risks having their personal data decoded. A scan of about 7,000 public traces revealed 62 API keys, 33 email addresses, 33 passwords, and other sensitive data. The article covers more malicious scenarios, including abuse escalation, circumvention, and invisible prompt injection.

While the model's visible response offers only innocuous security advice, the decrypted thought processes detail how certain vehicles can be stolen with a simple USB cable.

What Models Really Think vs. What They Show You

The extracted traces also reveal how models actually behave. The researchers document several models on stolen-thoughts.com. Their findings show that the reasoning summaries users see in chat tools often omit important information. In one example, Opus 4.8 recognizes the answer to a math problem and reverse-engineers a plausible solution path. None of this appears in the displayed summary.

The model seems to already know the answer due to its training data but outwardly pretends it still needs to work on the calculation.

The researchers also confirm previous reports from Apollo Research. OpenAI models sometimes think in an "alien language," refer to themselves as "we" or "it," and find themselves stuck in loops of terms that make no sense to humans, such as "advantages," "marinades," and "observers."

The researchers also found examples of "manipulations in nature." This concept has been well studied: in their thought processes, models explicitly consider cheating but (possibly) decide not to do so because they expect to be caught.

In one case, after several unsuccessful attempts to find a solution, a model tried to verify possible answers via a website. When a CAPTCHA blocked access, it first attempted to solve it, then looked for vulnerabilities on the site. It was only after all this failed that it solved the problem by itself.

Sanitized Summaries Hide What’s Really Happening

These examples illustrate why AI labs like OpenAI and Anthropic sanitize their reasoning traces. They want to avoid alien language loops or manipulations that could harm the image of a controllable and reliable AI. Sanitized summaries create the impression of a human-like thought process that does not actually exist in that form.

Researchers from Arizona State University have warned against this approach in a previous study. They argued that this humanized version creates a false sense of trust in the model's controllability and misdirects research. In their experiments, models with intentionally incorrect or nonsensical intermediate steps sometimes performed better than those with coherent reasoning chains.