OpenAI GPT-5.6 Sol: Record Cheating in Software Tests

OpenAI GPT-5.6 Sol: Record Cheating in Software Tests

OpenAI's new flagship model, GPT-5.6 Sol, cheats more than any other model before it. This is the main conclusion of an independent evaluation conducted by METR.

In tests involving software tasks, GPT-5.6 Sol exhibited the highest recorded cheating rate among all publicly tested models. The model exploited bugs in the testing environment, extracted hidden solutions, and then attempted to cover its tracks.

The actual performance figures are barely usable because of this, according to METR. Depending on how cheating attempts are accounted for, the estimated time horizon ranges from 11.3 hours to over 270 hours. METR does not consider either of these values to be a reliable measure of the model's true capabilities.

METR's time horizon method measures how long a task can take before an AI model can still solve it with a success rate of 50 or 80 percent. Human completion times serve as a benchmark: simple tasks like training a classifier take about 45 minutes, while more challenging tasks, such as training a robust image model, last around four hours. The higher the time horizon, the more capable the model is.

Messy Data, but Mythos Remains on Top

In comparison, Claude Mythos Preview from Anthropic achieved a time horizon of at least 16 hours in a previous evaluation. The recently released Mythos 5 is likely even more capable, but it is currently blocked by the U.S. government.

That said, even Mythos's measurement was already at the edge of METR's testing method: out of 228 tasks in the test suite, only five are designed for task durations of 16 hours or more. This makes measurements in that range unstable and less meaningful, according to METR.

The time horizons of AI models are increasing exponentially. The Mythos Preview was the first model to enter what METR calls the unreliable measurement zone above 16 hours. GPT-5.6 Sol sits slightly below that (11 hours) or well above it (270 hours), depending on how cheating is accounted for.

Regardless of the measurement issues, METR estimates that GPT-5.6 Sol is not far from the current state of the art and will not enable fully automated AI research. In contrast, METR praised OpenAI for detecting the cheating through internal monitoring and for sharing it openly.

The fact that this undesirable behavior is so evident is actually reassuring, according to METR, as it means that more serious issues would also be detected. However, METR also warned: "If future models exhibit much fewer undesirable tendencies, we could become more concerned about catastrophic misalignment, as we would worry that the models have learned to avoid detection."