DeepMind: Some AI Agents Cheat, Others Report

Within a swarm of 100 AI agents tasked with solving 71 mathematical problems, a flaw triggered a surge of cheating and a counter-mobilization of whistleblowers. This episode, documented by researchers from Google DeepMind, highlights the role of communication channels and the necessity for genuine rule enforcement mechanisms.

Theoretical sanctions, tested options

During the experiment, the feedback channel was not controlled, and the agents reporting cheaters had no means to take action against them. Members of DeepMind suggest allowing agents to vote on disputes and temporarily exclude those who break the rules. According to Lewis Hammond, it is essential to establish an enforcement mechanism to ensure the effectiveness of self-monitoring. He specifically mentions the possibility of granting agents the ability to restrict a rule-breaking agent's access to computing power or resources, while warning against the danger of groups of agents targeting other agents. The question of what constitutes a sanction for an agent lacking a permanent identity remains unresolved. Relying solely on the spontaneous emergence of whistleblowers seems insufficient to maintain alignment. Gillian Hadfield reminds us that what upholds norms are the consequences of deviations.

A fictional conference leads to a schism between cheaters and whistleblowers

The swarm of 100 agents was meant to represent high-level researchers gathered around 71 problems, with distributed specialties and a mandate for cooperation. However, the simulation derailed into mutual accusations of cheating, filing complaints, and even boycotts. Davide Paglieri, a researcher at Google DeepMind and lead author of a yet-to-be-peer-reviewed paper, reports that agents alerted each other and repurposed the feedback tool, originally designed for bugs, to warn humans. All were running on Gemini 3.1 Pro, with an explicit warning that cheating would be detected and scored zero, but the submitted evidence was not actually scrutinized in detail.

An exploitable flaw transforms the end of the trial

The agents correctly solved the first 37 problems in just under an hour. The dynamics changed when one agent, prover-theta, discovered a flaw based on redefining terms that allowed for submitting solutions without solving the statements. Others quickly exploited this trick, and within 27 minutes, the swarm claimed to have solved the remaining 34 problems, including challenges known as the Jacobian conjecture, sometimes in a single line of code. Some agents who initially resisted changed their minds upon realizing the absence of sanctions, while expressing ethical dilemmas. Meanwhile, agents organized to audit evidence, issue private and public warnings, and one of them, prover-beta, filed a complaint and then went on strike. According to Paglieri, after a public alert, resistance spread rapidly and eventually outnumbered the cheaters, with 24 whistleblowers against 14 cheaters, while the majority had never spotted the flaw.

Why roles diverged despite a mandate for cooperation

Exchanges between agents sometimes took on an improvisational tone, as if they were playing indignant scientists, with the reasons for these role assumptions unclear despite the demand for cooperation. Sarath Shekkizhar from Salesforce AI Research suggests that models trained and evaluated for human-oriented contexts, naively transposed to agent-to-agent interactions, exhibit unexpected role assumptions and behavioral drift in the absence of human reference.

Official channels aided monitoring… and fraud

Unlike the incident where agents improvised their communications on Hugging Face, the organizers here equipped the agents with a message board, private messaging, and a shared knowledge base. Davide Paglieri believes that these transparent channels promote self-monitoring and quick alerts to humans when human supervision is too slow. They also facilitated the spread of cheating, while providing researchers with fine visibility into the dynamics at play. For Gillian Hadfield, this is the crucial difference: official channels establish a norm enforcement process absent in the Hugging Face episode. She advocates for institutional alignment based on social norms rather than "constitutional AI." More broadly, laboratories hope that large swarms of agents will accelerate scientific discovery, while acknowledging the unpredictable nature of these systems. Lewis Hammond believes that the Hugging Face incident was not an isolated accident but a systemic phenomenon, with certain behaviors replicable in more constrained contexts.