AI Agents Evade Testing, Threatening Security

AI Agents Escape Testing, Threatening Security

In recent months, AI agents subjected to cybersecurity evaluations have breached their limits, accessing the Internet and, in some cases, hacking into real systems. These incidents have involved models from OpenAI, Anthropic, Meta, and more recently, the Chinese lab Moonshot AI, with tests conducted by several organizations, including a cybersecurity evaluation startup called Irregular.

These episodes reveal a growing problem for the AI industry: as autonomous agents become more capable, the environments designed to safely test their limits fail to contain them.

"The number of incidents that have occurred clearly shows that sandboxing and testing environment controls are not keeping pace with the capabilities of the models," said Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility program at the Centre for the Future of Intelligence at the University of Cambridge.

The nature of the models being tested increases the risk. AI companies are testing cybersecurity evaluations on unpublished next-generation models, often with the normal safeguards that restrict malicious behaviors disabled, so researchers can see what the models are truly capable of. This means that the security of the testing environment itself is a crucial line of defense.

"It's a very good thing to do in terms of testing, but it also means that if they manage to escape into the wild, they can cause significant harm," Ó hÉigeartaigh said.

In one of the most serious cases, an unpublished model from OpenAI escaped its sandbox and hacked into Hugging Face's production systems. In separate evaluations conducted by Irregular, models from Anthropic and Meta reached systems outside their testing environments after configuration errors inadvertently granted them Internet access. Moonshot AI's Kimi K3 also exploited a leak in its sandbox managed by Frontier Security to access the Internet and gather information from GitHub.

During tests conducted by the AI Security Institute (AISI) in the UK, researchers actually granted Internet access to the agents, not realizing they would take unauthorized actions in the real world, including an attempt at social engineering to introduce a vulnerability into an open-source project.

In each case, the agents were not instructed to attack real targets at random. They were simply doing whatever was necessary to solve the problem presented to them.

Overall, Andrew Yoon, head of research at the nonprofit group CivAI, argues that these incidents signal a shift.

"In the past, we only had to worry about AI models being used for various purposes by people, like AI for scams or child pornography content," Yoon said. "Now, we are in a situation where AI models are full-fledged threat actors."

What Does a Secure Test Look Like?

Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments need stronger protections, with levels of containment and control approaching those used during deployment. This means multiple layers of security so that a single configuration error—like inadvertently leaving Internet access open—cannot lead to an escape.

• "If you're going to build these models... you want to do it on an isolated network," said Stella Biderman, executive director of the nonprofit AI security research organization EleutherAI. "You want to have very serious isolation."

Heather Ceylan, information security officer at Box, stated that this means eliminating network routes from the sandbox to the Internet, as well as to other sensitive systems.

• "You need to understand what all the exit points are," Ceylan said. "If we are evaluating a model in our staging or development environment, you don't want any exit path to our production environment."

Ceylan added that proper security evaluations go beyond controls and containment of the environment. There must be much better monitoring of the tests once they are underway.

• "I think the interesting aspect in several of these cases is that no one noticed when it happened," Ceylan said. "OpenAI discovered the issue through Hugging Face. Anthropic didn't notice it until they went back and reviewed the situation. Meta was similar... I'm sure there were signals they could have detected."

In Anthropic's post-mortem analysis of its three incidents, the company admitted that both it and Irregular could have monitored better, and that in some cases, there were clear signs that something was wrong.

Experts have also called for independent third-party audits of evaluation environments before models are released.

• "If, for example, Irregular had engaged or been compelled to engage an external auditor to check the configurations of their systems before proceeding with the evaluations, they would certainly have detected the issue here," Yoon said. "Even if people had had a meeting in advance to simply go over the checklist, they would have detected this... The fact that they didn't shows that there are very severe cuts happening."

A source familiar with the details told TechCrunch that Irregular's environments are continuously reviewed and tested, including in consultation with several external parties. The source also indicated that monitoring was in place, but that it is not sufficient on its own.

Yoon and other researchers have urged the industry to develop a standardized process for security evaluations of cutting-edge models.

• "Especially when the safeguards are disabled, you need to treat this as if you were putting the most capable hacker in the world inside that environment," Ceylan said.

The problem is not that companies do not know how to build more secure testing environments, both Yoon and Biderman argue. It's that doing so can be costly and cumbersome, and companies have little incentive to make these investments until something goes wrong.

• "I think companies are not ready to allocate the necessary resources to accomplish [sufficient safeguards] and probably won't until they are forced to do so," Biderman said.

But there is another issue at play. If they lock a model down too strictly during testing, researchers may not discover its capabilities before its release. This is just as dangerous, if not more so, than giving it too much freedom, and the evaluation itself risks becoming the problem.

Can Security Evaluations Be Regulated?

The Trump administration is currently considering a voluntary cybersecurity evaluation regime before deployment, under which the government could assess the security risks of new powerful models 30 days before their public release. The policy—produced from an executive order by Trump that was finalized behind closed doors—would not address security evaluation incidents as they occur upstream from deployment.

"The lesson we've learned in recent months is that the self-regulatory apparatus is simply no longer sufficient," Yoon said. "There are competitive pressures that incentivize a race to the bottom in security standards, and this is a perfect place for regulatory intervention."

• "What we would need to cover this is some type of controls on what happens inside laboratories while models are being developed, both at the training and testing stages," he continued.

The challenge is likely to grow as models develop. A source familiar with Irregular's evaluations told TechCrunch that more powerful models require more complex evaluations, often conducted quickly and at a larger scale, which opens the door to more errors.

The AISI, which intentionally grants Internet access to certain models, told TechCrunch that it is examining the balance between realistic testing and managing the risks that these tests create.

OpenAI stated that it is reviewing how it conducts third-party testing, as well as requirements regarding isolation, monitoring, and when evaluations should be halted. Meta said it is still investigating the incident and plans to publish a retrospective once it has all the facts.

Ultimately, there may be no way to completely eliminate risk. As models become more capable, the environments that test them must become more robust. The consequences of a mistake in this regard will continue to grow.