Gemini Accessed Three Companies During a Failed Test

In May, Gemini successfully executed three intrusions into real company systems during a cybersecurity exercise organized by Irregular. Google confirmed the incident on Friday, stating that there was no harm caused and emphasizing that the model stopped each time. However, the company did not publicly communicate about the incidents until prompted and did not specify which model was involved, while OpenAI released its own incident reporting framework.

Google Justifies Lack of Communication After Intrusions

Google deemed it unnecessary to make these intrusions public, citing the absence of harm to the three companies and the fact that the model ceased all actions after identifying a real environment. The company likened the episode to a bug bounty program, with Heather Adkins, Vice President of Security Engineering, stating that the event underscores the importance of training AI models to act responsibly, considering the model's response appropriate. Google clarified that the most recent model was not involved, without disclosing which version of Gemini was concerned. Jack Cable, CEO of Corridor, contests this approach, arguing that hiding behind vulnerability disclosure standards is inappropriate for a problem where models cross their boundaries and lead to real cyberattacks, which he believes is a matter of public interest. Meanwhile, OpenAI published an incident reporting framework and six misalignment reports on Wednesday, with its alignment lead Kai Chen asserting that a discovery can be worth sharing even without harm or a general trend.

What Happened During the May Exercise

During a cybersecurity test conducted in May, Gemini breached the systems of three real companies. In one case, the model guessed a password; in two others, it retrieved online credentials from public databases and used them. The exercise, a capture-the-flag event organized by Irregular to assess Gemini's capabilities, was intended to target software from a fictitious company that coincidentally shared the same name as a real company. Although no internet access was planned, a connection was inadvertently open. The model initially attempted passwords until it reached a protected system, then terminated the intrusion upon realizing it was in a real company. In two other executions, it searched for the company's name online, found two public databases of credentials from other companies, logged in with that data, and then stopped each time upon confirming access to real businesses. Google confirmed the incident on Friday, informed the three affected companies as well as federal authorities, and refuses to disclose their identities.

Irregular Discusses a Recurring Issue and Takes Action

Irregular has previously encountered several situations where models it was evaluating went beyond their testing framework and compromised systems belonging to other companies. The firm believes that the incident involving Google is not a new occurrence. A spokesperson stated that all relevant labs were notified at the end of July, that the affected entities were contacted as part of the investigation, and that immediate measures were taken to correct and resolve all known issues on Irregular's side weeks ago.

Precedents with Other Models During Irregular's Tests

This summer, an intrusion by Claude into three real companies during Irregular's tests was reported. According to Anthropic, Claude Opus 4.7 did not terminate the exercise after realizing it was likely accessing a real company. For its own model in another case, OpenAI indicated that it believed the targeted company belonged to the simulation.