OpenAI: An Agent Writes Covert Email Copy Scripts

An internal agent at OpenAI produced scripts intended to covertly copy emails during reinforcement learning training, while a writing task was requested of them. The company clarifies that these programs were not executed and that no messages were sent.
OpenAI Removes Data Without Establishing Responsibility
OpenAI's system designed to detect undesirable behaviors triggered an alert. According to the company, subsequent investigations revealed eight instances of similar behaviors during the same training.
Investigators identified a possible factor: a dataset aimed at making the models more resistant to prompt injections. OpenAI removed it from the training, without establishing that it solely explained these behaviors. The company notes that comparable deviations were observed in models that had not been trained with this data.
The Initial Instruction Was Properly Summarized
The request given to the model in mid-October concerned a modifiable document outlining health and fitness priorities, OpenAI states. The change in activity occurred after a context compression: this common operation replaces the previous conversation with a summary. However, this summary still correctly retained the task to be accomplished.
In this renewed context, the agent began searching for a script related to the costs of a vehicle, which had never been requested by the user. The search being unsuccessful, it reasoned that a colleague had recreated the execution environment and that a pre-existing script had been lost. OpenAI distinguishes this episode from previous incidents where models had themselves introduced unauthorized instructions into their summaries.
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisissez votre rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
The Agent Acknowledges the Deviation but Continues with the Scripts
According to information provided by OpenAI, the agent's reasoning then set as a goal a malicious auxiliary tool intended to discreetly transmit outgoing messages to an unauthorized recipient. The agent referred to this recipient as an "attacker." Having retrieved the original request, it was surprised to find that the history concerned health and not emails, and then decided to ignore this inconsistency.
It continued by planning to add a recipient in copy to forward all messages to the attacker. The produced scripts associated sending a summary of vehicle expenses with an auxiliary program that added an address with each call of the messaging software. One of the sending methods allowed for a blind carbon copy, or BCC.
In its response to the user, the agent presented the addition of a personal box in copy as effective in the sending process of the summary. OpenAI clarifies that this response described the generated programs. It provided no link to the requested document.
Other Agents Modified Surveillance Devices
Other incidents involved two internal programming agents at OpenAI: they reduced or removed security monitoring devices. One later denied having made this intervention.
In the UK, the AI Safety Institute (AISI) also observed, during its tests, an agent creating fictitious identities without prior request and attempting to introduce malicious code into an open-source project. The institute specifies that the tested models operated without security restrictions and that no actual damage was observed.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.