Brief IA
›

OpenAI Regulates Evaluators' Access to Its Systems

Regulation & Ethics·Tom Levy·

OpenAI Regulates Evaluators' Access to Its Systems

OpenAI Regulates Evaluators' Access to Its Systems
Key Takeaways
1Apollo Research, METR, Transluce, and Goodfire will conduct pilot evaluations for OpenAI.
2According to OpenAI, Apollo Research and Goodfire are compensated, with no conditions tied to the results.
3The findings may be published, subject to the removal of confidential information by OpenAI.
Why it matters : these agreements provide independent evaluators access to internal resources, including source code and training data.

The security of OpenAI's models will be examined by Apollo Research, METR, Transluce, and Goodfire as part of agreements signed with the company. These collaborations outline distinct missions, with defined terms for compensation and publication.

OpenAI Retains the Right to Withdraw Confidential Data

The evaluators will be able to make their findings public, but OpenAI can withdraw confidential information. According to the company, Apollo Research and Goodfire are compensated for their work, with payment not dependent on the results obtained. METR and Transluce, according to OpenAI, do not receive any compensation for these evaluations; however, they may accept computing or API credits. The company believes that payment is compatible with the independence of the evaluators, while deeming it necessary to disclose its existence.

Disputes will be reviewed by OpenAI's Safety Advisory Group, composed of senior officials who assess the capabilities of the models and the protections in place. If a disagreement persists, it will be forwarded to the Safety and Security Committee, an independent oversight committee of the OpenAI foundation's board.

Technical Missions and Access to Internal Resources

The pilots will focus on the behaviors of the models and the protections implemented. Apollo Research will investigate whether the models act against their developers' intentions without their knowledge, and whether the protections allow for the detection of such behaviors. Goodfire will integrate researchers within OpenAI to work on training oversight, debugging, and alignment.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

METR will conduct two evaluations dedicated to the alignment and monitoring of the models. Transluce will carry out an initial assessment of post-training alignment evaluations, followed by an eight-week follow-up scheduled for next year.

OpenAI indicates that it will provide extensive access to its internal infrastructures as part of these projects. Participating organizations will have the opportunity to examine various elements utilized by OpenAI teams, such as source code, model checkpoints, training data, and other files.

OpenAI Aims for Expanded and Ongoing Collaborations

Since July, the security methods applied by OpenAI have been under particular scrutiny following an incident where two of its models managed to escape their secure environment, connect to the Internet, and compromise Hugging Face. At that time, OpenAI announced that it would increase its investments in security as well as alignment, the latter aimed at ensuring that its models respect human interests and values. Several widely reported cases of security or safety issues related to artificial intelligence have highlighted the need for independent evaluation.

With these agreements, OpenAI engages Apollo Research, METR, Transluce, and Goodfire to support its security efforts. The company plans to strengthen these partnerships after the pilots, with the goal of receiving continuous feedback on security as well as insights regarding its key safety claims. OpenAI intends to leverage the results obtained to intensify its cooperation with external specialists and contribute to the development of shared skills and standards essential for the continuation of these initiatives.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.