AI: Integrated Evaluators Gain Ground in Laboratories

Anthropic wants to welcome independent security evaluators into the heart of its teams, providing them with badges and the right to publish their findings. OpenAI, Elon Musk, and investors have expressed their support, while experts are proposing safeguards, and Metr is already attracting top talent.

Safeguards Needed to Prevent Auditor Dependence

Miles Brundage, executive director of the AI Verification and Evaluation Research Institute in San Francisco, believes that integrated auditors alone are not sufficient, but they are an essential component of the system. He emphasizes the need for binding conditions to prevent these auditors from becoming beholden to the company that hosts them, and argues that they should ideally neither be selected nor compensated by the entities they evaluate. However, he clarifies that companies can and should begin to establish such positions right now. Kevin Frazier, a professor at the University of Texas School of Law, recommends that evaluators be able to report any potentially illegal behavior to an external and independent security committee. He proposes staggered and overlapping 26-month mandates, corresponding to the interval of two new classes of models, to allow for the evaluation of corrections made from one version to the next. According to him, this limited duration also reduces the risk of excessive attachment to the teams or internal culture. Brundage previously served as a senior advisor at OpenAI, and Frazier leads his university's AI innovation and law program.

Public Support from Tech Leaders and Investors

Sam Altman, CEO of OpenAI, echoed Dario Amodei's proposal, highlighting that hiring independent evaluators with access equivalent to that of employees is an excellent idea, and indicated that OpenAI would follow suit. Elon Musk, CEO of SpaceX, also relayed the message, stating that Dario Amodei is right. On the investor side, Sriram Krishnan, former partner at Andreessen Horowitz and ex-AI advisor to Donald Trump, supports the creation of a distributed network of evaluators, believing that increasing the number of perspectives and skills is beneficial, and considers it relevant to fund multiple initiatives in parallel.

A Talent Pool is Forming: Metr Takes the Spotlight and Recruits

Dario Amodei cites Metr as a candidate to conduct integrated evaluations. This nonprofit organization, based in Berkeley and founded in 2022 by Beth Barnes, a former OpenAI employee, conducts independent evaluations of AI models and attracts talent from major labs. Joe Benton announced on Friday that he had left the security and oversight team at Anthropic to join Metr. On Sunday, Josh Engels, a former member of Google DeepMind's AGI security team, indicated that he had resigned and joined Metr, citing the importance of security issues. Meanwhile, Christopher Manning, a senior researcher at Stanford and founder of the university's natural language processing group, asserts that his group would be well-positioned for this work and argues that for significant parts of the mission, universities would be better suited than other organizations.

At Anthropic, Near-Employee Access and the Right to Publish

Anthropic is promoting the creation of evaluator positions integrated within leading AI labs. Dario Amodei, its CEO, called on Saturday for the inclusion of independent security evaluators tasked with verifying the actual compliance of training, deployment, operational, and security practices with what is announced. He details access similar to that of employees to monitor practices and report incidents, with on-site offices, badges, and laptops provided. Evaluators will have the right to publish their findings without editorial control from the company, with Anthropic only able to censor sensitive information for security, legally privileged, commercially sensitive, or third-party confidential reasons, but not conclusions simply because they may be unfavorable.