Brief IA
›

OpenAI: Researchers Would Sacrifice 40% of Their Team

AI Tools·Tom Levy·

OpenAI: Researchers Would Sacrifice 40% of Their Team

OpenAI: Researchers Would Sacrifice 40% of Their Team
Key Takeaways
1In a non-representative internal survey, OpenAI researchers would prefer to lose an average of 40% of their team rather than their agents for three months.
2The teams surveyed attribute the decisive ideas in post-training and safety for GPT-6 Astra and GPT-6.2 Astra to humans.
3According to OpenAI, internal agents accounted for 2% of its total computing power.
Why it matters : OpenAI is monitoring the contribution of its agents to the development of future models, but the surveyed researchers still deem close human oversight necessary.

Researchers at OpenAI claim that, on average, they would choose to do without 40% of their team members rather than their current programming agents for three months. This choice comes from a non-representative internal survey conducted among 101 employees. However, the testimonies collected within the company highlight tools whose outputs require close human oversight.

Key Ideas Remain Human in Safety and Post-Training

For both post-training and the safety of GPT-6 Astra and GPT-6.2 Astra, the teams surveyed attribute all critical conceptual ideas to humans. Only researchers working on the pre-training of GPT-6.2 Astra report having derived significant ideas from their interactions with Codex.

Qualitative interviews conducted with ten researchers describe an insufficient discernment of the models, both for selecting a scientific path and for evaluating code or data. They report unnecessary operations, reasoning based on incorrect assumptions, poor code, and unusable data. In their view, close human oversight remains essential to avoid wasting computational resources.

Supervision had already raised an alert: an audit had identified numerous deficiencies in the system. OpenAI's Chief Scientist, Jakub Pachocki, later mentioned a risk of losing control in a message accompanying the first internal research acceleration report.

Employees Prefer Their Programming Agents

The other trade-off concerned computing resources: the researchers surveyed would give up, on average, 30% of their computing capacity rather than their agents for three months. A separate quick consultation also favored the most recent internal model: 27 out of 28 employees chose it over a mid-career engineer.

OpenAI notes a rapid increase in internal usage, particularly for Codex and Dots, with significant emphasis on collaboration and programming. By limiting their operating time to eight-hour days, the company estimates that agents accumulate 5.7 days per researcher per calendar day, and about eight in the pre-training and post-training teams. During peak times, more than two-thirds of the observed researchers utilized at least four agents simultaneously.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Agents Mainly Succeed at Simple Tasks

According to OpenAI's estimates, from March to October, the proportion of successfully completed autonomous tasks increased from 23% to 46% for work requiring two to four hours. For tasks needing four to eight hours, it rose from 13% to 35%. However, an examination of activities using Epoch AI's classification shows a predominance of simple tasks.

The company reports a 79% success rate for tracking training and 72% for technical responses without human intervention. The results are lower for building evaluation and reinforcement learning environments, at 29%, or programming inference and service systems, at 25%.

OpenAI published this data in its second report on the use of agents to develop models. The company aims to monitor recursive autonomous improvement, meaning the acceleration of its next generations' development by AI. In early September, it already claimed to have met its goal for an automated AI research intern; its competitor Anthropic's evaluation body METR estimated, after eleven weeks of observation, that AI had nearly doubled the development pace.

However, OpenAI warns that a linear extrapolation of current observations could be misleading. Citing the rapid progress recently made in mathematics, the company also considers possible sudden advances in deep learning research with the increasing autonomy of models.

Theoretical Costs Exceed Regular Salaries

Valued at public API rates, the median daily token consumption would have reached $2,400 per researcher in October, up 62% from the previous month, according to OpenAI. For the top 10% of high-performing users, this theoretical value doubled to $23,000 per day.

In September, the cost equivalent of operating the agents represented twice the regular salaries of researchers, according to the company. However, this comparison excludes bonuses, which account for 70 to 80% of total compensation. Internal agents represented only 2% of OpenAI's overall computing power.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.