Brief IA
›

OpenAI: Over 1,700 Performance Surveys with Codex

Models & LLM·Tom Levy·

OpenAI: Over 1,700 Performance Surveys with Codex

OpenAI: Over 1,700 Performance Surveys with Codex
Key Takeaways
1OpenAI announces over 1,700 follow-up inquiries in just over three months with Performance Factory.
2Codex agents operate on the internal network with permissions tied to a service identity.
3Corrections are submitted to an engineer, and their effect is verified after deployment.
Why it matters : automation no longer stops at the diagnosis published in Slack, but also organizes the preparation of corrections and the monitoring of recovery.

OpenAI has built Performance Factory, a system that uses Codex and its models to address performance regressions, with over 1,700 investigations tracked in just over three months, according to the company. It reports more than 600 identified cases and 17 automated investigations per day. Agents prepare the fixes, which are submitted to an engineer, and then monitor performance after deployment.

Agents Access OpenAI's Internal Tools

The agents operate on OpenAI's network, in a cloud environment accessible via an internal API that triggers Codex executions. They have access to code repositories as well as the monitoring tools used by engineers: metrics, logs, traces, CPU profiles, and memory snapshots. A service identity governs their permissions and helps distinguish their actions.

According to OpenAI, this remote execution allows for parallel task processing without relying on engineers' laptops. It also avoids the manual administration of individual cloud development environments. The API is one of four components of the system, along with task tracking, a work assignment loop, and event subscriptions.

An agent does not need to remain active while waiting for a person or a deployment. The task can subscribe, for example, to review comments or failures from continuous integration checks associated with a GitHub pull request. When an event matches this subscription, the task becomes available again, and an agent can receive the next steps.

Tasks Transition from Agents to Engineers

Performance Factory organizes investigations and fixes around persistent tasks rather than around a conversation with a single agent. Engineers describe the processes in configuration files and write instructions for each step in Markdown. They specify possible states, allowed transitions, and mandatory human interventions. A loop periodically checks the available work and then launches Codex with instructions corresponding to the task's state.

The agent records its progress and advances the task when it completes its step. The status and conclusions remain stored independently of its context, allowing another agent or an engineer to pick up after a code review or deployment. Engineers can consult this tracking in Slack or in a dedicated interface; they can enrich the context and review suggested changes.

The first operational process, named Regression Workflow, opens a task when a configured performance alert is triggered. It distinguishes the investigation to be conducted, the identified cause, the proposed fix, and then the resolution, with a review and deployment before the final check. Closure occurs after verifying the return to initial performance levels.

OpenAI also presents the possibility of incorporating recurring optimizations that are executed regularly on its services. A process that detects a potential improvement can create a task in another process responsible for testing and implementing it.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

How Codex Investigates the Cause of a Slowdown

The diagnosis relies in part on perf-forensics, a shared skill in which OpenAI has formalized the method of its performance team. It provides Codex with the available tools, the conditions for their use, and an investigation guide. The agent searches for degraded metrics, the onset of their evolution, and the affected queries; the method also includes examining deployments and configuration changes that could explain the issue.

The timing of the alert is not sufficient to date the incident: it appears upon crossing a threshold, while the degradation may have begun several hours earlier, OpenAI clarifies. The agent therefore digs back through the data to find the actual start of the change. It then compares groups of queries to determine if comparable queries have become slower or if the service is receiving more requests that were already slow. According to the company, these situations lead to searching for different causes.

Once the problem is located in time and service, the agent examines deployments and features. OpenAI has tested and adjusted perf-forensics with assessments based on historical regressions. For targeted categories, including latency and availability of ChatGPT, the company claims to obtain a diagnosis from an alert in one pass, without guidance at each step.

From Manual Investigations to Autonomous Agents

Before the use of Codex, investigations combined reading metrics and logs, comparing traces, debugging on live machines, and reviewing code. OpenAI describes this as a lengthy process, often requiring in-depth knowledge of the service, where useful clues could escape engineers. Faced with recurring incidents, particularly the increased latency before the first token of ChatGPT, the team had developed specialized tools, including CPU profilers and allocation tools. Its arsenal exceeded 30 performance tools.

With the interactive assistance of Codex, an engineer could then investigate a slowdown, even in a system they were unfamiliar with. According to OpenAI, the model was capable of traversing vast codebases, identifying useful metrics and traces, retracing service interactions, and performing appropriate profiling. However, the direction of the investigation and the selection of tools remained in the hands of the engineers. The company believes this assistance made difficult cases accessible to more people, requiring less expertise and time.

With reusable skills, part of the human work was reduced to relaying the alert to Codex with the correct instructions and then reporting its response in Slack. The transition of investigations to the cloud automated the initiation of the session from the alert and the publication of conclusions in the corresponding conversation. However, this version stopped at the diagnosis: an engineer still had to take over, develop the fix, and verify its effect after deployment. The accumulation of alerts also required sorting through Slack conversations to determine which regressions needed intervention.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.