WebSockets Enhance Codex: Accelerated Response API

Codex and Workflow Optimization with the Response API

When a user requests Codex to fix a bug, the process involves a series of complex steps. Codex analyzes the code, identifies relevant files, establishes context, makes necessary modifications, and tests the corrections made. This process relies on numerous requests to the Response API, with each step requiring interaction with the model to determine the next action, execute tools on the user's computer, and return results to the API.

These repeated interactions can lead to significant delays, forcing users to wait while Codex executes complex tasks. In terms of latency, Codex's agent loop spends a large portion of its time on three main steps: processing requests through API services, model inference, and client time for tool execution and context building. Inference, which occurs on GPUs, has historically been the slowest step, masking the overhead of API services. However, as inference speeds up, the cumulative cost of the API becomes more apparent.

In this article, we detail how we succeeded in making agent loops using the API 40% faster end-to-end. This has allowed users to benefit from an increase in inference speed, going from 65 to nearly 1,000 tokens per second. We achieved this goal through techniques such as caching, eliminating unnecessary network hops, improving our security stack to quickly report issues, and most importantly, establishing a persistent connection with the Response API instead of multiplying synchronous API calls.

The Response API, a Bottleneck

Previous models of the Response API, such as GPT-5 and GPT-5.2, operated at around 65 tokens per second (TPS). With the launch of GPT-5.3-Codex-Spark, a fast coding model, our goal was to achieve a speed exceeding 1,000 TPS, thanks to the use of specialized Cerebras hardware optimized for LLM inference. For users to truly benefit from the speed of this new model, it was imperative to reduce the overhead of the API.

In November 2025, we launched a performance sprint on the Response API, resulting in numerous optimizations to reduce the latency of the critical path of a single request:

• Caching tokens and model configuration in memory to avoid costly tokenization and network calls during multi-turn responses.
• Reducing latency from network hops by eliminating calls to intermediary services and directly calling the inference service.
• Improving our security stack to execute certain classifiers and report conversations more quickly.

These improvements led to a nearly 45% reduction in time to the first token (TTFT), reflecting the responsiveness of the API. However, even with these enhancements, the overhead of the Response API remained too significant compared to the speed of the model, forcing users to wait for CPUs to execute our API before they could utilize the GPUs serving the model.

The problem was structural: each Codex request was treated as independent, requiring the processing of conversation state and other reusable contexts in each follow-up request. Even when most of the conversation remained unchanged, we still had to process the entire history, which became costly as conversations lengthened.

Towards a Persistent Connection

To optimize our design, we rethought the transport protocol: could we maintain a persistent connection and cache state, rather than establishing a new connection via HTTP and sending the entire conversation history for each follow-up request? The idea was to send only the new information requiring validation and processing, while caching reusable state in memory for the duration of the connection. This would reduce the overhead from redundant work.

We explored several approaches, including WebSockets and bidirectional gRPC streaming. We chose WebSockets because, as a simple message transport protocol, it allowed users to keep their input and output formats for the Response API unchanged. This solution was developer-friendly and integrated with our existing architecture with minimal disruption.

The first WebSocket prototype changed our perception of what was possible regarding the latency of the Response API. An engineer from the Codex team, an expert in the entire API stack, developed a prototype by running a Codex agent overnight.

In this prototype, agent deployments were modeled as a single long response. Using asyncio features, the Response API would asynchronously block in the sampling loop after a tool call had been sampled, and the Response API would return a response.done event to the client. After executing the tool call, clients would return a response.append event with the tool's result, which would unblock the sampling loop and allow the model to continue.

An analogy here is treating the local tool call as a hosted tool call. When the model calls web search, the inference loop blocks, calls a web search service, and places the service's response into the model's context. In our design, we did the same; but instead of calling a remote service, we sent the model's tool call back to the client via WebSocket. When the client responded, we integrated the client's tool call response into the context and continued sampling.

This design was extremely efficient as it eliminated repeated API work across an agent deployment. We could perform pre-inference work once, pause for tool execution, and perform post-inference work once at the end.

Unfortunately, this came at the cost of a less familiar and more complicated API form. We wanted developers to be able to integrate WebSocket support without having to rewrite their API integration around a new mode of interaction.

Maintaining API Familiarity While Making the Stack Incremental

For the version we launched, we returned to a familiar form: continuing to use response.create with the same body, and using previous_response_id to continue the conversation context from the previous response's state.

On a WebSocket connection, the server maintains an in-memory cache of the previous response's state, specific to the connection. When a follow-up response.create includes previous_response_id, we retrieve that state from the cache instead of rebuilding the entire conversation from scratch.

This cached state includes:

• The previous response object
• The previous input and output items
• Tool definitions and namespaces
• Reusable sampling artifacts, such as previously rendered tokens

By reusing the previous response state in memory, we were able to achieve several major optimizations:

• Ensuring that some of our security classifiers and request validators only process new inputs, not the entire history each time.
• Maintaining an in-memory cache of rendered tokens that we complete to avoid unnecessary tokenization.
• Reusing our successful model resolution/routing logic across requests.
• Overlapping non-blocking post-inference work like billing with subsequent requests.

The goal was to get as close as possible to the low-overhead prototype while maintaining an API form that developers already understood and around which they had built.

Establishing a New Benchmark for Speed

After a two-month sprint building the WebSocket mode, we launched an alpha version with key coding agent startups so they could integrate it into their infrastructure and gradually increase traffic safely. Alpha users loved it, reporting up to 40% improvements in their agent workflows. Given the positive feedback from the alpha, we were ready to launch.

The results of the launch were immediate. Codex quickly shifted the majority of its traffic from the Response API to the WebSocket mode, seeing significant latency improvements. For GPT-5.3-Codex-Spark, we achieved our goal of 1,000 TPS and observed peaks of up to 4,000 TPS, demonstrating that the Response API could keep up with much faster inference in real production traffic. The impact also quickly manifested in the developer community:

• Codex rapidly shifted the majority of its traffic to WebSockets.
• Codex users utilizing the latest models such as GPT-5.3-Codex, GPT-5.4, and beyond all benefit from the acceleration of the WebSocket mode.
• Vercel integrated the WebSocket mode into the AI SDK and saw latency reductions of up to 40%.
• Cline's multi-file workflows are 39% faster.
• OpenAI models in Cursor have become up to 30% faster.

The WebSocket mode is one of the most significant new capabilities of the Response API since its launch in March 2025. We moved from idea to production in just a few weeks thanks to close collaboration between OpenAI's API teams and Codex. This not only dramatically improves the latency of agent deployments but also supports a growing need for creators: as model inference becomes faster, the services and systems surrounding inference must also speed up to transfer these gains to users.