GPT-Live Revolutionizes Voice AI with Unprecedented Responsiveness

A Complex Challenge for Voice AI

In the field of voice artificial intelligence, determining the ideal moment to intervene in a conversation is a task more complex than it seems. Humans naturally pass the floor in a fraction of a second, but traditional voice AI systems have long struggled to keep pace. These systems relied on turn-based architectures, using small models called turn detectors to decide the right moment to respond. A premature decision could interrupt the user, while a late response gave an impression of slowness. Only after the turn detector made its decision could the larger language model (LLM) begin its processing.

GPT-Live, our third-generation voice system, was designed to overcome these limitations. By removing the turn detector, the voice model becomes full duplex, capable of listening and speaking simultaneously. This eliminates the need for a separate detector, making conversations more immediate and natural. For tasks requiring deeper reasoning or the use of tools, GPT-Live can also consult advanced models like GPT-5.5, without interrupting the flow of conversation. This combination gives GPT-Live unmatched responsiveness and conversational intelligence.

An Architecture Optimized for Latency

To deliver this experience at scale, a new system architecture has been developed, optimized for low latency. Unlike traditional request-response inference, our system streams incoming audio to the voice model and outgoing audio to the user, while managing delegation on a separate asynchronous path. Over the past six months, we have reworked model inference, context management, and multimedia transport to maintain a smooth end-to-end speech flow.

This architecture also creates a clear distinction between the core voice path and application logic, facilitating the customization of application behavior without affecting responsiveness. This foundation powers an increasing range of capabilities in ChatGPT Voice, including the new feature that allows you to control your computer and coordinate your agents in the ChatGPT desktop application.

From Turn-Taking to Continuous Streaming

Previous voice architectures were inherited from textual LLMs, with each turn represented by a discrete audio blob rather than text. In these cascading systems, speech-to-text conversion, the LLM, and text-to-speech conversion operated in series, adding latency and ignoring cues such as tone and rhythm.

Speech-to-speech models improved this approach by directly processing audio. By training the model to understand and generate speech natively, it became possible to preserve details lost during transcription and respond more quickly. However, the system still relied on the turn detector to decide when inference could begin. Although the model managed more interactions, these remained turn-based.

GPT-Live places the voice model at the heart of the conversation: audio enters and exits the model, while deeper reasoning and tool usage occur asynchronously. The main challenge of the system is to maintain an uninterrupted multimedia loop. Other tasks, such as invoking advanced models and maintaining conversation persistence, take place outside the live path.

Ensuring Continuous Inference

Maintaining this uninterrupted multimedia loop is not always straightforward. Any delay in transport, processing, or inference can result in a pause or audible artifact. A turn-based system could tolerate some variations in the timing of an audio blob's arrival. However, a live multimedia system must deliver each audio frame on time.

Previous work on ChatGPT Voice and the real-time API provided an important foundation. We had already rebuilt our voice infrastructure to stream audio and video directly in and out of our systems with lower and more predictable latency. GPT-Live has pushed this design further, streaming media to the model via a new stateful inference system designed for continuous conversation.

Streaming inference was only part of the solution. To function well in production, we also needed to ensure reliable audio delivery from the client to the inference stack and address the challenges of statefulness.

Accelerating Multimedia Flow

An early decision we made was to specifically separate the multimedia flow from application and business logic. Audio flows between the client and the voice model on a dedicated fast path. Delegation, tool usage, and other application work occur behind an asynchronous RPC boundary. A slow tool call or backend service can delay its own result but cannot block the multimedia flow.

This separation also gives the system a clear boundary for customization. Applications can change their tools, policies, and backend behaviors without affecting the multimedia frontend responsible for maintaining audio movement. The live path remains small, predictable, and focused on the work that needs to be done in real-time.

We wrote the multimedia frontend and inference logic in Go, replacing a previous implementation in Python asyncio. This significantly improved the smoothness of frame delivery, with the new system achieving a p95 that corresponds to the p50 of the old system.

WebRTC provides the transport foundation. It is designed for low-latency media and can continue to function in the event of packet loss, clock drift, and client connection changes. If packets arrive late, WebRTC can subtly stretch the audio to avoid gaps, then briefly speed up playback to catch up to real-time.

By minimizing buffering and blocking throughout the system, we can offer the sub-second responsiveness that humans expect from a conversation.

Maintaining the Conversation (Stateful)

Stateful inference has its own operational trade-offs. A voice session can remain active for a long time, but its context continuously grows, and model instances start and stop based on demand.

To address these concerns, we built a seamless transfer mechanism between model instances. When a transition is necessary, we can pre-warm a replacement model instance alongside the existing one, pre-filling it with the current session context, running inference on both in parallel, and switching when the new instance is fully ready.

The same underlying mechanism also supports dynamic context compaction. As a conversation progresses, its accumulated context may eventually exceed the model's context limit. Compaction can reduce the context size to fit within the limit, but the operation takes time. And because it modifies past context, it also invalidates the model's key-value (KV) cache, which stores attention keys and values of previously processed tokens. Rebuilding this state requires a new pre-filling, introducing additional latency.

Instead, we treat compaction as another managed transition. While the original model instance continues to converse, the system compacts the context and prepares a replacement model instance with the new context. Once that instance is ready, we can switch without multimedia interruption. This allows the system to support long-duration calls, compacting whenever necessary.

The heavy lifting remains outside the live path, so even during a transfer, the conversation never misses a beat.

Delegation Without Blocking the Conversation

GPT-Live's ability to invoke existing advanced models gives it considerable power, effectively decoupling "speaking" from deeper "thinking." But making this two-model architecture feel like a single system required solving two related engineering problems.

First, results must return quickly enough to be useful in the ongoing exchange, so we needed to minimize latency across the entire delegation path, from routing and prompt processing to inference and tool calls. At the same time, systems elsewhere in the product still need discrete messages, so we had to represent the ongoing conversation in a form they could understand.

Making Delegation Fast Enough to Feel Natural

When a delegation is sent, we optimize the time until the advanced model produces something useful for the conversation. The voice model can briefly keep the exchange moving while an advanced model reasons or uses tools, but it cannot mask an arbitrarily slow response. Therefore, we treated the entire delegation loop—routing, prompt processing, inference, and tool calls—as part of the responsiveness budget.

The first optimization is to set up the advanced model and all the tools it needs before the delegation is requested. When the voice session starts, the application server creates an inference session for the advanced model and pre-fills it with the initial conversation context, ensuring that the prompt has been fully processed before the first delegated request.

We then keep this inference session available for the duration of the voice conversation and use stable session affinity for successive requests. Coupled with prompt caching, these techniques improve latency while allowing easy recovery in case of a worker failure.

The reasoning effort, output limits, tool patterns, and model-tool round trips also affect when the conversation receives a useful result, and we have adjusted these levers to achieve faster responses. By minimizing the work required on the delegation path, we have enabled the voice model to quickly incorporate results from our advanced models.

Deriving Discrete Turns from Continuous Speech

While the voice model operates on continuous streams of speech, many surrounding systems still function on user and assistant turns, including the ChatGPT conversation interface and certain parts of our analysis and security infrastructure. Thus, the application server separates the overlapping, sometimes ambiguous conversation into discrete messages.

As audio arrives, the server uses partial transcriptions and timing signals to deduce which speaker has the floor and build a queue of messages.