OpenAI just detailed how they built GPT-Live, their third-generation voice system, and the architecture is legitimately fascinating. This isn't incremental improvement—it's a fundamental rethink of how voice AI works.
The core insight: previous voice systems were turn-based, waiting for a tiny "turn detector" model to guess when you'd stopped speaking before the real LLM could start working. Guess too early and you get interrupted; guess too late and the conversation feels sluggish. GPT-Live throws out the turn detector entirely and makes the voice model full-duplex—it listens and speaks simultaneously.
What makes this announcement compelling isn't just the product—it's that OpenAI shared the actual engineering. Stateful inference, dynamic context compaction, asynchronous delegation to frontier models, WebRTC transport optimization. This is a rare window into production voice AI architecture at scale.
The turn-based bottleneck
Earlier voice systems inherited the discrete turn-taking of text LLMs. Cascaded architectures ran speech-to-text, then the LLM, then text-to-speech in series—slow and lossy. Speech-to-speech models improved latency by processing audio natively, but still relied on turn detection to decide when inference could begin.
The turn detector was a lose-lose proposition. It had to make split-second guesses about speaker intent with incomplete information. Any delay became audible lag; any eagerness caused interruptions. And until it fired, the big model sat idle.
GPT-Live's architecture is different: audio streams continuously in and out of the voice model. Deeper reasoning and tool use happen on a separate asynchronous path. The system's job is sustaining an uninterrupted media loop, not orchestrating turns.
Streaming inference all the way down
Keeping that media loop uninterrupted is harder than it sounds. Any delay in transport, processing, or inference becomes an audible pause. A turn-based system could tolerate variation in when audio blobs arrived. A live system must deliver every audio frame on schedule.
OpenAI had rebuilt voice infrastructure for ChatGPT Voice and the Realtime API to stream audio and video with lower, more predictable latency. GPT-Live pushed further, streaming media all the way to the model through a new stateful inference system.
The key architectural decision: separate media flow from application logic. Audio moves between client and voice model on a dedicated fast path. Delegation, tool use, and backend services happen behind an asynchronous RPC boundary. A slow tool call can delay its own result but cannot stall media flow.
This separation creates a clean customization boundary. Applications can change tools, policies, and backend behavior without affecting the media frontend. The live path stays small, predictable, and focused on real-time work.
They rewrote the media frontend and inference logic in Go, replacing a Python asyncio implementation. The improvement: the new system's p95 latency matched the old system's p50. WebRTC provides the transport foundation—designed for low-latency media, it continues operating through packet loss, clock drift, and connection changes.
Stateful inference is operationally weird
Voice sessions run for a long time. Context continuously grows. Model instances spin up and down based on demand. How do you keep a conversation going across all that churn?
OpenAI built seamless handoff between model instances. When a transition is needed, they warm a replacement instance alongside the existing one, prefill it with current context, run inference against both in parallel, and cut over when the new instance is ready.
The same mechanism handles dynamic context compaction. As conversations grow, they eventually exceed the model's context limit. Compaction reduces context size but takes time and invalidates the key-value cache, requiring a new prefill.
Instead of blocking, they treat compaction as a managed transition. The original instance keeps chatting while the system compacts context and prepares a replacement. When ready, they switch over without media interruption. Long-running calls get compaction whenever necessary, but the conversation never pauses.
Delegation without feeling like delegation
GPT-Live can invoke frontier models like GPT-5.5 for deeper reasoning or tool use while maintaining conversational flow. Making this two-model architecture feel like one system required solving latency and representation problems.
Making delegation fast enough
When delegation is dispatched, they optimize for time-to-useful-result. The voice model can briefly keep conversation moving, but can't hide arbitrarily slow responses. The full delegation loop—routing, prompt processing, inference, tool calls—is part of the responsiveness budget.
First optimization: set up the frontier model before delegation is requested. When a voice session starts, the application server creates an inference session for the frontier model and prefills it with initial context. The prompt is fully processed before the first delegated request.
They keep that inference session available for the conversation's duration with stable session affinity for successive requests. Together with prompt caching, this improves latency while keeping worker failures recoverable.
Reasoning effort, output limits, tool schemas, and model-tool round trips also affect response speed. They tuned these levers to get faster results. By minimizing work on the delegation path, the voice model can quickly incorporate frontier model outputs.
Deriving discrete turns from continuous speech
Here's a gnarly problem: the voice model operates on continuous streams, but ChatGPT's conversation UI, analytics, and safety systems still expect discrete user/assistant turns. So the application server has to tease apart overlapping, occasionally ambiguous conversation into discrete messages.
As audio arrives, the server uses partial transcripts and timing signals to infer who has the floor and build a message queue. The newest message remains provisional—text, timing, and speaker assignment can all change as more speech arrives. Once a speaker has sustained the floor long enough for reliable attribution, the server finalizes the message.
Speaker overlap complicates this. A brief acknowledgment from the assistant ("mm hmm," "okay") while the user talks shouldn't necessarily become its own message. But a substantive interjection often should. They prioritize coherence in displayed assistant responses even when the user speaks in the middle.
Every segmentation policy trades freshness for certainty. Committing too early produces fragmented history and unstable ordering; waiting too long delays transcripts and dependent features. The system maintains two views: a speculative view of current state and an authoritative record of what was said. The UI uses the speculative view; analytics use the final transcript.
Protocol-level startup optimization
Responsiveness starts when you click the button. GPT-Live must establish the media path and begin feeding audio through the model before conversation begins. Every part of the startup sequence is on the critical path.
WebRTC provides a strong real-time foundation, but standard connection establishment involves multiple round trips—offer/answer exchange, ICE candidate gathering, DTLS handshake. Each round trip adds latency.
They optimized by frontloading work. The client initiates media path setup during the button press, before the user speaks. ICE candidates are gathered early. Server-side resources are provisioned in parallel. By the time the user finishes their first utterance, the model is already listening.
What this architecture enables
This foundation now powers growing capabilities in ChatGPT Voice, including the newly launched ability to control your computer and coordinate agents in the ChatGPT desktop app. The clean boundary between core voice path and application logic makes customization easy without affecting responsiveness.
The six-month timeline is notable. Building stateful streaming inference, context compaction, async delegation, and protocol optimization isn't trivial. They shipped a production system that handles continuous conversation at scale while maintaining sub-second responsiveness.
The broader pattern
GPT-Live represents a specific architectural choice: prioritize the media loop, push everything else off the critical path, and build clean boundaries between real-time and non-real-time work. That pattern generalizes beyond voice—any real-time AI system faces similar latency budgets and streaming constraints.
The willingness to rewrite in Go for p95/p50 improvements, to build stateful handoff mechanisms, to treat compaction as a managed transition—these are production engineering decisions, not research moves. It's the unglamorous work that makes responsive AI actually ship.
What I'd love to see next: how they handle failure modes, what the monitoring looks like, how they debug latency regressions in a system this stateful. The architecture is impressive, but operations at scale is where things get really interesting.
For now, this is one of the clearer pictures we've gotten of modern voice AI architecture. Full-duplex isn't just a feature—it's a complete rethink of the inference stack.