Dissecting GPT-Live

This post was jointly written by Desh and Claude, based on Desh’s notes.

OpenAI released GPT-Live-1 in the API yesterday, with launch partners already using it in production: Hatch for customer service calls, Yelp for restaurant reservations, and Speak for language tutoring. It is the API version of the model that powers voice in ChatGPT, Codex, and Work.

OpenAI has not published anything about the architecture, but we can still infer a lot from the API, the pricing, the docs, and third-party evals. In this post, let us go through what we know and what we can guess.

Full duplex

From the launch post, GPT-Live-1 is “a single model that reasons over incoming and outgoing audio together.” Kwindla Kramer calls it “the first production speech-to-speech model that can talk and listen at the same time.”

The API reflects this. Justin Uberti noted that it is very different from the Realtime API. There are no turn or audio-done events. You stream audio in, audio streams out, and the model decides when to speak.

Frontend and backend

The bigger change is that GPT-Live only handles the interaction. GPT-Realtime is a single model that does the conversation, the reasoning, and the tool calls. GPT-Live “can delegate reasoning and tool calls to a backend text model like GPT-6 Astra or a third-party model.”

There are two delegation modes:

1. Responses delegation. An OpenAI backend model gets the conversation context through the Responses API.
2. Client delegation. GPT-Live emits a delegation event and your code handles it, so you can route to your own agents.

Client delegation in particular is very powerful: it lets you add full-duplex behavior on top of an existing system of agents. Kwindla: “We can now build things like state machine conversation graphs, threaded/backtracking modes, and structured data input that were difficult with the first generation of speech-to-speech APIs.”

This kind of frontend-backend separation is pretty much accepted as the correct way of building voice agents.✦guessOne could imagine the frontend being small enough to run on-device, talking to a large server-side backend. What is new is that the behavior is trained into the model. Kwindla again: “It’s great to see this pattern trained into a production model, so that we can leverage it natively at the prompt level.”

Details from the docs:

• In client delegation, the frontend sends no task description to the backend. From the delegation guide: “The delegation object contains metadata, not task text.” You collect user and agent transcript deltas, and the backend infers the task from them. “A transcript fragment is not a complete user turn, and transcripts may contain mistakes.”
• Backend results go into the frontend’s context in three ways: session.instructions.append (“Stop speaking about that request.”), session.commentary.append for things to be spoken (“Your appointment is confirmed for Thursday at 2:00 PM”), and session.thinking.append for internal context (“Checking Thursday availability. No appointment has been booked.”).
• Each append is limited to 500 tokens. The frontend acknowledges each one with a corresponding *.appended event.
• The frontend verbalizes structured results itself. From the delegation guide: “A concise tool result doesn’t need an additional model call to rewrite it for speech.”
• Images go to a “vision-capable backend”, and typed exact values such as an order number go “to the backend that handles the task” (delegation guide). UI context is passed to the frontend only as a concise text summary of the current page. The frontend itself only handles voice and text.

Training for delegation

The frontend has to learn three things:✦guessHow would you train this? In SFT, with sandboxed backends and scripted responses. In RL, with environments containing real or mocked backends, which is expensive.

1. When to delegate a task to the backend
2. How to carry on the conversation while the task runs asynchronously
3. How to gracefully incorporate the backend response

Open question: What did the training setup for delegation look like? In particular, how were asynchronous backends, with realistic latency and occasional bad responses, simulated at RL scale? This is the most valuable thing to know for anyone trying to build a comparable model.

Prompting the frontend

The prompting guide is mostly about turn-taking behavior. Through the frontend prompt you control response length, language and pronunciation of names (via IPA), interpreter mode, when to respond (e.g. only to certain kinds of queries), handling of background noise, and asking for clarifications.

What the frontend models

Audio tokens directly. You can “shape an agent’s tone, pace, and conversational style through the system prompt,” e.g. “Speak warmly and naturally, at an unhurried pace.” You cannot change the voice. There are 12 fixed voices (Quartz, Ripple, Vesper, Willow, Stone, Gleam, Meridian, Bossa, Tempo, Beacon, Delta, Cinder), disjoint from the 10 Realtime voices. This is consistent with a single model generating acoustic tokens rather than a TTS stage.✦guessThe promotion and evals focus on delegation and turn-taking, so most post-training effort probably went into the frontend’s conversational behavior, with the audio encoder and vocoder kept close to GPT-Realtime’s. The voice sets differ, so the vocoder was at least retrained. Kundan Kumar from OpenAI: “we did specific training to further improve the controllability and instruction following of the model.”

Open question: Did the audio encoder and vocoder change from GPT-Realtime? The completely new voice set says the vocoder was at least retrained. Nothing public says anything about the encoder.

User and agent text. GPT-Live-1 “natively provides ASR transcripts and response text” and “supports keyword biasing.” The model predicts user text as it hears audio, and generates agent text before agent audio. We did the same for Moshi in our paper, which let us match S2S intelligence with T2T.

A reasoning trace. The backend can inject text through session.thinking.append, which is described as context for the model’s internal reasoning. This means the frontend also maintains a reasoning trace, in addition to the user and agent audio streams and the user and agent text streams.

Turn tokens. “Although GPT-Live-1 is not a turn-based model, it natively supports turn detection.” So it is trained with special tokens for user turn start and end.

Putting this together, Figure 2 shows what the frontend’s sequence might look like, including a delegation. We have drawn the streams as separate rows for clarity, but the model would put them all in a single interleaved sequence, with special tokens to demarcate the boundaries. Note that the frontend keeps consuming user audio in chunks throughout, including while the backend is working.

Latency and chunking

Turn-taking latency on Full-Duplex-Bench v1 is 0.8 s. Moshi gets 0.25 s and PersonaPlex under 0.2 s. But those are multi-stream models operating on an 80 ms clock (see my earlier article on multi-stream v/s interleaved models). This suggests that GPT-Live-1 is operating on a larger clock.✦guessMost likely it still uses 80 ms tokens, but prefills user audio in chunks of 2 to 5 tokens (160 to 400 ms) and decodes between chunks. TML uses 200 ms chunks, for comparison.

Two things that are not evidence for the chunk size:

• The eval harness streams audio in 20 ms PCM appends. That is a transport choice. Audio is buffered on both sides.
• The RUN harness “builds a 400-millisecond playback reserve.” In the code, this is a jitter buffer on the output side: GPT-Live’s output audio deltas have no timestamps, so the relay holds up to 400 ms before forwarding. It says something about how bursty the output is, not about input chunking.

Open question: What are the actual frame rate and chunk size? The 0.8 s turn-taking latency and the per-minute pricing both fit chunked prefill on an 80 ms clock, but a coarser tokenizer would fit too. A direct answer would settle most of the guesses in this post.

Can we infer model size from token pricing?

GPT-Live-1 costs $0.05 per minute, billed per second. The backend is billed separately, and rate limits are per concurrent session rather than per token. There is no per-token price for the frontend, but we can try to back one out. Some assumptions:

• The audio tokenizer runs at 12.5 Hz (80 ms frames), which is the most common rate for full-duplex models. The user and agent streams then produce 750 tokens per minute each.
• User audio is prefilled in chunks of 5 tokens, so one minute needs 150 prefills. Agent audio is decoded.
• Text (user transcript, thinking, agent text) adds about 1 token per 5 audio tokens, i.e. ~300 decoded tokens per minute.

That gives 750 prefilled and ~1000 decoded tokens per minute. Let us model the token pricing under these assumptions and see which GPT model would charge $0.05 for that mix. The sliders change the assumptions.

At 12.5 Hz, the implied price is close to GPT-5.5 / GPT-5.6 Sol. At 25 Hz, it is close to GPT-5.4. But serving an S2S model is more expensive than serving a text model of the same size:

• Prefix caching is a no-op. Every audio input is different.
• Prefills are tiny (a few tokens per chunk), so disaggregated prefill/decode buys little. A 5-token prefill is a decode step as far as the GPU is concerned.
• Requests cannot wait in the continuous-batching queue without hurting real-time latency.

So for 12.5 or 25 Hz, the frontend seems to be similar in size to GPT-5.6 Terra or GPT-5.4.✦guessProbably initialized from the same backbone as Terra or GPT-5.4, with the price premium over those models paying for real-time serving.

Open question: How big is the frontend, really? The pricing argument above depends on the frame rate and on how much of the $0.05 is margin for reserved real-time capacity rather than compute. The gap between Sol-priced and Terra-sized is exactly the cost of real-time serving, and we do not know that number.

Context also fills fast: ~2000 tokens per minute plus ~500 per backend call. The prompting guide notes that “the live model has a small context window.”

Evals

τ-Voice

τ-Voice is built on τ²-bench’s customer support tasks across retail, airline, and banking. It has 278 scenarios with a dynamic simulated user, and correctness is judged by comparing the final DB state to the expected state. OpenAI’s launch post reports 86.2%. The τ-bench leaderboard and Artificial Analysis report lower numbers:

Open questions:

• Why is AA’s number so different from the leaderboard and the blog? There is randomness from the dynamic user, but does that explain 20 points?
• How much of the score is the backend? Astra-medium vs Sol-low is 8.6 points, even though the tasks are simple. How do Astra and Sol score on text τ²-bench, which is the source data? That would isolate the frontend’s contribution.
• What kind of errors improved over GPT-Realtime? Did delegation fix task execution, or did the interaction quality also change outcomes?

Full-Duplex-Bench

Artificial Analysis’s conversational dynamics index is a weighted average of pause handling, turn-taking, interruption handling, and backchannel handling on Full-Duplex-Bench v1 and v1.5 (Candor, ICC, and synthetic). This is saturated at the top. The FDB v1.5 interactivity subset (background speech, speech to another person, listener backchannels, interruptions), which OpenAI reports in the launch post, is more discriminative:

Agent-directed speech is handled correctly: this is the clearest quantitative evidence that the model knows whether it is being spoken to.

Evaluation cookbook

Anyone building evals for full-duplex voice agents should read OpenAI’s evaluation cookbook. They published the full harness they use, same protocol as for GPT-Realtime. Three modes separate the failure sources:

The harness reports one primary metric, task completion, and a set of diagnostic metrics that explain why a task failed. The diagnostics split into task-completion metrics (did the agent reason, delegate, and call tools correctly?) and experience metrics (did it sound like a good conversation?). Hover over a metric for its definition.

plain applies in all modes   blue most useful in RUN   amber only when a delegation or interruption occurs. Definitions condensed from the cookbook's metrics table.

Links

• Launch post
• Live API guide, delegation, prompting
• Model page
• Evaluation cookbook and harness source
• τ-Voice leaderboard, Artificial Analysis, Full-Duplex-Bench

Some of the design elements on this page (margin notes for guesses, figures wider than the text column, and the note boxes) are inspired by Edward Yang’s interactive adaptation of “How to Parallelize a Transformer for Training”.