A caller can't see your model's benchmark scores. They can hear the gap before it answers.
That gap is the whole game. As of June 2026, an end-to-end p50 below 800ms is the working definition of a "human-feeling" voice agent, and anything north of 1,200ms measurably reads as robotic, according to production-grounded benchmarks from Trillet and Tested.media.
Cross that 1.2-second line and callers start talking over the agent, repair latency spikes, and CSAT in A/B tests drops 10 to 20 points even when the answer is correct.
Which means the constraint that actually picks your architecture and your vendor isn't model quality. It's milliseconds.
TL;DR
Voice agent latency, not model IQ, is the binding constraint in production. Hit p50 ≤ 800ms and p95 ≤ 1,200ms or the conversation feels broken. A five-stage cascaded pipeline realistically sums to 1,255, 1,780ms, so you either collapse stages with a native speech-to-speech model or apply streaming-partials plus TTS-on-partials to claw a cascaded stack down to ~465ms.
Everything below flows from that one number.
Key takeaways
- 800ms is the human-feel bar; 1,200ms is the robotic cliff. Target p50 ≤ 800ms, p95 ≤ 1,200ms.
- Independent benchmarks beat vendor claims. Retell ~620, 680ms, Vapi 720, 900ms, Bland ~850ms p50. Vapi's sub-500ms and Bland's 412ms are vendor-reported and don't reproduce.
- Native speech-to-speech is the only architecture that consistently clears 800ms (Hume EVI 3 ~300ms), but it costs 5, 50x more per minute and locks you in.
- Streaming partials + TTS-on-partials is the best cascaded lever, measured at ~465ms p50 on Vapi by AssemblyAI in April 2026.
- Plan for p95 ≈ 2x p50. Vendors publish medians; the tails are yours to discover.
- No first-party Claude voice API exists as of June 2026. Chain STT + Claude + TTS, or use an aggregator.
Why 800ms is the production voice agent latency benchmark
Here's the answer first, because this is the number worth quoting: a voice agent feels human at or below 800ms median end-to-end latency, and starts feeling robotic above about 1,200ms. That crossover is a conversational-design heuristic validated against 2025, 2026 production traces from Vapi, Retell, and Bland.
The mechanism is turn-taking. Human conversation tolerates a sub-second gap before silence reads as "the other person isn't there." Tested.media calls the failure point the 1.2-second cliff: past it, callers begin overlapping the agent, the agent barges over the caller, and the repair spiral begins. Microsoft's Copilot Studio guidance and LiveKit's pipeline docs corroborate the same threshold.
So the operating target is concrete: p50 ≤ 800ms, p95 ≤ 1,200ms, with sub-500ms reserved for the highest-stakes flows like sales and support escalations.
One catch that shapes everything downstream: vendors publish p50 and hide p95. The Cekura benchmark across 5M+ minutes (published June 13, 2026) and independent reviews consistently find median latency reported behind a "production" qualifier with tails undisclosed. Budget for p95 ≈ 2x p50, and ~3x on tail turns that fire tool calls.
Vapi vs Retell vs Bland: the independent latency benchmark
The third-party platforms diverge on three axes: pipeline architecture, independently measured latency, and all-in cost per minute. The numbers below come from independent benchmarks with vendor claims flagged inline.
| Platform | Architecture | p50 E2E (independent) | p95 E2E | All-in cost/min |
|---|---|---|---|---|
| Hume EVI 3 (EVI 3, May 2025, still current) | Native speech-language model | ~300ms | not published | ~$0.07, 0.10 |
| ElevenLabs Conversational AI (Eleven v3 GA Feb 2026) | Co-located single-vendor | 300, 800ms | not published | $0.10, 0.30 |
| Retell AI (GPT-5.4 Fast support, Q2 2026) | BYO orchestrator + native TTS | 620, 680ms | ~1.2, 1.6s | $0.11, 0.31 |
| Vapi (Voices v2, week of June 1 2026) | BYO orchestrator | 720, 900ms | ~1.4, 1.8s | $0.13, 0.33 |
| Bland AI (self-hosted, May 2026 docs) | Proprietary self-hosted stack | ~850ms (412ms vendor) | ~1.6s+ | $0.11, 0.14 |
The vendor-vs-reality gap matters when you plan. Bland has the widest divergence: 412ms claimed, ~850ms measured. The likely explanation is the native-path-versus-BYO-path distinction, where Bland's own dedicated servers run faster than a customer's self-assembled stack on top. Vapi claims sub-500ms (a mid-2025 figure); 2026 independent tests show 720, 900ms, so treat the sub-500ms number as superseded for planning. Retell, ElevenLabs, and Hume land within ~50, 100ms of their own claims.
What moved in the last 30, 60 days: Vapi added xAI Grok STT/TTS and Vapi Voices v2 and announced a ~$500M Series B on May 12, 2026; Bland shipped Watchtower analytics and iMessage Enterprise; ElevenLabs brought Eleven v3 GA on Feb 2, 2026 and reached an $11B valuation, with Flash v2.5 hitting ~75ms time-to-first-byte.
First-party real-time voice APIs: who is actually fastest
A new category of foundation-model speech-to-speech APIs shipped between March and May 2026 and now rivals the orchestrator stacks. They differ sharply on speed and price.
| Model | Released | Independent TTFT | Audio pricing |
|---|---|---|---|
| xAI grok-voice-think-fast-1.0 | Apr 25, 2026 | 0.78s (fastest) | $0.05/min |
| OpenAI gpt-realtime-2 | May 7, 2026 | 1.12s minimal / 2.33s high | $32/$64 per 1M audio tokens |
| AWS amazon.nova-2-sonic | May 14, 2026 | 1.14s | $0.0034/$0.0136 per 1k speech tokens |
| Google gemini-3.1-flash-live | Mar 26, 2026 | 2.98s (slowest) | $3/$12 per 1M audio tokens |
| Anthropic | , | no first-party voice API | , |
On Artificial Analysis measurements, xAI's Grok Voice is the only first-party model that consistently hits sub-second first-audio, because it runs "background reasoning" off the audio critical path. OpenAI's gpt-realtime-2 is the trade-off model: it's the sole first-party option pairing GPT-5-class reasoning with native speech-to-speech, but its reasoning_effort lever swings TTFT from 1.12s at minimal to 2.33s at high.
That's a deliberate dial, not a flaw. Reasoning-heavy claims processing is exactly where you accept 2.33s to avoid a confident wrong answer at 800ms.
The pricing spread is brutal: roughly 50, 100x from Grok Voice ($3/hour) to gpt-realtime-2 at heavy audio use ($5, 10/hour).
And the conspicuous gap is real. Anthropic has no first-party real-time voice API as of June 19, 2026. Claude Voice Mode is a consumer mobile beta with no api.anthropic.com surface. To use Claude in a voice agent, chain STT + Claude API + TTS, or reach for an aggregator like Inworld Realtime, which exposes anthropic/claude-sonnet-4-6.
Where the milliseconds actually go in a real-time voice agent
A canonical 2026 voice agent is a five-stage cascade: VAD/endpointing → STT → LLM → TTS → network. Decompose it and the budget becomes obvious.
| Stage | Latest component | p50 |
|---|---|---|
| VAD / endpointing | Silero VAD v5 | 30, 80ms |
| STT (streaming) | Deepgram Nova-3 | 200, 250ms |
| LLM TTFT | GPT-5-mini / Haiku 4 (small) | 250, 450ms |
| LLM TTFT | GPT-5-class (large) | 600, 1,100ms |
| TTS first-byte | Cartesia Sonic-3.5 / Flash v2.5 | 75, 100ms |
| Network | WebRTC intra-region | 30, 80ms |
Two things jump out. The LLM TTFT stage dominates at large models; 1,100ms p50 on GPT-5-class reasoning is normal. And TTS is no longer the bottleneck. The Flash-v2.5 / Sonic-3.5 generation pushed first-byte synthesis to ~75ms, the first time TTS stopped being the wall.
Sum it up and the cascade is unforgiving:
- Baseline, sequential, no streaming: VAD 80 + STT 250 + LLM 1,100 + TTS 250 + network 100 ≈ 1,780ms. Outside the target.
- Optimized, streaming partials: VAD 30 + STT 150 + LLM 875 + TTS 100 + network 100 ≈ 1,255ms. Inside the human-feeling zone but still above target.
- Native speech-to-speech collapse: 300, 800ms E2E. The only architecture that consistently clears 800ms in production.
VAD is nearly free but it's where the architecture choice bites hardest. Turn detection decides whether you start STT on a partial transcript or wait for the user to finish, a 200, 500ms swing on every turn.
The single biggest cascaded latency win: TTS-on-partials
If you stay cascaded, do this one thing first. Start TTS synthesis on the LLM's first complete semantic chunk instead of waiting for the full generation. AssemblyAI measured this at ~465ms p50 end-to-end on Vapi in April 2026, using Deepgram Nova-3 streaming STT plus Cartesia Sonic-3.5 plus TTS-on-partials.
Their published build is the named production test.
It's a single-deployment case study, so it won't generalize to every Vapi config. But it's the practical sweet spot: sub-500ms without abandoning the cascade's observability. The cost is premature endpointing on roughly 3, 8% of turns, which you tune per domain.
Here's the lever table, ranked by what it buys and what it costs:
| Lever | Latency gain | Main cost |
|---|---|---|
| Native speech-to-speech | −300 to −600ms p50 | 5, 50x per-minute cost, vendor lock-in |
| Streaming partials + TTS-on-partials | −300 to −700ms perceived | Premature endpointing 3, 8% of turns |
| Speculative/predictive TTS | −200 to −500ms perceived | 10, 25% wasted TTS compute |
| Model cascade (small→large routing) | −200 to −400ms on simple turns | Routing classifier 92, 97% accurate |
| Semantic VAD vs energy-based | −100 to −250ms on barge-in | Domain-specific tuning |
When latency is the wrong thing to optimize
The 800ms frame is wrong for a meaningful slice of production agents. Three cases flip the priority.
Quality beats raw speed when wrong answers are expensive. In medical intake, financial KYC, and multi-step reasoning, a hallucinated reply costs more than 200ms. Hume EVI 3 wins blind preference tests 55, 60% of the time against faster cascaded competitors because empathic prosody is the product, not the latency. Run gpt-realtime-2 at reasoning_effort=high and accept 2.33s when the alternative is a confident wrong answer in 800ms.
Endpointing beats first-token speed for live support. The turn-taking problem is where "the agent felt slow" actually lives in CSAT. Over-tuned VAD cuts users off mid-sentence, and that UX failure drops scores faster than a 100ms latency win raises them.
Cascaded pipelines still win six ways speech-to-speech can't: stage-level transcript logging, no model lock-in, mature multilingual coverage, per-customer fine-tuning, regulatory audit trails, and cost at scale. Past roughly 1M minutes/month, a cascaded stack on GPT-5-mini plus Flash v2.5 at ~$0.05, 0.10/min is materially cheaper than native speech-to-speech, and streaming partials close most of the latency gap.
What this means for you
Pick your architecture from the constraint, not the brochure. A quick decision matrix:
| If you need… | Choose |
|---|---|
| Lowest latency + emotional quality | Hume EVI 3 (~$0.07, 0.10/min) |
| Sub-500ms in a cascaded stack | Vapi + Nova-3 + Cartesia Sonic-3.5 + TTS-on-partials |
| Cheapest first-party real-time voice | xAI Grok Voice ($0.05/min) |
| GPT-5-class reasoning in voice | OpenAI gpt-realtime-2 |
| Self-hosting, audit, custom voices | Bland ($0.11, 0.14/min) |
| Claude inside a voice agent | Aggregator (Inworld Realtime → claude-sonnet-4-6) |
Then ship against a checklist. Pin every model version with its release date in your SLO doc, because anything older than ~3 months is superseded. Measure p50 and p95 in your own traffic.
Default to streaming partials + TTS-on-partials. Reach for native speech-to-speech only when latency or emotion is literally the product. And treat every vendor latency number as a target, never a production p50.
What to watch next: the first-party speech-to-speech models all have under 90 days of production track record as of June 2026, so expect pricing and behavior to shift as providers tune for load. The two signals worth tracking are whether Anthropic finally ships a voice API surface, and whether xAI's off-critical-path reasoning design (0.78s TTFT) gets copied. If it does, sub-second first-party voice stops being a one-vendor advantage and becomes the floor.
Sources
- Retell vs Vapi vs Bland vs Synthflow: 2026 Head to Head, Tested.media
- Voice AI Latency Benchmarks 2026, Trillet
- How to build the lowest latency voice agent in Vapi (~465ms), AssemblyAI
- Benchmarking Language Models for Real-World Voice Agents, Cekura
- Mastering OpenAI's Realtime API, The New Stack
- OpenAI API Changelog
- Real-Time Speech AI Gets Serious, Towards AI
- ElevenLabs drops latency to 75 milliseconds, DeepLearning.AI
- Turn-Taking in Voice AI: The Hidden Problem, Retell AI
- Best Voice AI May 2026, Future AGI
- Hume AI Review 2026, ToolChase
- ElevenLabs launches Flash, The Decoder