← Blog

Turn-Taking for Voice Agents: What Actually Works in 2026

End-of-turn detection, semantic VAD vs server VAD, barge-in, and backchanneling for voice agents - five turn-taking systems measured on 200 real clips.

Turn-taking is everything a voice agent decides about when to speak: whether the caller is done (end-of-turn detection), whether to stop talking because the caller wants in (barge-in), and whether a sound from the caller is a turn at all (backchannels, coughs, line noise). Get it wrong in one direction and the agent talks over people mid-thought. Get it wrong in the other and it sits in dead air.

This guide defines each piece and attaches the numbers we have published: the five-system turn-taking board (English, 200 real human clips), a 158-trial VAD matrix against gpt-realtime-2, a four-stack barge-in test, and how Speko runs turn detection on live calls. Where we have no measurement, the section says so instead of estimating.

If you take one number away: a plain VAD-plus-silence-timer baseline scored a 100% false-cutoff rate on our benchmark. The best measured system scored 1.0% on the same 200 clips.

Last updated: August 14, 2026.

What is turn-taking in a voice agent?

Turn-taking is the set of real-time decisions that determine when a voice agent starts and stops speaking in a conversation with a human. It decomposes into three separable problems, and they fail in different ways:

  • End-of-turn (EOT) detection: after every pause, decide whether the caller finished or is still mid-thought. Failure mode: the false cutoff.
  • Barge-in: while the agent is speaking, decide whether incoming caller audio is a real interruption. Failure mode: the agent either plows on over the caller or halts for a cough.
  • Backchannel handling: recognize that “uh huh”, “okay”, and “mm hmm” are encouragement, not a turn-take. Failure mode: the agent cancels its own answer because the caller was politely listening.

Most “the bot feels robotic” complaints trace to one of these three, not to the voice itself.

End-of-turn detection: five systems measured

End-of-turn detection is the decision, after every pause in caller audio, of whether the turn is complete (END) or the caller is still thinking (WAIT). Our turn-taking board scores five systems on 200 real human English clips, each at its best operating threshold. The board’s own metric definitions: EOT accuracy is “Correct END-vs-WAIT decisions across 200 real clips, at each model’s best operating threshold (min false-cutoff s.t. END-recall >= 85%)”, and false-cutoff is “WAIT clips wrongly ended = talked over the caller mid-thought. The costly live-call error.”

RankSystemInputEOT accuracyEND-recallFalse-cutoffInference (warm CPU)
1Smart Turn v3.2 (Pipecat, Whisper-tiny)audio (prosody)94.0%89.0%1.0%32ms (49ms p90)
2LiveKit Intl turn-detector v0.4.1text (transcript)87.0%86.0%12.0%29ms (57ms p90)
3Turnsense SmolLM2-135Mtext (transcript)86.0%88.0%16.0%127ms (193ms p90)
4LiveKit EN turn-detector v1.2.2text (transcript)82.0%87.0%23.0%5ms (24ms p90)
5VAD + silence timer baselineaudio (energy)46.9%100%100%~1ms

Two caveats travel with the table. First, the eval set is Smart Turn’s home distribution, so treat 94% as a ceiling, not a universal claim. Second, the text-based models additionally wait on the STT transcript (~0-125ms in this setup) before they can decide, which the inference column does not include.

The scoped takeaway: on real human English clips, the audio-prosody model leads on the error that matters (1.0% false-cutoff against 12-23% for the transcript models), and every trained model beats the silence timer by a wide margin.

False cutoffs: the error that makes agents feel robotic

A false cutoff is a mid-thought pause that the agent wrongly treats as the end of the turn, so it starts talking over the caller. It is the costly live-call error because the caller experiences it as being interrupted, every time it happens.

The baseline row explains why silence timers cannot fix this. A VAD-plus-silence-timer reaches 100% END-recall, because it ends every turn eventually, and a 100% false-cutoff rate, because a person pausing to think and a person who finished both produce silence. That combination is the floor: 46.9% EOT accuracy.

In production the threshold matters as much as the model. Speko runs Smart Turn v3.2 on live calls at a decision threshold of 0.85 instead of the stock 0.5, leaning the system toward patience. Over 3,742 live turns in a 30-day window, 6.5% of turns fell in the band the stock threshold would have committed on (a cutoff) but the raised threshold held. The full configuration, including per-question threshold shifts and the guards, is in How We Stopped Cutting Callers Off.

Where our own table flips: the synthetic set

The board also scores the same systems on a synthetic (TTS-generated) clip set, and the ranking inverts:

SystemSynthetic EOT accuracySynthetic false-cutoff
Turnsense SmolLM2-135M98.8%0.0%
LiveKit Intl turn-detector v0.4.190.6%6.2%
LiveKit EN turn-detector v1.2.279.2%26.2%
Smart Turn v3.262.4%57.7%

Smart Turn, the leader on real clips, drops to last with a 57.7% false-cutoff rate on synthetic ones. The mechanism: TTS renders even an unfinished sentence fragment with finished-sounding prosody, so a model that listens to pitch and energy hears “done”. A model that reads the transcript is immune to that trick. If your test harness feeds synthesized speech to an audio-based turn detector, your lab numbers will slander a model that works on real callers, and vice versa. We publish both tables because either one alone misleads.

Semantic VAD vs server VAD

Server VAD is an energy detector: it fires when microphone volume crosses a threshold, with no knowledge of the words. Semantic VAD is a classifier that scores the probability the user finished speaking based on the words they uttered. On OpenAI’s Realtime API these are the two turn-detection modes, and the docs recommend the semantic one for natural conversation.

We ran 158 trials against gpt-realtime-2 across both modes (server_vad thresholds 0.5/0.8/0.9; semantic_vad eagerness low/medium/high/auto) and seven stimulus types. Results:

  • server_vad at threshold 0.8 or 0.9 absorbed the under-breath “mm hmm”. semantic_vad cancelled the response at every eagerness setting.
  • Every conversational-volume backchannel (“uh huh”, “okay”, “yeah”, full-voice “mm hmm”) cancelled the response at every configuration in both families.
  • A 400ms cough was absorbed in 14 of 14 trials, across every configuration in both families.
  • A real interrupt (“Wait, stop”) cancelled correctly in both modes.

There was no configuration in the matrix where semantic_vad outperformed tuned server_vad. Our recommendation for gpt-realtime-2 under these conditions is server_vad with threshold 0.8. And the recommendation loses too: at conversational volume, every backchannel still cancels the agent on both modes, so neither mode is good enough for production agents serving backchannel-heavy 1:1 dialogue. The fix is a turn-detection model trained on the backchannel-vs-turn-take distinction, not a VAD setting.

Backchanneling: the stimulus nobody stress-tests

A backchannel is a short vocal acknowledgment (“uh huh”, “mm hmm”, “okay”) a listener produces to encourage the speaker to continue, without taking the turn. Voice agent demos stress-test the cough; the cough is a solved problem (14/14 absorbed in our matrix). The polite worded backchannel is what breaks responses.

The latency data shows the semantic classifier can tell the difference and cancels anyway: a real interrupt trips cancellation in 350-430ms while backchannels take 500-1100ms, 1.5-3x slower. The classifier hesitates, then commits to the same cancellation, and the API surfaces none of that internal uncertainty.

Two directions of backchanneling matter for builders:

  • Inbound (caller backchannels at the agent): measured above. The only surviving configuration was server_vad 0.8/0.9 on a quiet “mm hmm”.
  • Outbound (agent says “mm hmm” while the caller talks): we have not measured this and do not publish numbers on it. The prerequisite is a turn model with an explicit HOLD state, because an acknowledgment inserted at the wrong moment is the agent barging in on the caller.

Barge-in: the defense stack

Barge-in is the caller interrupting the agent mid-response, and handling it means answering two questions fast: is this real speech, and if so, how quickly can the agent shut up. We tried to break four production stacks with a 400ms cough at -6 dB, injected 1.2 seconds into the agent’s response: an OpenAI Realtime stock config, a tuned Realtime config, and two cascaded pipelines. All four absorbed it. Zero false interrupts.

The engineering numbers that make that work:

  • Asymmetric VAD hysteresis: Silero-style ~80ms to confirm speech onset, ~400ms to confirm offset. Fast to notice a voice, slow to declare it gone.
  • Interrupt debounce: 200-400ms of STT round-trip to confirm the sound produced actual words before killing the agent’s audio.
  • No tail replay: when the interrupt is real, kill the agent’s audio at the source and never replay the buffered half-sentence afterward. Replayed fragments are the “why is it still talking?” moments.

And from the VAD matrix: a genuine “Wait, stop” cancels in 350-430ms across every configuration tested. Real interrupts are not the hard case. The hard case is everything that sounds like one.

Endpointing latency: how fast is fast enough

Endpointing latency is the time from the caller actually stopping to the agent committing to speak. It stacks the VAD silence window, the turn model’s inference, and, for transcript-based models, the wait for the STT to finalize.

Measured inference on the board (warm CPU): LiveKit EN 5ms (24ms p90), LiveKit Intl 29ms (57ms p90), Smart Turn v3.2 32ms (49ms p90), Turnsense 127ms (193ms p90), silence-timer baseline ~1ms. Note our production pick is not the fastest: LiveKit EN infers in 5ms against Smart Turn’s 32ms. Nobody perceives 27ms; everyone perceives a false cutoff, which is why the board ranks on accuracy at a recall floor, not on speed.

The transcript wait is the bigger hidden cost for text models: ~0-125ms in the board’s setup, and on our separate STT board (FLEURS read speech, n=30, us-east4) measured finalize p50s span 66ms to 1124ms across 17 streaming models. Pick a slow-finalizing STT and your text-based turn detector inherits every millisecond.

The speed-vs-patience trade has a principled answer: fix a recall floor, then minimize the damaging error. That is how the board picks operating points (min false-cutoff subject to END-recall >= 85%), and it is how a compliance-bound agent, like a collections bot that must never talk over a customer, should tune: raise the threshold, cap the hold (Speko caps at 1.5s, because longer holds created dead air that confused callers), and measure the band between your threshold and the stock one on live traffic.

Noisy calls: what holds up and what we have not measured

Noise attacks the two turn-taking inputs differently. Energy-only VAD degrades first by construction: it cannot distinguish background chatter from the caller, which is one reason the silence-timer baseline is the floor on our board. Trained models read richer signals. On the board’s real clips, the prosody separating done from thinking is large: final F0 slope averages -119.7 Hz/s on complete ENDs vs +42.9 Hz/s on incomplete WAITs, and final energy slope -14.9 dB/s vs -2.5 dB/s.

What is measured under adverse audio today:

  • Transient noise: the 400ms cough was absorbed 14/14 in the VAD matrix and by 4/4 production stacks in the barge-in test.
  • The transcript path: text-based turn detectors are only as good as the words they receive. In our production-conditions streaming STT study (6 models, 60 clips per condition, treat +-1 point as noise), word error rates on degraded audio:
ModelTelephony WERNoisy WER
Qwen3-ASR7.8%7.3%
Smallest Pulse8.4%8.4%
Cartesia Ink-210.2%9.7%
ElevenLabs Scribe v2 Realtime10.6%10.2%
Deepgram Nova-312.0%12.9%
Soniox stt-rt-v515.7%16.6%

What is not measured: the turn-taking board itself has no noise condition, so we do not publish noisy-call EOT accuracy for any of the five systems, and we will not estimate one here. If your deployment lives on noisy telephony, run the candidate turn model on your own recorded turns before trusting anyone’s clean-audio number, ours included.

Choosing a platform for turn-taking

Turn-taking quality is a property of the turn-detection model and the runtime knobs around it, not of a platform’s brand. We have not measured Vapi, Retell, or Bland platform-level turn-taking, so we publish no ranking of them. What we can give you is the component evidence above and the questions that separate platforms in practice:

  1. Which end-of-turn model does it run, and can you swap it? The measured spread is 46.9% to 94.0% EOT accuracy; this choice dominates everything else.
  2. Can you set the decision threshold and hold behavior? Stock thresholds are tuned for demos, not patience.
  3. How does barge-in resume? If it replays the buffered tail of the interrupted sentence, callers will hear the agent talk past them.
  4. What happens on a backchannel? Ask for the vendor’s measured behavior on “uh huh” at conversational volume; the measured default on the Realtime API is cancellation, every time.

For the platform-vs-runtime-vs-router taxonomy itself, see voice agent platforms compared.

FAQ

What is the best end-of-turn detection for voice agents?

Smart Turn v3.2 (Pipecat) has the best measured EOT accuracy on our board: 94.0%, with a 1.0% false-cutoff rate, on 200 real human English clips. The result is conditional: the eval set is Smart Turn’s home distribution, so treat 94% as a ceiling, and on synthetic TTS clips the ranking inverts with Turnsense at 98.8%. Full table at benchmarks.speko.ai/turntaking.

How do I stop my voice agent from interrupting callers?

Replace the silence timer with a trained end-of-turn model, then raise its decision threshold. A VAD-plus-silence-timer baseline scored a 100% false-cutoff rate on our benchmark; trained models on the same clips scored 1.0% (Smart Turn v3.2) to 23.0% (LiveKit EN). In production, Speko runs Smart Turn at threshold 0.85 instead of the stock 0.5; over 3,742 live turns, 6.5% were holds the stock threshold would have turned into cutoffs.

Semantic VAD vs server VAD for voice agents: which should I use?

server_vad with threshold 0.8, if you are on OpenAI’s Realtime API. In 158 trials against gpt-realtime-2, server_vad at 0.8 absorbed the quiet backchannel that semantic_vad cancelled on at every eagerness setting, and the two modes tied on everything else: loud backchannels (both fail), the cough (both absorb it), and the real interrupt (both cancel correctly, in 350-430ms). There was no setting where semantic_vad won.

Which voice agents support backchanneling like mm-hmm sounds?

No configuration we measured survives a conversational-volume “mm hmm”. Across 158 trials on gpt-realtime-2, every full-volume backchannel (“uh huh”, “okay”, “yeah”, “mm hmm”) cancelled the agent’s response at every server_vad threshold and every semantic_vad eagerness; only an under-breath “mm hmm” survived, and only on server_vad 0.8 or 0.9. Absorbing real backchannels requires a turn model trained on the backchannel-vs-turn-take distinction; our turn-taking board measures end-of-turn, not backchannel absorption, so we do not publish a backchannel leaderboard yet.

Which voice AI platforms have the most natural turn-taking?

We publish component measurements, not platform-brand turn-taking rankings, and we have not measured Vapi, Retell, or Bland on this axis. The component that dominates conversational feel is the end-of-turn model (measured spread: 46.9% to 94.0% EOT accuracy), so evaluate platforms on whether they let you choose that model and tune its threshold, hold time, and barge-in resume behavior. The four questions to ask are in the section above.

What are the top voice AI components for handling barge-in?

A VAD with asymmetric hysteresis plus an interrupt debounce, in front of a clean audio kill. The measured stack: Silero-style hysteresis (~80ms speech onset, ~400ms offset), a 200-400ms STT round-trip debounce to confirm words before interrupting, and no replay of the buffered tail after the interrupt. Four production stacks built this way absorbed a 400ms, -6 dB cough with zero false interrupts, while real “Wait, stop” interrupts cancelled in 350-430ms.

What voice activity detection holds up best on noisy phone calls?

Energy-only VAD is the first component to degrade in noise, because it cannot tell background chatter from the caller; it is also the floor of our board at 46.9% EOT accuracy even on clean clips. Transient noise is handled: a 400ms cough was absorbed in 14/14 trials and by 4/4 production stacks. For transcript-based turn detection, noise arrives through the STT: measured noisy-audio WER in our streaming study runs from 7.3% (Qwen3-ASR) to 16.6% (Soniox stt-rt-v5). We have no noise-condition numbers for the turn-taking board itself and do not estimate them.

How do teams balance end-of-turn speed against false-interruption risk?

Fix a recall floor, then minimize false cutoffs, and spend your latency budget on patience rather than model inference. Our board scores every system at “min false-cutoff s.t. END-recall >= 85%”, and the inference cost of the accuracy leader is 32ms, which no caller perceives. For a compliance-bound agent that must never talk over a customer: raise the threshold (Speko uses 0.85 vs the stock 0.5), cap the hold at something like 1.5s so patience does not become dead air, and log the band between the two thresholds as your measured saves.

Why does my outbound dialer’s agent cut off callers who pause to think?

Because a silence timer cannot tell thinking from finished: both are silence. That baseline ends every turn on our benchmark, a 100% false-cutoff rate. Swapping in a trained end-of-turn model drops false cutoffs to between 1.0% and 23.0% depending on the model, at 5-127ms of added inference, and raising the decision threshold above stock buys further patience for a few hundred milliseconds of hold on borderline turns.

How do I make an AI receptionist feel like a real conversation instead of a script?

Three measured levers, in order of impact: stop cutting callers off (a trained EOT model at 1.0% false-cutoff vs the timer’s 100%), stop cancelling on their acknowledgments (server_vad 0.8 or 0.9 were the only Realtime API configs that absorbed a quiet “mm hmm”), and make barge-in clean (kill audio in 350-430ms on real interrupts, never replay the tail). Having the agent produce its own “mm hmm” acknowledgments is unmeasured territory; the prerequisite is a HOLD-capable turn model so the acknowledgment does not land as an interruption of the caller.

Callers keep talking over my support bot mid-sentence. What actually fixes talk-over?

Treat talk-over as barge-in to handle, not to prevent. The measured recipe: detect real interrupts fast (350-430ms in our matrix), debounce transients for 200-400ms so a cough does not halt the agent (4/4 stacks passed), kill the agent’s audio at the source without replaying the buffered tail, and after the barge-in, wait for the caller’s real turn-end instead of pouncing on their first word. Callers often talk over agents that ramble past the answer; shorter agent turns reduce the trigger, though that is design advice, not a measurement.

Sources