Voice Agent Turn-Taking, Measured: Barge-In, Backchannels and End-of-Turn Detection

A turn-taking failure is the moment a voice agent stops sounding like a product and starts sounding like a machine. The caller pauses to think of an account number and the agent answers a half-asked question, or the caller finishes and the line just hangs there.

Every one of those moments is one of three failure modes. End-of-turn detection that is too slow produces dead air: the caller finished, the agent is still waiting for more. Detection that is too eager produces cut-offs: the caller pauses mid-thought and the agent talks over them. And backchannels, the “mm-hmm” and “yeah” a listener drops in to say keep going, get mistaken for turns, so the agent stops mid-answer to respond to encouragement. We covered the second failure in our barge-in stress test and the third in the backchannel trap; the turn-taking board measures the detectors that decide all three.

What we measured

The board scores end-of-turn detection on 200 real human clips in English, drawn from pipecat’s smart-turn-v3.1-test set, with transcripts for the text-based detectors produced by Deepgram nova-3. Each clip has a ground-truth label: END (the speaker finished) or WAIT (the speaker paused mid-thought). Each model is scored at its best operating threshold, chosen as the minimum false-cutoff rate subject to END-recall of at least 85%, so no model gets to look good by refusing to end turns. Accuracy is the share of correct END-vs-WAIT decisions; false-cutoff is the share of incomplete turns wrongly ended, the error a caller experiences as being talked over.

The results

Smart Turn v3.2, pipecat’s audio-prosody model on a Whisper-tiny backbone, scored 94.0% accuracy with 89.0% END-recall and a 1.0% false-cutoff rate (one false cutoff in 100 incomplete clips), at 32ms inference on a warm CPU. One caveat the board carries: this eval set is Smart Turn’s home distribution, so treat 94% as a ceiling rather than a field expectation.

LiveKit’s Intl turn-detector v0.4.1, a text model that reads the transcript, scored 87.0% accuracy with 86.0% END-recall and a 12.0% false-cutoff rate, at 29ms inference plus the wait for the STT transcript to arrive.

Turnsense (SmolLM2-135M), also transcript-based, scored 86.0% accuracy with 88.0% END-recall and a 16.0% false-cutoff rate; at 127ms it is the slowest as shipped, because it pads every input to 256 tokens regardless of length.

LiveKit’s English turn-detector v1.2.2 scored 82.0% accuracy with 87.0% END-recall and a 23.0% false-cutoff rate, at 5ms the fastest forward pass on the board.

The VAD-plus-silence-timer baseline scored 46.9% accuracy: 100% END-recall, because a silence timer ends every turn eventually, and a 100% false-cutoff rate, because it also ends every pause. That is the floor semantic detection has to beat.

The ranking flips on synthetic speech

The same models on synthetic (TTS) clips produce a different board. Turnsense scored 98.8% accuracy with 0.0% false-cutoff, LiveKit’s multilingual detector 90.6% with 6.2%, LiveKit’s English detector 79.2% with 26.2%, and Smart Turn dropped to 62.4% accuracy with a 57.7% false-cutoff rate. The reason is the signal each model type reads: TTS renders a finished-sounding pitch contour regardless of the words, so an audio model cannot hear incompleteness that only exists in the text.

On real clips, that acoustic signal is real and measurable. Across the eval set, completed turns fall in pitch (final F0 slope of -119.7 Hz/s) while incomplete ones hold or rise (+42.9 Hz/s), and completed turns taper in energy (-14.9 dB/s against -2.5 dB/s). Completion is acoustically visible, which is why a prosody model can decide before the transcript exists. How Speko runs Smart Turn on live calls covers what it takes to use that in production.

When to use which

For best overall accuracy on real human callers, the measurements favor Smart Turn v3.2: 94.0% accuracy, a 1.0% false-cutoff rate, and it runs in parallel with STT instead of waiting on the transcript. If your traffic includes synthetic or replayed speech, or you want a detector that reads the transcript, LiveKit’s Intl v0.4.1 is the strongest text-based option measured, at 87.0% on real clips and 90.6% on synthetic. And if you are choosing between a bare VAD timer and anything semantic, the 46.9% baseline is the whole argument: every semantic detector on the board beats it by 35 points or more.

The full table, including END-recall and inference latency per model, is on the turn-taking board, and the raw numbers are in the published dataset.