The Pause Is the Hard Part: How We Stopped Cutting Callers Off

Ask anyone who has talked to a voice agent what makes it feel like a machine. It is almost never the voice. It is the interruption — you take a breath in the middle of a sentence and the agent barges in over you, answering a question you had not finished asking.

That failure has a name: a false cutoff. It is the most damaging error in a live call, and it is entirely a turn-taking problem. The agent has to answer one deceptively hard question after every pause: is the caller done, or just thinking?

This is how we answer it.

Why the obvious approaches don’t work

The naive version is a silence timer: wait N milliseconds of quiet, then talk. This is what a raw VAD (voice-activity detector) gives you, and it is hopeless. A person pausing to think and a person who has finished both produce silence. Tune the timer short and you trample every hesitation; tune it long and the agent feels laggy and still cuts off the slow talkers. On our own turn-taking benchmark a VAD-plus-timer ends every turn — a 100% false-cutoff rate. It is the floor, not a solution.

The next idea is to read the words. Feed the running transcript to a small language model and ask “does this look like a finished sentence?” This works surprisingly well for clear incompletes — “I was thinking maybe we could” is obviously unfinished — and models like LiveKit’s turn-detector and Turnsense score well on it. But it has two problems in a real call. It can only run after speech-to-text produces the transcript, so it inherits the STT’s latency. And words lie: “Okay, so my account number is four one—” is a complete-looking clause right up until it isn’t.

The signal that actually separates done from thinking is prosody — the pitch, energy, and final lengthening of human speech. Humans hold the turn on sound, not grammar. So that is what we run.

What we actually run

Speko uses Smart Turn v3.2, Pipecat’s open-source audio turn-detection model. Three decisions matter:

It runs in-process. The model is an ~8 MB ONNX file baked into the worker image and executed via onnxruntime-node — no network hop. About 55 ms of local CPU per turn. We used to host it as a remote endpoint and paid 200–800 ms of round-trip for the privilege; moving it in-process deleted that.

It listens to audio, not text. We tee the caller’s audio stream into a rolling 8-second, 16 kHz buffer. When the VAD detects end-of-speech, the model reads that buffer and returns one number: P(the turn is complete). Because it works on the audio directly, it runs in parallel with STT instead of waiting behind it — it decides first.

It gates, it doesn’t trigger. VAD detecting silence starts the clock; Smart Turn decides what to do with it. If the probability clears the threshold, the turn commits. If it doesn’t, we HOLD — keep the mic open and extend the wait.

The stack that kills false cutoffs

Getting from “we run a good model” to “callers stop getting cut off” took a series of specific, boring fixes. Each one is a lever against the same failure.

The threshold is 0.85, not 0.5. This is the single biggest one. The model’s stock decision boundary is 0.5. On live calls that was a disaster: on 2026-06-17 we watched borderline turns score p ≈ 0.76–0.83 and commit — the model was 76% sure the caller was done, so it talked over the 24%. Clearly-finished turns sit above 0.95. Raising the bar to 0.85 leans the whole system toward patience.

The threshold bends to the question. A flat threshold is still wrong, because the right amount of patience depends on what you just asked. After a yes/no question we drop the bar (−0.25) — the answer ends cleanly, so commit fast. After an open-ended question we raise it (+0.05), and after one that invites a number or an address — “what’s your phone number?” — we raise it more (+0.07), because people deliver those in chunks with pauses in between. This context-conditioning is on by default.

Under the threshold, we wait — up to a point. A held turn extends toward a maxDelay of 1.5 s rather than committing at the ~600 ms floor. That ceiling used to be 3 s, which created so much dead air that callers started talking again out of confusion — so patience has a limit too.

Guards catch the model’s own mistakes. Smart Turn under-scores some genuinely complete utterances — a crisp “305, this is Jenny.” can land at p ≈ 0.63. So a text guard commits a clearly-finished sentence anyway — unless the audio strongly vetoes it (p below 0.35), which means a real mid-thought continuation is happening and the words are lying.

Barge-in is a clean interrupt. When a caller talks over the agent, we kill the agent’s audio at the source and — critically — never replay the buffered tail. The framework default pauses and resumes, replaying the half-sentence the caller just interrupted; those replayed fragments are exactly the “why is it still talking?” moments. We turn that off. And after a barge-in, we suppress the fast-commit guards so the agent waits for the caller’s real turn-end instead of pouncing on their first word.

Does it actually work? The numbers.

Mechanisms are easy to describe and easy to fool yourself about. So we measured it on live traffic.

Every turn logs the probability and the verdict. A turn where the model scores between 0.5 and our threshold is one the stock model would have committed on — a cutoff — but ours HOLDs. That is a directly measurable “false cutoffs prevented” rate.

Over 3,742 turns across live calls in the last 30 days: 6.5% fell in that danger band — roughly one turn in fifteen that the stock 0.5 threshold would have cut off, held instead. Another 23% are genuine mid-thought holds (both thresholds agree to wait), and 5.7% are the opposite save — turns the model under-scored that our guards committed, preventing dead air rather than a cutoff.

The threshold distribution shows the context-conditioning working in the wild: most English turns run at 0.85, but 772 of them were raised to 0.90–0.92 (open and number-taking questions) and a slice dropped to 0.5–0.67 (closed questions and Spanish).

One turn in fifteen sounds small until you are the caller who got cut off. Across every call, on every pause, it is the difference between an agent that listens and one that interrupts.

The honest caveat

If you look at our turn-taking benchmark, Smart Turn does not top the table — the text models do. That is not a contradiction; it is the measurement being honest. Our benchmark clips are synthesized speech, and text-to-speech renders even an unfinished fragment with finished-sounding prosody. An audio model correctly hears “done” because the audio sounds done — so a controlled dataset built from TTS structurally favors models that read the words. On real speech, where a mid-thought pause actually trails off, the gap narrows and prosody earns its keep. We run the audio model on live calls for the thing the benchmark can’t stage: the human pause.