What a Voice Agent Actually Hears

Almost every speech-to-text leaderboard is run on clean, read speech — audiobook audio in a quiet room. It’s a useful number, and it’s the wrong one for a voice agent. A real call is band-limited to 8 kHz down a phone line, there’s a fan running or traffic outside, the caller has an accent and says “um,” and the words are “metoprolol” and “confirmation code four-two-nine.” None of that is in the benchmark.

So we measured the other number. Six streaming STT models, fed the same clips through the same gateway, across five conditions built to look like production. We scored Word Error Rate on the streamed final — the exact text a live agent consumes as the caller talks, not a tidy batch re-transcription you’d never run on a call.

The leaderboard

Two things stand out. The models in front — Qwen3-ASR and Smallest Pulse, tied at 9.9% — aren’t the ones with the loudest launch posts; they’re a general multilingual model and a small, cheap streaming model. And a model that shipped recently claiming the #1 streaming word error rate lands fourth on this mix.

That last part deserves a fair word. Cartesia’s Ink-2 is a genuinely strong model, and their launch post is refreshingly honest that clean benchmarks mislead — they test on accented call-center audio precisely because it’s harder. We simply tested on a wider slice of production. “Best on fourteen English accents” and “best on the full mess a phone agent sees” turn out to be different sentences. The model that wins depends entirely on which one you’re asking about.

Where models pull apart

The surprise is how little the phone line and the noise floor cost. Telephony and noisy stay close to accented for almost every model — Cartesia and Deepgram are essentially flat, and Deepgram even ticks down slightly on the phone. The acoustic conditions everyone worries about turn out to be the ones modern streaming models handle best.

The real difficulty is content, not channel. Medical vocabulary opens a gap — the field spreads from Qwen’s 8.2% to Soniox’s 19.1% — and spontaneous speech is the great equalizer: every model’s WER roughly doubles and the ranking reshuffles. The models that win on clean audio are not the ones that hold up when a real human is talking over themselves. Smallest Pulse is the flattest curve on the hardest condition; Qwen wins the vocabulary test.

Two models weren’t broken — we were

On the first run, two providers looked terrible. Deepgram Nova-3 returned a full sentence as “Than euthanasia.” Qwen3-ASR trailed off in the middle of a word. If we had published those numbers, we would have libeled two good models.

So we did the thing a benchmark is supposed to do before it ships a ranking: we tried to break our own result. We called each vendor’s API directly, bypassing our gateway, with the same audio. Both returned the full transcript. The bug was ours — the same bug wearing two coats. Deepgram streams a sentence as interim hypotheses and commits it with an empty final marker, and our aggregator was discarding the interim the instant that empty final arrived. Qwen3-ASR uses server-side voice-activity detection to decide a turn is over, and only finalizes once it hears trailing silence — and we were handing it a clip that ended without any.

Neither model was wrong. Our gateway was feeding both of them in a way real calls never do. Both fixes are merged (#1350, #1352), and every number here is the corrected one. We’re telling you about the bugs because the bugs are the point: a benchmark’s job isn’t to produce a ranking, it’s to produce a ranking you can trust — which means being at least as suspicious of your own pipeline as you are of the models.

How we ran it

Six models, five conditions, sixty clips each — 1,800 streamed transcriptions through one socket, one WER normalizer applied identically to every hypothesis, one machine. Accented is VoxPopuli; telephony and noisy are the same clips degraded, so the deltas between those three are pure condition, not content; medical is MultiMed; spontaneous is People’s Speech.

A word on limits, honestly: sixty clips ranks the field but won’t split two models a point apart — treat ±1 point as noise. WER counts every word equally, so a fumbled “um” costs the same as a wrong drug name even though only one breaks a call. Everything here is English. And this measures accuracy, not latency or turn-taking — which, for a live agent, matter just as much and live on their own boards.

The full interactive breakdown — every model, every condition — is at /streaming-stt. The takeaway isn’t “use model X.” It’s that there’s no single #1, only a #1 for your distribution — and the only way to know is to measure on that, not on read speech in a quiet room.

FAQ

Which streaming STT model is most accurate on real phone-call audio?

Qwen3-ASR and Smallest Pulse, tied at 9.9% average word error rate across five production conditions (accents, telephony, noise, medical vocabulary, spontaneous speech) in our six-model streaming test. Neither is the loudest launch; both beat the model claiming the #1 streaming WER, which landed fourth on this mix.

Which STT is most accurate for medical terms?

Qwen3-ASR, at 8.2% WER on medical vocabulary (MultiMed clips) in our streaming test - the field spread all the way to 19.1% for Soniox on the same condition. Content, not channel, is what separates streaming STT models: medical terms opened a wider gap than phone-line audio or background noise did.

Does telephone audio ruin STT accuracy?

Less than you’d expect. Telephony and noisy conditions stayed close to the accented baseline for almost every model we tested - Cartesia and Deepgram were essentially flat, and Deepgram even ticked slightly down on the phone. The hard conditions are content: spontaneous speech roughly doubled every model’s WER and reshuffled the ranking.

Is Cartesia Ink-2 the best streaming STT?

Not on the full production mix. Ink-2 claims the #1 streaming word error rate and is genuinely strong on accented call-center audio, but it landed fourth in our five-condition test, behind Qwen3-ASR and Smallest Pulse at 9.9% WER. Best on fourteen English accents and best on the full mess a phone agent hears are different claims.