What a voice agent actually hears

Six streaming speech-to-text models, fed the same clips through the same gateway, across five conditions built to look like a real call — accents, telephone audio, noise, medical vocabulary, and spontaneous speech. We measured Word Error Rate on the streamed final: the exact text a live agent consumes as the caller talks.

Why this benchmark exists

A voice agent is only as good as the words it hears. Speech-to-text is the first step in the pipeline, and if it gets the transcript wrong, everything downstream inherits the mistake: the language model answers a question the caller never asked, the agent interrupts at the wrong moment, the caller sighs and repeats themselves. It is the one component with no recovery path — a wrong word early becomes a wrong turn later.

So which streaming STT model belongs in front of an agent? Every vendor has a benchmark that crowns them first. The problem is that almost all of those numbers come from clean, read speech — audiobook-quality audio recorded in a quiet room. That is not a phone call. On a real call the audio is band-limited to 8 kHz, there is a fan running or traffic outside, the caller has an accent and says “um,” and the words are “metoprolol” and “confirmation code four-two-nine.” Clean benchmarks measure none of that. We wanted the other number: how these models do on the audio an agent actually hears.

The five conditions

We didn’t invent audio. Each condition is a real English corpus (or a controlled degradation of one) chosen to stress a different thing a phone agent runs into every day:

• Accented — spontaneous multi-accent English from parliamentary recordings (VoxPopuli). Real speakers, real accents, but reasonably clean audio. This is roughly the best case a streaming model sees.
• Telephony — the exact same accented clips, band-limited to a 300–3400 Hz phone channel at 8 kHz. Because the words are identical to the accented set, any change in WER is purely the phone line, not different content.
• Noisy — the same clips again, under an added noise floor. Same trick: the delta isolates robustness to background noise.
• Medical — clinical lectures and consultations (MultiMed). Dense domain vocabulary and named entities — drug names, anatomy, acronyms — where a general model’s language prior stops helping.
• Spontaneous — unscripted, in-the-wild English (People’s Speech). Disfluencies, false starts, overlapping speech, uneven recording quality. This is the hardest thing on the list and the closest to a messy live call.

Every model heard the same clips, fed the same way through the same /v1/transcribe/stream socket. Only the model changed.

The leaderboard

Two things stand out. First, the models in front — Qwen3-ASR and Smallest Pulse, tied at 9.9% — are not the ones with the loudest launch posts; they’re a general multilingual model from Alibaba and a small, cheap streaming model. Second, a model that shipped recently claiming the #1 streaming word error rate lands fourth on this mix.

That last part deserves a fair word. Cartesia’s Ink-2 is a genuinely strong model, and their launch post is refreshingly honest that clean benchmarks mislead — they test on accented call-center audio precisely because it is harder. We simply tested on a wider slice of production. “Best on fourteen English accents” and “best on the full mess a phone agent sees” turn out to be different sentences. The model that wins depends entirely on which sentence you’re asking about.

Where models pull apart

Clean-ish accented speech barely separates the field — the top five sit within a few points of each other. The surprise is how little the phone line and the noise floor cost: telephony and noisy stay close to accented for almost every model (Cartesia and Deepgram are essentially flat; Deepgram even ticks down slightly on the phone). The acoustic conditions everyone worries about turn out to be the ones modern streaming models handle best.

The real difficulty is content, not channel. Medical vocabulary opens a gap — the field spreads from Qwen’s 8.2% to Soniox’s 19.1% — and spontaneous speech is the great equalizer: every model’s WER roughly doubles, and the ranking reshuffles. The models that win on clean audio are not the ones that hold up when a real human is talking over themselves.

Every number

The models, one at a time

Two models weren’t broken — we were

On the first run, two providers looked terrible. Deepgram Nova-3 returned a full sentence as “Than euthanasia.” Qwen3-ASR trailed off in the middle of a word. If we had published those numbers, we would have libeled two good models.

So we did the thing a benchmark is supposed to do before it ships a ranking: we tried to break our own result. We called each vendor’s API directly, bypassing our gateway, with the same audio. Both returned the full transcript. The bug was ours — and it was the same bug wearing two different coats. Deepgram streams a sentence as interim hypotheses and then commits it with an empty final marker; our aggregator was discarding the interim the instant that empty final arrived, keeping only the next fragment. Qwen3-ASR uses server-side voice-activity detection to decide a turn is over, and it only finalizes once it hears trailing silence — and we were handing it a clip that ended without any. Neither model was wrong. Our gateway was feeding both of them in a way real calls never do.

A benchmark’s job isn’t to produce a ranking. It’s to produce a ranking you can trust — which means being at least as suspicious of your own pipeline as you are of the models.

Both are small fixes, and both are now merged (#1350, #1352). Every number on this page is the corrected one. We’re telling you about the bugs because the bugs are the point: the difference between a benchmark and a marketing chart is what you do when a number looks wrong.

How we ran it

Six models, five conditions, sixty clips each — 1,800 streamed transcriptions, all through wss://api-staging.speko.dev/v1/transcribe/stream, one machine, one word-error-rate normalizer applied identically to every hypothesis (lower-casing, punctuation and number normalization, so “twenty” and “20” aren’t counted as different words). No per-model tuning beyond feeding each one the way it expects to be fed. Same clips, same path, same math; the only variable is the model. Because telephony and noisy are degradations of the accented clips, the deltas between those three columns are pure condition, not content.

Limitations, honestly. Sixty clips per condition is enough to rank the field but not to split hairs between two models a point apart — treat ±1 point as noise. WER is a blunt instrument: it counts every word equally, so a fumbled “um” costs the same as a wrong drug name, even though only one of those breaks a call. Everything here is English; multilingual is a separate study. And this measures accuracy, not latency or turn-taking — which, for a live agent, matter just as much.

So which one should you use?

The honest answer is: it depends on what your callers sound like. If your traffic is clean, accented, or over the phone — think structured call-center flows — several models are within a point of each other and Cartesia and Deepgram’s acoustic robustness is a real advantage. If your calls are open-domain and messy — real people talking over themselves — Smallest Pulse’s flat curve on spontaneous speech is the safer bet, with Qwen3-ASR close behind and cheaper on clean audio. If you handle domain vocabulary like medicine, Qwen’s lead on terminology is decisive.

The larger point is that there is no single #1 — only a #1 for your distribution. The only way to know is to measure on that, not on read speech in a quiet room. That’s the number we care about, and it’s the one we’re building Speko to measure — honestly, for every model, out loud.