The truth about voice AI benchmarks (and how to run your own)

The #2 text-to-speech model in the world, by arena Elo, is the only top-three model that falls apart on a ten-minute take: 353 severe-tier failure windows in one 8:34 render, where the other two logged zero. The Elo score is real. It just measures 30-second clips, and nobody ships 30-second clips.

That is not an argument that the leaderboard is wrong. It is an argument that the leaderboard answers a different question than the one you are asking when you pick a model for production. This post documents five specific ways the published voice AI benchmarks mislead, each with a measured counter-example (named model, number, date), then lays out what a trustworthy benchmark actually requires, and how to run your own in an afternoon. Every number below links to a published post with the raw method; nothing here is a claim you have to take on faith.

Part 1: Five ways voice benchmarks mislead

1. They score 30-second clips. You ship minutes.

The Artificial Analysis Speech Arena is the closest thing TTS has to a neutral ranking: blind human A/B votes on paired samples, aggregated into Elo. The methodology is sound. The quiet assumption, not written on the methodology page, is that every clip is roughly thirty seconds long.

In May 2026 we took the Arena’s top three at the time - Cartesia Sonic 3.5 (Elo 1218), Google Gemini 3.1 Flash TTS (Elo 1209), ElevenLabs Eleven v3 (Elo 1184) - and rendered the same 9,184-character script through each, pinned to dated model snapshots, then scored the audio with twelve signals per 3-second window: speaker-embedding similarity, no-reference PESQ/STOI/SI-SDR, ASR confidence, jitter, shimmer, harmonic-to-noise ratio, and more (“Artificial Analysis Ranks Gemini 3.1 Flash TTS #2. We Asked It for Ten Minutes.”, 2026-05-23).

At three minutes, all three looked fine - minor wobbles, nothing publishable. At ten minutes the data separated completely. Counting windows in the severe tier (summed z-score of 30 or more, multi-signal breakdown):

• Eleven v3: 0
• Sonic 3.5: 0
• Gemini 3.1 Flash TTS: 353

The model ranked #2 in the world stops sounding like itself around minute one and never recovers. You do not need our analyzer to hear it; Whisper independently corroborates it, splitting Gemini’s take into 249 segments against 108 and 93 for the other two, because the audio forced 2-3x more detected discontinuities. The failure reproduces run-to-run at the same approximate timestamps, so it is not noise.

Both numbers are correct: on a 30-second clip Gemini really is second-best in the world, and on a ten-minute take it really is the only one of the three that breaks. A benchmark that only tests the first length cannot see the second failure, and almost every public voice leaderboard only tests the first length.

2. They report one averaged number. Production fails on specifics.

Word error rate and pass rate are averages, and averages are where the failures that end a phone call go to hide.

Concrete case: in July 2026 we stress-tested the two cheapest streaming TTS providers in our gateway on an invoice line, “The invoice total came to $2,349.99 plus a 15 euro shipping fee” (“The .99 Problem”, 2026-07-09). On clean prose, Alibaba’s qwen3-tts-flash is genuinely good - round-trip character error rate near zero. On the invoice line, across eight takes, it never once read $2,349.99 correctly. The cents came back as “.999” five times, “.9” once, a clean “.99” exactly once, and the dollar sign turned into a euro in half the takes. xAI’s grok-tts read the same line correctly four out of four times. Word-level timestamps show qwen emits no clean “.99” token at all - just a 0.7-second smear that the transcriber best-guesses as “and” at 57% confidence. An aggregate WER-style score over mixed text would rate this model excellent. Your customer, quoted the wrong invoice total, would not.

The same flattening happens with LLM pass rates. Two Cerebras-hosted models on our LLM board, gpt-oss-120b and gemma-4-31b, both post a perfect 100/100/100 on overall, tool-calling, and grounded reliability - on the leaderboard they are twins (“Both Models Scored 100%. Then We Stress-Tested Them.”, 2026-07-09). Stress-tested past the score, at temperature 0, three runs each: gpt-oss-120b invented store hours (“Mon-Fri 9-6, Sat 10-4”) and shipping times it was never given, 3 runs out of 3, despite an explicit never-invent instruction - and invented a different set of hours in a later test, which is the tell. It also went completely silent on 16 of 27 adversarial tool-calling turns, firing the tool with zero spoken words while the caller listens to dead air; gemma went silent on 2. Both models are 100% by the scoring rule. A pass rate is a verdict with the evidence thrown away.

If a benchmark does not test entities - money, dates, confirmation codes, drug names - and does not show you per-run transcripts, it cannot distinguish a model that is safe to put on a phone from one that reads your prices wrong at a confidence you will find persuasive.

3. They test demo conditions. Calls happen on phone lines, with humans.

Almost every STT leaderboard is scored on clean, read speech - audiobook audio in a quiet room. A production call is band-limited to 8 kHz, has a fan or traffic in the background, and features a real human saying “um” halfway through a drug name.

In July 2026 we ran six streaming STT models through one gateway across five conditions built to look like production: accented speech (VoxPopuli), the same clips degraded to telephony and to noisy, medical vocabulary (MultiMed), and spontaneous speech (People’s Speech) - 1,800 streamed transcriptions total, scored on the streamed final text a live agent actually consumes, not a tidy batch re-transcription (“What a Voice Agent Actually Hears”, 2026-07-11).

Two results matter for this argument. First, a model that shipped claiming the #1 streaming word error rate landed fourth on this mix. To be fair to that vendor (Cartesia’s Ink-2): their launch post is refreshingly honest that clean benchmarks mislead, and they test on accented call-center audio precisely because it is harder. “Best on fourteen English accents” and “best on the full mess a phone agent sees” simply turn out to be different sentences. Second, and more surprising: the acoustic conditions everyone worries about barely moved the needle - telephony and noise cost almost nothing for most models. What separated the field was content. Medical vocabulary spread the models from 8.2% to 19.1% WER, and spontaneous speech roughly doubled every model’s WER and reshuffled the ranking. The models that win on clean audio are not the ones that hold up when a real human talks over themselves.

The condition set of a benchmark is a claim about what production looks like. If that claim is “read speech, quiet room, full bandwidth,” the ranking is answering a question no phone call will ever ask.

4. They score the brand. You deploy one specific voice.

TTS arenas aggregate votes at the brand level: a vote for one Cartesia clip is a vote for Cartesia. But Cartesia Sonic 3.5 ships 378 English voices, and they do not behave alike.

We sampled 50 of those 378 voices at random and ran a 90-second drift probe on each - a sliding speaker-embedding window, cosine similarity back to the start of the take (“The Cartesia Drift”, 2026-05-25). Only 5 of 50 stayed above the perceptual same-speaker threshold (cosine 0.70 at 60 seconds). 33 were borderline. 12 sounded like a different person within the minute. The default voice - the one you get if you do not choose - drifted at beta = -0.30 cosine per minute, roughly four times steeper than the next provider’s default, and it drifted by about the same amount in every one of five languages we tested. Meanwhile the “British Lady” voice on the identical model and identical text held at 0.73: same model, different catalog entry, opposite production verdict. The catalog UI does not tell you which kind you picked.

Control tags cut the same way. Speechify’s simba-3.2 sits at the top of our board on clarity and steadiness - and a single SSML attribute, prosody pitch=“+15%”, halves speaker similarity (ECAPA cosine 0.63 baseline down to 0.28), while plus-or-minus 40% takes it to roughly zero: in embedding space, a different person, on a one-second utterance (“The Pitch Trap”, 2026-07-07). Rate and volume tags sat at baseline; only pitch swapped the speaker. No leaderboard scores identity-preservation under a provider’s own control tags, which means a top-of-board model can fail on the very first sentence you author expressively.

A brand-level Elo cannot warn you about either failure, because both live one level below the thing being ranked. When 10% of a sampled catalog is long-form-safe, “Cartesia is #1” and “the Cartesia voice you picked is production-safe” are unrelated statements.

5. The benchmark publisher usually sells something the benchmark flatters.

This criticism is not ours alone, and credit where due. Coval has published a series arguing that vendor TTS benchmarks are, in their words, marketing copy with measurements attached: real numbers under conditions picked to flatter the system being measured (coval.ai/blog, “Best TTS Providers 2026: Why Vendor Benchmarks Lie”). They back it with public head-to-heads and a methodology repo. Telnyx published “Low Latency Voice AI: Why Provider Benchmarks Cannot Be Trusted,” making a point that deserves wider circulation: changing one configuration variable swings measured latency more than the difference between any two providers.

Both critiques are correct, and both come with the same footnote: Coval sells voice-agent evaluation, and Telnyx sells the low-latency infrastructure its latency benchmark flatters. That is not an accusation of dishonesty - it is the observation that in this field the entities with the resources to run benchmarks are the entities with a position in the outcome. We are not exempt: Speko sells a voice gateway, and our benchmarks run through it. The incentive problem does not go away because your numbers are real. Coval’s own point applies to everyone, including the people making it.

You cannot fix incentives with promises. You can fix them with exposure: publish the methodology, the model snapshots, the raw per-run data, and the failure clips, so that a motivated stranger can catch you. Here is what that looks like in practice: on the first run of our STT benchmark, two providers looked broken - Deepgram Nova-3 returned a full sentence as “Than euthanasia,” and Qwen3-ASR trailed off mid-word. Before publishing, we called both vendors’ APIs directly with the same audio, bypassing our own gateway. Both returned perfect transcripts. The bugs were ours: our aggregator discarded interim hypotheses when Deepgram committed with an empty final marker, and we fed Qwen clips with no trailing silence when its server-side VAD needed some to finalize. We would have libeled two good models with a benchmark that ran through our own product. Both fixes are public PRs on the platform repo (#1350, #1352) and every published number is the corrected one. The lesson generalizes: be at least as suspicious of your own pipeline as you are of the models, because the most likely source of a dramatic benchmark result is a bug in the benchmark.

Part 2: What a trustworthy benchmark requires

A checklist. A benchmark missing more than one or two of these is a marketing page with axes.

1. Published methodology, down to the scoring rule. Not “we evaluated quality” - the exact signals, thresholds, and aggregation. Ours is twelve signals per 3-second window with MAD-based z-scores capped at 10 sigma; you may think that is the wrong design, and you can only think that because it is written down.
2. Exact, dated model identifiers. “Cartesia” is not a model. sonic-3.5-2026-05-04 is - a pinned snapshot, so the run reproduces even after the vendor ships a new model under the rolling alias. A benchmark that names brands instead of snapshots cannot be re-run, and a benchmark that cannot be re-run cannot be checked.
3. Raw data you can download. Every take, every transcript, every per-run number - not a summary chart. If the publisher will not give you the losing takes, assume the winning ones were selected.
4. Production-shaped conditions. Telephony bandwidth, background noise, spontaneous speech, entity-dense text (money, codes, names), takes as long as your real sessions, and the same network path you would actually ship through. Test length is a variable, not a constant.
5. Repeats, with spread. Single runs are anecdotes. At 100 stacked rules, gpt-5-nano scored 23%, 58%, and 98% across three otherwise-identical runs (“Your System Prompt Has a Budget”, 2026-07-10) - a model can be unpredictable rather than bad, and only repeated runs with reported spread can tell you which. Report percentiles, not just means.
6. Dated runs, with an expiry mindset. Providers ship weekly. A number without a date is a rumor. Every board entry should say when it was measured and be re-measured on a schedule.
7. Published failures, not just rankings. The most useful artifacts in every post above are the failure clips - the audible Gemini breakdown at 8:08, the qwen “.999” takes, the silent tool turns. Failures are checkable; rankings are only sortable.
8. A public account of the benchmark’s own bugs. If a benchmark has never published a correction, it has either never made a mistake or never looked.

Part 3: Run your own in an afternoon

You do not need our harness. The minimum viable version of everything above fits in an afternoon and a few hundred API calls.

Step 1: Write a script that looks like your traffic. Take 10-20 real utterances from your domain (or realistic fakes) and make sure they include the things averages hide: a currency amount with cents, a confirmation code, a phone number, two or three domain-specific proper nouns, and at least one sentence a human would say messily. If you ship long-form audio, include one script at your real production length - not the demo length.

Step 2: Test through the path you ship. Call the providers the way your product does - same region, same streaming mode, same gateway or SDK. If your callers are on phones, band-limit the test audio to 8 kHz and add moderate noise before scoring STT; both are one-line ffmpeg filters. Numbers from a vendor console in ideal conditions measure the console.

Step 3: Run repeats. Five to eight runs per item, at the temperature and settings you would actually ship. You are looking for two different things: the mean, and whether the model is stable run-to-run. One run cannot show you the second, and the second is what pages you at 2am.

Step 4: Score the thing that breaks calls, not just an aggregate. For TTS: round-trip each take through a neutral open ASR (faster-whisper large-v3 is free and reproducible) and check the entities character-by-character - did the cents survive, did the code read back right - alongside any overall error rate. For long-form TTS: run a speaker-embedding window (Resemblyzer or ECAPA-TDNN, both open) across the take and watch cosine-to-start; reject voices that fall below about 0.7 within the first minute. For STT: score the streamed finals, per condition, and keep per-clip results. For LLMs: read the transcripts, not just the pass column - check whether the model invented facts and whether it said anything at all before calling tools.

Step 5: Write everything down. Exact model IDs and snapshots, date, region, settings, raw audio, raw transcripts, per-run scores. This is the difference between a benchmark and a vibe, and it is also what lets you re-run the same probe next quarter when the vendor ships a new snapshot under the same name.

Step 6: Compare against an open baseline. Everything we publish - every board, every metric, every dated run - is exported as one static JSON document at benchmarks.speko.ai/data.json, the same object the boards render from, so it cannot drift from the site. The interactive boards are at benchmarks.speko.ai (/stt, /tts, /llm, and /streaming-stt for the per-condition STT breakdown), and each post above ships its clips and per-run tables. If your afternoon run disagrees with our numbers, one of us has a bug or a different distribution - both are worth knowing, and we would genuinely like the bug report.

FAQ

Why do vendor benchmark numbers disagree with production behavior?

Because they measure flattering conditions: short clips, clean read speech, hand-picked voices, single runs, and averaged scores. Production adds length, telephony bandwidth, noise, spontaneous speech, and entity-dense content. In our measurements those variables flipped rankings outright - the arena’s #2 TTS model was the only top-three model to fail a ten-minute take.

Is the Artificial Analysis Speech Arena wrong?

No. Blind human A/B with Elo is sound methodology for what it measures: which 30-second clip sounds better. It is answering “which demo wins,” not “which model survives minute eight” or “which catalog voice is stable.” The error is not in the arena; it is in using a short-clip ranking to make a long-form production decision.

What is the minimum viable benchmark before picking a voice stack?

One afternoon: 10-20 utterances shaped like your real traffic including money, codes, and names; rendered or transcribed through the exact path you ship; five-plus repeats per item; entity-level scoring plus a speaker-drift check at your real session length; everything logged with dated model snapshots. That single afternoon catches every failure mode described in this post.

Where can I get raw benchmark data to compare against?

Speko publishes its full dataset - every board, metric, and dated run - as one static JSON file at benchmarks.speko.ai/data.json, rendered from the same object as the public boards. Coval publishes head-to-head results at benchmarks.coval.ai with a public methodology repo. Both are open; disagreements between your runs and either dataset are worth investigating in both directions.