Everyone Measures the Clip. Nobody Measures the Call.
WER, MOS and time-to-first-token each score one component doing one thing once. The number that actually decides a voice agent is what one solved call costs — end to end, with the caller talking back.
Every voice-AI leaderboard measures a clip. Word Error Rate on a clean, scripted sentence. MOS on a single utterance. Time-to-first-token on one prompt. Cost-per-minute on a rate card. Each of those scores one component doing one thing once, on audio that behaves.
A real call doesn’t behave. The caller interrupts. The line goes silent. The name is spelled out loud. The model has to decide whether to book the appointment or keep asking questions. None of that shows up on a component leaderboard — and it’s the entire thing you’re actually buying.
So we measured the call.
The fabrication trap
The obvious way to score an agent is cost ÷ resolution: how many dollars per solved problem? Run that naively and you crown a liar. The cheapest-looking stack in our booking test resolves in 1.7 turns — because the model confidently says “you’re booked!” and hangs up without ever calling the scheduling tool. Nothing was booked. The bill is tiny. The metric loves it.
The fix is to make “solved” mean something. We wire a real schedule_appointment tool and count a solve only when the tool actually fires. Grounded resolution, not asserted resolution.
Once you do that, the ranking flips:
- The cheapest honest stack — Deepgram nova-3 · Cerebras gpt-oss-120b · Cartesia sonic-3.5 — solves for $0.058 per booking, grounded 100% of the time.
- The all-brand premium stack — ElevenLabs · GPT-4.1 · ElevenLabs — costs $0.703. About 12× more for the same outcome.
- Two stacks that looked cheap per-call get flagged: one grounds only 33% of the time, another never books at all. Their low bill is meaningless.
What “cheap” hides
The spread isn’t about the sticker price of any one model. It’s about how many turns the agent burns, whether the LLM fires the tool under a latency cap, and whether the STT heard the caller in the first place. A “premium brain” that clarifies for ten turns is more expensive than a fast one that books in four — even if its per-token price is lower.
That’s why we publish the components and the call. The STT, TTS, LLM and S2S boards tell you where each piece stands. The Cost / solve board tells you what happens when you wire them together and let a caller talk back.
Read the dial yourself
Speko operates the gateway these tests run through — we build voice-AI infrastructure. So every provider is measured by the same rig, on the same audio, with no preferred path, and we publish the method and the raw deltas. Where we can’t measure something honestly, it shows a dash, not a guess. The point of an instrument is that anyone can read the dial.
Everyone measures the clip. We measure the call.