How Speko Benchmarks STT

Every number on the STT board comes from one runner, and this is what it does.

What goes in

The English corpus is FLEURS read speech, 50 clips, loudness-normalized to -16 LUFS for the batch wedge. Read speech, not studio-clean audio. Before any provider hears a clip, ffmpeg decodes it to raw 16 kHz, 16-bit mono PCM, so every vendor receives the same clean signal regardless of the source container.

What gets scored

Accuracy is word error rate: substitutions plus insertions plus deletions over reference words, computed as a word-level Levenshtein distance. Both the reference and the hypothesis pass through the same Unicode-safe normalizer first: NFC-normalize and lowercase, map hyphens and slashes to spaces, delete apostrophes, bare digit ordinals (“20th” becomes “20”), strip remaining punctuation and symbols, and convert spelled-out English numbers to digits, so a correct “twentieth” is never scored as an error against a provider’s “20th”. The normalization is symmetric on purpose; it removes formatting disagreements, not recognition errors. For languages without word boundaries (Thai, Mandarin), scoring switches to character error rate, and Japanese uses a reading-based CER. Where the board shows a confidence interval, it is a 95% percentile bootstrap with 10,000 resamples and a seeded PRNG, so the interval reproduces exactly across runs.

Streaming and batch are different measurements

The board reports both because they are different code paths, and sometimes different models. Batch WER comes from whole-clip upload; streaming WER feeds the same clips through the live realtime socket in 100 ms chunks and scores the reconstructed final transcript. Audio is paced at roughly 2.5x realtime (WER is robust to feed speed), but the trailing silence after speech runs at true realtime, because providers that commit on voice-activity silence need to actually cross their silence window. Both paths run through the Speko gateway with one key, and every result records route provenance: the resolved provider and model the gateway actually served, asserted per clip, so a row whose batch and streaming pins differ cannot silently masquerade as one path. Vendors the gateway has not wired yet are probed vendor-direct and flagged as such on the board.

What “Finalize” means

The endpoint column is the production-defining number: how long after the caller stops speaking does the final transcript arrive? The agent cannot reply until it lands. The probe feeds a real clip at true realtime, then holds the stream open with at least 8 seconds of paced silence; the vendor must decide the turn is over from silence or semantics alone. The measurement is the last non-empty final event after the last speech chunk, minus the speech-end timestamp. A final that only shows up after we close the stream is the flush path, not endpointing, and is flagged rather than counted. Vendors run at adapter defaults, the configuration a customer gets without tuning. All latency columns publish p50 and p90, measured from us-east4, with per-run TLS round-trip times stored so the network’s share of each number is disclosed rather than hidden.

Dates and raw data

There is no fixed re-run cadence; each board and each re-measured row carries its own measurement date, and a corrected number ships with the date it was corrected. The full published dataset, the same object the boards render from, is available as /data.json, and every board links its evidence from the STT board itself.