Voice AI Glossary
The terms behind the boards, each tied to a measured value where we have one.
- WER (word error rate)
- The standard STT accuracy metric: substitutions plus insertions plus deletions, divided by the number of reference words. Lower is better; 0% is a perfect transcript. Casing, punctuation, and number formatting are normalized on both sides before scoring. AssemblyAI Universal-3.5 Pro measured 2.0% WER on FLEURS read English on the STT board .
- CER (character error rate)
- WER computed at the character level. Used for languages without word boundaries, such as Mandarin, Japanese, and Thai, where splitting on spaces is meaningless. ElevenLabs Scribe v2 measured 4.1% CER on Thai on the multilingual STT board .
- End-of-turn detection
- Deciding whether the caller has finished speaking or is only pausing mid-thought. Too slow reads as dead air; too eager cuts the caller off. Smart Turn v3.2 measured 94.0% accuracy on 200 real human clips on the turn-taking board .
- Barge-in
- The caller speaking while the agent is talking, and the agent yielding the floor. A well-built agent absorbs coughs and short acknowledgements but yields to a real interruption. How four production stacks handled a 400 ms cough fired into the reply: the barge-in stress test .
- Backchannel
- Short listener signals like "mm-hmm", "yeah", or "right" that mean keep going, not stop. A turn detector that reads them as interruptions makes the agent stall mid-answer. Measured across 158 trials on OpenAI Realtime in the backchannel trap .
- Code-switching
- Two languages inside one utterance, common in real phone traffic. Recognizers that lock onto one language transcribe the second phonetically in the wrong language. AssemblyAI universal-3-5-pro measured 9.8% mixed-token error on Mandarin-English on the code-switching board .
- TTFT (time to first token)
- How long the LLM takes to emit its first token after the request lands; also called TTFB. In a voice agent the caller hears all of it as silence, so it dominates perceived responsiveness. Cerebras gemma-4-31b measured 192ms p50 on the LLM board .
- First-audio latency
- Time from sending text to a TTS engine (or from end of caller speech, for speech-to-speech) to the first audio byte back. The last leg of the voice latency budget. Palabra palabra-tts-v1 measured 72ms p50 to first audio on the TTS board .
- Streaming STT
- Transcription over a live socket while audio is still arriving: partial hypotheses stream in and are later committed as finals. This is the path a voice agent actually runs on, and most models lose accuracy on it. AssemblyAI Universal-3.5 Pro measured 2.0% streaming WER on the STT board .
- Batch STT
- Whole-file transcription after upload. With no latency pressure, batch accuracy is usually a model's ceiling, and most vendors price it below their streaming rate. OpenAI GPT-4o Transcribe measured 2.3% batch WER on the STT board .
- TTS naturalness
- How human a synthetic voice sounds. Speko measures it as an arena Elo from blind human A/B votes on phone-agent lines (field mean 1500), not a vendor-reported MOS. Gemini gemini-3.1-flash-tts-preview measured Elo 1591 and ElevenLabs eleven_v3_conversational 1590 at the top of the TTS board .
- Voice cloning
- Synthesizing speech in a specific person's voice from reference audio. The quality axes are similarity to the reference and stability of that identity across generations. Measured on the voice cloning board .
- Speech-to-speech (S2S)
- A single model that takes caller audio in and produces reply audio out, with no separate STT, LLM, and TTS stages. It removes pipeline latency but makes behavior harder to control per stage. xAI grok-voice-think-fast-2.0 completed 100% of its scripted concierge calls on the S2S board .
- Cascaded pipeline
- The standard voice-agent architecture: STT transcribes the caller, an LLM decides the reply, TTS speaks it. Each stage is swappable and measurable on its own. The cheapest fully grounded stack measured $0.058 per call on the cost-per-solve board .
- Endpointing
- The STT-side decision that speech has ended, which commits the final transcript. Endpoint latency, end of speech to final, gates how soon the agent can start replying. Measured p50s span 66ms to 1124ms on the STT board .
- VAD (voice activity detection)
- An energy detector that flags whether anyone is speaking. Cheap and fast, but it cannot tell a mid-thought pause from a finished turn. A VAD-plus-silence-timer baseline measured 46.9% end-of-turn accuracy with 100% false-cutoff on the turn-taking board .
- Diarization
- Splitting a transcript by who spoke, assigning each segment a speaker label. Needed for meeting notes and QA of two-sided calls; vendors usually bill it as an STT add-on.
- Semantic turn detection
- End-of-turn detection that uses the content of speech, transcript text or audio prosody, rather than silence alone, so an unfinished sentence keeps the floor even through a pause. The best text-based detector, LiveKit turn-detector v0.4.1, measured 87.0% accuracy on the turn-taking board .
- RTF (real-time factor)
- Processing time divided by audio duration. Below 1.0, the system keeps up with live audio; batch throughput is often quoted this way. Streaming voice agents care about per-turn latencies instead.
- Telephony audio (8 kHz)
- Phone-network audio sampled at 8 kHz, half the bandwidth of the 16 kHz most speech models train on. Consonant detail above 4 kHz is simply not in the signal. Streaming WER under a telephony condition is measured in what a voice agent hears .