LiveKit Inference for Voice Agents: Fast First Token, and the Goodbye Most Models Skip
LiveKit shipped an inference host. We ran it through a voice-agent lens from us-east4 — first-token latency and six dial-discipline behaviors. It's the new speed leader (gemma-4-31b-it at 92ms), and on what matters — tool-calling, not fabricating, scope, honesty, injection resistance — nearly every model is clean. The one shared blind spot is ending the call: most models hang up silently instead of saying goodbye.
LiveKit shipped an inference host — a fast, worker-level way to run open and hosted models. We put it through a voice-agent lens from us-east4, two ways: how fast it returns a first token, and how it behaves on the small disciplines a live call actually depends on. This is the LiveKit inference host measured on its own — not blended into any gateway ranking.
First token: the new speed leader
| Model (on LiveKit) | TTFT p50 | p90 |
|---|---|---|
| gemma-4-31b-it | 92ms | 112ms |
| gpt-oss-120b | 118ms | 132ms |
| kimi-k2.5 | 167ms | 232ms |
| gemini-2.5-flash-lite | 241ms | 373ms |
| gpt-4o-mini | 366ms | 464ms |
| gpt-4.1 | 422ms | 477ms |
| gpt-4.1-mini | 524ms | 583ms |
gemma-4-31b-it posts the fastest first token we’ve measured anywhere — 92ms — and gpt-oss-120b serves at 118ms, beating the same model on Cerebras (~195ms). For a live turn, every millisecond before the agent starts speaking is silence the caller hears, so this is the difference between snappy and sluggish.
Dial discipline: one shared blind spot
Speed is only half the turn. We ran six dial-discipline scenarios — the behaviors a phone agent lives or dies on — three times each (18 runs per model):
| Model | tool trigger | don’t fabricate ID | farewell + end_call | scope refusal | robot honesty | injection resist | total |
|---|---|---|---|---|---|---|---|
| kimi-k2.5 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| gemini-2.5-flash-lite | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| gemini-2.5-flash | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| gpt-5 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| gemma-4-31b-it | 3/3 | 3/3 | 1/3 | 3/3 | 3/3 | 3/3 | 16/18 |
| gpt-oss-120b, gpt-4o-mini, gpt-4.1, gpt-4.1-mini, gpt-5-mini | 3/3 | 3/3 | 0/3 | 3/3 | 3/3 | 3/3 | 15/18 |
Read down the columns, not across the rows. On five of the six behaviors — firing the right tool, not fabricating an ID, refusing out-of-scope requests, staying honest about being a bot, and resisting prompt injection — every model scores a clean 3/3. On the things that actually get an agent in trouble, LiveKit’s models are disciplined.
The entire spread comes from one column: farewell + end_call. kimi-k2.5, both Gemini Flash variants, and gpt-5 speak a proper closing line before hanging up (3/3). gemma-4-31b-it slips once (1/3). And five models — gpt-oss-120b, gpt-4o-mini, gpt-4.1, gpt-4.1-mini, gpt-5-mini — miss it entirely (0/3): they fire end_call with empty content, hanging up in silence instead of saying goodbye. gpt-4o-mini fails a different way — it writes the tool call out as literal text.
What that means
It’s a UX wart, not a safety hole. The model does the right thing — it just doesn’t announce it, so the caller hears a click instead of “thanks, goodbye.” In production you don’t leave that to the model: the worker enforces a farewell so a closing line always plays, whichever model is behind it. But it’s worth knowing — wire a bare model straight to end_call and most of them will hang up on your caller without a word.
The verdict
LiveKit Inference is fast, and on the disciplines that matter it’s clean. For the latency floor, reach for gemma-4-31b-it (92ms) or gpt-oss-120b (118ms). Just don’t count on the model to say goodbye — enforce the farewell downstream.
Method: measured from us-east4 — first-token latency over 15 iterations per model; dial discipline over six scenarios, three reps each (18 runs). One measurement date, a snapshot on our rig; we publish the counts so you can weight it.