Request a new model

Tell us which model you want measured. We review every request.

Speech-to-Speech

Action

higher is better
InworldInworldgpt-4.1-nano (cascade)1.00
OpenAIgpt-realtime-20.87
OpenAIgpt-realtime0.83
OpenAIgpt-realtime-2.1-mini0.83
GeminiGemini3.1-flash-live0.78
OpenAIgpt-realtime-mini0.70
xAIgrok-voice-think-fast-2.00.67
OpenAIgpt-realtime-2.10.56
xAIgrok-voice-fast0.44
xAIgrok-voice-think-fast-1.00.22
0.000.250.500.751.00

Call completion

higher is better
xAIgrok-voice-think-fast-2.0100%
xAIgrok-voice-think-fast-1.096%
xAIgrok-voice-fast95%
OpenAIgpt-realtime-2.1-mini93%
GeminiGemini3.1-flash-live92%
OpenAIgpt-realtime-2.190%
OpenAIgpt-realtime88%
OpenAIgpt-realtime-281%
OpenAIgpt-realtime-mini74%
InworldInworldgpt-4.1-nano (cascade)51%
0%25%50%75%100%

Task capability

higher is better
OpenAIgpt-realtime-mini0.91
xAIgrok-voice-fast0.91
OpenAIgpt-realtime0.87
GeminiGemini3.1-flash-live0.86
InworldInworldgpt-4.1-nano (cascade)0.86
xAIgrok-voice-think-fast-2.00.86
OpenAIgpt-realtime-20.84
OpenAIgpt-realtime-2.1-mini0.79
xAIgrok-voice-think-fast-1.00.77
OpenAIgpt-realtime-2.10.76
0.000.250.500.751.00

Latency

lower is better
OpenAIgpt-realtime494ms
OpenAIgpt-realtime-mini614ms
xAIgrok-voice-think-fast-2.0820ms
xAIgrok-voice-think-fast-1.01.01s
OpenAIgpt-realtime-2.1-mini1.01s
OpenAIgpt-realtime-21.10s
OpenAIgpt-realtime-2.11.10s
GeminiGemini3.1-flash-live1.11s
xAIgrok-voice-fast1.11s
0ms500ms1.00s1.50s2.00s

What we tested

ScenarioThe story it forcesTool
defaultbook a flight to your 3pm-meeting city, then flip the daybook_flight
travel_supportinfer the wedding city, book there, flip Thu→Sat, spell an emailbook_flight
restaurant_calendarinfer the anniversary night, make the reservation, move 7→8pmcreate_event
finance_currencyinfer trip budget + currency, convert it, then bump the budgetconvert_currency
refund_negativenegative-heavy support; infer the one free pickup day, schedule it, spell an order codecreate_event
multiconstraint_scheduleinfer the summit city, book a flight, then change destination Berlin→Munichbook_flight
Per-scenario scores · 0–1 · higher is better · “—” = dropped every session
ModelDefaultTravelRestaurantFinanceRefundMulti-con.
grok-voice-think-fast-2.00.990.870.840.850.600.62
Geminigemini-3.1-flash-live0.920.990.740.840.870.32
gpt-realtime0.960.950.780.750.430.67
gpt-realtime-2.1-mini0.970.880.600.830.760.47
grok-voice-fast0.660.540.860.990.730.56
grok-voice-think-fast-1.00.590.700.440.910.390.38
Inworldgpt-4.1-nano (cascade)0.59—0.490.74——

FAQ

What is the best speech-to-speech model for a voice agent?
The board ranks each parameter on its own and publishes no single winner. Among native speech-to-speech models OpenAI gpt-realtime-2 scored 0.87 on confirmed write actions, a 0-1 rate, and xAI grok-voice-think-fast-2.0 completed 100% of its scripted calls.
Which realtime S2S API has the lowest latency?
OpenAI gpt-realtime measured 494ms p50 from end of speech to first audio, the lowest on the board; OpenAI gpt-realtime-mini follows at 614ms.
Do speech-to-speech models actually execute tool calls?
Unevenly. Scored as a 0-1 rate of firing the write tool after the caller confirmed, OpenAI gpt-realtime-2 reached 0.87 while xAI grok-voice-think-fast-1.0 reached 0.22 and mostly narrates instead.
Is a native S2S model more reliable than a cascaded pipeline?
On this scenario set, yes: the one cascade measured (Inworld running gpt-4.1-nano) completed 51% of calls, while xAI grok-voice-think-fast-2.0 completed 100%.
How are S2S models scored on this board?
Every system runs the same six scripted concierge scenarios at n=3, and each parameter is measured and ranked on its own: confirmed tool actions, call completion, task capability, and end-of-speech-to-first-audio latency. Measured latencies span 494ms (OpenAI gpt-realtime) to 1111ms (xAI grok-voice-fast).