Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

xAIGrok-4.391.23%

OpenAIgpt-4.190.35%

OpenAIgpt-4.1-mini88.89%

OpenAIgpt-5.6-luna85.96%

OpenAIgpt-5.6-terra85.09%

Gemini3.5-flash-lite83.77%

Cerebrasgpt-oss-120b83.33%

Cerebrasgemma-4-31b81.58%

AnthropicClaude Haiku 4.579.39%

OpenAIgpt-5-mini79.24%

Baseteninkling76.17%

AnthropicClaude Sonnet 576.02%

BasetenDeepSeek-V4-Flash-073175.73%

BasetenGLM-4.774.56%

OpenAIgpt-573.98%

BasetenNemotron-3-Ultra69.01%

Baseteninkling-small63.16%

OpenAIgpt-5-nano31.87%

TogetherLlama-3.3-70B31.58%

Alibabaqwen-turbo12.43%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

BasetenGLM-4.7100%≥82%

OpenAIgpt-5100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

OpenAIgpt-5.6-terra100%≥82%

Baseteninkling100%≥82%

OpenAIgpt-4.198%

Cerebrasgemma-4-31b95%

OpenAIgpt-5-nano93%

AnthropicClaude Haiku 4.591%

Alibabaqwen-turbo90%

Baseteninkling-small86%

Gemini3.5-flash-lite72%

BasetenNemotron-3-Ultra46%

TogetherLlama-3.3-70B19%

Cerebrasgpt-oss-120b16%

OpenAIgpt-4.1-mini13%

xAIGrok-4.312%

BasetenDeepSeek-V4-Flash-073111%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

Llama-3.3-70B13%

Grok-4.315%

gpt-4.117%

gpt-5.6-luna17%

gemini-3.5-flash-lite17%

gpt-oss-120b17%

gpt-4.1-mini20%

gpt-5.6-terra24%

gpt-524%

DeepSeek-V4-Flash-073126%

inkling30%

Claude Sonnet 531%

Claude Haiku 4.533%

gemma-4-31b35%

gpt-5-mini37%

GLM-4.739%

inkling-small48%

Nemotron-3-Ultra61%

gpt-5-nano85%

qwen-turbo89%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.51%1% of tool calls1 of 146 turns

GLM-4.71%1% of tool calls1 of 156 turns

DeepSeek-V4-Flash-073115%21% of tool calls27 of 186 turns

qwen-turbo15%79% of tool calls11 of 74 turns

gpt-5-nano37%100% of tool calls ≥78%35 of 95 turns

Claude Sonnet 539%57% of tool calls64 of 166 turns

gpt-4.1-mini41%69% of tool calls61 of 148 turns

gemma-4-31b55%99% of tool calls76 of 137 turns

Grok-4.357%92% of tool calls90 of 158 turns

Nemotron-3-Ultra57%100% of tool calls ≥82%79 of 139 turns

gpt-4.160%97% of tool calls93 of 156 turns

gpt-5.6-terra64%100% of tool calls ≥82%107 of 167 turns

gpt-5-mini65%100% of tool calls ≥82%110 of 170 turns

inkling-small66%92% of tool calls132 of 201 turns

gpt-oss-120b69%100% of tool calls ≥82%128 of 185 turns

gpt-5.6-luna70%100% of tool calls ≥82%140 of 199 turns

gemini-3.5-flash-lite70%100% of tool calls ≥82%124 of 178 turns

inkling71%96% of tool calls141 of 199 turns

gpt-573%100% of tool calls ≥82%137 of 187 turns

Llama-3.3-70B94%100% of tool calls ≥82%240 of 256 turns

0%25%50%75%100%

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

7 models0 of 30≤31% · 10 independent0%

Claude Sonnet 5gemma-4-31bgpt-5.6-lunagpt-5.6-terraGrok-4.3inklinginkling-small

qwen-turbo21 of 3070%

GLM-4.716 of 3053%

gpt-4.1-mini15 of 3050%

gpt-512 of 3040%

gpt-oss-120b9 of 3030%

Nemotron-3-Ultra9 of 3030%

gemini-3.5-flash-lite6 of 3020%

gpt-5-nano6 of 3020%

gpt-4.15 of 3017%

gpt-5-mini5 of 3017%

DeepSeek-V4-Flash-07313 of 3010%

Claude Haiku 4.51 of 303%

Llama-3.3-70B1 of 303%

A tool returns $75.50 or a date; does the agent say that exact value out loud? Judge-scored on the 2026-07 model set, so it covers three of these languages — three vendors independently turn $75.50 into $57.50 in Arabic, because Arabic reads units before tens.

Cerebrasgemma-4-31b100%

OpenAIgpt-4.1100%

OpenAIgpt-4.1-mini100%

OpenAIgpt-5100%

OpenAIgpt-5-mini100%

Cerebrasgpt-oss-120b100%

xAIGrok-4.3100%

TogetherLlama-3.3-70B100%

Alibabaqwen-turbo86%

0%25%50%75%100%