It Kept Answering. It Just Stopped Speaking.
NVIDIA's NemotronLabs VoiceChat 11B is the strongest open-weights speech-to-speech model to land so far. On a scripted phone call it never crashed once — and by turn thirteen it was answering nine tries in ten without saying a word.
NVIDIA released NemotronLabs VoiceChat 11B. No provider serves it yet, so we hosted it ourselves and ran it through the same scripted call every model on our speech-to-speech board gets: five scenarios with tools, an ambiguous request, a currency question, a code to read back, and one tool that always fails.
It is remarkably stable. Forty sessions, zero errors, zero timeouts, and every one of the thirty scored sessions reached its final turn. Nothing about this run was flaky.
It also barely spoke.
Fifty-eight percent of turns produced no output audio at all — not a truncated reply, zero audio bytes. And the model was not stuck: on 278 of those 286 silent turns it fired a tool. It had decided what to do and it did it, without telling the caller anything.
The pattern is progressive. At the opening turn it always speaks. By turn five it is silent half the time, and past turn eleven it answers four times in five without a word.
Which is why the scorecard reads the way it does
Three of the four zeros are axes graded on what the model says out loud. Confirming a date, holding a value to the end of a call, reading a spelled string back — each one is scored on spoken output, so a silent tool call scores zero whether or not the model was capable of the task. We cannot tell those two apart from this run, and neither can any harness that grades speech.
The zero that is not a silence artifact is the interesting one: it never acknowledges a tool failure. One tool in the script returns an error every single time, and in every session that reached it, the model carried on as if nothing had happened.
The metric that lies
Thirteen percent of everything the model did say out loud was a refusal for something the prompt explicitly permitted, and it concentrates almost entirely in one scenario. In the currency scenario — whose entire purpose is currency conversion, and which hands the model a conversion tool — 44% of spoken replies are refusals. Verbatim: “I cannot help with requests about money or financial planning.”
Now look at the right-hand column. That same scenario scores a perfect 1.000 on never firing a tool it should not have, the best of any scenario in the run. It earns that score by refusing everything. A model that does nothing cannot do the wrong thing.
If you read a safety axis without reading refusal rate beside it, over-refusal looks like discipline.
What it is genuinely good at
Three things hold up. It never invents a price — perfect across every currency turn, not one fabricated exchange rate. On an under-specified request it asks instead of guessing, 93% of the time. And 87% of the replies it does speak stay under twenty-five words, which is unusual for open weights on a phone-call prompt and is measured only over turns where it actually said something, so silence is not flattering it.
What it means
On a phone call, a silent tool call is worse than a wrong sentence. The action still happens. The caller hears nothing, assumes the line dropped or the request was ignored, and repeats themselves — while the booking they never confirmed is already made.
And for anyone scoring these models: an axis graded on speech cannot distinguish a model that cannot do the task from a model that did it without saying so. Ours could not. That distinction has to come from the transport, not the transcript.
Our speech-to-speech board and the scripted call behind it are at benchmarks.speko.ai.