Code-Switching Speech Recognition: Which STT Models Survive Two Languages in One Sentence

Code-switching is a speaker mixing two languages inside a single utterance. “Yes, Thursday works, pero solo por la tarde, after three” (a constructed illustration, not a benchmark clip) is one sentence, two languages, and completely ordinary phone traffic for any business whose callers are bilingual.

It breaks speech recognition because most streaming recognizers lock onto one language per stream, so the words in the second language come back transcribed phonetically in the first.

What the board measures

The code-switching board tests four language pairs over the live streaming path: Mandarin-English (from the ASCEND corpus, scored as mixed-token error rate), and Spanish-, German-, and French-English (FLEURS concatenations, scored as WER). Each cell also carries a both-language coverage figure, and a model “handles” a pair only when that coverage clears 50%, because a transcript that silently drops one language can still post a survivable error rate. The ranking is simply how many of the four pairs the second language survives. Only streaming-measured providers are shown; Deepgram is absent because its multi-language mode returned empty transcripts over the gateway stream.

The results

AssemblyAI universal-3-5-pro handled 4 of 4 pairs and posted the strongest line: 9.8% mixed-token error on Mandarin-English with 100% both-language coverage, 3.5% WER on English-Spanish (100% coverage), 3.7% on English-German (98%), and 2.6% on English-French (93%).

Soniox stt-rt-v5 also handled 4 of 4: 11.8% on Mandarin-English (88% both-language), 4.9% on English-Spanish (97%), 5.1% on English-German (100%), and 8.8% on English-French (89%).

ElevenLabs scribe_v2_realtime handled 3 of 4. On Latin-script pairs it is competitive: 3.8% on English-Spanish (100% coverage), 3.1% on English-German (98%), 7.8% on English-French (89%). On Mandarin-English it anglicizes: 75.0% error with only 34% both-language coverage, meaning the Mandarin mostly comes back as English words.

That last row is the failure mode worth understanding. It is not a noisy-audio problem; the same model scores 3.1% one column over. When the second language shares no script with English, a language-locked recognizer does not degrade gracefully. It drops the language entirely.

The monolingual baseline

The same vendors are strong when the languages arrive one at a time. On the multilingual STT board, measured monolingual Spanish (FLEURS es_419, 250 clips) came in at 1.4% WER for OpenAI GPT-4o Transcribe, 1.8% for ElevenLabs Scribe v2, and 2.0% for Alibaba Qwen3-ASR. Those runs use a different corpus, sample size, and (for ElevenLabs) the batch model rather than the realtime one, so they are context rather than a direct comparison, but the shape is clear: monolingual Spanish sits in the 1-3% range, while the best code-switched Spanish-English figure on the streaming board is 3.5%. Mixing languages mid-sentence costs real accuracy even for the models that survive it.

Verdicts by pair

For Mandarin-English, AssemblyAI universal-3-5-pro is the only model measured that kept full both-language coverage, at 9.8% error; Soniox is usable at 11.8% with 88% coverage; ElevenLabs scribe_v2_realtime is not, at 75.0%. For Spanish-English, all three hold: AssemblyAI at 3.5%, ElevenLabs at 3.8%, Soniox at 4.9%. For German-English the spread is similarly tight (3.1% to 5.1%), with ElevenLabs posting the lowest error. For French-English, AssemblyAI’s 2.6% leads ElevenLabs’ 7.8% and Soniox’s 8.8% by a wider margin than any other Latin-script pair.

If your traffic code-switches into Mandarin, the pair table is the whole decision. If it stays within Latin-script languages, any of the three works, and the choice can ride on the rest of the stack: end-of-turn latency and streaming WER on the STT board, and per-language accuracy on the multilingual board. Per-model pages and the raw numbers are linked from the code-switching board and published in the dataset.