Seven researchers at Hume AI have published a set of diagnostic experiments covering 11 leading automatic speech recognition (ASR) models. Their question was simple: does the system with the lowest word-error-rate (WER) on public benchmarks actually hear audio more accurately? The results point the other way — the top-scoring models are also the ones most likely to be echoing reference transcripts that are themselves wrong.
The lineup included OpenAI's Whisper-large-v3, Qwen3-ASR, Nvidia's canary-qwen and parakeet, Microsoft's Phi-4, Mistral's Voxtral-Mini, Moonshot's Kimi-Audio, IBM's granite-speech, CohereLabs' cohere-transcribe, along with bosonai's higgs-audio and Moonshine's streaming medium model. The researchers call this pattern "benchmark overfitting": a model's score can rise because it has memorized a dataset's transcription conventions, not because its listening actually improved.
What the three probes measure
The first is a reference-mismatch probe. Public speech datasets don't have perfectly clean reference transcripts — in the VoxPopuli test clips the team analyzed, roughly 40% show signs of a reference error, touching about 3% of all words. The probe locates spots where audio and transcript disagree and checks which one the model follows. Models showing benchmark overfitting reproduced the wrong reference text 18% to 30% of the time — choosing the dataset's version over what was actually said.
The second is masked-entity retrieval. The team silenced numbers in the audio and asked the models to transcribe anyway. Several models reinserted the missing numbers on LibriSpeech 30% to 40% of the time. The sound was gone; the word came back regardless.
The third tests orthographic switching — the same word written differently across datasets, like "Mr." versus "Mister," or "any one" versus "anyone." Some models hit close to 90% accuracy at guessing which spelling convention a given dataset expects — too high to be coincidence.
How models recognize which exam they're taking
The clearest evidence comes from a resynthesis experiment. One VoxPopuli reference transcript is missing the opening "Thank you." Tested on the original recording, 6 of the 11 models dropped it too. Resynthesizing the same sentence with the original speaker's cloned voice brought that down to 5. Swapping in a voice clone of a different legislator entirely brought it down to just 1.
The words didn't change. The meaning didn't change. Only the source of the voice changed. The researchers' read is that these models are capable of faithfully transcribing what's literally spoken, but they're also using surrounding acoustic cues to infer what context they're in, then deciding whether to follow the audio or a dataset's transcription policy.
"Models are able to faithfully transcribe the literal spoken words, but are using surrounding acoustic context to decide whether to follow the audio or a benchmark-specific transcription policy."
This is the same phenomenon as benchmark contamination in the large-language-model world, just harder to spot in speech. Leaked test questions in text benchmarks can at least be caught with string matching. Acoustic-level "exam recognition" leaves no visible trace — nothing on a WER table looks unusual.
How much decision-making can one error-rate column support
The scatter plot is the more unsettling part: the models with the lowest error rates are also the ones most likely to reproduce wrong reference transcripts. Anyone selecting a vendor by WER alone is ranking in a direction that happens to coincide with who's best at gaming the test.
The researchers aren't arguing for scrapping public leaderboards — their point is that these boards are transparent, reproducible, easy to run, and well understood by the community. Their prescription is practical: use fully held-out evaluation sets; avoid naive same-distribution random splits in favor of splitting by time, speaker, or metadata; add a "benchmark-fit" tab to leaderboards that tracks reference-error rates and orthographic switching over time; and open-source evaluation scripts along with raw, unnormalized model output so others can audit it.
For teams shipping speech products domestically, the practical takeaway is one extra step in vendor selection. Gathering 200 to 500 real recordings from your own business and transcribing them to your own conventions as a held-out set costs roughly a few person-days — and running it against a few vendors' public rankings quickly shows whether the ordering still holds. Call-center audio, meeting notes, and in-car voice commands carry accents, noise profiles, and terminology far removed from LibriSpeech; a few tenths of a percentage point of difference on a leaderboard will likely be swamped by that domain gap.
Sources: Hume AI research team's published probe experiment records, Hugging Face community technical blog, CocoLoop, official model cards; cross-checked against the tested-model list, the three probes' reported ratios, and the resynthesis experiment's counts.