The Open ASR Leaderboard Adds Its First Global South Language
Oleh Zai · Let's Make It Easy
Dipublikasikan 2026-08-29
Automatic speech recognition systems are often compared with one number: word error rate, or WER. The Open ASR Leaderboard update from Hugging Face and VoiceArena shows why that...
Automatic speech recognition systems are often compared with one number: word error rate, or WER. The Open ASR Leaderboard update from Hugging Face and VoiceArena shows why that number needs context. The new evaluation work adds Hindi and Indian English datasets, with speaker and regional variation represented in the test design.
1. Apa yang berubah
The update introduces two evaluation sets called Monsoon en-IN and Monsoon hi-IN. The first covers Indian English and the second Hindi. The article describes Hindi as the first Indic language added to a multilingual tab that previously focused on European languages. That matters because a leaderboard influences which models people adopt and improve. If a language is absent from the benchmark, errors in that language receive less attention.
The collection is designed with more metadata than a simple audio and transcript pair. The four splits are speaker-disjoint, covering 4,888 speakers, and the dataset records 12 speaker attributes for each speaker. The article also describes variation across geography, orthography, and other properties of Hindi speech. Those details help researchers see whether a model performs consistently or only works well for a narrow slice of speakers.
2. Public and private splits
The public splits can be used for self-scoring. A private split is withheld from model developers, which makes it harder to tune directly against the test examples. This does not make a benchmark perfect, but it reduces one common failure mode: a model appearing better because its pipeline was optimized for a known evaluation set.
The Open ASR Leaderboard repository also documents evaluation across short-form, long-form, and multilingual datasets. Its evaluation scripts report WER and real-time factor metrics. The repository notes that Hugging Face Jobs can run evaluations on consistent hardware, which helps reduce differences caused by local drivers or machine configurations.
3. What the number still cannot tell you
WER is useful, but it is not a full description of usability. Transcription quality can change with accents, background noise, code-switching, microphone quality, and domain vocabulary. A model that leads on one public set may still struggle with customer calls or Indonesian mixed-language meetings.
The practical lesson is simple: when comparing ASR models, inspect the language coverage, speaker diversity, normalization rules, and hidden-test design before trusting the rank. The benchmark is part of the result, not just the scoreboard. That habit is useful even when the model will only run on a small internal dataset, because the evaluation setup shapes what the team notices and fixes.
Referensi: https://huggingface.co/blog/open-asr-leaderboard-global-south