Benchmarks Decide What Gets Built
Here's the uncomfortable truth about speech recognition leaderboards: they measure one number (word error rate), optimized on audio from people who look and sound like the people who built the models. A system can post a 5% WER and still be twice as bad for Black speakers as white speakers, or fail entirely on accents from outside the training distribution.
The Open ASR Leaderboard just added its first Global South language—Hindi, spoken by more than half a billion people—alongside Indian English. But the real story isn't the language. It's that Hugging Face and Voice Arena built the evaluation set to expose the failure modes that aggregate metrics hide.
This is what happens when you design a benchmark that actually varies along the axes where systems break in production.
What Makes Monsoon Different
Most ASR benchmarks are built from whatever audio was readily available—studio recordings, read speech, small numbers of speakers recorded at length. The new Monsoon datasets (en-IN and hi-IN) were built backward from nine explicit axes of variation: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and multiple valid transcripts.
Each axis is a way an aggregate WER can be right on average and wrong for a particular population.
The collection method follows from that design:
- Geography comes from recruiting across 202–428 districts rather than recording longer sessions in fewer places
- Devices and acoustic conditions come from contributors using their own handsets (315–582 distinct device models) and connections, indoors and out, rather than supplied hardware in a quiet room
- Vocabulary and speech type come from prompts pushing toward opinion, disagreement, narration—where named entities, numbers, and unrehearsed phrasing appear
- Age and gender are recorded per speaker and verified
The result: four splits (public and private for each language), 4,888 speakers total, with 12 speaker attributes recorded for each.
Small in Hours, Large in Speakers
Monsoon is 12.6 hours of Hindi and 11.2 hours of Indian English. That's tiny by ASR corpus standards. But it's 3,888 speakers across the four splits, with a median of just 8–12 seconds of audio per speaker.
This is the design, and it's where the value sits.
More than half of all speakers appear exactly once. The ten largest contributors account for just 2.8–6.8% of total duration. No single device model exceeds 2.1% of segments. The Indian English public set draws on 428 native districts across 30 states and union territories.
A result on Monsoon is an average over hundreds of distinct voices, not a small number of talkers recorded at length. Test sets of comparable duration are usually constructed the other way.
Indian English Is Not One Accent
The en-IN sets span all six zones of India: 35% of segments from the South, 18% from the East, 18% from Central, 16% from the North, 11% from the West. This is English as it is spoken across the country, not the English of one region or one socioeconomic class.
The Hindi sets concentrate more tightly in the Hindi belt—Uttar Pradesh accounts for roughly 40% of speakers, which is what a Hindi corpus sampled by population should look like. But they still span 202–295 districts.
All of this variation is recorded in the metadata rather than asserted. District and state carry real accent signal, which is why these fields are released rather than summarised away.
Eighteen Metadata Columns Per Segment
Monsoon ships:
- Segment:
id,audio,audio_length_s,language - Reference:
lattice(Hindi) ortext(Indian English) - Speaker:
speaker_id,gender,date_of_birth - Background:
occupation,educational_background,marital_status,income - Geography:
native_district,native_state,current_city,years_spent_in_current_district - Recording:
device_manufacturer,device_model
Most public ASR test sets ship an identifier, a transcript, and a duration. This one ships the demographic and environmental context that lets you disaggregate error rates by the dimensions where disparities actually appear.
Prior work has shown district-level error rates for Indian ASR spanning roughly 4–44%, with underrepresented regions well behind the Hindi belt and the metros. Those runs were on closed benchmarks. Monsoon makes the same class of analysis possible on a public leaderboard test set.
Lattice References for Hindi Orthographic Variation
The English sets use standard string references, where the leaderboard's normalizer collapses most spelling variation. Hindi has far more of it, and no normalizer can resolve it—the variants are not a fixed mapping between two conventions.
The Hindi sets therefore ship a lattice: for each span of the transcript, a list of the spellings that are accepted as correct. This is the right approach for a language where orthographic variation is systematic and culturally conditioned, not noise.
Quality Control at Scale
Broad geographic coverage requires recruitment across hundreds of districts rather than longer sessions from fewer speakers. Distributed recruitment at this scale introduces failure modes that a smaller collection doesn't face: contributors gaming the task, played-back audio submitted as live speech, inattentive annotation.
The collection pipeline addresses each with explicit checks:
- Contributors completed language proficiency screening before being granted recording access
- Per-speaker duration caps prevented small numbers of speakers from dominating the corpus
- Dual-channel peer-to-peer conversations on assigned everyday topics, segmented so each clip carries one speaker
- Audio quality and transcription accuracy checks at multiple stages
Contributors used their own handsets and connections. Many of those are low-end devices on unstable bandwidth. That condition is present in the released audio rather than filtered out, because that's the deployment environment for half a billion Hindi speakers.
Why This Matters
Benchmarks decide what gets built. A model that scores well on the leaderboard gets adopted and iterated on. Capabilities the leaderboard does not measure tend not to improve.
The Open ASR Leaderboard has spent the past year making WER harder to game: held-out private splits, benchmark-fitting analysis to quantify how much models are reproducing reference transcripts rather than transcribing from audio, closing gaps in normalizers.
All of that makes one number more trustworthy. It's still one number. And a long line of work—racial disparities in automated speech recognition, gender and age and accent disparities—has shown that ASR error rates are not evenly distributed across the people using them.
None of that is visible on a leaderboard, not because the leaderboard is hiding it, but because the test sets record what was said and almost nothing about who said it.
Monsoon changes that. It's still an aggregate metric at the top of the leaderboard, but now the metadata exists to disaggregate it by district, by device, by age, by gender, by income band. The failure modes are no longer invisible.
What Gets Measured Gets Fixed
Adding Hindi to a multilingual leaderboard that currently covers only European languages is symbolically important. But the real shift is methodological: designing evaluation sets to vary along the axes where deployed systems fail, and releasing the metadata that makes those failures visible.
This is expensive to build and expensive to run inference on (nearly 5,000 speakers means nearly 5,000 distinct acoustic conditions). But it's what fair evaluation looks like when you're serious about serving populations beyond the ones that built the models.
The leaderboard just got harder to top. That's the point.