The open ASR evaluation just got less WEIRD
The Open ASR Leaderboard added two new accents to its private evaluation split: Egyptian 🇪🇬 and Nigerian 🇳🇬 English. This marks another step in the leaderboard's ongoing effort to move beyond the usual US/UK benchmarking monoculture and actually test whether speech recognition works for the billions of people who speak English with accents from the Outer and Expanding Circles.
The new data comes from Liva AI, who contributed 1.34 hours of Egyptian-accented English and 1.55 hours of Nigerian-accented English—both conversational, spontaneous speech with punctuation, casing, and disfluencies intact. It's the kind of messy, real-world data that exposes whether your ASR model was trained on a representative slice of humanity or just whatever was easiest to scrape from YouTube.
This is part of a broader trend on the leaderboard. A few months back, they added 11 private splits from Appen and DataOceanAI covering five accents. Then came Indian-accented English via Voice Arena. Now Egyptian and Nigerian. The pattern is clear: the community wants held-out eval sets that can't be gamed, and they want those sets to reflect linguistic diversity that matters.
Why private data matters (and why it keeps growing)
The Open ASR Leaderboard has been running private evaluation splits for a while now, and the reasoning is straightforward: public benchmarks get optimized into oblivion. When your eval set is public, you can't tell whether a model genuinely understands speech or whether someone just threw more Egyptian TED Talks into the training mix after seeing the leaderboard.
Private splits—data that model developers can't train on because they've never seen it—are the only way to get a trustworthy signal about generalization. The leaderboard now pulls from multiple data providers (Appen, DataOceanAI, Voice Arena, and now Liva AI), which makes it harder for any single entity to leak or contaminate the eval set.
The community response has been enthusiastic. As the announcement notes, many users have highlighted "the importance of held-out sets for trustworthy evaluations." In an era where benchmark saturation is a real problem—where models can hit near-perfect scores on public sets while still failing in production—having fresh, private, diverse eval data is the difference between a leaderboard that's useful and one that's just a vanity scoreboard.
Kachru's Three Circles: a better way to slice accent performance
With Egyptian and Nigerian accents in the mix, the leaderboard is now adopting a more sophisticated breakdown inspired by Kachru's Three Circles model (kudos to Hume AI for pioneering this approach in their leaderboard). The new reporting structure distinguishes:
- US vs. non-US (UK, Australia, Canada, India, Nigeria, Egypt)
- Inner Circle (US, UK, Australia, Canada) vs. Outer & Expanding Circles (India, Nigeria, Egypt)
This is a much smarter framing than the usual "US vs. everyone else" or "native vs. non-native" dichotomies. The Inner Circle consists of countries where English is the dominant native language. The Outer Circle includes postcolonial nations where English serves as an official or institutional second language. The Expanding Circle covers countries where English is learned as a foreign language.
For ASR, this distinction matters because accent variation, code-switching patterns, and phonological features differ systematically across these circles. A model that aces US English might completely fall apart on Nigerian English—not because Nigerian speakers are "doing it wrong," but because the model was never exposed to the legitimate linguistic patterns of Outer Circle English.
As more accents join the leaderboard, the plan is to eventually split out the Outer and Expanding Circles into finer-grained categories. That's the right move. Collapsing all non-Inner-Circle English into a single bucket would obscure massive performance gaps between, say, Singaporean and South African accents.
What Liva AI brings to the table
Liva AI's approach is interesting. According to the announcement, they "collect multimodal interaction data through consumer products used by real people." Because they own both the collection environment and the distribution, they can design products around specific model gaps—targeting underrepresented domains, languages, modalities, and interaction types.
This gives them access to "natural, high-context behavior that emerges through real product use, rather than artificially elicited or task-based interactions." That last distinction is crucial. A lot of ASR training data comes from read speech, scripted prompts, or task-completion scenarios. Conversational, spontaneous speech—the kind where people interrupt themselves, hesitate, code-switch, and use colloquialisms—is much harder to capture at scale.
The fact that Liva can instrument real consumer products to gather this data is a structural advantage. You're not asking someone to sit in a booth and read sentences; you're capturing the messy, contextualized speech that happens when people are actually trying to communicate something.
The slow, grinding work of representation
It's worth zooming out for a second. The addition of Egyptian and Nigerian English is great, but let's be clear about scale: 1.34 hours and 1.55 hours respectively. That's a few thousand utterances, not a massive dataset. The leaderboard's private splits are still dominated by Inner Circle accents in terms of sheer volume.
But the symbolic and methodological importance outweighs the raw hours. Every new accent signals to model developers: this is what we're evaluating on now. If you want to top the leaderboard, you need to handle Egyptian English, not just optimize for LibriSpeech.
The slow accumulation of these eval splits also puts pressure on training data. If Nigerian English shows up in evals, researchers will start hunting for Nigerian English in their training pipelines. That creates a feedback loop where underrepresented accents get more attention, more data collection, and eventually better performance.
It's not fast. It's not glamorous. But it's the kind of infrastructure work that actually moves the needle on fairness and representation in deployed systems.
What's next
The Open ASR Leaderboard is clearly committed to this trajectory: more accents, more languages, more private data providers. The next logical steps would be expanding into non-English languages (they've already added their "first Global South language" according to related blog posts) and adding even finer-grained accent breakdowns within the Outer and Expanding Circles.
There's also room to think about other axes of variation—age, gender, speaking style, acoustic environment. ASR doesn't just fail on accent; it fails on kids, elderly speakers, people with speech impediments, noisy backgrounds, and domain-specific jargon.
But for now, Egyptian and Nigerian English joining the private eval is a win. It's another data point proving that the ASR community is serious about building systems that work for everyone, not just the easiest-to-benchmark populations. And it's a reminder that trustworthy evaluation—private, diverse, continuously updated—is just as important as the models themselves.