10 posts
Most safety alignment treats harm as topic-level: refuse all weapons prompts, all political content. A new paper shows why this blunt approach quietly breaks real deployments—and how to fix it.
Allen AI's new auditing tool reveals that many LLM benchmarks mix multiple capabilities into a single score—and shows which questions actually matter.
Hugging Face's ASR leaderboard just added Hindi and Indian English with metadata on 4,888 speakers across hundreds of districts. Benchmarks decide what gets built—this one might actually be fair.
Researchers introduce three probes that catch speech models reproducing benchmark transcripts—even when the audio contradicts them, words are silenced, or spellings should vary randomly.
Allen AI's new benchmark reveals that LLMs trained to be helpful assistants struggle with tutoring's core trade-off: knowing when students need support versus when they need productive struggle.
HuggingFace's new agent benchmark doesn't just ask if the model got the right answer—it measures how much work it took to get there, across models, library versions, and task tiers.
ServiceNow's new voice-agent benchmark spans airlines, IT, and healthcare—with joint-generation pipelines, adversarial scenarios, and a coming multilingual expansion.
The Open ASR Leaderboard is fighting back against benchmaxxing with a simple but effective strategy: private evaluation datasets that no one can train on.
TII launches QIMMA, a rigorous quality-focused leaderboard for Arabic LLMs that goes beyond translation metrics to measure genuine language understanding and cultural nuance.
Hugging Face just dropped Ecom-RLVE, a reinforcement learning framework that trains e-commerce agents in realistic but controllable environments. This is how we move from chatbots to actually useful shopping assistants.