One Model, Four Tasks, Natural Language Control
NVIDIA just released Nemotron 3.5 Transcribe, an open-weight speech model that does transcription, speaker diarization, translation, and audio question-answering in a single checkpoint. The model handles 40 languages for transcription, translates 31 of them to or from English, and diarizes 13 languages.
What makes this interesting isn't just the feature list—it's that you can steer all of it with natural language prompts. Want speaker labels? "Transcribe this recording and label each speaker." Need translation? "Translate this audio into English." Domain-specific terminology? Drop context into the prompt: "Possible terms: HbA1c, metformin. Use these terms only when supported by the audio."
The model ships on Hugging Face and ModelScope with full fine-tuning recipes. NVIDIA is also releasing Granary v2, the multilingual training data and curation pipeline they used, plus NeMo Speech tooling to adapt the model to new domains. In a medical case study, fine-tuning cut word error rate by 45.8% relative to the best prompt alone.
Leaderboard Sweep: Three Benchmarks, Three Wins
Nemotron 3.5 Transcribe isn't just claiming competitive performance—it's topping multiple leaderboards outright.
Hugging Face Open ASR: First Among Open Weights
On the Hugging Face Open ASR Leaderboard English track, Transcribe ranks first among open-weight models at 5.33% average word error rate. It also leads the multilingual track at 4.62% mean WER across seven languages and the long-form track at 10.98% mean WER, according to the model card's comparisons.
Artificial Analysis: Pareto-Optimal on Accuracy vs Active Parameters
Transcribe uses a hybrid Mamba–Transformer mixture-of-experts language model, activating only about 4 billion of its 30 billion parameters per token. On Artificial Analysis's speech-to-text leaderboard, it ties for first among open-weight models at 2.8% AA-WER while using roughly one-sixth the active parameters of the model it ties (4B vs 24B).
On AA-AgentTalk, the held-out voice-agent dataset that comprises 50% of the AA-WER index, Transcribe scores 1.8% WER versus 2.3% for the next-best open-weight model. This lands it on the Pareto frontier of accuracy versus active parameters—and ahead of several proprietary APIs on the all-model chart.
Diarization: Double Win on Speaker Attribution
For speaker attribution, Transcribe claims two outright wins.
On David AI's DAI-ASR-I18N benchmark, it posts the lowest diarization error rate across 21 languages: 18.26%, edging out even NVIDIA's dedicated Nemotron 3 Diarization model at 18.38%. The combined transcription-and-diarization track ranks it first among open-weight models and second overall.
On VoiceArena's Diarization-Bench, Transcribe leads all 14 evaluated systems on English full sessions with 32.4% concatenated minimum-permutation WER, ahead of the NVIDIA Multitalker Parakeet + Nemotron 3 Diarization pipeline at 34.2%. It also has the lowest deletion rate: 8.32% of reference words dropped, versus a median around 31% for other systems.
The Sortformer Trick: Diarization During Generation
Most speech models treat diarization as a post-processing step—transcribe first, then try to figure out who said what. Transcribe bakes speaker cues into the generation process using what NVIDIA calls "Sortformer speaker cues," which reach the language model alongside acoustic features. This lets the model tag speakers turn-by-turn during decoding rather than bolting labels on afterward.
The result: better handling of overlapping speech and faster-than-realtime inference that doesn't require a separate diarization pass. NVIDIA demonstrated this on a clip from the 1940 screwball comedy His Girl Friday, famous for its rapid-fire, heavily overlapping dialogue—Transcribe captioned it with speaker labels intact.
Granary v2: Open Training Data and Adaptation Pipeline
Here's where this gets interesting for practitioners: NVIDIA isn't just shipping the model. They're releasing Granary v2, the multilingual training dataset, plus the full curation pipeline and fine-tuning recipes via NeMo Speech.
The medical case study is the proof of concept: fine-tuning on domain data cut WER on held-out speakers by 45.8% relative to the best prompt. That's the delta between "try harder in the prompt" and "actually adapt the model to your distribution."
This matters because speech models have historically been black boxes you either use as-is or ignore. Shipping the training pipeline and data means you can define the written form you need—medical terminology, legal formatting, technical jargon—and adapt recognition to the accents and vocabulary your application actually encounters.
Deployment: From RTX to Blackwell
Transcribe handles recordings up to an hour long and labels up to eight speakers. NVIDIA says quantized versions are coming soon, targeting local deployment on RTX Spark, DGX Spark, or Jetson AGX Thor, with scale-up on Blackwell data-center GPUs.
Agora, whose real-time voice platform powers live conversations in apps, built a classroom demo using word-level timestamps: Transcribe turns a recorded class into a transcript shown alongside replay, so students can see what was said and when. Agora's blog walks through the implementation.
What This Signals
Three things stand out here.
First, promptable speech models are starting to work. The ability to guide output with natural language—"label speakers," "translate to English," "use medical spelling"—collapses what used to be separate pipelines into a single prompt-steered checkpoint. That's a usability win, but it also changes how you think about building speech applications: less pipeline orchestration, more prompt engineering.
Second, NVIDIA is competing on openness. Shipping the model, the training data, the curation pipeline, and the fine-tuning recipes is a direct play for developer mindshare. If you can adapt the model to your domain and run it on your infrastructure, that's a different value proposition than API-only alternatives—especially for regulated industries or data-sensitive applications.
Third, the leaderboard performance is legitimately strong. Sweeping three benchmarks—transcription accuracy, speaker attribution across languages, and overlapping-speech handling—suggests this isn't a narrow optimization. The Pareto position on Artificial Analysis (competitive accuracy at one-sixth the active parameters) is particularly notable: it means you're getting frontier performance without proportional compute overhead.
The medical fine-tuning case study is the tell: 45.8% WER reduction over prompting alone means there's real headroom for domain adaptation. If the fine-tuning recipes actually work at scale, this becomes infrastructure you can build on, not just a model you consume.