NVIDIA Nemotron 3.5 Transcribe and Granary v2 Explained: Prompts, Benchmarks, and Domain Adaptation

A transcript is rarely the end of the request. People listening back want names attached to each line, an English rendering of a foreign-language session, or drug names written the way clinicians expect them. Those needs shift with the setting — a multilingual support call, a fast-paced meeting, or dictated medical notes.

NVIDIA Nemotron 3.5 Transcribe is built as one open-weight, promptable checkpoint that covers all four jobs: transcription, speaker attribution (diarization), translation, and audio question answering. Its published capability envelope is 40 languages transcribed, 31 translated to or from English, and 13 supported for speaker attribution; recordings up to an hour long are accepted, with up to eight speakers labeled. Distribution runs through Hugging Face and ModelScope, natural-language prompts steer the output, processing location stays a developer decision, and recognition can be tuned to domain vocabulary and accents.

The benchmark picture: Transcribe leads open-weight models for English on the Hugging Face Open ASR Leaderboard and sits on Artificial Analysis’s accuracy–active-parameter frontier. For speaker attribution it posts the lowest error rate on David AI’s 21-language diarization leaderboard and ranks first in VoiceArena’s 14-system comparison — the model takes the top spot on two speaker-attribution benchmarks outright. Quantized versions are coming soon, which opens local use on RTX Spark, DGX Spark, or Jetson AGX Thor, and larger deployment on Blackwell data-center GPUs.

Control is not limited to inference. Granary v2 contributes multilingual training data (also on Hugging Face and ModelScope) plus a reusable curation pipeline, and NeMo Speech supplies the fine-tuning tools. In the medical case study, fine-tuning cut word error rate (WER) on held-out speakers by 45.8% relative to the best prompt.

Prompting Instead of Rewiring

One recording, one checkpoint, different instructions and therefore different deliverables:

• Transcribe this recording and label each speaker.
• Translate this audio into English.
• What decisions were made in this meeting?

To show the labeling behavior under pressure, the release uses a scene from His Girl Friday, the 1940 screwball comedy famous for rapid-fire, heavily overlapping dialogue; Nemotron 3.5 Transcribe keeps speaker tags attached even when voices collide.

Unfamiliar terminology is better handled with context before training. A healthcare example:

Transcribe this medical discussion in the original language. Possible terms: HbA1c, metformin. Use these terms only when supported by the audio. Preserve standard medical spelling and write numbers in conventional written form.

The model card ships recommended presets — including reasoning for audio questions — plus the full prompt guide and a task-by-language matrix. Support varies by task, and audio question answering is trained in English. Word-level timestamps are available. As with any speech model, names, speaker changes, and answers should be spot-checked against the recording.

Agora used word-level timestamps in a classroom demo. Agora’s real-time voice and video platform powers live conversations in apps; pairing a recorded class with Nemotron 3.5 Transcribe produced a transcript displayed alongside the replay, so students could see what was said and when. Agora’s blog covers the build.

Reading the Leaderboard Results

Two threads run through the published numbers: content accuracy among open-weight releases, and speaker identity.

three tracks on the Hugging Face Open ASR Leaderboard

Public test sets are scored by word error rate (WER) across three tracks:

• English: 5.33% average WER, first among open-weight models (Figure 1).
• Multilingual: 4.62% mean WER across seven languages, the lowest among evaluated open-weight models in the model card’s Open ASR comparison.
• Long-form: 10.98% mean WER, also the lowest in that comparison.

Figure 1. Excluding API models, Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR Leaderboard (English) at 5.33% average WER; lower WER means fewer transcription errors.

Artificial Analysis Speech-to-Text Leaderboard

Artificial Analysis ranks speech-to-text models independently with AA-WER, its word error rate index. Nemotron 3.5 Transcribe reaches 2.8% AA-WER, tying for first among open-weight models with about 4B active parameters against 24B for the model it ties. On AA-AgentTalk, the held-out voice-agent speech set that makes up 50% of the AA-WER index, it scores 1.8% WER versus 2.3% for the next-best open-weight model. Architecturally, a speech encoder is paired with a hybrid Mamba–Transformer mixture-of-experts language model, so only about 4B of its 30B parameters are active per token — roughly one-sixth — placing it on Artificial Analysis’s open-weight Pareto frontier of accuracy versus active parameters. On AA’s all-model chart it ranks ahead of several widely used proprietary speech APIs.

Figure 2. Leading AA accuracy at about one-sixth the active parameters: 2.80% AA-WER with approximately 4B active parameters.

Speaker attribution through overlap

Sortformer speaker cues reach the language model together with the acoustic features, so Transcribe tags each turn during generation instead of depending only on a later pass to attach speakers to words.

David AI DAI-ASR-I18N Leaderboard

On the David AI DAI-ASR-I18N Leaderboard, Transcribe ranks first among evaluated open-weight models on the combined transcription-and-speaker-attribution track and second overall. Its diarization track, which scores who spoke when across 21 languages, shows the lowest diarization error rate of the 14 systems tested, slightly ahead of Nemotron 3 Diarization.

Figure 3. NVIDIA models hold the top two places on David AI’s multilingual diarization leaderboard: Nemotron 3.5 Transcribe at 18.26% DER across 21 languages, then Nemotron 3 Diarization at 18.38%. DER measures speaker-time errors; lower is better.

VoiceArena Diarization-Bench

On the English full-session benchmark of VoiceArena Diarization-Bench, Transcribe posts the lowest concatenated minimum-permutation WER (cpWER) of the 14 systems and ranks first in both in-person and online settings. It also drops the fewest words: deletions are 8.32% of reference words, against a median of about 31%.

Figure 4. NVIDIA models again take the top two of 14 systems: Nemotron 3.5 Transcribe leads at 32.4% cpWER, followed by NVIDIA Multitalker Parakeet with Nemotron 3 Diarization at 34.2%. Diarization-Bench v1 covers English full sessions, overlapping speech and 2–8 speakers. cpWER counts word errors after matching speaker labels; lower is better.

Further evaluations and throughput

Ocular AI’s Converse-STT evaluation ranks Transcribe the top open model, and Josh Talks’ Voice of India leaderboard reports its results across Indian languages in India.

Benchmarking processed 100 hours of audio in under three minutes on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU, using the forthcoming NVFP4 quantized version, measured on 30-second clips at high concurrency.

The released checkpoint can be served today with vLLM. Local deployment with NeMo-Speech.cpp is coming soon, alongside the quantized versions. Transcribe is built for recorded audio, not live streaming.

Specialized recordings still call for adaptation to domain vocabulary and transcript conventions — the job Granary v2 and NeMo Speech fine-tuning address, starting from the data behind the model.

Granary v2: The Open Data Curation Layer

A promptable speech model needs training data with unambiguous formatting targets. Granary v2 forms part of the Nemotron speech collection on Hugging Face and ModelScope. Its reusable pipeline was built with NVIDIA NeMo Curator and prepared multilingual speech used in Nemotron 3.5 Transcribe’s training, producing aligned transcript variants, task labels, and inspectable quality metadata. The public manifest covers 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1. Those figures describe the dataset release, not the model’s full training mixture or its sampling weights.

Each recording yields three transcript forms (Table 1), matching the verbatim and clean styles developers can request through a prompt. Beyond transcription the pipeline also generates short audio captions, code-switching restoration of loanwords, speech-grounded Q&A pairs, contextual ASR biasing terms, and translations with automatic quality scores.

Training target	Illustrative transcript
Text-normalized, verbatim	I paid, uh, twenty-five euros.
Inverse text normalization, with disfluencies	I paid, uh, €25.
Inverse text normalization, without disfluencies	I paid €25.

Table 1. Granary v2 training targets for the same utterance.

The pipeline splits into four reusable stages:

1. Audio Preparation (Metadata): raw or long-form audio is segmented, speakers are tracked across the whole recording, and acoustic events and language are tagged. Granary v2’s sources were already segmented, so this release skipped the stage.
2. Transcription (ASR): each language is routed to primary and recovery ASR models, falling back when the primary fails a hallucination check. Human reference transcripts preserve trusted spellings while adding spoken forms and disfluencies.
3. Task Label Generation (Text): transcript variants are derived, then character rate, hallucination patterns, and model or human-reference agreement are checked. Only accepted records receive contextual-ASR terms and translations.
4. High Fidelity (HiFi): noisy, clipped, or multi-speaker segments are filtered out, leaving a smaller high-quality set usable beyond ASR. On VoxPopuli it reduced 354k hours to 111k hours across 22 European languages.

Figure 6. Granary v2’s curation pipeline follows one sample from raw audio through the Metadata, ASR, and Text stages into a training example.

Teams can run the NeMo Curator pipeline with Ray and NVIDIA Dynamo backends over their own recordings or authorized public corpora, or build on the Granary v2 public manifests. Those manifests carry the three transcript forms as tn_raw, itn_raw, and itn_clean, the labels above, entity hints, derived instruction pairs, provenance, and a per-record quality flag. Source audio is downloaded separately under each corpus’s original license, following the release’s published terms.

Adapting Recognition to Domain Vocabulary

A development-set prompt should fix language, script, and transcript format. In the case study below, naming English and the Latin alphabet in the prompt cut full-benchmark WER from 30.8% to 23.8%. For the domain errors that remain, the NeMo Speech fine-tuning workflow pairs representative speech with explicit written targets.

The progression comes from the NeMo Speech medical ASR case study on Eka Care’s open benchmark of recordings from Indian doctors and medical students: 3,619 clips, with no designated training split. Adaptation data is generated from medical text and public medicine catalogs, then combines approximately 153 hours of synthetic speech, 29 hours of accented English from Common Voice, and 63 hours of medical recordings from MultiMed.

Figure 7. On the Eka Care Indian medical speech benchmark full test set, a better prompt moves WER from 30.8% to 23.8%, and fine-tuning brings it to 11.8% — about 62% fewer transcription errors than out of the box. The full set includes the 855 tuning clips used to choose the prompt and checkpoints; the held-out results below are the cleaner test.

Synthetic generation separates the written target from its spoken rendering: “HbA1c” is the label, while “H B A one C” is sent to the synthesizer. Multiple TTS voices broaden the audio, and real recordings add accents and recording conditions. Training text is filtered for exact matches and shared eight-word sequences with benchmark references.

Following Granary v2’s explicit targets, training examples vary digits, unit spacing, company-name expansion, and brand-name hyphens, with instructions stating the required convention. Some prompts add term lists with near-miss distractors so the model applies a spelling only when the audio supports it. The final system uses one fixed convention prompt and one model pass, without retrieval or transcript post-processing.

LoRA adapts the language model while the recipe trains the speech encoder and projection. Two runs use different data mixtures; after adapter merging, selected checkpoints are uniformly averaged into one final model. Checkpoint selection relies on decoded development-set results rather than training loss alone. The full adaptation, from text generation through evaluation, took about 14 GPU-hours on one NVIDIA GB300.

Gains measured on held-out speakers

Splitting the benchmark by speaker produces 855 tuning clips, used to choose the prompt and checkpoints, and 2,764 held-out clips, decoded once per final system. With Eka Care’s KARMA scoring pipeline, fine-tuning cuts held-out WER from 22.44% for the best-prompt baseline to 12.17% — a 45.8% relative reduction, and the clearest sign the gains generalize to new speakers. Keyword WER falls from 9.2% to 6.5%, semantic WER from 10.0% to 7.6%. On the same held-out speakers and scorer, the adapted model’s 12.17% WER beats an open medical-specialist model (13.79%) and a widely used general-purpose ASR model (14.92%).

For general English, LibriSpeech test-clean WER rises slightly, from 1.54% to 1.69%. Exact matches for brand-name drugs and diagnostics are still led by the medical specialist, and the study points to real recordings of clinicians saying brand names as the next step. For your own domain, keep at least 100 real, labeled clips for checkpoint selection, and evaluate the entity types your application depends on alongside general-speech retention.

Commands to reproduce each step ship with the case study. The fine-tuning tutorials cover recipes, resource requirements, and tools for training, export, and evaluation, including new-language and code-switching ASR. An accompanying agent skill lets a coding agent drive the workflow, from baseline selection and data checks through training validation, adapter merging, and evaluation. For specialized multimodal applications, ModelScope also enables fine-tuning of Nemotron 3.5 Transcribe through its ms-swift framework.

Trying It: Demos, Endpoints, and Adaptation Paths

Nemotron 3.5 Transcribe can be tried in the Hugging Face demo or on ModelScope Studios, then the checkpoint downloaded from Hugging Face or ModelScope and served with the vLLM deployment guide.

Teams that skip self-hosting can use hosted endpoints from Baseten, DeepInfra, or Simplismart.

Domain adaptation can begin with the Granary v2 Manifests on Hugging Face or ModelScope and the medical ASR case study, or with a coding agent driving the workflow through the fine-tuning agent skill.

Further resources named in the release: the NVIDIA Nemotron 3.5 Transcribe checkpoints and interactive demos on Hugging Face and ModelScope; the Granary v2 Manifests and High Fidelity Subset on Hugging Face and ModelScope; the NVIDIA NeMo Speech GitHub, NeMo Curator Granary v2 Pipeline, vLLM Deployment Guide, Speech Fine-Tuning Guide, and Fine-Tuning Agent Skill; partner blogs from Agora, Inflection AI, VoiceArena, Simplismart, and DeepInfra; and local AI updates for NVIDIA RTX PCs, laptops, and DGX Spark.

For ItInsightsWorld readers the practical takeaway is a sequence rather than a single feature: prompt first, because language, script, and formatting instructions already move the error rate; adapt second, using Granary v2’s explicit targets and the NeMo Speech recipes, and judge the result on held-out speakers — where the 45.8% figure, not the demo, is the number worth benchmarking your own data against.