NVIDIA Nemotron 3 Tops VoiceArena's Diarization-Bench

NVIDIA's Nemotron 3 diarization model takes the top spot on VoiceArena's Diarization-Bench with a DER of 14.72%, ahead of the next system at 19.3%. Designed for both offline and streaming applications, it manages up to eight speakers and offers adjustable latency settings. Its 31-layer Transformer architecture and post-processing choices aim for stable speaker attribution, even in cases of overlap.

A public ranking details 14.72% DER over 22 hours

In the initial results from VoiceArena's Diarization-Bench, Nemotron 3 Diarization ranked first among 12 systems and 17 evaluated configurations. The corpus included 139 English conversations, totaling approximately 22 hours. With overlapping speech accounted for, a VAD generated by the systems, and no edge tolerance, NVIDIA's Nemotron 3 Diarization shows a diarization error rate of 14.72%, compared to 19.3% for the next system, representing a relative reduction of about 24%. Thus, the model stands as number one in this ranking.

Streaming: stabilizing labels and choosing latency

Streaming complicates the preservation of speaker attribution, as the system only processes small chunks of audio with limited context, whereas an offline mode examines the entire recording. Without a memory mechanism, a speaker can switch channels from one chunk to the next, despite the correct detection and attribution sought by any diarization system. Nemotron 3 Diarization employs the Sortformer approach: output channels follow the order of arrival of voices, with the first becoming channel 1, the next channel 2, stabilizing generic labels and avoiding the need to resolve permutations with each chunk. The model anticipates recommended input buffer latencies of 30.4 s, 1.04 s, 0.64 s, and 0.32 s. Shorter buffers speed up response times, while a broader context generally improves accuracy and throughput.

Architecture: 31-layer Transformer and probability-based outputs

The model ingests a mono stream at 16 kHz, converted into Mel features with a 10 ms step. These features are stacked in groups of eight to form 80 ms frames that feed into a 31-layer Transformer encoder equipped with rotary positional embeddings. A Conv1D layer then raises the resolution of predictions to match that of the inputs. By default, the output is a floating tensor [T, 8] where each value represents the probability of a speaker's activity at a given moment. This representation naturally handles overlaps, with multiple active channels in a single frame, followed by post-processing that generates generic labels with start and end timestamps. Up to eight speaker channels are supported; these channels remain anonymous and do not constitute identities, with the model able to indicate that "speaker_2" spoke during a period without determining who they are.

Training and data gain from David AI

According to NVIDIA, the training combines public and licensed data, including annotated real multi-speaker conversations. The addition of data provided by David AI is said to have reduced the composite diarization error rate by 0.77 absolute points, from 11.19% to 10.42%.

What is diarization used for and how is error measured

Diarization indicates who speaks when, including when multiple people speak simultaneously, and its timestamps can be combined with ASR to produce attributed transcriptions. Its utility is tangible in research, summaries, action items, conversation analyses, and agent memos, which suffer from a lack of reliable attribution. Accuracy is assessed via the DER, defined as the sum of missed speech, false positives, and speaker confusion, divided by the total reference time of the speakers. A missed speech corresponds to undetected reference activity; a false positive to detected activity with no reference equivalent; and confusion to detected activity but attributed to the wrong speaker.