DiariZen Explained: A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline

Abstract: Speaker diarization (SD) is the task of answering "who spoke when" in a multi-speaker audio stream. Classically, an SD system clusters segments of speech belonging to an individual speaker's identity. Recent years have seen substantial progress in SD through end-to-end neural diarization (EEND) approaches. DiariZen, a hybrid SD pipeline built upon a structurally pruned WavLM-Large encoder, a Conformer backend with powerset classification, and VBx clustering, represents the leading open-source state of the art at the time of writing across multiple benchmarks. Despite its strong performance, the DiariZen architecture spans several repositories and frameworks, making it difficult for researchers and practitioners to understand, reproduce, or extend the system as a whole. This tutorial paper provides a self-contained, block-by-block explanation of the complete DiariZen pipeline, decomposing it into seven stages: (1) audio loading and sliding window segmentation, (2) WavLM feature extraction with learned layer weighting, (3) Conformer backend and powerset classification, (4) segmentation aggregation via overlap-add, (5) speaker embedding extraction with overlap exclusion, (6) VBx clustering with PLDA scoring, and (7) reconstruction and RTTM output. For each block, we provide the conceptual motivation, source code references, intermediate tensor shapes, and annotated visualizations of the actual outputs on a 30s excerpt from the AMI Meeting Corpus. The implementation is available at https://github.com/nikhilraghav29/diarizen-tutorial, which includes standalone executable scripts for each block and a Jupyter notebook that runs the complete pipeline end-to-end.

• The paper introduces DiariZen, a modular framework combining end-to-end neural diarization and VBx clustering to achieve high-accuracy speaker segmentation.
• It details a step-by-step pipeline including aggressive window overlap, pruned WavLM feature extraction, Conformer-based powerset classification, and robust overlap-add aggregation.
• Empirical results on benchmarks like AMI and DIHARD-III demonstrate DiariZen’s efficiency, reproducibility, and state-of-the-art performance in real-world diarization tasks.

DiariZen: An Expert Summary of a State-of-the-Art Speaker Diarization Pipeline

Introduction and Motivation

"DiariZen Explained: A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline" (2604.21507) provides a comprehensive technical exegesis of the DiariZen system, establishing it as the canonical open-source reference for high-accuracy speaker diarization. DiariZen synthesizes front-end end-to-end neural diarization (EEND) and back-end clustering (VBx) within a rigorously modular framework. This integration circumvents historical EEND limitations (notably speaker count fixedness) while eliminating the error propagation endemic to fully modular heuristic pipelines.

Figure 1: The sequential blocks illustrating the complete DiariZen pipeline with each principal component and data transformation.

This paper’s focus is not to introduce a new architecture but rather to decompose DiariZen into transparent, well-documented stages with reproducible code, annotated tensor shapes, and careful intermediate output visualizations. The publication thus resolves the fragmentation and opacity of the original multi-repo implementation, offering a pedagogically oriented yet technically precise resource for research and advanced development.

System Pipeline Overview

DiariZen modularizes diarization into seven interdependent blocks that receive, transform, and annotate audio with robust speaker labels in RTTM format:

1. Sliding Window Segmentation: Audio is dissected into highly overlapping 16 s windows (90% overlap), creating redundancy and statistical stability for downstream predictions.
2. WavLM Feature Extraction: Each window’s audio is encoded using a structurally pruned WavLM-Large self-supervised model, leveraging a learned weighted combination of all 25 internal representations to optimize speaker-relevant information flow.
3. Conformer-Based Powerset Classification: Extracted features traverse a 4-layer Conformer, outputting per-frame log probabilities over an 11-class powerset—each class denoting a unique configuration of up to 2 active speakers among 4 candidate slots.
4. Overlap-Add (OLA) Aggregation: The per-chunk segmentation predictions are combined by OLA, with temporal median filtering and speaker counting to realize stable, continuous diarization tracks.
5. WeSpeaker Embedding Extraction: For each (chunk, speaker) tuple, overlap-excluded embeddings are computed, ensuring that only clean, unblended speech contributes to clustering.
6. VBx Clustering: Local embeddings are globally clustered via PLDA affinity and variational Bayes HMM refinement to resolve speaker label permutations and achieve cross-window/global consistency.
7. RTTM Reconstruction: The mapping from local-to-global speaker identities is finalized, post-processed, and rendered to standard RTTM output.

Model Design Choices

Aggressive Window Overlap and Pruned Self-Supervised Encoding

The decision to adopt a 90% window overlap is empirically justified by substantial noise reduction in segmentation, albeit at increased computational cost. The backbone WavLM encoder is pruned by 80% (316M to 63M parameters), disabling 4 of 24 transformers and reducing head and FFN size without affecting output dimensionality. The pipeline deploys a learned layer-weighted scheme (as per SUPERB benchmark practice) to optimize for speaker identity—yielding large positive weights for early and final transformer layers, confirming that these layers mediate most speaker-specific information.

Figure 2: Left—Learned weights for WavLM layers (earliest and last dominate); Right—L2 norms show feature stability across windows.

Conformer Powerset Architecture

The backend Conformer enables modeling of long-range temporal dependencies and outputs an 11-category powerset describing all silences, single, and two-speaker overlaps (with $S=4$, $O=2$). The exclusion of higher-order overlaps (3+ speakers) produces a controlled hypothesis space that improves discriminability and computation. This powerset approach, in contrast to independent multi-label predictions, strictly constrains the output space to plausible meeting/telephonic scenarios while rejecting rare high-order combinations.

Figure 3: Top—Powerset class posteriors for a test chunk, exhibiting confident transitions; Middle—Per-speaker activity over the session; Bottom—Framewise powerset argmax.

Cross-Chunk Aggregation and Activity Smoothing

The OLA process is critical for denoising and aggregating redundant predictions arising from overlapping windows, followed by median filtering for temporal smoothing. Analysis demonstrates that speaker overlap is accurately modeled; in the AMI test session, the system identifies 8.0% silent frames, 64.0% single-speaker activity, and 27.9% two-speaker overlap.

Figure 4: Top—Coverage by overlapping chunks; Second—Post-aggregation activity; Third—Pre-aggregation chunk view; Bottom—Final speaker count per frame.

Supervised Embedding and Probabilistic Global Clustering

Clean (non-overlapped) speech frames for each (chunk, speaker) pair are embedded via WeSpeaker ResNet34, producing L2-normalized 256-dimensional vectors appropriate for PLDA modeling. The embeddings’ pairwise cosine similarity matrix reveals diagonal and near-diagonal high-similarity blocks, mapping directly to true speaker identity structure for clustering.

Figure 5: Left—Raw embeddings (chunk 0); Middle—Cosine similarities (same-speaker block pattern); Right—L2 norms near unity.

Global speaker assignment is executed with VBx, combining AHC with PLDA scoring and variational Bayes HMM refinement. The matrix of (chunk, local speaker) to global speaker assignments visualizes this permutation resolution.

Figure 6: Matrix of local-to-global speaker assignments from VBx—identity consistency is maintained across windows despite permutation.

RTTM Output and Segment Structure

The final step reconstructs the diarization annotation, aggregates across overlapping windows, binarizes, and emits standardized RTTM. In the AMI evaluation, the pipeline produced 4 speakers and 13 distinct segments, precisely matching conversational structure.

Figure 7: Output annotation—13 segments with clear dominance by individual speakers reflecting meeting conversational flow.

Empirical Performance and Claims

The pipeline processes a 30-second four-party session in under 60 seconds on a single H200 GPU, maintaining high alignment to manual ground truth annotation. DiariZen’s integration of pruned WavLM and EEND-VC with rigorous modularization achieves leading open-source accuracy on AMI, VoxSRC, and DIHARD-III benchmarks, as determined by reference [han2025leveraging, han2025fine, han2025efficient]. The explicit claim is that DiariZen is the strongest freely-available diarization system at the time of publication.

Theoretical and Practical Implications

This work represents an important step in reproducible, transparent, and accessible high-performance SD. The formalization and modularization detailed here facilitate research extensibility (e.g., incorporating novel clustering algorithms or alternate self-supervised encoders). Practically, DiariZen is directly deployable in large-scale ASR preprocessing, meeting analytics, and surveillance contexts due to its efficiency and minimal dependency profile.

From a theoretical perspective, the demonstrated success of powerset-based output modeling, paired with learned layer weighting, validates design directions in robust, low-error diarization for low-resource and high-overlap scenarios. The pruning results encourage future model compaction and hardware approximation research.

Future Directions

The paper hints at ongoing research including integration of adaptive affinity pruning (SC-pNA) into the clustering backend and extensive evaluations on additional challenging corpora such as DIHARD-III and DISPLACE-M. These developments reflect an agenda focused on further robustness and adapting the system to even more spontaneous conversational domains and adverse acoustic conditions.

Conclusion

This essay has summarized the structural, algorithmic, and empirical contributions of DiariZen as detailed in (2604.21507). DiariZen presents a modular blueprint for high-performance speaker diarization, resolving longstanding practical obstacles to system reproducibility and extensibility. Its open-source, pedagogically guided implementation will facilitate further research, comparative benchmarking, and deployment in advanced spoken language systems.