DiariZen: Hybrid Speaker Diarization
- DiariZen is a family of speaker diarization systems that determine 'who spoke when' using a blend of neural diarization and global clustering techniques.
- It encompasses diverse applications—from judicial speaker verification to joint diarization-identification—leveraging state-of-the-art architectures like WavLM and Conformer.
- DiariZen also supports downstream tasks such as speaker-attributed ASR and spatial integration, demonstrating robust performance across multilingual and challenging environments.
DiariZen is a name used in several strands of speaker-processing research to denote systems for speaker diarization and closely related tasks such as speaker identification and speaker-attributed automatic speech recognition. In the broadest sense, these systems address the canonical diarization problem of determining “who spoke when,” but the literature uses the label for materially different formulations: a reference-dependent judicial-speech workflow for legal proceedings, a modular toolkit for joint diarization and identification, and, in later diarization work, a hybrid end-to-end neural diarization pipeline built around WavLM representations, a Conformer backend, and clustering for global speaker consistency (Tumminia et al., 2021, Morrone et al., 2024, Han et al., 2024, Raghav, 23 Apr 2026).
1. Terminological scope and lineages
The term does not denote a single immutable architecture. Instead, the literature attaches the name to several systems with overlapping goals but different assumptions, components, and deployment targets.
| Usage | Paper | Defining characteristics |
|---|---|---|
| Reference-dependent legal diarization | "Diarization of Legal Proceedings. Identifying and Transcribing Judicial Speech from Recorded Court Audio" (Tumminia et al., 2021) | Reference Audio Library, GE2E-trained d-vectors, open-set identification of judges |
| Modular joint diarization-identification toolkit | "A Toolkit for Joint Speaker Diarization and Identification with Application to Speaker-Attributed ASR" (Morrone et al., 2024) | SAM, Audioma, FlyScribe, EEND-vector clustering, configurable SD/VAD/SI/ASR pipeline |
| Hybrid WavLM-based open-source diarization pipeline | "Leveraging Self-Supervised Learning for Speaker Diarization" (Han et al., 2024); "DiariZen Explained: A Tutorial for the Open Source State-of-the-Art Speaker Diarization Pipeline" (Raghav, 23 Apr 2026) | WavLM front end, Conformer, powerset classification, speaker embeddings, clustering, RTTM output |
In the legal-proceedings work, DiariZen is effectively a reference-dependent speaker verification system for partially known speakers rather than a fully generic clustering-only diarizer. In the toolkit paper, it becomes a configurable end-to-end application stack for diarization, identification, and speaker-attributed transcription. In later diarization-focused papers, DiariZen is described as a hybrid EEND-VC-style pipeline and is treated as a strong open-source baseline or open-source state-of-the-art system, depending on the benchmark and publication date (Lanzendörfer et al., 30 Sep 2025, Raghav, 23 Apr 2026).
A common misconception is to treat these usages as identical. The papers suggest instead that “DiariZen” names a family of approaches whose shared concern is speaker-structure inference, while the operational problem may range from open-set judicial speaker recovery to multilingual long-form diarization to speaker-attributed transcription.
2. Canonical hybrid diarization architecture
In the diarization literature from 2024 onward, DiariZen is most clearly specified as a hybrid pipeline combining local neural diarization with global clustering. The tutorial paper decomposes the system into seven stages: audio loading and sliding-window segmentation, WavLM feature extraction with learned layer weighting, Conformer backend and powerset classification, segmentation aggregation via overlap-add, speaker embedding extraction with overlap exclusion, VBx clustering with PLDA scoring, and reconstruction into RTTM output (Raghav, 23 Apr 2026).
The inference procedure begins by segmenting mono 16 kHz audio into overlapping 16-second chunks with a step ratio of 0.1, which yields 90% overlap. The tutorial’s AMI example uses ten chunks for a 30-second excerpt. Each chunk is passed through a structurally pruned WavLM-Large encoder reduced from 316M to 63M parameters, with about 80% parameter reduction, while preserving a 1024-dimensional interface. The encoder produces 25 hidden representations—one CNN feature extractor output and 24 transformer layer outputs—which are combined by a learned weighted sum in a SUPERB-style aggregation scheme (Raghav, 23 Apr 2026).
The aggregated representation is projected and processed by a 4-layer Conformer backend, followed by a powerset classifier. The powerset formulation constrains frame-level outputs to speaker-combination classes rather than independent per-speaker decisions. With a maximum of speakers per chunk and at most simultaneous speakers, the number of classes is
This gives one silence class, four single-speaker classes, and six two-speaker overlap classes (Raghav, 23 Apr 2026). The same 11-class design appears in the earlier WavLM-based DiariZen paper, which describes the system as replacing Pyannote’s local EEND front end with WavLM features, a linear projection, layer normalization, a 4-block Conformer, and a powerset classifier (Han et al., 2024).
Chunk-level predictions are then merged by overlap-add aggregation, after which median filtering smooths the framewise speaker activity timeline. For global identity consistency, embeddings are extracted only from clean single-speaker regions when possible; overlap frames are excluded from the embedding mask, and overlap-inclusive fallback is used only when too few clean frames remain. The tutorial describes a 256-dimensional WeSpeaker ResNet34 embedding extractor, L2 normalization, LDA projection, PLDA scoring, agglomerative initialization, VB-HMM refinement, and final reconstruction into RTTM segments (Raghav, 23 Apr 2026). In benchmark-oriented descriptions, DiariZen is summarized more compactly as a hybrid system combining EEND-like local diarization, WavLM-based speaker embeddings, and Pyannote clustering components (Lanzendörfer et al., 30 Sep 2025).
This architecture is significant because it joins two historically separate diarization paradigms. The local EEND component models overlap-aware frame activity, while the clustering stage restores recording-level speaker identity consistency across long inputs.
3. Empirical performance and benchmark position
The first major DiariZen results are reported on far-field AMI, AISHELL-4, and AliMeeting. In that study, the WavLM-updated configuration outperforms a Pyannote baseline under both collar settings. At 0 s collar, the reported DERs are 15.4 on AMI, 11.7 on AISHELL-4, and 17.6 on AliMeeting, versus 21.1, 13.9, and 22.8 for the Pyannote baseline. At 0.25 s collar, the corresponding results are 9.8, 5.9, and 10.2, versus 13.7, 7.7, and 13.6 (Han et al., 2024). The same paper reports that WavLM-based variants are markedly more robust under data scarcity: at 5% of the compound real-data training set, Fbank degrades by about 60% relative DER, whereas WavLM systems degrade by less than 30%, and WavLM-updated trained on 14.4 hours still beats the full-data Pyannote baseline (Han et al., 2024).
A later multilingual benchmark evaluates DiariZen out of the box across four diarization datasets spanning five languages and 196.6 hours of audio. The reported overall DER is 13.3%, compared with 11.2% for PyannoteAI, and the paper characterizes DiariZen as a competitive open-source alternative. One particularly strong result is 7.1% DER in the 5+ speaker setting, which the benchmark highlights as evidence of good high-speaker-count behavior (Lanzendörfer et al., 30 Sep 2025).
Challenge results further illustrate the system’s operating regime. In DISPLACE-M Track 1, targeting naturalistic medical conversations in noisy rural-healthcare scenarios, the best submitted configuration uses the Diarizen baseline with AHC and median filtering with context window 29, achieving 10.37% DER on development and 9.21% on evaluation. The post-evaluation analysis reports about a 39% relative DER reduction over a SpeechBrain ECAPA-TDNN baseline with estimated VAD (Raghav et al., 2 Mar 2026).
These results establish DiariZen as a high-performing open-source diarization framework, but they also delimit its strengths. The benchmark literature places it close to the best reported commercial system in some settings, yet not uniformly superior across domains or evaluation protocols (Lanzendörfer et al., 30 Sep 2025).
4. Domain assumptions, adaptation, and failure modes
The legal-proceedings variant demonstrates how strongly task assumptions shape the meaning of “DiariZen.” That system, called Reference-Dependent Speaker Verification, constructs a Reference Audio Library from annotated judge speech and compares d-vectors from target cases against this library. It uses a pretrained Resemblyzer Voice Encoder trained with GE2E loss, computes cosine similarity over unit-normalized embeddings, and applies score_thresh = 0.85 and sim_thresh = 0.10 to decide whether a segment belongs to a referenced judge or an unreferenced speaker. On a held-out test set of 25 SCOTUS cases, the method reports a mean DER of 13.8% for speakers covered by the reference library, using pyannote.metrics with a 0.5-second collar (Tumminia et al., 2021). This is a fundamentally different operating point from unconstrained meeting diarization, because the candidate judges are known in advance and lawyer speech is collapsed into an “unreferenced” category.
A contrasting picture emerges in low-resource long-form Bengali speech. In the DL Sprint 4.0 competition setting, DiariZen was evaluated alongside ECAPA-TDNN, Pyannote 3.1, and Pyannote Community-1 on a 22-hour hidden test set of long audio. The DiariZen configuration with retraining via Bengali-trained WavLM integration and post-processing reports public/private DERs of 0.28077 and 0.27893, with RTF 0.020. It is not the best system on either leaderboard, and the paper concludes that DiariZen is a weak fit for this low-resource long-form Bengali setting (Hasan et al., 26 Feb 2026).
The Bengali study is especially informative because it documents failed adaptation attempts. Integrating a Bengali-trained WavLM into DiariZen produced no performance shift; Demucs noise removal worsened diarization; fine-tuning segmentation on muffled audio for 20 epochs yielded no observable DER improvement; and extensive retraining more generally was described as yielding negligible improvement (Hasan et al., 26 Feb 2026). The winning strategy instead relied on heuristic post-processing—“Strict Gap Post-Processing”—applied most effectively to Pyannote Community-1 rather than to DiariZen. The specified thresholds were
This suggests that DiariZen’s strong benchmark position should not be interpreted as universal transferability. The papers indicate that performance is highly contingent on domain structure, annotation quality, overlap patterns, speaker-count regime, and whether the task is genuinely generic diarization or a partially known-speaker identification problem.
5. Spatial and multi-channel extensions
Several papers treat DiariZen as a base system for incorporating spatial information into foundation-model-based diarization. One line of work conditions the single-channel WavLM-based DiariZen model with multi-channel spatial cues derived from inter-channel phase differences and magnitude spectrograms, injected through FiLM layers before the Conformer and before each Conformer block. In this study, the single-channel baseline yields 12.2% macro DER under oracle clustering and 14.8% under VBx clustering. The best spatially conditioned version, DiariZen + Spatial Diarization with joint fine-tuning, reaches 11.6% macro DER under oracle clustering and 14.1% with VBx (Deegen et al., 5 Jan 2026). The gains are consistent but small, and the paper explicitly notes that the expected dramatic improvement in overlapping speech does not materialize.
The authors’ interpretation is that the learnable weighted sum across all WavLM layers may already encode much of the information needed for accurate speaker discrimination, including in overlap regions (Deegen et al., 5 Jan 2026). This is an important result because it tempers the assumption that spatial cues must produce large gains whenever microphone arrays are available.
A second line of work makes DiariZen itself multi-channel aware by inserting cross-channel communication modules into the early WavLM layers. Two mechanisms are explored: ChannelAttention and TAC. The system remains agnostic to the number of channels and array topology, and the clustering stage uses VBx. In realistic full-diarization experiments, the reported macro DERs are 18.6% for the single-channel baseline, 17.4% for DOVER-Lap, and 16.7% for the best proposed configuration, namely ChannelAttention with attentive weighted fusion of multi-channel speaker embeddings. On CHiME-6, this setting achieves 27.5% DER (Han et al., 16 Oct 2025). The same paper emphasizes that embedding extraction is the main computational bottleneck and that attention-based channel selection or fusion improves both efficiency and accuracy relative to per-channel processing with DOVER-Lap (Han et al., 16 Oct 2025).
Together, these studies position DiariZen as a stable platform for spatial experimentation: one may either condition a strong single-channel foundation-model diarizer on auxiliary spatial features or alter the foundation model so that cross-channel information is processed internally.
6. Tooling, joint identification, and diarization-aware transcription
Beyond pure diarization, DiariZen also appears as an application framework for joint speaker analysis and transcription. The toolkit paper defines three main components: the Speaker Analysis Microservice (SAM), which exposes speaker diarization, voice activity detection, and speaker identification through gRPC; Audioma, which handles request management, scheduling, queuing, parallelization, ASR processing, and communication with SAM; and FlyScribe, a web-based graphical interface for job submission and result inspection (Morrone et al., 2024). The diarization side follows an EEND-vector clustering framework and can replace internal speaker embeddings with Wespeaker extractors trained on VoxCeleb2 for improved performance on long recordings with more than five speakers. The identification side supports both closed-set and open-set operation, with unknown speakers optionally labeled as Speaker1, Speaker2, ..., SpeakerN. Configuration files determine whether preprocessing uses SD or VAD, which SI model and registered speaker set are used, and which ASR engine is selected. On a Linux server with an Intel Core i7-9800X CPU at 3.80 GHz, one CPU thread, and no GPU, the paper reports an RTF of 0.18 for a transcription job (Morrone et al., 2024).
DiariZen also functions as an upstream structural prior in multi-speaker ASR. DM-ASR reformulates transcription as multi-turn dialogue generation conditioned on diarization results. The paper explicitly compares diarization-assisted setups using front ends such as DiariZen and S2SND, and reports that a DM-ASR system using DiariZen with CN 1300h training and a 0.6B decoder achieves cpCER/tcpCER of 21.60/22.23 on AliMeeting and 19.69/20.40 on AISHELL-4 (Li et al., 24 Apr 2026). The broader conclusion is that diarization systems provide reliable speaker identities and segment-level boundaries, while LLMs provide linguistic modeling and long-context reasoning (Li et al., 24 Apr 2026).
This downstream usage underscores a broader point: DiariZen is not only an end-task diarization system but also a structural module for larger speech pipelines. In practice, its outputs are consumed by speaker-attributed ASR, subtitle generation, indexing, institutional workflows, and domain-specific speaker identification, making the system family relevant well beyond standalone DER reporting.