Overlap-aware Speaker Diarization

• Overlap-aware speaker diarization is a methodology that models and resolves overlapping speech using power-set encodings and permutation-invariant training.
• The approach integrates specialized loss functions, neural architectures, and graph-based clustering to accurately assign multiple speakers per frame in complex audio environments.
• Empirical results show significant reductions in diarization error rates in meetings, broadcasts, and podcasts, making the technique robust for real-world multi-party scenarios.

Overlap-aware speaker diarization comprises a set of methodologies, algorithms, and neural architectures that directly model, detect, and resolve regions of simultaneous speech (overlap) between multiple speakers within multi-party audio recordings. Unlike conventional diarization, which typically assumes single-speaker dominance per frame and often mislabels or ignores overlap, overlap-aware approaches are specifically optimized to answer the question "who spoke when" robustly—even in highly conversational, realistic meeting or broadcast conditions where overlapping speech is frequent and non-negligible. Modern overlap-aware diarization systems tightly integrate separation, detection, and speaker attribution, leveraging power-set encodings, permutation-invariant objectives, graph-based community detection, and overlap-specialized loss functions to achieve state-of-the-art diarization error rates in the presence of substantial overlap.

1. Problem Formulation and Significance

Classical speaker diarization outputs a segmentation of an audio stream into speaker-homogeneous segments, each attributed to a single speaker. This paradigm fails under high speaker overlap, where multiple speakers may be active in the same short time window. Overlap-aware diarization resolves this by allowing the system to assign, for each frame or time–frequency unit, one or more speaker labels, encoding the joint activity pattern across all possible speaker subsets.

Let $y_{t,s} \in \{0,1\}$ be the frame-level speech activity indicator for speaker $s$ at time $t$. Overlap-aware models predict $\mathbf{y}_t = [y_{t,1}, ..., y_{t,S}]$, supporting $||\mathbf{y}_t||_1 > 1$ in the case of overlap. Diarization Error Rate (DER) is computed over the full time–speaker occupancy matrix, counting misses, false alarms, and confusion—including multi-label errors for overlap regions (Jalal et al., 8 Aug 2025).

Overlap phenomena are especially pronounced in naturalistic meetings, podcasts, and casual conversations, where interruptive, collaborative, or backchannel speech can constitute 10–40% of total speech time. Overlap is also highly correlated with the hardest diarization confusions (Confusion errors), as classic embedding–clustering pipelines are fundamentally limited by single-speaker segment assumptions (Horiguchi et al., 30 May 2025).

2. Model Architectures and Overlap-aware Training Schemes

Power-set and Multi-label Encodings

Traditional approaches model each speaker's activity independently (multi-label classification). More recent systems recast the frame labeling as power-set (PSE) single-label classification: for $N$ enrolled speakers and up to $K$ simultaneous speakers, all valid speaker subsets form $C = \sum_{k=0}^K \binom{N}{k}$ classes. Multi-label $\rightarrow$ single-label mapping allows explicit modeling of speaker dependencies, removes threshold-tuning problems, and assigns combinations of speakers as atomic classes (Wang et al., 2023, Du et al., 2022, Du et al., 2022). For example, the EEND-OLA, SEND, and SOND models all employ PSE for direct overlap modeling.

Neural Overlap-aware Diarization Architectures

The prevailing neural overlap-aware diarization systems include:

• End-to-End Neural Diarization (EEND): Predicts per-frame multi-label activity using permutation-invariant training (PIT). Permutational invariance ensures correct speaker assignment regardless of output channel order (Horiguchi et al., 30 May 2025, Bredin et al., 2021).
• Power-set Enhanced Neural Diarization (EEND-OLA, SEND, SOND, TOLD): Integrates PSE, speaker context dependencies, and multi-stage refinement (e.g., SOAP Networks in TOLD) (Wang et al., 2023, Du et al., 2022, Du et al., 2022).
• Cluster-based + Overlap Assignment: Combines frame/segment-level embeddings with overlap detection (often by neural OSD modules) and uses clustering/discrete assignment approaches for speaker label estimation, e.g., overlap-aware spectral clustering (Raj et al., 2020), CDGCN (Wang et al., 2023), or OCDGALP (Li et al., 3 Jun 2025).
• Spatial or Array-based Systems: Exploit spatial information through beamforming and direction-of-arrival estimation, paired with deep encoders and overlap-aware OSD, achieving fine-grained overlap resolution in multi-microphone setups (Wang et al., 2022, Zheng et al., 2021).
• Enrollment-free Targeted Systems: Automatically derive target speaker embeddings from mixtures, relying on dual-stage pipelines blending VAD pretraining and separation under overlap-specific losses (see Section 3) (Jalal et al., 8 Aug 2025).

Pretraining for Overlap-awareness

Large-scale pretraining on multi-speaker identification tasks using fully overlapped mixtures has proven data- and storage-efficient, and yields overlap-aware encoders with strong generalization (Horiguchi et al., 30 May 2025). This contrasts with previous approaches that depend on large amounts of simulated conversational data for overlap coverage.

3. Overlap-specific Loss Functions and Detection Modules

High performance in overlap-aware diarization is strongly linked to dedicated overlap-sensitive loss terms and specialized overlap detection modules.

Overlapping Spectral Loss and Frame Weighting

Loss augmentation strategies include overlap-weighted penalties that increase reconstruction or separation loss in time–frequency bins where multiple speakers are active. For example, the overlap weight mask

$$w(f, t) = \frac{\sum_{k=1}^K |Y_k(f,t)|}{\max_k |Y_k(f,t)| + \epsilon}$$

assigns higher emphasis to frames with simultaneous speech. The total loss combines base L1/L2 frame reconstruction with an overlap-aware penalty, modulated by a tunable $\lambda$ (Jalal et al., 8 Aug 2025).

Within hybrid architectures, overlap detection heads are trained using binary cross-entropy per frame, with OSD output either gating the overlap-aware assignment mechanism or directly altering clustering constraints (Wang et al., 2022, Bullock et al., 2019).

Permutation-invariant and Conditional Losses

Permutation-invariant objectives (PIT) minimize the loss over all possible matchings between model outputs and reference speakers for each window/chunk, supporting overlap by resolving speaker assignment ambiguity (Horiguchi et al., 30 May 2025, Bredin et al., 2021). In conditional multitask EEND, diarization is conditioned on subtask predictions (VAD, OSD) via the probabilistic chain rule, enabling explicit dependency modeling (Takashima et al., 2021).

Resegmentation and Post-processing

Overlap-aware resegmentation modules refine primary diarization outputs by integrating OSD and frame-level posterior matrices, attributing overlapped frames to both the most probable and secondary speakers as per posteriors (Bullock et al., 2019, Bredin et al., 2021). Two-best or softmax-based assignments have become standard, with statistical smoothing used to reduce spurious overlap labels.

4. Overlap-aware Clustering and Graph-based Methods

Recent research extends overlap-aware diarization to the clustering step, introducing graph-theoretic formulations:

• Overlap-aware Spectral Clustering: Modifies the assignment matrix constraint from sum-to-one (per-segment) to sum-to-two in overlap regions, solved via convex relaxations and eigen-decomposition. Overlap assignment is finalized using modified non-maximal suppression, allowing multi-label segment assignment (Raj et al., 2020).
• Community Detection GCNs (CDGCN, OCDGALP): Constructs speaker graphs where each node is a segment embedding, edge weights are similarity-derived, and both community (speaker) assignment and second-label (overlap) assignment are learned via GCNs or graph attention plus label propagation. Label propagation strategies explicitly allow multi-community (multi-speaker) membership during overlap (Li et al., 3 Jun 2025, Wang et al., 2023).
• Graph-PIT: Generalizes permutation-invariant training to graph coloring, enforcing no two overlapping utterances are assigned to the same channel in utterance-based diarization. Clustering with cannot-link constraints enables consistent global speaker tracks (Kinoshita et al., 2022).

5. Practical Implementations, Adaptivity, and Latency Considerations

Enrollment-free and Adaptive Systems

Overlap-aware diarization can eliminate dependence on pre-recorded speaker enrollment. By clustering embeddings derived from non-overlapped segments or entire utterances, systems can estimate the number and identity of speakers in an unsupervised, fully automatic manner (Jalal et al., 8 Aug 2025, Tanveer et al., 2022).

Hybrid adaptive pipelines select optimal diarization branch—conventional clustering or end-to-end segmentation—based on measured overlap rates within each meeting. For example, when overlap ratio $R_{ov}$ is below 1%, traditional clustering is used; otherwise, an overlap-robust end-to-end path is applied (Huang et al., 28 May 2025). This yields significant DER improvements (42.8% reduction in challenging evaluation sets), demonstrating the importance of adaptive overlap-handling.

Online and Low-latency Operation

Overlap-aware designs can be implemented in online or streaming settings by processing short rolling buffers with high time resolution (e.g., 5 s chunks, 16 ms frames), and by supporting incremental embedding extraction and constrained clustering. Assigning cannot-link constraints prevents fusion of simultaneously active (overlapping) embeddings, with the Hungarian algorithm enforcing one-to-one assignments within each buffer (Coria et al., 2021). Latency-accuracy trade-offs are explicitly measured, with performance gracefully degrading as output latency is reduced from seconds to hundreds of milliseconds.

Data and Annotation Issues

Boundary precision is critical for overlap-aware diarization. Training on loosely aligned ASR-style segments (including internal pauses) leads to erroneously inflated DER and poor out-of-domain generalization. Forced alignment and pause-absorption into tight, diarization-style boundaries prevent this degradation, ensuring training consistency and accurate overlap handling in both diarization and ASR downstream tasks (Horiguchi et al., 12 Jul 2025).

6. Performance Metrics, Ablation Results, and Empirical Advances

Performance in overlap-aware diarization is mainly evaluated using DER (including in-overlap errors), confusion rate, and, for ASR-coupled scenarios, cpWER/cpCER.

Key empirical highlights:

System/Study	Dataset	DER Baseline	DER Overlap-aware	Rel. Gain
Robust ECAPA (V1-V4, OL) (Jalal et al., 8 Aug 2025)	LibriCSS (single-mic)	14.69%	4.21%	71%
TOLD (EEND-OLA+SOAP) (Wang et al., 2023)	CALLHOME	15.29% (EEND-EDA)	10.14%	34%
SOND (PSE+SOND) (Du et al., 2022)	AliMeeting	6.92% (TSVAD)	4.46%	36%
OCDGALP (GAT+LPANNI) (Li et al., 3 Jun 2025)	DIHARD-III Eval	23.0% (AHC)	15.94%	30%
CDGCN (full) (Wang et al., 2023)	DIHARD-III	20.75% (TDNN+AHC)	13.72%	34%

Ablations across systems consistently show that each of the following contributes markedly to final performance: explicit overlap labeling, overlap-specific loss weighting, power-set or multi-label formulations, robust embedding extraction (especially under augmentations), high-quality overlap detection, and graph-based constraints. Failure to accurately detect and assign overlap is the main contributor to speaker confusion and missed speech in modern systems.

7. Limitations, Challenges, and Future Directions

Despite substantial progress, several open problems remain. Current overlap-aware diarization methods are still limited by:

• Multi-speaker scalability: Power-set encodings grow combinatorially with the number of speakers and maximal allowed overlap, limiting size for high-$N$ or high-$K$ scenarios (Wang et al., 2023, Du et al., 2022).
• Reliance on accurate OSD: Errors in the overlap detector cascade into clustering and assignment, particularly when overlapping frames are misclassified as single-speaker (Jalal et al., 8 Aug 2025, Bullock et al., 2019).
• Joint training and integration: Most systems keep VAD, overlap detection, embedding, and clustering modules decoupled; promising directions include joint end-to-end optimization of all components and tighter integration of graph-based and neural modules (Wang et al., 2023, Wang et al., 2023).
• Streaming and real-time: While several online implementations exist, there is a trade-off between latency and overlap-resolving power, with the need for robust streaming architectures that maintain accuracy in highly dynamic, overlapping environments (Coria et al., 2021).
• Boundary annotation variance: Models are sensitive to the style and precision of segment boundaries in training data; standardizing on tight, diarization-style boundaries improves generalizability and accuracy (Horiguchi et al., 12 Jul 2025).

Anticipated future developments include adaptive overlap weighting, fine-grained loss design, universal pretraining across arbitrary overlap conditions, dynamic estimation of simultaneous speakers, and integration of multimodal cues (particularly spatial information) for further improvements in challenging, real-world environments.