Diarization-Conditioned Architecture

• Diarization-conditioned architectures are systems that integrate speaker diarization into core model training, embedding speaker segmentation and clustering directly in the optimization process.
• They leverage self-attention, metric learning, and conformer modules to model both local and global temporal dependencies, significantly reducing diarization error rates.
• These architectures extend to multitask paradigms, conditioning on auxiliary tasks like speech activity detection and overlap handling, thereby improving robustness in complex acoustic conditions.

A diarization-conditioned architecture refers to any system in which the process of speaker diarization—segmenting and clustering audio into homogeneous speaker regions—is explicitly incorporated as a conditioning factor within the core architecture or optimization target, in contrast to traditional pipelines where diarization is performed as a loosely-coupled post-processing or pre-processing stage. Across the neural diarization literature, diarization-conditioning mechanisms range from end-to-end attention-based embedding extraction and metric learning (Song et al., 2018), through permutations of clustering supervision, to tightly coupled conditioning for robust multi-speaker activity modeling and downstream applications. The evolution of such architectures has substantially advanced diarization error rates (DER), interpretability, and the ability to handle overlapping and variable speaker conditions.

1. Unified End-to-End Architectures with Embedded Diarization Objectives

Diarization-conditioned systems frequently unify embedding extraction and metric learning within a single end-to-end network trained to directly minimize diarization-specific objectives. The original triplet attention network (Song et al., 2018) integrates a stack of self-attention layers operating on MFCC (plus delta) sequences, yielding fixed-dimensional embeddings via temporal pooling. Metric learning is imposed via a triplet loss: $$l(x_p, x_r, x_n) = \max(0, \|A(x_r) - A(x_p)\|_2^2 - \|A(x_r) - A(x_n)\|_2^2 + \alpha)$$ where anchor, positive, and negative segments are sampled to enforce intra-speaker compactness and inter-speaker separation. Crucially, the attention mechanism is incorporated in the embedder itself, yielding context-aware representations without dependence on separately pre-trained i-vectors or GMM-UBM models.

Later works (Narayanaswamy et al., 2018) explored the pipeline design space further, showing that, regardless of the feature extractor, careful loss function selection (triplet/quadruplet), negative sampling strategies (distance-weighted sampling, semi-hard), and discriminative margin tuning are critical for representation discriminability. Fine-grained validation analyses demonstrated substantial DER improvements with attention-based embeddings and advanced negative sampling (e.g. best DER of 12.44% on CALLHOME), but also documented sharp performance drops when increasing language/domain mismatch or speaker count.

2. Multi-head Self-Attention and Temporal Modeling

A key property of diarization-conditioned architectures is the capacity to model both local and global temporal dependencies. Early models leveraged stacked self-attention layers with positional encoding, while EEND (End-to-End Neural Diarization) (Fujita et al., 2020) formalized diarization as a frame-wise multi-label classification: $z_t = [P(y_{t,1}|X), ..., P(y_{t,C}|X)]$ with a permutation-free cross-entropy loss: $$J^{(PF)} = \frac{1}{T\cdot C} \min_{\phi \in \text{perm}(C)} \sum_{t} \text{BCE}(l_t^\phi, z_t)$$ This directly optimizes the diarization alignment over all label permutations, enabling the model to handle speaker overlap and label ambiguity natively. Analysis of attention heads revealed that some focus on global speaker characteristics (vertical pattern), while others attend to local speech activity boundaries (diagonal pattern), thereby distinguishing between global consistency and local segmentation.

The Conformer-based extension (Liu et al., 2021, Palzer et al., 5 Jun 2025) injects convolution layers into the attention mechanism, further enhancing the capture of locality in speech dynamics. Empirical results show that replacing vanilla transformers with conformers leads to significant reductions in DER (e.g., from 9.96% to 6.98% on CALLHOME (Palzer et al., 5 Jun 2025)) due to superior modeling of turn transitions and missed speech.

3. Conditioning on Auxiliary Tasks and Multi-task Paradigms

Recent diarization-conditioned architectures make explicit use of hierarchical, multitask, and chain-rule factorized models to condition the diarization objective on the outputs of related subtasks. The conditional multitask learning of (Takashima et al., 2021) introduces a probabilistic factorization: $$P(y_1,\ldots,y_S,\ u_1,\ldots,u_K| X) = \prod_k P(u_k | u_{<k}, X)\prod_s P(y_s | y_{<s}, u_{1:K}, X)$$ where $u_{1:K}$ denote auxiliary tasks (such as speech activity detection or overlap detection), and $y_{1:S}$ denote diarization labels. The architecture (transformer encoder plus uni-directional LSTM chain) thus learns dependencies that enable SAD and OD outputs to guide the diarization process. Conditioning on auxiliary subtasks (SAD-first, SAD+OD-first) reduces DER (for example, a relative improvement from 15.57% to 15.32% in variable-speaker CALLHOME experiments), particularly improving robustness in overlapping and complex conditions.

4. Robustness to Overlap and Variability

Diarization-conditioned designs explicitly address two major challenges: overlapping speech and intra-speaker variability. Overlap-aware systems (Bullock et al., 2019) augment diarization pipelines with neural LSTM-based overlap detectors and use the detected overlapping regions to assign a second speaker label per frame. The final diarization hypothesis is constructed as: $$H(t) = \begin{cases} s_t^{(1)}, & \text{if no overlap}\ s_t^{(1)}\cup s_t^{(2)}, & \text{if overlap} \end{cases}$$ This approach achieved up to 20% relative DER reduction on AMI corpora, particularly by lowering missed detection errors in overlapped regions.

Recent work further addresses intra-speaker variability by augmenting diarized speech segments with synthetic, style-diverse versions, using global style tokens and TTS techniques to blend embeddings across stylistic variations (Kim et al., 18 Sep 2025). This reduces speaker splitting errors and yields up to 49% DER reduction in high-variability simulated data, demonstrating how conditioning architectures can be expanded with augmentation modules to directly counteract adverse acoustic variability.

5. Downstream Conditioning and Advanced Representation Learning

A growing trend involves leveraging diarization-conditioned architectures as modules within larger tasks such as joint diarization-separation (Boeddeker et al., 2023), diarization-conditioned ASR (Polok et al., 30 Dec 2024), and error correction (Han et al., 2023). For instance, the DiCoW approach (Polok et al., 30 Dec 2024) conditions a Whisper-based ASR model on time–frame diarization masks via frame-level transformations (FDDT) and query-key biasing (QKb), enhancing target-speaker resilience and overlapping-speech robustness. Similarly, post-processing with transformer-based DiaCorrect (Han et al., 2023) takes acoustic features and initial speaker activity predictions as two parallel input streams, refining diarization output by fusing feature and activity information.

Another class of architectures employs attractor mechanisms, where speaker (or speaker attribute) attractors are refined through either auto-regressive multi-head cross-attention (Palzer et al., 5 Jun 2025) or iterative clustering within conformer blocks (Palzer et al., 5 Jun 2025). These allow the diarization network to internally “condition” audio representations on latent speaker prototypes, yielding high DER improvements (e.g., 4.99% on CALLHOME for a compact attractor-conformer model (Palzer et al., 5 Jun 2025)) while maintaining scalability.

6. Empirical Performance and Validation

Diarization-conditioned architectures have consistently demonstrated empirical gains over classical pipelines. Key measurements include the diarization error rate (DER), speaker confusion (CF), and error breakdown by overlap and missed speech. For example, in end-to-end self-attention models (Fujita et al., 2020), DERs drop from 19%–28% (x-vector baseline, β = 2 simulated overlap) to 6–7% (SA-EEND). For metric-learning-based attention architectures (Song et al., 2018), DERs were reduced from 18.7% (i-vector+cosine) to 12.7% (triplet attention network). Extensions leveraging robust embedding architectures (ECAPA-TDNN), mixed-data augmentation, and mixture-of-experts modules all demonstrate enhanced generalization across challenging datasets such as CHiME-6, DiPCo, Mixer 6, DIHARD III, and AMI (Dawalatabad et al., 2021, Yang et al., 17 Jun 2025).

Validation protocols increasingly include cross-lingual evaluation, artificially increased speaker count, and overlap robustness, highlighting the importance of holistic system testing.

7. Future Trajectories and Open Challenges

Diarization-conditioned architectures are moving toward greater integration of multitask learning, variable speaker handling, joint separation-diarization-ASR objectives, and advanced augmentation for intra-speaker robustness. Open challenges remain in the scalability of conditioning mechanisms to very large and spontaneous meeting datasets, training with imperfect diarization labels (semi-supervised or weakly supervised conditioning), and generalization across domains and unseen speaker and overlap patterns.

A plausible implication is that future diarization systems will increasingly unify all subtasks—embedding extraction, segmentation, clustering, overlap/speech activity detection, and downstream task adaptation—within a single diarization-conditioned architecture, trained end-to-end to optimize for holistic accuracy and interpretability across complex acoustic environments.