Conformer-based EEND Model

• Conformer-based EEND is a speaker diarization approach that interleaves convolution and self-attention to capture both local acoustic events and global speech context.
• It utilizes a four-stage sandwich architecture combining feed-forward layers, multi-head self-attention, and convolution modules to effectively model speaker segments and overlaps.
• The model demonstrates state-of-the-art performance with reduced DER by leveraging advanced feature integration, robust data augmentation, and innovative powerset training.

A Conformer-based End-to-End Neural Diarization (EEND) model applies convolution-augmented Transformer architectures to sequence-level speaker diarization, capturing both global contextual dependencies and local acoustic events directly from the input speech signal. By interleaving multi-head self-attention and convolutional modules, Conformer-based EEND systems overcome the limitations of purely self-attentive or recurrent encoders, achieving improved frame-level speaker segmentation and robust performance even in conversational scenarios characterized by frequent speaker overlap or complex turn-taking dynamics.

1. Architectural Principles

The Conformer encoder is central to EEND, providing joint modeling of long-range (global, cross-segment) dependencies and short-range (local, temporal) events. Each Conformer block consists of a four-stage “sandwich” structure:

1. Feed-forward module: $z_1 = x + \frac{1}{2}\mathrm{FFN}(x)$
2. Multi-head self-attention (MHSA): $z_2 = z_1 + \mathrm{MHSA}(z_1)$
3. Convolution module: $z_3 = z_2 + \mathrm{Conv}(z_2)$
4. Second feed-forward module and layer normalization: $\mathrm{LN}(z_3 + 0.5\,\mathrm{FFN}(z_3))$

This structure enables the encoder to simultaneously learn speaker-specific temporal patterns (e.g., turn boundaries, overlap regions) and global conversation context. The convolutional subsampling layer preceding the encoder reduces sequence length while retaining local structures via two-dimensional separable convolutions, critical for accurate speaker change detection (Liu et al., 2021).

2. Feature Engineering and Front-End Integration

Input to the Conformer-based EEND model typically consists of log-mel filterbank features or, in more recent frameworks, expressive self-supervised representations such as WavLM (Pálka et al., 22 Oct 2025). Preprocessing steps may include:

• SpecAugment: Time and frequency masking applied directly on acoustic features, mitigating overfitting and increasing representation robustness (Liu et al., 2021).
• Convolutional subsampling: Efficient reduction of sequence length with local pattern preservation, replacing simple stacking and dropping (Liu et al., 2021).
• WavLM feature integration: Use of domain-robust features extracted by large-scale self-supervised models, proven to increase generalization without dataset-specific fine-tuning (Pálka et al., 22 Oct 2025).

When ASR-derived features are available, integration occurs via early fusion (concatenation with acoustic features), late fusion (embedding concatenation prior to classification), or contextualized self-attention, where ASR features are concatenated to the query and key inputs of each Conformer layer (Khare et al., 2022).

3. Frame-Level Speaker Activity Prediction and Powerset Training

The Conformer encoder’s output sequence is projected via a fully connected layer (with sigmoid or softmax activation) to frame-level speaker activity posteriors. In multi-speaker EEND systems, “powerset training” is used: the classification head predicts all possible speaker combinations (active/inactive sets) per frame, enabling direct modeling of overlaps (Pálka et al., 22 Oct 2025). The loss function is typically binary cross-entropy over each output dimension or powerset label.

Table: Frame-level activity prediction with Conformer-based EEND

Module	Input	Output
Conformer encoder	Features F	Hidden state h
Linear classifier	Hidden h	Speaker probs

4. Data Augmentation, Training Strategies, and Generalization

Robustness and generalization of Conformer-based EEND models are critically linked to data augmentation and training strategies:

• SpecAugment is indispensable for reducing development and test error.
• Exponential moving average (EMA) and variational noise injection increase training stability and speed (Karita et al., 2021).
• Mixed training data (real + simulated) aligns temporal statistics of training/testing sets (e.g., overlap/silence durations), mitigating performance loss when simulated data poorly matches conversational structure (Liu et al., 2021).
• Multi-task learning with ASR feature auxiliary loss enforces the encoder to encode both diarization and lexical/phonetic cues, improving speaker discrimination (Khare et al., 2022).

Quantitative improvements (e.g., a 24% DER reduction over SA-EEND and 10% over transformer-based TB-EEND on CALLHOME) are consistently reported when deploying Conformer-based encoders with these strategies (Liu et al., 2021).

5. Integration with Post-Processing and Clustering Techniques

Global speaker identification and count estimation in open-domain or long-form recordings utilize a two-stage EEND-VC (“EEND with vector clustering”) approach:

1. Short-window Conformer-based EEND with WavLM features yields robust frame-level speaker probabilities and embeddings in local regions (Pálka et al., 22 Oct 2025).
2. Global speaker identities are deduced via clustering algorithms (e.g., VBx clustering), with unreliable embeddings filtered and reassigned for improved performance when speaker counts are large and the data domain is variable.

This hybrid pipeline demonstrates state-of-the-art generalizability and error rates across diverse domain benchmarks (AMI, AISHELL-4, AliMeeting, DIHARD3, VoxConverse) without fine-tuning or per-dataset parameter search (Pálka et al., 22 Oct 2025).

6. Advantages, Limitations, and Generalization Challenges

Advantages of Conformer-based EEND models include combined local/global representation, efficient training (high parallelism), and resilience to overlapping speech. However, generalization from simulated to real conversational data remains challenging unless training data temporal statistics match test data distributions (Liu et al., 2021). While convolution enhances local structure modeling, increases in model complexity and sensitivity to data simulation recipe can yield degradation in mismatched scenarios.

Recent work incorporating advanced self-supervised features (WavLM), domain-mixed training, and carefully designed clustering in the global post-processing stage demonstrably improves generalization and robustness without exhaustive hyperparameter search or model adaptation (Pálka et al., 22 Oct 2025).

7. Future Directions and Extensions

Promising directions involve cross-domain training robustness, joint ASR-diarization modeling, and adaptation to streaming or low-latency settings through causal masking and efficient context carry-over (Pálka et al., 22 Oct 2025). Theoretical extensions include:

• Contextualized attention for integrating semantic or prosodic cues
• Streaming EEND implementations with causal Conformer blocks
• **Memory augmentation for long-form diarization (external memory modules as in Conformer-NTM), enabling persistence and retrieval of long-range speaker information

A plausible implication is that domain-robust front ends, convolutional subsampling, and dynamic context integration will remain at the core of future Conformer-based EEND models, especially as requirements for deployment in real-world, open-domain, and multi-speaker scenarios proliferate.

---

Overall, the Conformer-based EEND architecture—augmented by data-driven strategies, sophisticated feature engineering, and robust clustering—currently represents the state-of-the-art in speaker diarization, balancing efficiency, accuracy, and cross-domain generalizability (Pálka et al., 22 Oct 2025).