---
title: End-to-End Neural Diarization (EEND)
url: https://www.emergentmind.com/topics/end-to-end-neural-diarization-eend
type: topic
---

# End-to-End Neural Diarization (EEND)

End-to-End Neural Diarization (EEND) is a paradigm for speaker diarization that reformulates the task as direct, frame-level multi-label classification using deep neural networks. EEND approaches have demonstrated substantial superiority over clustering-based pipelines, particularly in the presence of overlapped speech, and enable the integration of advanced deep learning architectures and end-to-end optimization of Diarization Error Rate (DER) [2003.02966]. The following sections provide a comprehensive overview of EEND and its major contemporary variants, including LS-EEND, covering architectural principles, streaming and long-form diarization, training methodologies, empirical performance, and system-level implications.

## 1. Core Principles and Canonical EEND Framework

The foundation of EEND is the direct mapping from input acoustic features, typically log-Mel filterbanks with context stacking, to frame-wise speaker activity posteriors $\hat y_{s,t} \in [0,1]$ for a set of candidate speakers $s$, at every frame $t$. The network structure consists of:

- A deep encoder, originally BLSTM or Transformer-based, that produces frame-level embeddings $e_t$.
- A multi-label, permutation-invariant training objective (PIT), which considers all $S!$ permutations of $S$ output streams and minimizes binary cross-entropy with respect to the optimal reference ordering for each utterance, thus resolving the “label permutation” ambiguity:
  $$
  \mathcal{L}_{\rm PIT} = \frac{1}{S T} \min_{\phi \in \rm{Perm}(S)} \sum_{t=1}^T \sum_{s=1}^S -y_{\phi(s),t} \log \hat y_{s,t} - (1-y_{\phi(s),t}) \log (1-\hat y_{s,t}).
  $$
The output $\hat{\mathbf{Y}} = [\hat{y}_{s,t}]$ is interpreted as frame-level multi-label speaker activity, naturally handling speech overlaps—i.e., multiple concurrent active speakers [2003.02966].

## 2. Architectural Evolution and Attractor Mechanisms

A central challenge in EEND is supporting variable and flexible numbers of speakers. The Encoder-Decoder Attractor (EDA) module enables this by producing a variable set of $S^*$ “attractor” vectors $\mathbf{a}_s$ via a recurrent sequence-to-sequence architecture (LSTM or Transformer variant), conditioned on the encoder output [2106.10654, 2312.06253]. For each attractor, the model computes $\hat{y}_{s,t} = \sigma(\mathbf{a}_s^\top e_t)$. A speaker-existence mechanism (sigmoid-activated scalar $\hat z_s$ per attractor) is used to dynamically select the number of speakers during inference.

Notable architectural variants and enhancements include:
- **Transformer and Conformer Encoders:** Employ multi-head self-attention (Transformer), or hybrid convolutional-attention blocks (Conformer) to capture both long-range speaker relations and local context. Conformer-based EEND further reduces DER, especially when conversational statistics match target data [2106.07167, 2410.06670].
- **Attention-based Decoder Replacements:** Transformer-based attractor decoders (as in AED-EEND, EEND-TA) improve both convergence and efficiency compared to LSTM-based EDA [2309.06672, 2312.06253].
- **Embedding Enhancement:** Embedding enhancer modules (cross-attention refinements post-attractor extraction) further improve discrimination and robustness to unseen speaker numbers [2309.06672].

## 3. Streaming and Long-Form Diarization (LS-EEND and Related)

Standard self-attention incurs quadratic complexity in sequence length, limiting conventional EEND to short to medium-length audio. LS-EEND achieves true online, frame-synchronous diarization with linear temporal complexity, enabling diarization of hour-long recordings in streaming mode [2410.06670].

- **Causal Conformer Encoder:** Each block uses multi-head "retention" (a recurrence-compatible attention surrogate), causal convolutions (preserving causality), and L2-normalization of embeddings.
- **Online Attractor Decoder:** Maintains per-speaker attractors $a_{s,t}$, updated at every frame using along-time retention (O($T D^2$)), and cross-attractor self-attention in the speaker dimension for enhanced speaker separation.
- **Retention Mechanism:** Retention, replacing self-attention softmax with an exponential decay mask, allows efficient accumulation of past context in both training (parallel recurrent updates) and inference (sequential recurrence).
- **Frame-in-Frame-out Processing:** Every input frame directly updates all speaker attractors and yields a diarization prediction with minimal delay (≤1 s latency with lookahead).
- **Progressive Training:** Multi-stage curriculum over increasing speaker count and audio length, chunk-wise retention during long-form adaptation, and output-anchored losses for direct scale-up.
- **Performance:** LS-EEND achieves new state-of-the-art online DER on CALLHOME (12.11%), DIHARD II (27.58%), DIHARD III (19.61%), and AMI (20.76%) with linear time complexity (RTF ≈ 0.028), outperforming buffer-based and block-wise streaming systems by 3–7 DER points while reducing computational cost by an order of magnitude [2410.06670].

## 4. Training Methodologies and Data Simulation

EEND training is characterized by large-scale simulation, domain adaptation, and recent advances in "teacher forcing" and enhanced data synthesis:

- **Data Simulation:** Both simulated mixtures (random pause distributions) and more realistic simulated conversations (empirical pause/overlap statistics) are used. The latter better emulates target data conversational structure, reducing reliance on fine-tuning and improving generalization [2204.00890].
- **Curriculum Learning:** Progressive increase in the number of speakers and utterance length, as in LS-EEND's staged training [2410.06670].
- **Loss Functions:** PIT binary cross-entropy for diarization, auxiliary losses (e.g., embedding similarity, intermediate attractor losses), and cross-entropy over auxiliary subtasks (speech activity detection, overlap detection) in multitask setups [2309.06672, 2106.04078].
- **Pseudo-label and Semi-supervised Training:** Iterative pseudo-labeling and committee-based fusion allow effective adaptation to unlabeled domains, yielding up to 37.4% relative DER reduction without ground-truth frame annotations [2106.04764].

## 5. Empirical Performance and Benchmarking

EEND and its derivatives establish state-of-the-art results across diarization benchmarks. Representative best accuracies as reported:

| System                | CALLHOME | DIHARD II | DIHARD III | AMI   | RTF   |
|-----------------------|----------|-----------|------------|-------|-------|
| LS-EEND [2410.06670]  | 12.11%   | 27.58%    | 19.61%     | 20.76%| 0.028 |
| AED-EEND+Enh/Conformer[2309.06672]| 10.08%| 24.64%| 13.00%| —   | —     |

LS-EEND and AED-EEND variants, without oracle SAD, robustly outperform prior online and offline systems, including those based on offline EEND-EDA, FLEX-STB, and buffer-based inference [2410.06670, 2309.06672].

## 6. Extensions: Multi-Channel, Unlimited-Speaker, and System Calibration

- **Multi-Channel Diarization:** EEND extends naturally to distributed microphone settings via spatio-temporal and co-attention encoder variants, leveraging spatial diversity for superior DER, even in asynchronous or spatially ambiguous conditions [2110.04694].
- **Unlimited-number-of-Speakers:** Offline and online block-wise inference coupled with clustering over local attractors allows EEND-GLA to handle recordings with more speakers than seen during training, by relaxing the cap on output tracks via post-hoc clustering [2206.02432].
- **Calibration and Fusion:** Probability-level calibration (joint or per-speaker), probability-space fusion, and "Fuse-then-Calibrate" schemes significantly reduce DER and enable use of EEND outputs in risk-aware diarization or system combination. Soft probability fusion outperforms hard-segmentation methods such as DOVER-Lap [2511.22696].

## 7. Methodological Implications and Future Directions

EEND and LS-EEND unify embedding extraction, attractor generation, and diarization into a single, end-to-end optimized, causal network, enabling deployment in real-time, low-latency scenarios such as meeting transcription, conferencing, and streaming ASR front-ends. While current architectures cap maximum speakers by decoder dimension or block design, ongoing advances in unsupervised attractor clustering, dynamic attractor selection, semi-supervised adaptation, and multimodal conditioning continue to extend EEND's flexibility and accuracy envelope [2410.06670, 2511.22696, 2206.02432].

Recent results confirm that self-attention and retention-based architectures, progressive and adversarial training, probability calibration, and multi-task learning (including ASR feature conditioning) all contribute significantly to state-of-the-art diarization performance, robust to long-session, multi-speaker, and overlapping-speech conditions.

Source: https://www.emergentmind.com/topics/end-to-end-neural-diarization-eend