---
title: 'EEND-DEMUX: End-to-End Diarization'
url: https://www.emergentmind.com/topics/eend-demux
type: topic
---

# EEND-DEMUX: End-to-End Diarization

EEND-DEMUX is an end-to-end neural speaker diarization model designed to expose speaker-specific embedding streams directly from audio mixtures, enabling direct multi-speaker activity prediction without any external clustering, voice activity detection (VAD), or auxiliary embedding extractors in inference. The key innovation involves demultiplexing latent speaker information and using a multi-head cross-attention mechanism to associate frame-level latent representations with inferred speaker identities. EEND-DEMUX achieves significant diarization error rate (DER) reductions over previous end-to-end approaches, demonstrating efficacy in both fixed and flexible speaker-count settings [2312.06065].

## 1. Model Architecture

EEND-DEMUX systematically processes an input log-Mel feature sequence through four principal modules:

1. **MixtureEncoder**: Receives input $\mathbf X = [\mathbf x_1, \ldots, \mathbf x_T] \in \mathbb R^{F \times T}$ and produces mixture embeddings $\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$ via four Transformer encoder blocks (four heads, $D = 256$).
2. **Demultiplexer**: Transforms $\mathbf E$ via $S$ parallel 2-layer 1D-CNN branches (kernel size 5, Conv $\to$ BatchNorm $\to$ ReLU per layer), producing $\hat{\mathbf E}_s = [\hat{\mathbf e}_{1,s},\ldots,\hat{\mathbf e}_{T,s}]\ \forall\, s=1,\ldots,S$, yielding a tensor $[\hat{\mathbf E}_1, \ldots, \hat{\mathbf E}_S] \in \mathbb R^{D \times T \times S}$.
3. **AttractorDecoder**: Computes prototype embeddings by temporal average pooling, $\hat{\mathbf e}_{\mu,s} = \frac{1}{T}\sum_{t=1}^{T} \hat{\mathbf e}_{t,s}$, refines speaker separation using a multi-head self-attention (MHSA) block, and finally, via multi-head cross-attention (MHCA) between prototypes and the original mixture ($Q \leftarrow$ prototypes, $K,V \leftarrow \mathbf E$), obtains speaker attractors $\mathbf A = [\mathbf a_1, \ldots, \mathbf a_S] \in \mathbb R^{D \times S}$.
4. **Activity Predictor**: For each frame $t$ and speaker $s$, produces posteriors $\hat y_{t,s} = \mathrm{sigmoid}(\hat{\mathbf e}_{t,s}^T \mathbf a_s)$. A speaker-existence probability $\hat p_s$ is also predicted; inference discards speakers with $\hat p_s < 0.5$.

The model's data flow can be summarized as:
$\mathbf X \rightarrow$ MixtureEncoder $\rightarrow$ $\mathbf E \rightarrow$ Demultiplexer $\rightarrow$ $\{\,\hat{\mathbf E}_s\}$ $\rightarrow$ AttractorDecoder ($\mathbf E$, prototypes) $\rightarrow$ $\{\mathbf a_s\}$ $\rightarrow$ dot-product + sigmoid $\rightarrow$ framewise speaker activity $\hat y_{t,s}$.

## 2. Demultiplexed Speaker Embeddings

EEND-DEMUX introduces the demultiplexed embedding paradigm, learning to separate, or "demultiplex," speaker-specific factors directly from mixture embeddings. Each of the $S$ parallel CNN branches serves as a routing operator for a candidate speaker, yielding a distinct channel $\hat{\mathbf e}_{t,s}$. Unlike previous methods dependent on external speaker clustering or heuristics, EEND-DEMUX learns this separation implicitly.

The formal demultiplexing operation for each speaker $s$ is
\[
\hat{\mathbf E}_s
= \mathrm{Conv}_2^{(s)}\bigl(\mathrm{ReLU}(\mathrm{BN}(\mathrm{Conv}_1^{(s)}(\mathbf E)))\bigr),\qquad \hat{\mathbf E}_s \in \mathbb R^{D \times T},
\]
where layer weights and statistics are not shared across speaker channels. Stacking over $s = 1, \ldots, S$ yields the global demultiplexed embedding tensor.

This design allows the latent space to explicitly disentangle speaker identity at each frame, facilitating direct per-speaker activity estimation.

## 3. Multi-Head Cross-Attention Mechanism

The attractor formation subnetwork crucially employs both multi-head self-attention (MHSA) and multi-head cross-attention (MHCA) for robust separation of speaker identities. After MHSA refines the $S$ prototype embeddings in $\mathbb R^{D \times S}$, MHCA uses these as queries ($Q$), while the original mixture embeddings act as keys ($K$) and values ($V$):
\[
Q = W^Q \hat{\mathbf E}_{\mu},\qquad K = W^K \mathbf E,\qquad V = W^V \mathbf E.
\]
Cross-attention operates as
\[
\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T,
\]
aggregating over multiple heads then projecting to $\mathbb R^{D}$.

This operation bridges the demultiplexed and original mixture spaces, focusing each attractor on the relevant portion of the mixture and sharpening speaker/activity alignment.

## 4. Loss Functions and Latent Constraints

EEND-DEMUX combines conventional permutation-invariant diarization losses with three auxiliary constraints designed to stabilize and refine the demultiplexed embedding space:

- **Diarization Loss**: PIT-based binary cross-entropy across all permutations of active speaker heads $\mathcal S$:
  \[
  \mathcal L_{\mathrm{diar}}
  = \frac{1}{T|\mathcal S|} \min_{\phi \in \Phi(\mathcal S)} \sum_{t=1}^{T} \sum_{s \in \mathcal S} -y_{t,s}^{\phi} \log \hat y_{t,s} - (1-y_{t,s}^{\phi}) \log (1 - \hat y_{t,s}).
  \]
- **Existence Loss**: Binary cross-entropy between true speaker presence $p_s$ and existence predictions $\hat p_s$ for all $s$.
- **Matching (Distillation) Loss**: $L_2$ distance between each $\hat{\mathbf e}_{t,s}$ and an "oracle" single-speaker embedding from a frozen SpeakerEncoder, aligned under PIT.
- **Orthogonality Loss**: Promotes intra-speaker cohesion and penalizes intra-frame inter-speaker similarity based on cosine similarity metrics, enhancing separability in embedding space.
- **Sparsity Loss**: $L_1$ penalty to encourage sparse activations in each demultiplexed vector.

Loss weights are empirically tuned ($\lambda_{\text{diar}} = 1$, $\lambda_{\text{ext}} = 1e$-2, $\lambda_\text{dis} = 2.5$, $\lambda_\text{ort} = 1e$-3, $\lambda_\text{spa} = 1e$-5).

## 5. Training Regime, Inference, and Evaluation

**Training** uses WSJ/WHAM!-derived Libri2Mix (two speakers) and Libri3Mix (three speakers) with batch-wise PIT for label permutation. Optimization utilizes Adam with Noam scheduling, an effective batch size of 128, a peak learning rate of $5 \times 10^{-4}$, and 30 epochs of warmup.

**Inference** does not require any VAD, clustering, or extra embedding extraction: 
- The demultiplexer and attractor modules operate as in training.
- A speaker head is considered active if $\hat p_s \geq 0.5$.
- $\hat y_{t,s}$ directly assign “who-when” diarization tracks.

**Evaluation** employs Diarization Error Rate (DER) as the principal metric:
\[
\mathrm{DER} = \text{FA} + \text{Miss} + \text{Confusion}
\]
with collar tolerance 0 s to ensure strict alignment.

| Experiment                 | Baseline DER | EEND-DEMUX DER | Relative Reduction |
|----------------------------|:------------:|:--------------:|:-----------------:|
| 2 speakers (Libri2Mix)     |    6.13%     |    3.79%       |      -24.5%       |
| 3 speakers (Libri3Mix)     |    6.50%     |    4.91%       |      -18.2%       |
| Flex scenario (min mode)   |    4.48%     |    4.39%       |      Best so far  |

Further, ablation confirms that the three demultiplexing losses offer complementary improvements. USED[+spk] is outperformed in both min and max flexible scenarios.

## 6. Analysis, Advantages, and Limitations

EEND-DEMUX offers several empirical and functional advantages:
- End-to-end inference is free of external clustering, VAD, or embedding extractors, reducing potential system fragility and cascading errors.
- Explicit latent disentanglement via the demultiplexed speaker representations results in tighter, more robust speaker separation, as supported by consistent DER reductions across speaker number conditions.
- The MHCA attractor mechanism further refines alignment between mixture representations and per-speaker activity.

Limitations include the requirement to fix a maximum number of speaker heads $S$ in advance, with fully unbounded diarization still unaddressed. Integration into joint ASR-diarization or multi-speaker source separation pipelines, and evaluation on real conversations or low-resource scenarios, represent open research directions.

## 7. Outlook and Research Directions

EEND-DEMUX's success demonstrates the feasibility of direct end-to-end diarization with latent demultiplexing and cross-attention mechanisms. Future work may focus on:
- Removing the fixed $S$-speaker assumption.
- Coupling with multi-speaker ASR or audio separation for unified modeling.
- Extending methodology and evaluation to conversational, telephony, and other real-world datasets, especially those with variable/unknown speaker counts or limited data.

The model’s paradigm of learnable demultiplexed embeddings provides a template for further developments in end-to-end multi-source sequence modeling [2312.06065].

Source: https://www.emergentmind.com/topics/eend-demux