EEND-DEMUX: End-to-End Diarization

• The paper demonstrates a novel end-to-end diarization approach that demultiplexes latent speaker embeddings to achieve up to 24.5% DER reduction in controlled settings.
• EEND-DEMUX systematically integrates a MixtureEncoder, Demultiplexer, AttractorDecoder, and Activity Predictor to perform direct multi-speaker activity prediction without additional VAD or clustering modules.
• Extensive evaluation on Libri2Mix and Libri3Mix benchmarks confirms that the model’s innovative use of multi-head cross-attention and auxiliary losses yields robust performance across fixed and flexible speaker scenarios.

EEND-DEMUX is an end-to-end neural speaker diarization model designed to expose speaker-specific embedding streams directly from audio mixtures, enabling direct multi-speaker activity prediction without any external clustering, voice activity detection (VAD), or auxiliary embedding extractors in inference. The key innovation involves demultiplexing latent speaker information and using a multi-head cross-attention mechanism to associate frame-level latent representations with inferred speaker identities. EEND-DEMUX achieves significant diarization error rate (DER) reductions over previous end-to-end approaches, demonstrating efficacy in both fixed and flexible speaker-count settings (Mun et al., 2023).

1. Model Architecture

EEND-DEMUX systematically processes an input log-Mel feature sequence through four principal modules:

1. MixtureEncoder: Receives input $$\mathbf X = [\mathbf x_1, \ldots, \mathbf x_T] \in \mathbb R^{F \times T}$$ and produces mixture embeddings $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$ via four Transformer encoder blocks (four heads, $D = 256$).
2. Demultiplexer: Transforms $\mathbf E$ via $S$ parallel 2-layer 1D-CNN branches (kernel size 5, Conv $\to$ BatchNorm $\to$ ReLU per layer), producing $$\hat{\mathbf E}_s = [\hat{\mathbf e}_{1,s},\ldots,\hat{\mathbf e}_{T,s}]\ \forall\, s=1,\ldots,S$$, yielding a tensor $$[\hat{\mathbf E}_1, \ldots, \hat{\mathbf E}_S] \in \mathbb R^{D \times T \times S}$$.
3. AttractorDecoder: Computes prototype embeddings by temporal average pooling, $$\hat{\mathbf e}_{\mu,s} = \frac{1}{T}\sum_{t=1}^{T} \hat{\mathbf e}_{t,s}$$, refines speaker separation using a multi-head self-attention (MHSA) block, and finally, via multi-head cross-attention (MHCA) between prototypes and the original mixture ($$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$0 prototypes, $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$1), obtains speaker attractors $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$2.
4. Activity Predictor: For each frame $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$3 and speaker $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$4, produces posteriors $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$5. A speaker-existence probability $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$6 is also predicted; inference discards speakers with $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$7.

The model's data flow can be summarized as: $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$8 MixtureEncoder $$\mathbf E = [\mathbf e_1, \ldots, \mathbf e_T] \in \mathbb R^{D \times T}$$9 $D = 256$0 Demultiplexer $D = 256$1 $D = 256$2 $D = 256$3 AttractorDecoder ($D = 256$4, prototypes) $D = 256$5 $D = 256$6 $D = 256$7 dot-product + sigmoid $D = 256$8 framewise speaker activity $D = 256$9.

2. Demultiplexed Speaker Embeddings

EEND-DEMUX introduces the demultiplexed embedding paradigm, learning to separate, or "demultiplex," speaker-specific factors directly from mixture embeddings. Each of the $\mathbf E$0 parallel CNN branches serves as a routing operator for a candidate speaker, yielding a distinct channel $\mathbf E$1. Unlike previous methods dependent on external speaker clustering or heuristics, EEND-DEMUX learns this separation implicitly.

The formal demultiplexing operation for each speaker $\mathbf E$2 is

$\mathbf E$3

where layer weights and statistics are not shared across speaker channels. Stacking over $\mathbf E$4 yields the global demultiplexed embedding tensor.

This design allows the latent space to explicitly disentangle speaker identity at each frame, facilitating direct per-speaker activity estimation.

3. Multi-Head Cross-Attention Mechanism

The attractor formation subnetwork crucially employs both multi-head self-attention (MHSA) and multi-head cross-attention (MHCA) for robust separation of speaker identities. After MHSA refines the $\mathbf E$5 prototype embeddings in $\mathbf E$6, MHCA uses these as queries ($\mathbf E$7), while the original mixture embeddings act as keys ($\mathbf E$8) and values ($\mathbf E$9): $S$0 Cross-attention operates as

$S$1

aggregating over multiple heads then projecting to $S$2.

This operation bridges the demultiplexed and original mixture spaces, focusing each attractor on the relevant portion of the mixture and sharpening speaker/activity alignment.

4. Loss Functions and Latent Constraints

EEND-DEMUX combines conventional permutation-invariant diarization losses with three auxiliary constraints designed to stabilize and refine the demultiplexed embedding space:

• Diarization Loss: PIT-based binary cross-entropy across all permutations of active speaker heads $S$3:

$S$4

• Existence Loss: Binary cross-entropy between true speaker presence $S$5 and existence predictions $S$6 for all $S$7.
• Matching (Distillation) Loss: $S$8 distance between each $S$9 and an "oracle" single-speaker embedding from a frozen SpeakerEncoder, aligned under PIT.
• Orthogonality Loss: Promotes intra-speaker cohesion and penalizes intra-frame inter-speaker similarity based on cosine similarity metrics, enhancing separability in embedding space.
• Sparsity Loss: $\to$0 penalty to encourage sparse activations in each demultiplexed vector.

Loss weights are empirically tuned ($\to$1, $\to$2-2, $\to$3, $\to$4-3, $\to$5-5).

5. Training Regime, Inference, and Evaluation

Training uses WSJ/WHAM!-derived Libri2Mix (two speakers) and Libri3Mix (three speakers) with batch-wise PIT for label permutation. Optimization utilizes Adam with Noam scheduling, an effective batch size of 128, a peak learning rate of $\to$6, and 30 epochs of warmup.

Inference does not require any VAD, clustering, or extra embedding extraction:

• The demultiplexer and attractor modules operate as in training.
• A speaker head is considered active if $\to$7.
• $\to$8 directly assign “who-when” diarization tracks.

Evaluation employs Diarization Error Rate (DER) as the principal metric: $\to$9 with collar tolerance 0 s to ensure strict alignment.

Experiment	Baseline DER	EEND-DEMUX DER	Relative Reduction
2 speakers (Libri2Mix)	6.13%	3.79%	-24.5%
3 speakers (Libri3Mix)	6.50%	4.91%	-18.2%
Flex scenario (min mode)	4.48%	4.39%	Best so far

Further, ablation confirms that the three demultiplexing losses offer complementary improvements. USED[+spk] is outperformed in both min and max flexible scenarios.

6. Analysis, Advantages, and Limitations

EEND-DEMUX offers several empirical and functional advantages:

• End-to-end inference is free of external clustering, VAD, or embedding extractors, reducing potential system fragility and cascading errors.
• Explicit latent disentanglement via the demultiplexed speaker representations results in tighter, more robust speaker separation, as supported by consistent DER reductions across speaker number conditions.
• The MHCA attractor mechanism further refines alignment between mixture representations and per-speaker activity.

Limitations include the requirement to fix a maximum number of speaker heads $\to$0 in advance, with fully unbounded diarization still unaddressed. Integration into joint ASR-diarization or multi-speaker source separation pipelines, and evaluation on real conversations or low-resource scenarios, represent open research directions.

7. Outlook and Research Directions

EEND-DEMUX's success demonstrates the feasibility of direct end-to-end diarization with latent demultiplexing and cross-attention mechanisms. Future work may focus on:

• Removing the fixed $\to$1-speaker assumption.
• Coupling with multi-speaker ASR or audio separation for unified modeling.
• Extending methodology and evaluation to conversational, telephony, and other real-world datasets, especially those with variable/unknown speaker counts or limited data.

The model’s paradigm of learnable demultiplexed embeddings provides a template for further developments in end-to-end multi-source sequence modeling (Mun et al., 2023).