EEND-VC: Neural Embedding & Constrained Clustering

• EEND-VC is a diarization paradigm that segments long multi-speaker audio into manageable blocks, extracts speaker-specific embeddings, and links local outputs globally using constrained clustering.
• This method employs a neural front-end with transformer-based feature extraction and weighted embedding pooling, enforcing unique speaker assignments with blockwise cannot-link constraints.
• EEND-VC demonstrates state-of-the-art performance in overlapping speech scenarios, offering scalable, efficient diarization with improved speaker tracking and lower error rates.

Blockwise neural embedding with constrained clustering, widely known as End-to-End Neural Diarization with Vector Clustering (EEND-VC), is a diarization paradigm that unifies the robustness and interpretability of vector clustering with the overlap-handling and global permutation invariance of the EEND framework. EEND-VC processes long, multi-speaker audio by segmenting it into blocks, extracting speaker-specific embeddings and local diarization outputs per block, and linking these local decisions into global speaker identities using constrained clustering algorithms that enforce blockwise cannot-link constraints. This solves the label permutation issue inherent in blockwise EEND and allows accurate speaker tracking in sessions with arbitrary numbers of speakers and variable overlap.

1. Model Principles and Motivation

Traditional clustering-based diarization (e.g., x-vector + PLDA + agglomerative clustering) robustly handles arbitrarily long recordings and arbitrary speaker counts but lacks overlap resolution. In contrast, end-to-end neural diarization (EEND) directly models multitarget speech-activity but suffers from label permutation ambiguities between blocks, memory scaling issues on long content, and limited speaker counts due to fixed output nodes. EEND-vector clustering integrates these paradigms by equipping local EEND blocks with embedding heads trained for global cluster-friendliness, then employing a global clustering backend to assign block-local outputs to consistent global identities (Kinoshita et al., 2020, Kinoshita et al., 2021, Serafini et al., 2023).

This hybrid structure enables (1) end-to-end handling of overlaps and speaker transitions, (2) blockwise processing for tractability and scalable label alignment, and (3) handling of sessions with more global speakers than local EEND output slots by leveraging the clustering's ability to merge slots representing the same speaker across blocks.

2. Neural Front-End and Blockwise Embedding Extraction

EEND-VC commences by partitioning a long audio input into non-overlapping or partially overlapping blocks of $T$ frames (typ. 15–80 s) (Kinoshita et al., 2021, Tawara et al., 2023). Each block is transformed into a feature tensor $\mathbf{X}_i \in \mathbb{R}^{T \times F}$. A multi-head self-attention network (Transformer/Conformer) with $L$ layers and $H$ heads, often preceded by a pretrained self-supervised model (e.g., WavLM), ingests $\mathbf{X}_i$ and outputs:

• Frame-wise speaker-activity posteriors: $\widehat{\mathbf{y}}_{t,i,s}$, $s=1\dots S_\text{Local}$, via sigmoid-activated linear heads on hidden states.
• Speaker embeddings per local output: For each slot $s$, a frame-level embedding $\mathbf{z}_{t,i,s}$ is produced and pooled across frames with weights proportional to $\widehat{y}_{t,i,s}$. The pooled embedding for block $\mathbf{X}_i \in \mathbb{R}^{T \times F}$0, slot $\mathbf{X}_i \in \mathbb{R}^{T \times F}$1 is $\mathbf{X}_i \in \mathbb{R}^{T \times F}$2 (Kinoshita et al., 2020, Kinoshita et al., 2021, Serafini et al., 2023).

The network is trained end-to-end with a weighted sum of PIT-based binary cross-entropy diarization loss and an embedding discriminability loss (contrastive, triplet, or cross-entropy versus a dictionary), often with permutation-invariant alignment to ground-truth (Kinoshita et al., 2020).

3. Constrained Clustering Formulation

After blockwise embedding extraction, EEND-VC collects all embeddings $\mathbf{X}_i \in \mathbb{R}^{T \times F}$3 along with associated local output indices. The critical constraint is that any two embeddings from the same block must not be assigned to the same global speaker ("cannot-link") (Kinoshita et al., 2021, Serafini et al., 2023, Tawara et al., 2023). The clustering objective is:

$\mathbf{X}_i \in \mathbb{R}^{T \times F}$4

Two principal algorithmic classes are used:

• Constrained Agglomerative Hierarchical Clustering (cAHC): Distance between embedding pairs is set to a large value ($\mathbf{X}_i \in \mathbb{R}^{T \times F}$5) for within-block pairs, preventing mergers that violate cannot-link. Standard average linkage is used elsewhere. The process stops at the number of global speakers (Kinoshita et al., 2021, Delcroix et al., 2023).
• Constrained K-means (COP-Kmeans): Assignment and centroid update steps are alternated, with assignments forbidden if they violate cannot-link (Kinoshita et al., 2021, Tawara et al., 2023).

Variants including constrained spectral clustering (zeroing forbidden similarity edges) are also employed (Kinoshita et al., 2021).

4. Enhanced Bayesian Clustering and Spherical Models

Recent research has advanced the backends for the clustering stage. The classical pipeline uses Gaussian PLDA-based clustering with variational Bayesian inference (VBx). VBx models the embedding distribution per speaker and infers mixture assignments and cluster means iteratively (Pálka et al., 22 Oct 2025). Extensions adapted to EEND-VC include:

• Multi-stream VBx (MS-VBx): Each block's set of $\mathbf{X}_i \in \mathbb{R}^{T \times F}$6 embeddings (for $\mathbf{X}_i \in \mathbb{R}^{T \times F}$7 local streams) is jointly modeled in a factorial HMM, where each state corresponds to unique global speaker assignments per stream that automatically enforce cannot-link (Delcroix et al., 2023). This allows temporal continuity and avoids ad-hoc assignment/fixing per block, outperforming cAHC on both DER and speaker count estimation.
• SphereVBx and T-PSDA: To better match the angular geometry of modern embeddings, SphereVBx replaces Gaussian PLDA with a mixture of von Mises-Fisher distributions (T-PSDA) (Pálka et al., 23 Jun 2026). Variational inference in this space aligns naturally with cosine similarities, and a parameter-free variant reduces to normalized cosine clustering. The cannot-link constraint is enforced either by post-hoc assignment (Hungarian algorithm) or by full permutation marginalization per block, yielding close to optimal results for up to four simultaneous local streams.

5. Training and Inference Workflow

Training involves simulation of multi-speaker mixtures and joint optimization of local diarization and cluster-friendly embedding objectives. Hyperparameters are set for block duration $\mathbf{X}_i \in \mathbb{R}^{T \times F}$8 (typically 15–50 s), speaker embedding dimension (128–512), and silent thresholds for embedding exclusion (Kinoshita et al., 2021, Serafini et al., 2023).

Inference proceeds as:

1. Segment the audio, extract blockwise posteriors and embeddings, remove silent/low-activity embeddings.
2. Aggregate embeddings and cannot-link constraints.
3. Perform constrained clustering (cAHC, COP-Kmeans, MS-VBx, or SphereVBx), yielding global speaker assignments.
4. Re-label and stitch local diarization output rows according to cluster assignment, forming a global diarization matrix.
5. Optionally, refine results with self-supervised adaptation or external fusion (e.g., DOVER-LAP for multi-channel conditions) (Tawara et al., 2023).

6. Comparative Performance and System Ablations

EEND-VC consistently yields lower DER than both vanilla EEND and traditional clustering methods across multiple domains:

• On CALLHOME, constrained AHC achieves DER of 12.22% (oracle #speakers), outperforming x-vector clustering (18.98%) and EDA-EEND (15.43%) (Kinoshita et al., 2021).
• VBx integration further improves performance: on an eight-corpus benchmark, DER drops from 14.5% (pyannote AHC) to 13.0% (cVBx with filtering) (Pálka et al., 22 Oct 2025).
• MS-VBx and SphereVBx provide improvements of 1–2% absolute DER and halve speaker counting error compared to memoryless clustering (Delcroix et al., 2023, Pálka et al., 23 Jun 2026).
• The block size $\mathbf{X}_i \in \mathbb{R}^{T \times F}$9 provides a tradeoff: longer blocks yield better embedding statistics but risk intra-block permutation errors; optimal $L$0–$L$1 s (Kinoshita et al., 2021, Tawara et al., 2023).

Ablation studies reveal that constrained clustering is critical in high-speaker-count and high-overlap settings. Filtering short-duration embeddings prior to clustering reduces spurious clusters ("ghost speakers") (Pálka et al., 22 Oct 2025).

7. Practical Considerations and Limitations

EEND-VC is computationally efficient (RTF $L$2) and scales well with session length due to its blockwise structure (Serafini et al., 2023). Its limitations include:

• Data requirements: High-quality cluster-friendly embeddings require large labeled datasets; performance degrades more rapidly with training size reduction than self-attentive EEND (Serafini et al., 2023).
• Complexity of constrained clustering: COP-Kmeans and MS-VBx incur additional computational costs relative to unconstrained clustering but remain tractable due to blockwise granularity and small per-block output cardinality.
• Parameter-free clustering: SphereVBx-PF and direct cosine approaches eliminate the need for a PLDA backend, simplifying deployment at some cost in parameter tuning flexibility (Pálka et al., 23 Jun 2026).

References Table

Algorithmic Step	Key Techniques / Algorithms	References
Embedding Extraction	Blockwise Transformer/Conformer, weighted pooling	(Kinoshita et al., 2020, Kinoshita et al., 2021, Serafini et al., 2023, Tawara et al., 2023)
Constrained Clustering	cAHC, COP-Kmeans, cSC, one-to-one assignment	(Kinoshita et al., 2021, Serafini et al., 2023, Tawara et al., 2023)
PLDA/VBx Backend	Variational Bayes, Mahalanobis/cosine scoring	(Pálka et al., 22 Oct 2025, Delcroix et al., 2023)
Spherical/Angular Models	von Mises-Fisher mixture, SphereVBx	(Pálka et al., 23 Jun 2026)
Multi-Stream Clustering	MS-VBx, joint chunk assignment	(Delcroix et al., 2023, Pálka et al., 23 Jun 2026)
Self-Supervised Adapt.	DOVER-LAP, pseudo-label fine-tuning	(Tawara et al., 2023)

EEND-vector clustering realizes state-of-the-art diarization, particularly for long-form, multi-speaker, and highly overlapping conversational scenarios, by structurally combining neural overlap-resolution with global identity consistency through mathematically principled constrained clustering mechanisms (Kinoshita et al., 2021, Serafini et al., 2023, Tawara et al., 2023, Pálka et al., 22 Oct 2025, Pálka et al., 23 Jun 2026).