---
title: 'EEND-VC: Neural Embedding & Constrained Clustering'
url: https://www.emergentmind.com/topics/blockwise-neural-embedding-constrained-clustering-eend-vector-clustering
type: topic
---

# EEND-VC: Neural Embedding & Constrained Clustering

Blockwise neural embedding with constrained clustering, widely known as End-to-End Neural Diarization with Vector Clustering (EEND-VC), is a diarization paradigm that unifies the robustness and interpretability of vector clustering with the overlap-handling and global permutation invariance of the EEND framework. EEND-VC processes long, multi-speaker audio by segmenting it into blocks, extracting speaker-specific embeddings and local diarization outputs per block, and linking these local decisions into global speaker identities using constrained clustering algorithms that enforce blockwise cannot-link constraints. This solves the label permutation issue inherent in blockwise EEND and allows accurate speaker tracking in sessions with arbitrary numbers of speakers and variable overlap.

## 1. Model Principles and Motivation

Traditional clustering-based diarization (e.g., x-vector + PLDA + agglomerative clustering) robustly handles arbitrarily long recordings and arbitrary speaker counts but lacks overlap resolution. In contrast, end-to-end neural diarization (EEND) directly models multitarget speech-activity but suffers from label permutation ambiguities between blocks, memory scaling issues on long content, and limited speaker counts due to fixed output nodes. EEND-vector clustering integrates these paradigms by equipping local EEND blocks with embedding heads trained for global cluster-friendliness, then employing a global clustering backend to assign block-local outputs to consistent global identities [2010.13366][2105.09040][2305.18074].

This hybrid structure enables (1) end-to-end handling of overlaps and speaker transitions, (2) blockwise processing for tractability and scalable label alignment, and (3) handling of sessions with more global speakers than local EEND output slots by leveraging the clustering's ability to merge slots representing the same speaker across blocks.

## 2. Neural Front-End and Blockwise Embedding Extraction

EEND-VC commences by partitioning a long audio input into non-overlapping or partially overlapping blocks of $T$ frames (typ. 15–80 s) [2105.09040][2309.12656]. Each block is transformed into a feature tensor $\mathbf{X}_i \in \mathbb{R}^{T \times F}$. A multi-head self-attention network (Transformer/Conformer) with $L$ layers and $H$ heads, often preceded by a pretrained self-supervised model (e.g., WavLM), ingests $\mathbf{X}_i$ and outputs:

- **Frame-wise speaker-activity posteriors:** $\widehat{\mathbf{y}}_{t,i,s}$, $s=1\dots S_\text{Local}$, via sigmoid-activated linear heads on hidden states.
- **Speaker embeddings per local output:** For each slot $s$, a frame-level embedding $\mathbf{z}_{t,i,s}$ is produced and pooled across frames with weights proportional to $\widehat{y}_{t,i,s}$. The pooled embedding for block $i$, slot $s$ is $\hat{\mathbf{e}}_{i,s} = \frac{\sum_{t=1}^T \widehat{y}_{t,i,s}\mathbf{z}_{t,i,s}}{\|\sum_{t=1}^T \widehat{y}_{t,i,s}\mathbf{z}_{t,i,s}\|}$ [2010.13366][2105.09040][2305.18074].

The network is trained end-to-end with a weighted sum of PIT-based binary cross-entropy diarization loss and an embedding discriminability loss (contrastive, triplet, or cross-entropy versus a dictionary), often with permutation-invariant alignment to ground-truth [2010.13366].

## 3. Constrained Clustering Formulation

After blockwise embedding extraction, EEND-VC collects all embeddings $\mathcal{E} = \{\hat{\mathbf{e}}_{i,s}\}$ along with associated local output indices. The critical constraint is that any two embeddings from the same block must not be assigned to the same global speaker ("cannot-link") [2105.09040][2305.18074][2309.12656]. The clustering objective is:

\[
\min_{\{C_k\},~\{\mu_k\}} \sum_{k=1}^{S_\text{Global}} \sum_{\mathbf{e} \in C_k} \|\mathbf{e} - \mu_k\|^2 
\qquad \text{s.t.}~
\forall i,~s \neq s':~\hat{\mathbf{e}}_{i,s},\hat{\mathbf{e}}_{i,s'} \notin C_k~\forall k
\]

Two principal algorithmic classes are used:

- **Constrained Agglomerative Hierarchical Clustering (cAHC):** Distance between embedding pairs is set to a large value ($\kappa \gg 0$) for within-block pairs, preventing mergers that violate cannot-link. Standard average linkage is used elsewhere. The process stops at the number of global speakers [2105.09040][2305.13580].
  
- **Constrained K-means (COP-Kmeans):** Assignment and centroid update steps are alternated, with assignments forbidden if they violate cannot-link [2105.09040][2309.12656].
  
Variants including constrained spectral clustering (zeroing forbidden similarity edges) are also employed [2105.09040].

## 4. Enhanced Bayesian Clustering and Spherical Models

Recent research has advanced the backends for the clustering stage. The classical pipeline uses Gaussian PLDA-based clustering with variational Bayesian inference (VBx). VBx models the embedding distribution per speaker and infers mixture assignments and cluster means iteratively [2510.19572]. Extensions adapted to EEND-VC include:

- **Multi-stream VBx (MS-VBx):** Each block's set of $C$ embeddings (for $C$ local streams) is jointly modeled in a factorial HMM, where each state corresponds to unique global speaker assignments per stream that automatically enforce cannot-link [2305.13580]. This allows temporal continuity and avoids ad-hoc assignment/fixing per block, outperforming cAHC on both DER and speaker count estimation.
  
- **SphereVBx and T-PSDA:** To better match the angular geometry of modern embeddings, SphereVBx replaces Gaussian PLDA with a mixture of von Mises-Fisher distributions (T-PSDA) [2606.24528]. Variational inference in this space aligns naturally with cosine similarities, and a parameter-free variant reduces to normalized cosine clustering. The cannot-link constraint is enforced either by post-hoc assignment (Hungarian algorithm) or by full permutation marginalization per block, yielding close to optimal results for up to four simultaneous local streams.

## 5. Training and Inference Workflow

**Training** involves simulation of multi-speaker mixtures and joint optimization of local diarization and cluster-friendly embedding objectives. Hyperparameters are set for block duration $T$ (typically 15–50 s), speaker embedding dimension (128–512), and silent thresholds for embedding exclusion [2105.09040][2305.18074].

**Inference** proceeds as:

1. Segment the audio, extract blockwise posteriors and embeddings, remove silent/low-activity embeddings.
2. Aggregate embeddings and cannot-link constraints.
3. Perform constrained clustering (cAHC, COP-Kmeans, MS-VBx, or SphereVBx), yielding global speaker assignments.
4. Re-label and stitch local diarization output rows according to cluster assignment, forming a global diarization matrix.
5. Optionally, refine results with self-supervised adaptation or external fusion (e.g., DOVER-LAP for multi-channel conditions) [2309.12656].

## 6. Comparative Performance and System Ablations

EEND-VC consistently yields lower DER than both vanilla EEND and traditional clustering methods across multiple domains:

- On CALLHOME, constrained AHC achieves DER of 12.22% (oracle #speakers), outperforming x-vector clustering (18.98%) and EDA-EEND (15.43%) [2105.09040].
- VBx integration further improves performance: on an eight-corpus benchmark, DER drops from 14.5% (pyannote AHC) to 13.0% (cVBx with filtering) [2510.19572].
- MS-VBx and SphereVBx provide improvements of 1–2% absolute DER and halve speaker counting error compared to memoryless clustering [2305.13580][2606.24528].
- The block size $T$ provides a tradeoff: longer blocks yield better embedding statistics but risk intra-block permutation errors; optimal $T \approx 30$–$80$ s [2105.09040][2309.12656].

**Ablation studies** reveal that constrained clustering is critical in high-speaker-count and high-overlap settings. Filtering short-duration embeddings prior to clustering reduces spurious clusters ("ghost speakers") [2510.19572].

## 7. Practical Considerations and Limitations

EEND-VC is computationally efficient (RTF $\sim 10^{-3}$) and scales well with session length due to its blockwise structure [2305.18074]. Its limitations include:

- **Data requirements:** High-quality cluster-friendly embeddings require large labeled datasets; performance degrades more rapidly with training size reduction than self-attentive EEND [2305.18074].
- **Complexity of constrained clustering:** COP-Kmeans and MS-VBx incur additional computational costs relative to unconstrained clustering but remain tractable due to blockwise granularity and small per-block output cardinality.
- **Parameter-free clustering:** SphereVBx-PF and direct cosine approaches eliminate the need for a PLDA backend, simplifying deployment at some cost in parameter tuning flexibility [2606.24528].

## References Table

| Algorithmic Step         | Key Techniques / Algorithms                 | References           |
|-------------------------|---------------------------------------------|----------------------|
| Embedding Extraction    | Blockwise Transformer/Conformer, weighted pooling | [2010.13366][2105.09040][2305.18074][2309.12656] |
| Constrained Clustering  | cAHC, COP-Kmeans, cSC, one-to-one assignment| [2105.09040][2305.18074][2309.12656]   |
| PLDA/VBx Backend        | Variational Bayes, Mahalanobis/cosine scoring | [2510.19572][2305.13580]                 |
| Spherical/Angular Models| von Mises-Fisher mixture, SphereVBx         | [2606.24528]                           |
| Multi-Stream Clustering | MS-VBx, joint chunk assignment              | [2305.13580][2606.24528]                |
| Self-Supervised Adapt.  | DOVER-LAP, pseudo-label fine-tuning         | [2309.12656]                            |

EEND-vector clustering realizes state-of-the-art diarization, particularly for long-form, multi-speaker, and highly overlapping conversational scenarios, by structurally combining neural overlap-resolution with global identity consistency through mathematically principled constrained clustering mechanisms [2105.09040][2305.18074][2309.12656][2510.19572][2606.24528].

Source: https://www.emergentmind.com/topics/blockwise-neural-embedding-constrained-clustering-eend-vector-clustering