LSTM-Based D-Vector Systems

• The paper introduces an end-to-end LSTM framework with GE2E loss that generates discriminative d-vectors, achieving significant improvements over traditional i-vector methods.
• The system employs three stacked LSTM layers with a 256-dimensional projection and sliding-window normalization to robustly encode variable-length audio and text inputs.
• The refined spectral clustering pipeline and discriminative training ensure high cluster purity and low diarization error rates across diverse acoustic and textual domains.

LSTM-based d-vector systems refer to architectures and pipelines employing Long Short-Term Memory (LSTM) neural networks to generate robust fixed-dimensional vector representations (d-vectors) of variable-length sequential inputs, typically for speaker diarization or verification. These systems have demonstrated state-of-the-art performance by leveraging end-to-end embedding learning, discriminative loss functions, and advanced clustering procedures to separate speaker (or document) identity from acoustic or lexical variability. The concept has been extensively developed for both audio-based speaker analysis (Wang et al., 2017) and text-based document embeddings (Li et al., 2016).

1. LSTM-Based d-Vector Embedding Architectures

In LSTM-based speaker diarization systems, audio is processed through a series of standardized pre-processing steps. The signal is framed with a 25 ms window and a 10 ms hop length; each frame yields 40-dimensional log-Mel filter-bank energies. A Gaussian Mixture Model-based Voice Activity Detector (VAD) filters out non-speech regions. Speech frames are then grouped into short "segments" of up to 400 ms.

A sliding window of about 240 ms (step 120 ms) traverses the segment. Each window is passed through an architecture of three stacked LSTM layers, each containing 768 memory cells followed by a 256-dimensional projection layer. The last-frame output $h_t$ from the final LSTM is projected via a linear layer to produce a 256-dimensional d-vector. The LSTM cell employs the standard gating structure: $$\begin{aligned} i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) \ f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) \ o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) \ g_t &= \tanh(W_c x_t + U_c h_{t-1} + b_c) \ c_t &= f_t \odot c_{t-1} + i_t \odot g_t \ h_t &= o_t \odot \tanh(c_t) \ \end{aligned}$$ Each window embedding is $\ell_2$-normalized; then, embeddings within the same short segment are averaged and re-normalized to yield a single, fixed-length segment embedding: $$\hat{e} = \frac{1}{\|\frac{1}{K} \sum_{k=1}^K \tilde{e}_k\|_2}$$ where $\tilde{e}_k$ is the normalized per-window embedding.

This architecture exploits temporal modeling and projection to low-dimensional, discriminative d-vectors, allowing for accurate representation of speaker identity while suppressing irrelevant information (Wang et al., 2017).

2. Discriminative Training Objectives

The LSTM is trained in a text-independent speaker verification configuration using the Generalized End-to-End (GE2E) loss. Training batches sample $S$ speakers, $M$ utterances per speaker; each embedding $e_{i,j}$ (speaker $i$, utterance $j$) is compared to centroids

$c_i = \frac{1}{M}\sum_{j=1}^M e_{i,j}$

forming similarities

$s_{i,j,k} = w \cdot \cos(e_{i,j}, c_k) + b$

with $w$, $b$ learned. The GE2E loss

$$\mathcal{L} = -\sum_{i=1}^S\sum_{j=1}^M \log\frac{\exp(s_{i,j,i})}{\sum_{k=1}^S \exp(s_{i,j,k})}$$

encourages embeddings to cluster with their own centroid and separate from others. No margin or additional regularization is required, though L2 weight decay is an optional variant.

This structure produces embeddings inherently suited for speaker discrimination in downstream clustering and diarization tasks (Wang et al., 2017).

3. Clustering and Diarization Procedures

For diarization, segment embeddings are grouped according to speaker identity using clustering. Cosine similarity is employed: $\cos(x, y) = \frac{x^\top y}{\|x\| \|y\|}$ and cosine distance

$d(x, y) = \frac{1 - \cos(x, y)}{2}$

The leading approach is an offline spectral clustering pipeline:

• Build the raw affinity matrix $A_{ij}$ (cosine similarity, diagonal set to off-diagonal max).
• Refine $A$ with Gaussian blurring, row-wise thresholding, symmetrization ($X \gets \max(X, X^\top)$), matrix diffusion ($Y \gets X X^\top$), and row-wise max normalization.
• Eigen-decompose the refined affinity to obtain eigenvalues $\lambda_1 \geq \lambda_2 \geq \cdots$. Select the number of clusters $\hat{k}$ by the largest eigengap: $$\hat{k} = \arg\max_{1 \leq i < N} \frac{\lambda_i}{\lambda_{i+1}}$$
• Take rows of $V \in \mathbb{R}^{N \times k}$ (top-$k$ eigenvectors) and cluster with K-means.

A naïve online clustering is also described, maintaining speaker centroids and assigning new embeddings by thresholded cosine similarity.

The spectral clustering procedure delivers superior cluster purity and diarization error rates compared to K-means or naïve online methods, benefiting from affinity denoising and temporal smoothing (Wang et al., 2017).

4. Experimental Setup and Results

LSTM-based d-vector diarization systems are trained with approximately 36 million English voice-search utterances from about 18,000 speakers, providing extensive out-of-domain coverage. Evaluation is performed on standard public datasets:

• CALLHOME American English (LDC97S42+LDC97T14)
• 2003 NIST RT-03 English CTS (72 calls)
• 2000 NIST SRE CALLHOME multi-language (500 calls)

Performance is quantified by the Diarization Error Rate (DER): $$\mathrm{DER} = \frac{T_\mathrm{FA} + T_\mathrm{Miss} + T_\mathrm{Conf}}{T_\mathrm{Total}} \times 100\%$$ Key results for offline spectral clustering:

• CALLHOME American English: i-vector DER $\approx$ 20.5%, d-vector DER $\approx$ 12.5%
• RT-03: i-vector DER $\approx$ 21.1%, d-vector DER $\approx$ 12.3%
• NIST SRE 2000 CALLHOME: LSTM d-vectors achieve 12.0% DER (no in-domain data or VB resegmentation), outperforming i-vector baselines (13–14%).

These figures demonstrate substantial improvements over traditional i-vector-based pipelines in diverse real-world conversational conditions (Wang et al., 2017).

5. Architectural and Methodological Innovations

LSTM-based d-vector systems introduced several key innovations:

• The first demonstration that text-independent LSTM-based d-vectors, optimized with GE2E loss, can be directly substituted into diarization pipelines, surpassing i-vector performance without resegmentation or adaptation.
• A multi-stage spectral clustering process (Gaussian blur, thresholding, symmetrization, diffusion, normalization) that leverages temporal locality to denoise similarity matrices, enhancing cluster consistency and speaker purity.
• Strong generalization across linguistic and acoustic domains, attributed to robust GMM-VAD preprocessing and end-to-end embedding training, despite training exclusively on out-of-domain English data.
• A unified, coherent pipeline combining efficient sliding-window LSTM embedding extraction, discriminative training, refined spectral clustering, and automatic cluster number selection (eigengap), supporting low-latency, accurate diarization (Wang et al., 2017).

6. Connections and Extensions

The d-vector concept is adaptable beyond speech. In text, "DV-LSTM" approaches recast document representation using adapted LSTM-LM parameters. After parent LSTM-LM training, only gate and output biases are adapted per document; all adapted biases are $\ell_2$-normalized and concatenated into a fixed-length document vector. Empirical evaluation for text genre classification (PTB-4, Brown, BNC-Baby) shows DV-LSTM yields weighted F$_1$-scores superior to TF-IDF and Paragraph Vector alternatives in most cases (e.g., PTB-4: 0.8434 for DV-LSTM vs. 0.7996 TF-IDF-5gram, 0.8154 PV-DM; BNC-Baby: 1.0000 for DV-LSTM) (Li et al., 2016).

This suggests that LSTM-based d-vector architectures are effective in both continuous (audio) and discrete (text) domains, robustly encoding sequential and discriminative structure for downstream clustering or classification.