Speaker Embedding Interpolation

• Speaker Embedding Interpolation is a method for creating intermediate speaker representations by blending fixed vectors with context-driven adjustments.
• It leverages techniques including linear mixing, spherical interpolation, optimal-transport barycenters, and parameter merging to preserve perceptual similarity in overlapping speech.
• These approaches improve robustness in scenarios with incomplete or ambiguous speaker evidence, enhancing performance in verification, synthesis, and clustering tasks.

Speaker embedding interpolation denotes a set of methods that construct intermediate, context-conditioned, or distributionally blended speaker representations rather than treating every utterance or speaker as a single immutable point in embedding space. In the literature, this notion appears as linear mixing of speaker vectors, geodesic interpolation on a hypersphere, optimal-transport barycenters of Gaussian mixture models over speaker embeddings, parameter-space interpolation of entire text-to-speech models, and context-dependent “floating” embeddings whose state is fixed only after the comparison context is specified (Zheng et al., 2022, Cord-Landwehr et al., 2024, Murata et al., 2024, Watanabe et al., 2022, Liu et al., 26 Aug 2025).

1. Motivations and problem setting

A central motivation for interpolation-based methods is the inadequacy of fixed speaker vectors in conditions where speaker evidence is incomplete, ambiguous, or confounded. PRISM states that traditional speaker embeddings are represented as fixed vectors in high-dimensional space and that this could lead to biased estimations, especially when handling shorter utterances; it further attributes this to non-speaker properties such as content (Zheng et al., 2022). This directly motivates representations whose state depends on context rather than being finalized in isolation.

A second motivation is perceptual mismatch. Conventional DNN-based speaker embeddings such as a $d$-vector do not necessarily correlate with subjective inter-speaker similarity and are not necessarily appropriate speaker representations for open speakers whose speech utterances are not included in the training data (Saito et al., 2019). In this setting, interpolation is not merely a geometric convenience; it is expected to preserve perceived similarity structure if intermediate voices are to sound meaningful.

A third motivation arises in diarization and overlap handling. Segment-level clustering-based systems such as VBx and Spectral Clustering have difficulty in handling overlap, and the geodesic-interpolation literature argues that regions with two active speakers should occupy intermediate locations between the corresponding single-speaker embeddings on a sphere (Cord-Landwehr et al., 2024). In synthesis, attribute interpolation is a common approach to speaker generation and emotion intensity control, yet many prior methods require specific modules or training methods; this has motivated alternatives that operate by merging already trained models (Murata et al., 2024).

2. Interpolation geometries and representational forms

The most direct formulation is linear interpolation of speaker embeddings. In multi-speaker TTS, a conventional baseline interpolates speaker conditioning vectors as

$$\mathbf{e}_\alpha = (1-\alpha)\,\mathbf{e}_A + \alpha\,\mathbf{e}_B,$$

with $\alpha \in [0,1]$ controlling the degree of interpolation between two speakers (Murata et al., 2024). The same paper contrasts this with model merging, in which the interpolated object is not the conditioning embedding but the full parameter set of the synthesizer:

$$\boldsymbol{\theta}_\alpha = (1-\alpha)\,\boldsymbol{\theta}_A + \alpha\,\boldsymbol{\theta}_B.$$

The key distinction is that the generator itself is interpolated rather than only a latent condition vector (Murata et al., 2024).

When speaker embeddings are length-normalized or treated as lying on a hypersphere, spherical or geodesic interpolation is used instead of Euclidean averaging. INSIDE selects pairs of nearby speaker embeddings and computes intermediate embeddings using spherical linear interpolation:

$$\theta = \arccos \left( \frac{\mathbf{e}_i^\top \mathbf{e}_j}{\|\mathbf{e}_i\|\|\mathbf{e}_j\|} \right),$$

$$\mathbf{e}_{ij} = \frac{\sin((1 - \alpha)\theta)}{\sin(\theta)} \cdot \mathbf{e}_i + \frac{\sin(\alpha \theta)}{\sin(\theta)} \cdot \mathbf{e}_j.$$

The stated rationale is that speaker embeddings typically reside on or near a unit hypersphere and that SLERP preserves the geometric properties and unit norm (Liu et al., 26 Aug 2025).

A related but more structured geometry appears in frame-wise diarization for overlap regions. There, the target embedding for a frame with two active speakers is constrained to lie on the shortest path on a sphere between two teacher $d$-vectors, with the selected point chosen to be closest to the student estimate and then projected back onto the hypersphere scaled to average length $r$ (Cord-Landwehr et al., 2024). In this formulation, interpolation is local, frame-wise, and explicitly tied to overlap.

Interpolation can also be defined at the level of speaker distributions rather than individual vectors. Mid-attribute speaker generation models speaker embedding distributions by Gaussian mixture models conditioned on speaker attributes and then constructs new distributions by optimal-transport barycenter interpolation. For Gaussian components, the interpolated mean and variance are

$$\boldsymbol{\mu}' = \sum_{l=1}^L \lambda_l \boldsymbol{\mu}_l, \qquad \boldsymbol{\sigma}' = \sum_{l=1}^L \lambda_l \boldsymbol{\sigma}_l,$$

under a weighted Wasserstein-2 objective, and the barycenter of GMMs is a new GMM whose components are barycenters of components selected from the original attribute-conditioned GMMs (Watanabe et al., 2022).

A conceptually different formulation appears in PRISM. Rather than explicitly computing a convex combination between two endpoints, PRISM represents a speaker utterance as a “floating” vector whose state is indeterminate without knowing the context. The state is jointly determined by the utterance itself, other speech from the same speaker, other speakers it is being compared to, and the content of the speech (Zheng et al., 2022). This is not interpolation in the strict Euclidean or spherical sense, but it serves a similar role by letting ambiguous utterances move to contextually appropriate states.

A further, separate use of the term appears in Matryoshka speaker embeddings. M-Vec trains the first $m$ dimensions of a larger embedding to be discriminative for multiple nested dimensionalities, allowing dynamic extraction of 8, 16, 32, 64, 128, or 256 dimensions. The paper states that such embeddings naturally support “interpolation” across different embedding sizes and reports VoxCeleb1 EER values of 4.94 at 8 dimensions, 2.61 at 16, 1.57 at 32, 1.31 at 64, and 1.15 at 128 and 256 for a ResNet34-MRL model (Wang et al., 2024).

3. Context-dependent and overlap-aware interpolation in diarization

PRISM is a prominent example of context-conditioned speaker representation. Its architecture concatenates acoustic features and phonetic bottleneck features, uses 2 layers of TDNN and 6 dilated-convolutional transformer encoder layers, and produces a sequence of frame-level embeddings rather than a single vector per utterance (Zheng et al., 2022). The model is pre-trained with a frame-level contrastive loss whose exact loss and gradients for each frame depend on which other frames and speakers are present in the batch. For 1-1 speaker verification, the two utterances are concatenated, with padding to avoid cross-talk in convolutional layers, and poolings and losses are performed in context. For clustering, PRISM yields pairwise similarities rather than explicit fixed vectors, leading to graph-based clustering in which segments are nodes and PRISM similarities are edge weights (Zheng et al., 2022).

The reported empirical results support the usefulness of this context-sensitive interpolation behavior. On short-utterance speaker verification, PRISM achieves 10.79 EER, compared with 15.17 for x-vector, 14.45 for ETDNN, 18.24 for ResNet34, and 12.52 for ECAPA-TDNN. In clustering, PRISM-HDBSCAN reports 92.9% speaker-count accuracy, 96.7% precision, and 98.1% recall, compared with 35% and 35% speaker-count accuracy for k-means and Spectral, and 18% for AHC. In diarization on NIST SRE settings with 4 to 8 speakers, the proposed system reports DER values of 7.9, 9.8, 13.7, 16.1, and 18.4, outperforming x-vector+kmeans and EEND-EDA across all tested speaker counts (Zheng et al., 2022).

Geodesic interpolation of frame-wise speaker embeddings addresses a different but related problem: diarization of meeting scenarios containing partially overlapping speech. The teacher is a ResNet34-based $$\mathbf{e}_\alpha = (1-\alpha)\,\mathbf{e}_A + \alpha\,\mathbf{e}_B,$$0-vector model trained for speaker classification using single-speaker data, while the student replaces global pooling with a local pooling layer averaging over approximately 11 frames to produce frame-level embeddings. Both teacher and student embeddings are projected from 256 to 64 dimensions (Cord-Landwehr et al., 2024). The geodesic loss enforces embeddings from two-speaker overlap regions to lie on the shortest path on the sphere between the two teacher $$\mathbf{e}_\alpha = (1-\alpha)\,\mathbf{e}_A + \alpha\,\mathbf{e}_B,$$1-vectors, while non-overlapping frames are pulled toward the correct single-speaker teacher embedding.

The diarization results show that the choice of interpolation geometry is decisive. On the development and test sets, k-Means with MSE gives 90.78 and 87.59 DER, whereas k-Means with Geodesic gives 20.05 and 16.53. VMFMM with MSE gives 98.82 and 86.83, whereas VMFMM with Geodesic gives 24.07 and 25.23 (Cord-Landwehr et al., 2024). On LibriCSS test data, Spectral Clustering reports 14.86 average DER, SC + overlap detection 11.28, VBx 14.45, VBx + overlap detection 11.18, k-Means (E=64) 12.98, VMFMM + k-Means init (E=64) 11.07, and VMFMM + k-Means init + ov det 10.36. On DipCo, VMFMM with random initialization yields 24.4 dev DER and 25.6 eval DER, outperforming the CHiME7 baseline at 29.9 and 27.9 (Cord-Landwehr et al., 2024).

Related work on guided speaker embedding does not interpolate between speaker identities, but it uses explicit speaker activities to extract the target embedding directly from overlapping speech. Acoustic features are concatenated with target and non-target activity indicators, and attention weights for frames where the target speaker is inactive are set to zero and then re-normalized (Horiguchi et al., 2024). In one-vs-many overlapped verification, the proposed guided embedding reports 4.04 EER and 0.22 minDCF, compared with 12.84 and 0.36 for the baseline using pure single-speaker intervals and 14.11 and 0.39 for a baseline that allows overlap. In diarization, the macro average improves from 22.96/37.50 DER/JER to 22.34/35.67 (Horiguchi et al., 2024). This places guided extraction alongside interpolation-based methods as part of a broader shift away from fixed embeddings estimated only from clean segments.

4. Interpolation in speech synthesis and speaker generation

In speech synthesis, interpolation often aims to generate speech with intermediate speaker characteristics while preserving linguistic content. A standard baseline interpolates speaker embeddings and feeds them into a multi-speaker TTS model, but model merging instead interpolates the parameters of two base TTS models fine-tuned from a common multi-speaker pre-trained model (Murata et al., 2024). The reported evaluations compare the two strategies on speaker generation and emotion intensity control. For speaker generation, smooth, monotonic change in speaker similarity is observed as $$\mathbf{e}_\alpha = (1-\alpha)\,\mathbf{e}_A + \alpha\,\mathbf{e}_B,$$2 is swept from 0 to 1, measured by Speaker Encoder Cosine Similarity. Speech quality and intelligibility are comparable for same-gender merges: Female-Female gives MOS 4.25 and WER 11.5 for the speaker-embedding baseline versus MOS 4.24 and WER 10.9 for model merging; Male-Male gives MOS 3.92 and WER 10.4 versus MOS 3.91 and WER 9.5. For Female-Male, model merging reports MOS 3.45 and WER 11.2, compared with MOS 3.75 and WER 10.7 for speaker embedding interpolation, which the paper attributes likely to pitch normalization mismatches (Murata et al., 2024).

Optimal-transport-based interpolation of Gaussian mixture models targets “speaker generation,” defined as synthesis of a nonexistent speaker’s naturally sounding voice. The method learns separate GMMs over speaker embeddings conditioned on attributes and then interpolates them to represent mid-attribute speakers such as gender-neutral or intermediate language fluency (Watanabe et al., 2022). The experimental setup uses four attribute categories, 3-mixture GMMs over 256-dimensional embeddings, FastSpeech 2 with HiFi-GAN, and subjective evaluation with 750 listeners. The reported finding is that the method can control the generated speakers’ attributes by a continuous scalar value without statistically significant degradation of speech naturalness (Watanabe et al., 2022). The paper therefore treats interpolation not as pairwise vector arithmetic but as distributional control over a speaker-attribute-aware latent space.

Subjective-similarity-based embedding learning addresses a further issue: whether intermediate points in embedding space correspond to intermediate perceived voices. By using a crowdsourced inter-speaker similarity matrix of 153 Japanese female speakers, two DNN-based methods are trained either to predict similarity vectors or to match the Gram matrix of embeddings to the subjective similarity matrix (Saito et al., 2019). The paper reports that these algorithms learn embeddings highly correlated with subjective similarity and that similarity vector embedding improves synthetic speech quality for open speakers whose speech utterances are unseen during training. It further states that interpolating between embeddings of two speakers produces a voice that listeners perceive as somewhere between the two original speakers, a property conventional $$\mathbf{e}_\alpha = (1-\alpha)\,\mathbf{e}_A + \alpha\,\mathbf{e}_B,$$3-vector embeddings lack (Saito et al., 2019).

5. Interpolated speaker identities for data expansion

INSIDE treats interpolation as a mechanism for synthesizing new speaker identities for training data expansion. The pipeline begins with a pretrained speaker encoder from a controllable TTS model such as YourTTS, which extracts averaged speaker embeddings for real speakers in the training set. Speakers are grouped by gender, and interpolation pairs are selected either randomly within the same gender group or by a nearest-neighbor strategy based on cosine distance, with the latter designed to improve coverage of the embedding space and avoid duplicates or reverse pairs (Liu et al., 26 Aug 2025).

Intermediate embeddings are generated by SLERP and then fed directly to a frozen TTS model to synthesize speech waveforms for fictitious speakers. Text is drawn from a large-scale speech/text corpus such as LibriSpeech and filtered to match original utterance lengths; the resulting synthetic utterances are pooled with the original data for downstream training (Liu et al., 26 Aug 2025). The reported verification gains are +1.82% relative improvement for INSIDE-Syn, +2.76% for INSIDE-NN, and up to +5.24% for INSIDE ID-Exp with 40,000 synthetic speakers. For gender classification, INSIDE-NN yields +12.09% average improvement in error rate and INSIDE ID-Exp up to +13.44%. The method is also described as compatible with conventional augmentations such as reverberation, additive noise, and speed perturbation, which operate at the acoustic level but do not create new identities (Liu et al., 26 Aug 2025).

6. Encoded properties, perceptual alignment, and limitations

Interpolation quality depends on what the embedding actually encodes. A systematic probing study of i-vector, d-vector, and s-vector shows that i-vector has the highest accuracy for speaker classification, s-vector captures text content and word order effectively, and d-vector is balanced but loses sequential information through averaging. All embeddings achieve more than 97% gender accuracy, d-vector reaches 95% on speaking rate, s-vector is approximately 100% on word order, and all embeddings encode channel information far above the random baseline (Wang et al., 20 Dec 2025). This suggests that interpolation in a nominally “speaker” space may also traverse text, channel, or speaking-rate structure unless the embedding has been explicitly regularized against such factors.

A related limitation appears in target-speaker speech processing. A unified cross-task study reports that speaker verification performance is somewhat unrelated to target-speaker task performances, that the one-hot speaker code outperforms enrollment-based embeddings for TS-ASR, TSE, and p-VAD, and that the optimal embedding depends on the input mixture (Ashihara et al., 2024). Gradient-based optimization of the embedding yields WER reduction up to approximately 32% over initial one-hot or enrollment embeddings. This suggests that a universally fixed speaker embedding may be suboptimal when the task depends strongly on mixture context, and it places context-adaptive refinement in the same conceptual space as interpolation.

Not all speaker embedding spaces currently support explicit interpolation. Rhythm-based speaker embeddings extracted from phonemes and phoneme duration achieve 15.2% EER and show, through visualization and relation analysis, that the distribution of embeddings reflects subjective and objective similarity in speech rhythm (Fujita et al., 2024). However, direct interpolation or manipulation of embeddings in that rhythm-embedding space was not explored or demonstrated (Fujita et al., 2024). This marks an important boundary: a geometrically meaningful embedding space does not by itself establish that traversals through the space will yield controlled, perceptually valid intermediate speakers.

Taken together, the literature identifies several non-equivalent meanings of interpolation: convex mixing of speaker vectors, hyperspherical interpolation, barycentric interpolation of speaker distributions, parameter interpolation of generative models, and context-conditioned movement of utterance representations. Across these variants, the recurring technical issue is not only how to define an intermediate point, but also whether the underlying representation geometry aligns with speaker identity, overlap structure, and human perception.