Speaker-Agnostic Emotion Vectors

• Speaker-agnostic emotion vectors are representations designed to isolate emotional expression while stripping away speaker-specific traits.
• They are derived using diverse methods including task offsets, mean-pooled embeddings, vector quantization, and spherical coordinate transformations.
• Evaluation protocols measure naturalness, speaker identity preservation, and emotion intensity control using metrics such as MOS, SECS, and intensity-sorting accuracy.

A speaker-agnostic emotion vector is an emotion representation in which the encoded signal is intended to be “applicable to arbitrary speakers,” “computed solely from the text embedding” at sentence level, or “largely stripped of speaker identity,” depending on the task setting. In current work, the term covers parameter-space task vectors for text-to-speech, continuous speech embeddings, vector-quantized emotion codes, spherical emotion coordinates, and multimodal latent projections. Across these formulations, the recurring objective is to preserve emotional expression while avoiding the speaker mismatch that arises when emotion representations carry idiosyncratic pitch range, vocal fry, prosodic habit, or other speaker-specific cues (Murata et al., 4 Jul 2025, Li et al., 2021, Ma et al., 2023, Viswanath et al., 2023, Cho et al., 2024, Wang et al., 28 Apr 2026).

1. Conceptual scope and main formulations

The literature does not use a single canonical object for speaker-agnostic emotion. In some systems, it is a weight-space offset derived by subtracting a neutral TTS model from an emotional one. In others, it is a latent embedding extracted from speech, a quantized codebook entry, a low-dimensional spherical coordinate, or a projected multimodal representation. What is shared is the attempt to isolate “the common denominator” of emotional expression or to construct a representation in which only the emotion signal remains.

Formulation	Vector definition	Speaker-agnostic statement
SAEV (Murata et al., 4 Jul 2025)	$$\tau_{\mathrm{emo}}^{\mathrm{multi}}=\theta_{\mathrm{emo}}^{\mathrm{multi}}-\theta_{\mathrm{pre}}^{\mathrm{multi}}$$	“capture shared emotional expressions across multiple speakers”
emotion2vec (Ma et al., 2023)	$$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$	“rich in affective content yet largely stripped of speaker identity”
AffectEcho (Viswanath et al., 2023)	$e_{k^\*}$ from $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$	“speaker independent and language-agnostic emotion and affect transfer”
EmoSphere++ (Cho et al., 2024)	$s_i^k=(r_{IQR},\vartheta,\varphi)$	“control emotional style and intensity”
SEOV (Li et al., 2021)	$e_i=q_i\oplus q_i^\*$	“depends only on what is said”
ML-SAN (Wang et al., 28 Apr 2026)	$z_i=W_o\,v_i+b_o$	“no residual speaker identity ‘leaks’ into the emotion representation”

This diversity is significant because “speaker-agnostic” is task-dependent. In cross-speaker TTS, it denotes emotion transfer without degrading speaker identity. In conversation emotion recognition, it denotes sentence-level or multimodal emotion modeling that abstracts away from speaker-specific expressive style. In anonymization, it denotes preserving emotion while concealing the original speaker’s identity.

2. Task-arithmetic speaker-agnostic emotion vectors in cross-speaker TTS

The most explicit use of the term appears in “Speaker-agnostic Emotion Vector for Cross-speaker Emotion Intensity Control,” which treats the vector as a task offset learned from multi-speaker emotional fine-tuning rather than from a single speaker. In the single-speaker setting, emotion arithmetic defines

$$\tau_{\mathrm{emo}}^{\mathrm{spkA}}=\theta_{\mathrm{emo}}^{\mathrm{spkA}}-\theta_{\mathrm{pre}}^{\mathrm{spkA}},$$

but this vector “carries not only the expression of, say, ‘anger,’ but also whatever idiosyncratic pitch range, vocal fry or prosodic habit that particular speaker had when fine-tuned on angry data.” The multi-speaker alternative is

$$\tau_{\mathrm{emo}}^{\mathrm{multi}}=\theta_{\mathrm{emo}}^{\mathrm{multi}}-\theta_{\mathrm{pre}}^{\mathrm{multi}},$$

with synthesis at intensity $\alpha$ performed by

$$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$0

The paper states that, in practice, no new model is physically created for each $$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$1; a lightweight merge step adds $$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$2 into each pre-training weight in memory, and “no extra regularization or normalization of $$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$3 is required” because the scaling factor itself controls the effective norm (Murata et al., 4 Jul 2025).

The training pipeline builds on Conformer-FastSpeech2 conditioned on speaker embeddings. During both pre-training and fine-tuning, every input is tagged with an x-vector extracted from neutral speech for that speaker. Pre-training uses neutral utterances from 109 speakers in VCTK plus 10 speakers in ESD, with “the standard FastSpeech2 losses (mel-spectrogram reconstruction, duration, pitch, and energy losses).” The same model is then fine-tuned on ESD’s emotional subset—Angry, Sad, and Happy—from 10 speakers, again conditioning on each speaker’s neutral x-vector. The paper’s rationale is that, because the training set spans many voices, the offset is forced to capture only what generalizes: “the acoustic correlates of anger, sadness or happiness—rather than the quirks of any single individual” (Murata et al., 4 Jul 2025).

Evaluation covers three use-cases: same-speaker, seen cross-speaker, and unseen cross-speaker. The reported metrics are Mean Opinion Score on naturalness, Speaker Encoder Cosine Similarity between neutral and emotional outputs, and intensity-sorting accuracy for $$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$4. Averaged across Angry, Sad, and Happy, the key numbers are: MOS 2.3 vs. 3.7 in same-speaker baseline vs. SAEV; baseline failure to produce intelligible speech in cross-seen vs. SAEV 3.7; and SAEV 3.8 in cross-unseen. SECS is 0.87 for SAEV vs. 0.88 for baseline in same-speaker, with ground truth approximately 0.80; in cross-seen and unseen, SAEV is approximately 0.80, matching the ground-truth range 0.78–0.80. Intensity sorting accuracy is 0.74 in cross-seen and 0.67 in cross-unseen. The paper’s own summary is that these numbers show SAEV preserves naturalness and speaker identity even in zero-shot scenarios while listeners reliably perceive scaled emotion intensity (Murata et al., 4 Jul 2025).

3. Continuous, quantized, and spherical variants in speech synthesis

Alternative speaker-agnostic emotion vectors differ in whether they are continuous embeddings, codebook indices, or structured low-dimensional coordinates. emotion2vec is a universal speech emotion representation model trained by self-supervised online distillation on 262 hours and 169,053 utterances from five English corpora, including MSP-Podcast V1.8 with more than 10,000 speakers. Its inference vector is the mean of frame-level outputs,

$$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$5

and the paper explicitly frames its design as speaker-agnostic: large-scale multi-speaker pre-training, random masking across the entire signal, and utterance-level distillation encourage the model to ignore speaker-specific cues in favor of emotion-relevant ones. Extensive cross-speaker and cross-language results, together with UMAP visualizations, are presented as confirmation that emotion2vec yields a “768-dim vector that is rich in affective content yet largely stripped of speaker identity” (Ma et al., 2023).

AffectEcho adopts a discrete vector-quantized design. Each emotion embedding lives in $$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$6, with a single codebook $$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$7 arranged as 5 coarse emotion classes—angry, happy, neutral, sad, surprised—times 5 intensity levels per class. Given a reference utterance, the VQ-classifier outputs a prequantized vector $$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$8, and nearest-neighbor lookup in cosine similarity selects the quantized code:

$$e=\frac{1}{N_z}\sum_{i=1}^{N_z}Y_i^S\in\mathbb{R}^{768}$$9

The resulting code is then fed to a generator together with a JDC style embedding extracted from the source speech. The paper positions this as a model for “speaker independent and language-agnostic emotion and affect transfer,” with the quantized embeddings “implicitly derived from spoken speech samples,” eliminating the need for one-hot vectors or explicit strength embeddings (Viswanath et al., 2023).

EmoSphere++ uses an Emotion-Adaptive Spherical Vector rather than a task offset or a VQ code. A trained SER model predicts valence–arousal–dominance coordinates in Cartesian space; for each non-neutral emotion class, an emotion-adaptive centroid is computed; the VAD vector is shifted by that centroid and converted to spherical coordinates. The resulting vector is

$e_{k^\*}$0

where $e_{k^\*}$1 controls emotion intensity and the angular pair $e_{k^\*}$2 controls emotion style as a point on the unit sphere. EmoSphere++ combines this low-level emotion vector with a high-level emotion encoder and a WavLM-based speaker encoder in a multi-level style encoder, and it adds a normalized orthogonality loss to decorrelate emotion and speaker embeddings for zero-shot generalization (Cho et al., 2024).

A related multilingual line appears in METTS, which addresses cross-speaker and cross-lingual emotion transfer by introducing multi-scale emotion modeling that disentangles speech prosody into coarse-grained and fine-grained scales, “producing language-agnostic and language-specific emotion representations, respectively.” METTS further applies formant shift-based information perturbation to the reference signal and uses a vector quantization-based emotion matcher for reference selection (Zhu et al., 2023).

4. Speaker-invariant emotion representations beyond TTS

Speaker-agnostic emotion vectors also appear outside speech synthesis. In SEOVER, the Sentence-Level Emotion Orientation Vector is defined from text alone. A dialogue is encoded by a “transformer-emo” model into sentence embeddings $e_{k^\*}$3, each of which is linearly mapped into an emotion-orientation subspace by

$e_{k^\*}$4

and concatenated:

$e_{k^\*}$5

The paper emphasizes that SEOV is “computed solely from the text embedding $e_{k^\*}$6 of utterance $e_{k^\*}$7,” with no speaker identity token or speaker embedding injected at this stage. The resulting orientation depends only on what is said, not on who says it. These vectors are then fed into a downstream Dialogue Sentiment Analysis Model such as DialogueRNN, DialogueGCN, or bc-LSTM, and the whole model is trained end-to-end with a single cross-entropy loss. On IEMOCAP, DialogueRNN baseline weighted F1 is 62.75 versus 69.86 for SEOVER-RNN; on MELD, 55.90 versus 65.66, with an ablation drop of about 9 points on MELD when the $e_{k^\*}$8 branch is removed (Li et al., 2021).

ML-SAN formulates speaker-agnostic emotion as the final latent embedding of multimodal features after three adaptation stages. First, Input-level Calibration uses FiLM to transform raw audio and visual features into a neutral space unrelated to the speaker:

$e_{k^\*}$9

Second, Interaction-level Gating computes a speaker-conditioned modality trust gate

$E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$0

Third, Output-level Regularization adds an auxiliary speaker-ID loss so that no residual speaker identity leaks into the fused emotion representation. After fusion, the final speaker-agnostic emotion vector is

$E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$1

On MELD and IEMOCAP, ML-SAN reports gains over MultiEMO of +1.39% W-F1 and +1.26%, with ablation drops for removing FiLM, Gate, or Aux Loss. A t-SNE visualization is reported to show clean emotion clusters while speaker clusters no longer appear in the final embedding $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$2 (Wang et al., 28 Apr 2026).

In speaker anonymization, the issue is not only invariance but also privacy. A disentanglement-based system separately extracts content, speaker, prosody, and emotion. An anonymized speaker embedding $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$3 is produced by a speaker anonymizer, and emotion is reintroduced by moving that embedding along an emotion-specific SVM normal:

$E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$4

Emotion can also be injected through a pre-trained emotion encoder, emotion2vec, whose embeddings are 768-dimensional. The paper presents two strategies: integrating emotion embeddings from a pre-trained emotion encoder to help preserve emotional cues, and an emotion compensation strategy that reintroduces emotional traits lost during speaker embedding anonymization. The tradeoff is explicit: improved emotion preservation comes with some degradation in privacy metrics (Miao et al., 2024).

5. Evaluation protocols and observed empirical behavior

Across applications, the empirical question is whether speaker-agnostic emotion vectors maintain speaker consistency while preserving or controlling emotion. In cross-speaker TTS, the SAEV paper evaluates naturalness with MOS, speaker identity with SECS, and controllability with intensity-sorting accuracy. Its central comparative result is that the single-speaker emotion vector “collapses as soon as you swap speakers”: speech becomes muffled and speaker similarity crashes below 0.5, whereas SAEV remains stable across same-speaker, seen cross-speaker, and unseen cross-speaker cases (Murata et al., 4 Jul 2025).

AffectEcho evaluates emotion transfer with Mel cepstral distortion, structural similarity, speech emotion recognition accuracy, and MOS. In same-speaker English, Table 1 reports Angry: MCD 5.16, SSIM 0.69, SER 0.76, MOS $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$5; Happy: MCD 4.88, SSIM 0.71, SER 0.50, MOS $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$6; Sad: MCD 4.42, SSIM 0.72, SER 0.76, MOS $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$7; and Surprised: MCD 5.40, SSIM 0.68, SER 0.63, MOS $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$8. In speaker-independent same-language settings, performance drops by about 0.1–0.2 points in SSIM while MOS remains greater than 3.6. In cross-language transfer, MCD remains about 5.0–5.5, SER about 0.65–0.83, and EPS about 0.5–0.87. The paper further reports overall greater than 90% correct classification by its VQ-classifier, with slight confusions between Happy, Angry, and Surprised due to VAD proximity (Viswanath et al., 2023).

EmoSphere++ evaluates zero-shot TTS on held-out English speakers using WER, SECS, emotion-classification accuracy, and emotion-embedding cosine similarity. The reported averages are approximately WER 16.4%, SECS 0.76, ECA 94.6%, and EECS 0.936. The paper attributes part of this behavior to the orthogonality loss and to the use of a continuous 3-D EASV rather than discrete labels; ablations reported in Table V show that removing the disentangling loss causes drops in emotion-consistency and speaker-consistency on unseen speakers, and replacing EASV with a simpler mean-based spherical vector degrades intensity control accuracy in human preference tests (Cho et al., 2024).

emotion2vec evaluates a frozen upstream representation across speech emotion recognition, song emotion recognition, emotion prediction in conversation, and sentiment analysis. On IEMOCAP 4-way SER with leave-one-session 5-fold cross-validation, data2vec 2.0 weighted accuracy improves from 68.58% to 71.79% with emotion2vec; on leave-one-speaker 10-fold evaluation, emotion2vec reports 72.94% WA and 72.13% WF1. The paper also reports consistent gains across 9 cross-lingual SER settings and strong performance on conversation emotion prediction, where speech-only UAR improves from 65.01% to 77.19% and Macro-F1 from 65.91% to 76.71%. These evaluations support the claim that the learned embedding is speaker-independent and cross-lingually robust (Ma et al., 2023).

6. Limitations, misconceptions, and open questions

A recurring misconception is that “speaker-agnostic” means speaker information is absent everywhere in the model. The TTS task-arithmetic formulation does not remove speaker conditioning: during both pre-training and fine-tuning, every input is tagged with an x-vector extracted from neutral speech for that speaker, and the goal is to maintain speaker consistency while controlling emotion intensity. In SEOVER, speaker identity is not used in the sentence-level emotion orientation vector itself, but the downstream DialogueRNN formulation still concatenates the current speaker ID with the SEOV before context modeling (Murata et al., 4 Jul 2025, Li et al., 2021).

A second misconception is that a speaker-agnostic emotion vector is a single architectural device. The literature instead shows several incompatible but related mechanisms: weight subtraction in SAEV, mean-pooled self-supervised embeddings in emotion2vec, vector quantization in AffectEcho, spherical coordinates in EmoSphere++, SVM-boundary manipulation in anonymization, and FiLM-plus-gating-plus-auxiliary-loss projection in ML-SAN. This suggests that “speaker-agnostic emotion vector” functions more as a design objective than as a unique representation class.

The limitations are also task-specific. In anonymization, integrating emotion embeddings from a pre-trained emotion encoder can help preserve emotional cues, “even though this approach slightly compromises privacy protection,” and emotion compensation relies on a well-trained emotion encoder and emotion indicator; more fine-grained or continuous emotions would require multi-class or regression boundaries (Miao et al., 2024). In SAEV, the open questions listed by the authors are how the number of speakers in the multi-speaker pre-training set affects vector purity, whether one can learn disentangled per-emotion vectors without joint fine-tuning, how well SAEV extends to mixed or compound emotions, and whether normalizing $E=\{e_1,\dots,e_{25}\},\ e_i\in\mathbb{R}^{64}$9 by its $s_i^k=(r_{IQR},\vartheta,\varphi)$0 norm or by per-layer variances would tighten controllability ranges (Murata et al., 4 Jul 2025).

The broader record therefore supports a narrow conclusion. Speaker-agnostic emotion vectors are not defined by a single latent geometry, but by a shared attempt to separate emotion from speaker identity strongly enough to enable cross-speaker transfer, zero-shot generalization, multilingual or language-agnostic operation, privacy-preserving anonymization, or speaker-invariant conversation modeling. The exact form of the vector depends on whether the target operation is synthesis, control, recognition, anonymization, or multimodal adaptation, but the core requirement remains the same: the representation must preserve emotion without importing the speaker-specific traits that cause feature misalignment, ineffective fusion, foreign accent, or degraded speaker identity (Murata et al., 4 Jul 2025, Zhu et al., 2023, Miao et al., 2024, Wang et al., 28 Apr 2026).