Emotion2Vec Cosine Similarity

• Emotion2Vec cosine similarity quantifies the angular proximity between emotion embeddings, capturing affective similarity regardless of vector magnitude.
• The method leverages fine-tuning, normalization, and representational similarity analysis across text, speech, and language models to enhance emotion detection.
• Applications span emotion retrieval, classification, and clustering, while challenges in speech domains call for improved calibration and contrastive techniques.

Emotion2Vec cosine similarity refers to the use of vector-based geometric similarity, specifically cosine similarity, between emotion representations produced by models labeled as "Emotion2Vec" or similar affect-embedding frameworks. This measure underpins retrieval, clustering, and evaluation tasks across both text-based and speech-based emotion representation models, with applications in emotion detection, analysis, and generation assessment. The term encompasses both canonical Emotion2Vec models designed for text (e.g., word and sentence embeddings) and speech-focused encoders, as well as post hoc embedding comparisons in LLMs and speech evaluation pipelines.

1. Mathematical Basis: Cosine Similarity

Cosine similarity between two embedding vectors, $e_i$ and $e_j$, is defined as:

$$\mathrm{cosine\_sim}(\mathbf{e}_i,\mathbf{e}_j) = \frac{\mathbf{e}_i^{\top}\,\mathbf{e}_j}{\|\mathbf{e}_i\|\;\|\mathbf{e}_j\|}$$

where $\|\mathbf{e}\|$ denotes the $\ell_2$ norm. Cosine similarity quantitatively describes the angular proximity between embeddings in high-dimensional space, making it invariant to vector magnitude and sensitive to directional differences. This property is central in detecting whether two utterances, words, or sentences share similar affective content independent of their absolute activation levels (Ma et al., 2023, Seyeditabari et al., 2019, Xu et al., 2018, Jeong, 13 Apr 2026).

In most emotion embedding contexts, cosine similarity is computed after optional operations such as mean-centering or $\ell_2$-normalization, especially when combating embedding anisotropy and bias.

2. Emotion2Vec in Text: Construction and Evaluation

In the context of text, Emotion2Vec refers to specialized word or sentence embeddings trained to encode affective semantics.

Model Construction

• Two-stage fine-tuning is employed: starting from pre-trained embeddings (such as Word2Vec or GloVe), emotional content is “injected” via secondary training with labeled lexicons and psychological emotion models (e.g., Plutchik’s wheel).
• The fine-tuning objective includes:
  • Positive relation: words are pulled closer to their emotion prototype vectors using $1-\cos(u,v)$ (cosine distance) losses.
  • Negative relation: words are pushed away from their opposite emotion prototypes.
  • Vector-space preservation: terms ensure the new embedding geometry remains topologically similar to the original space locally (Seyeditabari et al., 2019).

Cosine Similarity in Evaluation

• After fine-tuning, cosine similarity between emotion-aware embeddings directly reflects emotional alignment.
• Empirical results show improvements: in-category cosine similarity increases by 13% (Word2Vec) to 29% (GloVe), while opposite category similarity decreases, sharpening emotional discrimination (Seyeditabari et al., 2019).
• Downstream applications include document-level emotion detection, lexicon expansion via nearest-neighbor search, and clustering for affective taxonomy (Xu et al., 2018, Park, 2018).

3. Emotion2Vec in Speech: Self-Supervised SER Embeddings

In speech domains, Emotion2Vec typically refers to universal emotion representation encoders such as "emotion2vec" (Ma et al., 2023), which utilize self-supervised online distillation to learn compact utterance- or frame-level vectors.

Training and Loss Functions

• Pre-training minimizes mean squared error (MSE) losses between student and teacher networks at both frame and utterance levels:

$$L_{\mathrm{Frm}} = \frac{1}{M}\sum_{i\in\mathbb{M}} \bigl\lVert Y_{i}^{\mathcal{T}} - Y_{i}^{\mathcal{S}} \bigr\rVert_{2}^{2}$$

$$L_{\mathrm{Utt}} = \bigl\lVert \bar{Y}^{\mathcal{T}} - \bar{U}^{\mathcal{S}} \bigr\rVert_{2}^{2}$$

$$L = L_{\mathrm{Frm}} + \alpha\,L_{\mathrm{Utt}}, \quad \alpha=1$$

Cosine-based objectives are not applied during training (Ma et al., 2023).

Downstream and Visualization Practice

• emotion2vec embeddings are frozen; simple linear or GRU classifiers are trained atop for supervised recognition tasks.
• Cosine similarity is not employed in evaluation, clustering, or reporting. No unit-norm projection or renormalization is conducted post-extraction, except for standard Transformer layer-normalization and mean pooling. The only visualizations reported use UMAP projections colored by discrete category or arousal, not by pairwise cosine (Ma et al., 2023).

4. Cosine Similarity in LLM Emotion Geometry

Analysis of emotion representation in LLMs uses cosine similarity to quantify the geometric relations among extracted “emotion vectors,” commonly via representational similarity matrices (RDMs) (Jeong, 13 Apr 2026).

Extraction and RDM Computation

• For a fixed set of emotions, multiple stimulus passages are encoded; at each model layer, vectors are averaged, mean-centered, and subjected to pairwise cosine similarity (or dissimilarity: $e_j$0).
• RDMs are constructed as 21×21 matrices for 21-emotion taxonomies:

$e_j$1

• Cross-model geometric alignment is quantified via Spearman $e_j$2 between the upper off-diagonal entries of different models’ RDMs.

Key Findings

• Major mature LLMs exhibit highly correlated emotion RDMs ($e_j$3–$e_j$4), confirming geometric universality of affective encoding across architectures and compliance regimes (Jeong, 13 Apr 2026).

5. Critical Perspective: Cosine Similarity for Speech-Emotion Evaluation

Recent work critically examines the deployment of cosine similarity as a zero-shot metric for emotional expressiveness in speech generation (Tsai et al., 29 Apr 2026). The "EMO-SIM" metric computes cosine similarity between latent vectors of reference and generated speech, sometimes after mean-centering to address anisotropy (vector norms cluster near 1 and raw cosine scores in [0.92, 0.98]).

Experimental Insights

• Categorical robustness: Even in matched speaker/text conditions, emotion2vec-based cosine similarity achieves only 60–70% accuracy on discriminating same-emotion from different-emotion pairs; performance degrades sharply with speaker or linguistic distractors, sometimes to sub-random levels ($e_j$53.4% in CREMA-D with linguistic distractors).
• Continuous emotion sensitivity: Cosine distance is uncorrelated with differences in continuous affect dimensions (valence/arousal), e.g., Spearman $e_j$6; shift discriminability remains at or near chance.
• Human alignment: Best EMO-SIM variants align with human judgments on 52-65% of forced-choice triplets, near chance.
• The metric is highly susceptible to spurious similarity caused by speaker or linguistic mimicry, not genuine affective proximity—a phenomenon labeled "false resonance."

Table: Robustness of Cosine-Based Emotion Similarity for Speech

Setting	EMO-SIM Accuracy (%)	Human Alignment (%)
Speaker+Text matched	60–70	52–65
Speaker distractor	<50 (degrades)	—
Linguistic distractor	≈3.4 (CREMA-D)	—

All numbers from (Tsai et al., 29 Apr 2026); “—” indicates measure not reported for that regime.

Recommendations

Researchers are advised to calibrate embedding geometry (mean-centering, sphere normalization), adopt contrastive loss objectives to promote emotion-selectivity, and employ more robust evaluation protocols (e.g., triplet-based robustness, multi-view metrics) in speech tasks.

6. Practical Usage and Guidelines

Text Embeddings

• After fine-tuning, cosine similarity between emotion2vec (or Emo2Vec/EVEC) vectors is directly usable for:
  • nearest-neighbor retrieval,
  • prototype scoring,
  • affect-aware clustering,
  • emotion lexicon construction,
  • concatenation with standard embeddings for classification (Seyeditabari et al., 2019, Xu et al., 2018, Park, 2018).
• Normalization (L2, mean subtraction) is generally recommended prior to similarity computation for robustness and bias mitigation.

Speech Embeddings

• emotion2vec embeddings, while effective under supervised evaluation with linear probes, are not currently reliable for direct cosine-based zero-shot assessment of emotion similarity in generation, due to entanglement with non-affective acoustic and linguistic content (Tsai et al., 29 Apr 2026).
• If cosine similarity is to be used in speech, embedding centering is mandatory, and contrastive fine-tuning is necessary to disentangle affective cues.

LLMs

• Post hoc cosine similarity of mean-centered emotion representation vectors reliably characterizes shared geometric structure of affect in LLMs and can be compared across architectures via representational similarity analysis (Jeong, 13 Apr 2026).

7. Open Challenges and Future Directions

Consensus is lacking on a “universal” cosine similarity regime for zero-shot emotional similarity, especially in speech domains. Major open problems include:

• Decoupling emotion-aligned and non-affective (linguistic, speaker) features in self-supervised audio embeddings.
• Developing embedding calibration and contrastive methods to enhance emotional separability.
• Standardizing robust triplet-based and human-aligned benchmarks for metric validation.
• Clarifying the impact of model pre-training objectives and input variability on the geometry of affective representations.

No single cosine similarity protocol currently suffices for all emotion representation domains or evaluation types. Future models will likely pair emotion2vec-style representations with explicit geometric or contrastive constraints, advanced embedding preprocessing, and multi-view metrics to ensure that cosine similarity accurately encodes affective proximity rather than superficial acoustic or linguistic similarity (Tsai et al., 29 Apr 2026, Jeong, 13 Apr 2026).