A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theories

Abstract: Text encoders are known for their utility in natural language processing, as they are able to efficiently compress inputs into dense vectors while preserving semantics. These models have been applied to affective computing, in particular to help with solving sentiment analysis and emotion recognition tasks. Nevertheless, it remains unclear to what extent the latent representations produced by modern text encoders capture well-defined psychological theories of affect. In this work, we investigate the affective capabilities of twelve recently released text encoders by probing their generated embeddings as input features for solving regression and classification tasks across three established emotion frameworks, using both word- and sentence-level data. Additionally, we apply a semantic data-leakage prevention technique to improve robustness in word-level evaluations. Our main findings show that the latent manifolds of the latest instruction-aware open-weight encoders enclose an equal or even a larger amount of affective information in comparison with proprietary counterparts when evaluated at word level. In contrast, embeddings of task-tuned and proprietary encoders reach the highest scores on sentence-level affective classification. Furthermore, a qualitative analysis of latent representations and their encoded affective cues is provided.

• The paper demonstrates that open instruction-aware encoders excel in word-level affective regression, achieving high R² and concordance compared to proprietary models.
• It employs rigorous semantic leakage prevention and evaluates embeddings across PAD, Plutchik, and Ekman's frameworks using both linear and nonlinear predictors.
• Results indicate that while open models generalize well in dimensional tasks, task-tuned and proprietary models perform comparably on sentence-level emotion classification.

A Comparative Analysis of Affective Information in Modern Text Embeddings Aligned With Psychological Emotion Theories

Introduction and Motivation

The paper "A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theories" (2606.29068) systematically investigates the extent to which contemporary text encoders represent affective concepts derived from established psychological models. While Transformer-based text encoders are widely adopted for a range of NLP tasks, their ability to encode fine-grained emotional cues—particularly those aligned with rigorous emotion theories—has remained an open question. This work directly addresses this gap by evaluating twelve recent text embedding models, considering both open-weight and proprietary, instruction-aware and task-tuned architectures, on affective regression and classification tasks grounded in three psychological frameworks.

Methodology Overview

The study probes the extracted embeddings using both word-level and sentence-level datasets, each aligned with a distinct emotion theory:

• NRC-VAD (Valence, Arousal, Dominance; Mehrabian and Russell)
• NRC-EIL (Emotion Intensities; Plutchik)
• GoEmotions (Ekman's six basic emotions plus neutral)

A major methodological strength is the rigorous splitting protocol that prevents both morphological and semantic leakage between train and test splits, implemented via clustering in semantic space using the Leiden algorithm and WordNet-based graph construction. Embeddings are computed and frozen per dataset-experiment combination and subsequently serve as input features for four classes of downstream predictors, including both linear and nonlinear models.

Figure 1: Pipeline for fitting and evaluating the emotion regression/classification procedure, exemplifying word-level regression with semantic leakage prevention.

Quantitative Analysis: Regression and Classification Results

Word-level Regression (NRC-VAD, NRC-EIL)

On the NRC-VAD (PAD) dataset, open-weight instruction-aware encoders, especially KaLM v2 (open, instruction-aware), achieve the strongest affective regression performance. With an MLP backend, KaLM v2 records a test-time $R^2$ of 0.677 and a concordance correlation $\rho_c = 0.811$, demonstrating substantial alignment of encoder latent space with PAD affective axes. Linq Embed Mistral, Qwen3 Embedding 8B, and LLaMA Nemotron 8B follow closely, all presenting $R^2 > 0.62$ and $\rho_c > 0.77$. Proprietary encoders, including OpenAI Text Embedding v3 Large, are outperformed under these conditions.

In NRC-EIL (Plutchik), the best open-weight models again excel, with KaLM v2 and Linq Mistral each attaining $R^2 \approx 0.54$ and $\rho_c \approx 0.73$. Performance differences among the top four open-weight models (KaLM v2, Linq Mistral, Qwen3 8B, EmbeddingGemma) are not statistically significant, but all demonstrably outperform proprietary baselines such as Gemini Embedding and OpenAI Text Embedding v3.

The results reveal that affective information is robustly available in latent spaces of open, instruction-aware encoders, provided the task is word-level and the emotional representation theory is well-defined.

Sentence-level Classification (GoEmotions)

The landscape shifts when moving to sentence-level emotion classification (GoEmotions dataset). Here, the proprietary Gemini Embedding 001 and open task-tuned EmbeddingGemma produce the best results, with macro-averaged $F_1$ scores of 0.60 and 0.59, respectively. Differences between these and the next-best models are not statistically significant, suggesting a performance plateau at this representation level and task formulation. Open instruction-aware models, which excelled at word-level PAD/Plutchik regression, do not universally outperform proprietary and task-tuned counterparts in sentence-level affective labeling tasks.

Model Type and Task Interaction

There is explicit empirical support for the paper's claims that:

1. Instruction-aware open models outperform proprietary models on word-level affective prediction, especially in regression aligned to dimensional theories (PAD).
2. Task-tuned and proprietary models are more competitive on sentence-level classification, with no clear dominance from instruction-aware open models.
3. Model performance is sensitive to emotion framework and task formulation; cross-dataset generalization is nontrivial.

Qualitative Analysis: Latent Space Structure

The UMAP analysis visualizes the clustering and separability of affective information in the frozen embeddings.

Figure 2: UMAP latent space projections for top-performing models, color-coded by categorical or continuous affect labels.

Findings include:

• NRC-VAD (PAD): None of the encoders produce clear clustering along PAD axes, suggesting current embedding spaces do not form strong topological correspondences to this theory beyond what is recoverable linearly/nonlinearly by downstream predictors. Arousal appears more separable than valence or dominance.
• NRC-EIL (Plutchik): Encoders can sharply separate positive from negative emotions at the word level, particularly in instruction-aware settings.
• GoEmotions (Ekman): Disgust, fear, and sadness are partially clustered, but other emotions scatter throughout the space, reflecting broader semantic variance within those categories.

Implications and Theoretical Insights

Practical Implications

• Instruction-aware open encoders are highly competitive for affective computing tasks requiring low-level affective feature extraction, including those involving dimensions of valence, arousal, and dominance.
• In sentence-level classification scenarios, task-tuned or proprietary models can match or slightly outperform instruction-aware models, signaling that training objectives geared toward downstream label prediction remain impactful.
• The semantic leakage-preventing split is crucial for robust assessment of generalization—studies omitting this are likely to overestimate encoder affective competency.

Theoretical Considerations

• Emotional structure is encoded in the latent manifolds of LLMs to a task- and theory-dependent degree; strong mapping exists for dimensional PAD models at the word level, weaker for multi-dimensional categorizations at the sentence level.
• The lack of emergent topological structure in UMAP for PAD affects suggests that substantial affective alignment may require additional regularization/supervision at pretraining or finetuning, despite reasonable downstream predictive performance.
• Nonlinear models (MLP) consistently extract more affective signal than linear or distance-based approaches, reflecting the complex, entangled representation of affect in current embedding spaces.

Future Directions

The paper highlights opportunities for extending the analysis to additional psychological frameworks (e.g., the Hourglass model [Cambria et al.]) and to multimodal embeddings. Given the trend of large models trained on multimodal corpora, evaluating whether multimodal embeddings further improve affective alignment and cross-framework mapping is a logical next step [see also relevant multimodal datasets such as IEMOCAP]. There is also substantial practical value in more granular prompt/architecture investigations for instruction-aware models and in developing benchmarks that better capture the nuances of emotional taxonomy.

Conclusion

This comparative study establishes that recent open instruction-aware text encoders encapsulate considerable affective information, outperforming proprietary alternatives specifically in word-level, dimensional emotion tasks, and supporting efficient zero-shot affective feature extraction. However, the translation of these strengths to sentence-level, categorical affect recognition is less clear, where proprietary or task-tuned models maintain parity. The findings underline that the intersection of embedding architecture, training paradigm, task-level supervision, and psychological framework is complex, but that modern encoders, especially those with instruction fine-tuning, are effective platforms for affective computing research and applications.