Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs

Abstract: Recent work identified emotion vectors in Claude Sonnet 4.5, which are internal representations that encode emotion concepts, causally influence behavior, and exhibit geometry mirroring human psychological structure. We test the generality of these findings in two open-weight models, Apertus-8B-Instruct-2509 and Gemma-4-E4B-it, extracting emotion contrast vectors across all layers, using two model-generated corpora. We recover valence geometry for both models, with peak PC1--valence correlations of $r = 0.76$ and $r = 0.83$, approaching the $r = 0.81$ reported for Claude.Beyond replication, we observe notable differences in how valence representations emerge across model depth. In Gemma-4-E4B-it, valence is strongly encoded in early layers but collapses towards later layers, whereas Apertus-8B-Instruct-2509 exhibits the opposite pattern, with valence representations absent in early layers, but emerging at mid depths. Arousal encoding, in contrast, is sensitive to the extraction corpus: both models show stronger PC2--arousal alignment with Gemma-generated stories ($r$ up to $0.45$) than Apertus-generated ones ($r \leq 0.21$), suggesting arousal-relevant cues are unevenly distributed across generated corpora. We open-source our experiment code and dataset for reproducible investigation of emotion representations across LLM architectures.

• The paper demonstrates that open-weight LLMs robustly encode linear emotion vectors, with the principal axis of valence correlating highly with human ratings (up to r=0.83).
• The paper reveals distinct layer-wise emergence patterns: GEMMA-4-E4B peaks early while APERTUS-8B exhibits mid-to-late emergence, highlighting architectural influences on emotion encoding.
• The paper finds that arousal encoding is highly sensitive to corpus design, emphasizing that linguistic cues and prompt methods play a key role in emotion extraction.

Formal Summary of "Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs" (2606.26987)

Background and Motivation

The study interrogates whether linear "emotion vectors"—identified in the internal activations of proprietary models such as Claude Sonnet 4.5—are a generalizable property across LLM architectures. Prior work demonstrated that such directions encode emotion concepts with geometry consistent with human psychological models, specifically the Russell valence-arousal circumplex. Importantly, steering along these directions induced causal and sometimes misaligned behavioral shifts, raising interpretability and alignment considerations. This paper extends the empirical investigation to open-weight models: APERTUS-8B and GEMMA-4-E4B, examining cross-layer emergence and corpus sensitivity.

Methodology

Two open-weight transformers—APERTUS-8B (32 layers) and GEMMA-4-E4B (42 layers)—were probed using synthetic corpora generated by each respective model, totaling 1,539 stories per corpus and spanning 171 emotion categories. Stories were designed to imply but never explicitly name the emotion, mirroring extraction methodology in Sofroniew et al. (2026) for control of confounds. Residual stream activations were averaged per story, and a neutral corpus was employed to construct and project out the emotion-agnostic subspace via PCA, yielding contrast vectors for subsequent analysis.

Principal Component Analysis (PCA) was employed on the emotion vector matrix per layer, followed by correlation with human-annotated valence and arousal ratings using the NRC VAD lexicon. Layer-wise and cross-layer geometry was tracked via Centered Kernel Alignment (CKA), and valence direction stability was quantified by cosine similarity across layers.

Key Empirical Findings

Replicability and Magnitude of Emotion Geometry

Both models reproduced valence-geometry as a principal axis: GEMMA-4-E4B achieved a peak PC1-valence correlation of $r = 0.83$ (layer 16, Gemma stories) and APERTUS-8B reached $r = 0.76$ (layer 31, Gemma stories), closely matching the $r = 0.81$ reference for Claude Sonnet 4.5. All peaks were statistically significant ($p < 10^{-3}$). This robust alignment establishes the generality of valence encoding across architectures and training regimens, negating the hypothesis that emotion vectors are an idiosyncrasy of proprietary models.

Divergent Layer-Wise Emergence

The two models exhibited pronounced divergence in the developmental profile of the valence axis:

• GEMMA-4-E4B: Valence encoding peaks early (layer 16) and collapses toward later layers, both for self-generated and Apertus-generated story corpora. Representational similarity (CKA $\geq 0.73$) remains high throughout, implying degradation of the valence axis is not rooted in global space reorganization.
• APERTUS-8B: Valence correlations are negligible across early layers and emergent in mid-to-late layers (becomes significant at layer 18 and stabilizes beyond layer 21), coincident with a phase transition in CKA metrics. Whole-space similarity shifts sharply across network depth; however, the valence direction itself remains unstable with inter-layer cosine similarity never exceeding $|0.49|$ in self-corpus conditions.

Peak valence correlations are similar in magnitude, but arise via different computational pathways, underscoring the need for architecture-driven analysis in interpretability and intervention.

Corpus Sensitivity of Arousal Encoding

Arousal (PC2) correlations were weaker and highly sensitive to story corpus. When the extraction corpus was Gemma-generated, both models achieved robust PC2-arousal alignments ($r = 0.45$ APERTUS-8B; $r = 0.41$ GEMMA-4-E4B, $p < 10^{-8}$), over twice the strength of Apertus-generated corpus ($r \leq 0.21$). This suggests that arousal-relevant linguistic cues vary with generator model and prompt methodology, corroborating the assertion that corpus composition is a significant methodological variable in emotion extraction.

Stability and Rotation of Emotion Directions

Representational geometry (CKA) is largely invariant across corpus conditions, but the valence axis itself rotates and loses alignment post-peak in GEMMA-4-E4B and is noisy across layers in APERTUS-8B. This disconnect indicates that valence information is preserved in the latent space, but the axis of extraction is unstable, cautioning against simple layer selection for steering or monitoring interventions.

Implications and Theoretical Perspectives

The replication and extension confirm that linear emotion vectors—and especially valence—are generic emergent properties in LLMs. The nonuniform layer-wise emergence opens avenues for mechanistic analysis relating geometry transitions to architectural or training choices. The corpus dependence for arousal urges further scrutiny into the impact of prompt engineering and synthetic stimuli. For safety, the capacity to monitor emotion representations in diverse LLMs enables proactive detection of internal misalignment states.

Practical implications include the necessity to select layers for steering interventions according to observed geometry, and the unreliability of a single layer-wise direction for control. Agentic LLM deployments should consider layer-dependent extraction for robust monitoring, and further research into cross-layer feature tracking and causal validation is warranted. Extension to multi-modal models could generalize valence-arousal geometry across modalities, contributing to unified interpretability frameworks.

Limitations

The study is partly constrained by reconstructed methodology from prior proprietary work, limited architecture sampling (two LLM families), and corpus generation effects that potentially confound model-intrinsic versus stimulus-induced geometry. Adoption of fully model-independent emotion stimuli sets would strengthen causal attribution in future analyses.

Conclusion

Valence geometry is robustly recoverable in open-weight LLMs, achieving PC1-valence correlations commensurate with proprietary references. The layer-wise emergence profile and axis stability are model-dependent, demonstrating that similar emergent structures can arise via distinct computational paths. The findings substantiate the generality of emotion vectors and delineate methodological factors critical for interpretability and intervention in LLM development.