AIPsy-Affect: A Keyword-Free Clinical Stimulus Battery for Mechanistic Interpretability of Emotion in Language Models

Abstract: Mechanistic interpretability research on emotion in LLMs -- linear probing, activation patching, sparse autoencoder (SAE) feature analysis, causal ablation, steering vector extraction -- depends on stimuli that contain the words for the emotions they test. When a probe fires on "I am furious", it is unclear whether the model has detected anger or detected the word "furious". The two readings have very different consequences for every downstream claim about emotion circuits, features, and interventions. We release AIPsy-Affect, a 480-item clinical stimulus battery that removes the confound at the stimulus level: 192 keyword-free vignettes evoking each of Plutchik's eight primary emotions through narrative situation alone, 192 matched neutral controls that share characters, setting, length, and surface structure with the affect surgically removed, plus moderate-intensity and discriminant-validity splits. The matched-pair structure supports linear probing, activation patching, SAE feature analysis, causal ablation, and steering vector extraction under a strong methodological guarantee: any internal representation that distinguishes a clinical item from its matched neutral cannot be doing so on the basis of emotion-keyword presence. A three-method NLP defense battery -- bag-of-words sentiment, an emotion-category lexicon, and a contextual transformer classifier -- confirms the property: bag-of-words methods see only situational vocabulary, and a contextual classifier detects affect (p < 10^-15) but cannot identify the category (5.2% top-1 vs. 82.5% on a keyword-rich control). AIPsy-Affect extends our earlier 96-item battery (arXiv:2603.22295) by a factor of four and is released openly under MIT license.

• The paper introduces a method to disentangle affective meaning from explicit lexical cues using keyword-free clinical vignettes.
• It employs a matched-pair design and multi-layer validation (VADER, NRC, transformer) to rigorously confirm affect detection independent of emotion-specific vocabulary.
• Strong numerical results show binary affect probes achieving an AUROC of 1.000 while top-1 emotion classification accuracy drops sharply, underscoring the method’s effectiveness.

Mechanistic Interpretability of Emotion in LLMs via AIPsy-Affect: A Keyword-Free Clinical Stimulus Battery

Motivation and Problem Formulation

The field of mechanistic interpretability for emotion processing in LLMs is heavily reliant on corpora where emotion is lexically marked. Probes, activation patching, SAE feature extraction, and causal ablation studies operate on benchmarks where the presence of emotion-specific vocabulary is confounded with the underlying emotional meaning. This conflation undermines empirical claims: detection of "anger" in a model when faced with the phrase "I am furious" is formally indistinguishable from lexical detection of the word "furious." The absence of resources that disentangle emotion from explicit vocabulary has impeded robust methodological progress.

AIPsy-Affect addresses this confound by offering a stimulus battery where emotional meaning is encoded situationally, with systematic removal of emotion-label words. The battery targets Plutchik's eight primary emotions, spanning six psychological domains, and is structured to support matched-pair contrasts, intensity gradients, and discriminant-validity splits. This design enables rigorous attribution of observed internal representations to affective constructs rather than lexical artifacts.

Dataset Architecture and Construction

AIPsy-Affect consists of 480 vignettes split across four categories: 192 clinical (peak-intensity, keyword-free emotion), 192 matched neutral controls (identical in domain, character, structure, minus affect), 48 moderate-intensity clinical vignettes (lower stakes, domains 1–3), and 48 complex_neutral items (narrative-rich, emotion-free for discriminant validity). Each clinical vignette is paired deterministically with a neutral counterpart, preserving all surface features except for the affective manipulation. The eight emotions, instantiated across six domains (including financial betrayal, professional violation, etc.), yield sufficient N for within-cell variance estimation and permutation tests at the cell level, enabling domain-versus-emotion variance decomposition.

Keyword exclusion is enforced through a multi-layer protocol: authoring by a clinical psychologist, lexicon-based audit (NRC Emotion Lexicon, Plutchik lists), and bag-of-words sentiment scoring (VADER). The resulting vignettes contain zero emotion-label words, confirmed by methodical validation.

Validation Methodology and Key Results

The dataset's keyword-free status is rigorously validated using a three-method NLP defense battery:

• Bag-of-words sentiment (VADER): Differentiates clinical from neutral primarily via situational, not emotional, vocabulary. Per-emotion medians are anti-correlated with target valence in six of eight categories. The difference is attributable to content such as "withdrawal," "death," rather than affect terms.
• Emotion-category lexicon (NRC): Clinical split contains fewer emotion-category words than neutral; within-clinical tests reveal no meaningful differentiation across emotions.
• Contextual transformer classifier (GoEmotions DistilRoBERTa): Accurately detects affect ($P(\text{emotional})$ clinical > neutral, $p < 10^{-15}$) but fails to classify category (5.2% top-1 accuracy vs. 82.5% for keyword-rich controls).

These results demonstrate that bag-of-words and lexicon-based sentiment systems are insensitive to the emotional structure of the battery. Contextual models can recover the presence of affect but not the category absent explicit keywords, reproducing the "whether, not which" dissociation. The matched-pair structure guarantees that any model-internal representation distinguishing clinical from neutral must be tracking affective meaning, not lexical artifacts.

Strong numerical results include:

• Binary affect probes reach AUROC 1.000 on keyword-free vignettes in prior work, saturating in early layers (2604.23719).
• Top-1 emotion classification accuracy drops precipitously for keyword-free vignettes (5.2%) compared to keyword-rich controls (82.5%).

Practical and Theoretical Implications

AIPsy-Affect enables mechanistic interpretability studies with a strong methodological guarantee: internal representations extracted via linear probes, activation patching, SAE features, causal ablations, or steering vectors cannot be ascribed to keyword presence. This advances the field's capacity to dissect whether emotion circuits and features genuinely encode affective constructs.

The battery supports:

• Intensity-gradient analyses (peak vs. moderate), facilitating dose–response studies.
• Discriminant-validity controls (complex_neutral), isolating affect from narrative complexity.
• Domain-versus-emotion dissociation via the six-domain factorial design, enabling robust representational geometry analyses.

Limitations include: English-only stimuli, Plutchik-centric emotion taxonomy, single-clinician authorship, and residual risks (notably for loathing via dehumanizing language correlates). Expansion to multi-lingual, multi-taxonomic, and multi-clinician datasets is planned.

Prospects for Future Research

The open release under the MIT license catalyzes independent validation, downstream mechanistic studies, and benchmarking across architectures. AIPsy-Affect sets methodological precedence for stimulus design in psychological construct studies, with future releases planned for constructs such as emotion regulation and cognitive distortions. The dataset invites analyses on layer-wise computation, representational scaling, and causal manipulation of affect-sensitive circuits in LLMs, unconfounded by lexical shortcutting.

Long-term, this paradigm informs both foundational and alignment-relevant research in affective NLP, guiding both mechanistic and behavioral investigations into how models internally process situational affect.

Conclusion

AIPsy-Affect provides a robust, keyword-free stimulus battery for the mechanistic interpretability of emotion in LLMs, systematically controlling for lexical confounds. It substantively strengthens methodology for attribution of internal representations to affective constructs rather than keyword presence. The dataset’s architecture and validation pipeline enable rigorous empirical interrogation of emotion circuitry in LLMs and lay groundwork for expanded research programs in clinical construct interpretability.