AttuneBench: EI Benchmark for LLMs

• AttuneBench is a comprehensive conversation-based benchmark that evaluates LLMs’ emotional intelligence through multi-turn human-model interactions.
• It employs granular turn-level annotations such as mood-shift tags, behavioral judgments, and response comparisons to capture real-time emotional dynamics.
• The benchmark uses detailed metrics including Emotion F₁, Valence-Arousal, and Composite Score to guide improvements in LLM emotional responsiveness.

AttuneBench is a comprehensive conversation-based benchmark designed to evaluate the emotional intelligence (EI) of LLMs through multi-turn, authentic human–model interactions. Unlike benchmarks that rely on synthetic prompts, single-turn assessments, or third-party annotation, AttuneBench assesses models’ ability to infer, track, and respond to a participant’s emotional state over the evolving course of real conversations. It provides a multi-dimensional framework for diagnosis and comparison of LLM behavior in emotionally salient contexts, incorporating granular turn-level and conversation-level metrics (Lubrano et al., 20 May 2026).

1. Dataset Construction

1.1 Multi-turn Human–Model Conversations

AttuneBench comprises 200 genuine dialogues, each with a minimum of five human–LLM exchanges (mean ≈ 6 turns). The participant pool—termed human participants (HPs)—includes 11 U.S.-based native English speakers, balanced by gender and mental health history (6 neurotypical, 5 with at least one clinical diagnosis). Conversation topics are assigned randomly from a 50-topic taxonomy (e.g., Money, Romantic Relationships, Hobbies), ensuring non-redundant and naturally varying emotional valence. The emotional targets are not artificially elicited; variation arises from participants’ spontaneous attitudes and mood states.

Eight original chat models serve as anonymized conversational partners (e.g., GPT-4 Turbo, Claude 3.5 Sonnet, Gemini 2.5 Flash, Inflection 3), with HPs blind to the model identity. All data are collected in single sessions to capture live emotional dynamics. Pre- and post-conversation moods are indexed using the Positive and Negative Affect Schedule (PANAS).

1.2 Turn-by-Turn Annotation Workflow

At every turn, HPs generate rich annotations:

• Mood-Shift Tags: One or more PANAS emotions, each rated on an intensity scale from 1 to 7.
• Behavioral Judgments: Binary responses to a selection from 36 Mayer-Salovey-Caruso Four-Branch Model–grounded questions, split into observed behavior (did the model acknowledge emotion?) and preferred behavior (would more/less acknowledgment be desired?).
• Response Comparison: After models adapt to HP preference, three responses (original model, preference-informed model, and a human-drafted “golden” reply) are compared in 15 pairwise judgments covering emotional understanding, guidance, safety, and adaptation.
• Golden Response: HPs author their own ideal reply referencing, but not copying, model outputs.

At session end, HPs provide post-conversation PANAS mood, Four-Branch evaluations, and free-response satisfaction assessments (conversation goals, emotional clarity, fit), including optional qualitative commentary on conversational “fit.”

2. Task Definitions and Evaluation Protocol

AttuneBench operationalizes EI along four principal axes, each capturing distinct subskills of the emotionally intelligent conversational agent:

Task	Input/Output Structure	Metric(s)
Emotion Recognition	HP's turn, model predicts mood-shift tags	Precision, Recall, F₁, Valence-Arousal VA
Behavioral Classification	Per turn, binary Q&A (observed/preferred)	Binary Acc. (OM & HP perspective)
Preference Prediction	Rank three response variants	Pairwise Acc., Kendall’s τ
Response Quality (Judge)	Model draft rated on 4 axes by judge LLM	Mean judge score (0–1 normalized)

Formal definitions:

• Emotion Recognition: For each turn $t$, ground truth $Y_t$ (subset of 20 PANAS tags), prediction $\hat{Y}_t$. $F_{1,t}$ averaged over turns yields Emotion F₁. Valence–Arousal partial credit computed via NRC VAD lexicon, capturing graded similarity between predicted and actual affect.
• Behavioral Classification: For question $q$ at turn $t$, ground truth binary observed $o_{tq}$ and preferred $p_{tq}$. Evaluated models (EMs) predict $\hat{o}_{tq}$, $\hat{p}_{tq}$. Accuracies reported separately for OM and HP perspectives.
• Preference Prediction: EM ranks the three candidate responses; correct pairwise ordering fraction and Kendall’s τ algorithmically evaluated against HP’s preference total order.
• Response-Quality Judgment: Judge LLM provides four 1–7 ordinal ratings, normalized and averaged.

3. Metrics and Composite Scoring

AttuneBench employs a rigorous, multi-axial scoring framework:

3.1 Primary Metrics

• Emotion F₁: Macro F₁ for emotion tag prediction.
• Emotion VA: Mean valence-arousal similarity.
• Binary OM / HP Acc: Accuracy for observed and preferred behavioral classification.
• Pairwise Acc / Kendall’s τ: Agreement in ranking response variants.
• Draft Judge: Judge LLM mean normalized rating.
• Four-Branch MAE: Mean absolute error on post-conversation Four-Branch branch scores.
• PANAS B-Adj: Adjusted mean absolute error for post-conversation mood, correcting for model-wide bias.
• Composite Score: Weighted sum for holistic evaluation:

$Y_t$0

The E, V, and H pillars contribute 24%, 49%, and 27%, respectively. High Composite scores indicate balanced EI: accurate perception, evaluation, and context-sensitive response production.

4. Model Evaluation and Empirical Findings

Eleven advanced LLMs are evaluated under uniform conditions (Default Mode, 200 conversations/model), including Anthropic Opus 4.6/4.7, Sonnet 4.6, Haiku 4.5; Google Gemini 3.1 Pro; OpenAI GPT-5.5/5.4; Mistral Large; xAI Grok 4; Alibaba Qwen 2.5 72B; and Xiaomi MiMo v2 Pro.

4.1 Turn- and Conversation-Level Results

• Composite scores range 50.1–54.3; Opus 4.6 leads (54.3), Grok 4 lags (50.1).
• Pairwise Acc shows highest discriminative power (η²=0.202): Opus 4.7 = 0.646, MiMo = 0.598. Five models display negative mean Kendall’s τ (preference rank anti-alignment).
• Behavioral Classification: Binary OM Acc uniformly ≥ 0.83 (Mistral Large = 0.862); Binary HP Acc varies 0.767–0.827, with the Opus family at the lower bound—demonstrating a “perspective gap.”
• Emotion Recognition: Emotion VA maximized by Gemini 3.1 (0.278), Haiku 4.5 (0.276). Emotion F₁ generally uninformative across top models (0.133–0.141); Qwen 2.5 outlier (0.106).
• Draft Judge: Most widely spread metric (η²=0.299), from Opus 4.6 (0.844) down to Qwen 2.5 (0.691).

4.2 Clustered and Conversation-Level Metrics

• Four-Branch MAE: Qwen 2.5, Mistral Large, and GPT-5.5 top this metric (≈81–82%); Opus 4.6/4.7 position lowest (73–76%), inverting overall Composite ranks.
• Free-Response Q1–Q3: Top performance (goal identification, emotion clarity, conversational fit) varies unpredictably with model and diverges from turn-level trends.

4.3 Decomposability of EI

Inter-task rank correlations are low:

• Emotion (F₁, VA): $Y_t$1
• Behavioral (Binary OM/HP): $Y_t$2
• Preference (Pairwise/Kendall): $Y_t$3
• Cross-cluster: near-zero or negative (e.g., Pairwise vs. Binary HP $Y_t$4).

This demonstrates that EI decomposes into recognition, behavioral inference, preference alignment, and language generation—skills that existing models manifest unevenly.

5. Diagnostic Insights: Failure Modes, Sensitivities, and Strengths

5.1 Failure Modes and Sensitivities

• Turn-drift: Binary OM Acc declines from early to late conversation (0.866→0.832; $Y_t$5), suggesting context maintenance challenges. Emotion VA remains stable or peaks mid-conversation.
• Participant Diagnosis: HPs with clinical mental health history elicit sharply lower Emotion VA (0.310→0.202) but higher Binary OM Acc (0.837→0.878 for AnxDep), indicating divergent patterns between affective inference and behavior judgment.
• Topic Difficulty: “Romantic Relationships” emerges as the most difficult topic (mean Composite ≈ 49.7), markedly lower than the next-most challenging (“Physical Health” ≈ 51.6).
• Mode Effects: Omniscient mode (providing HP psychometric profiles) offers negligible Composite improvement, except for Opus 4.7’s Pairwise Acc. Verbose (reasoning-trace) mode reduces Mistral Large’s Pairwise Acc.

5.2 Strengths and Attribution

AttuneBench facilitates model-level capability profiling, subgroup analyses (e.g., clinical vs. neurotypical, stable vs. mood-shifting conversations), and error decomposition (misrecognition of emotion, misalignment with HP preference, poor draft quality).

6. Limitations and Perspectives

6.1 Benchmarking Limitations

• Single-turn and synthetic benchmarks inadequately capture conversational mood shifts, adaptive tone, and real-time user preference shifts.
• Synthetic vignettes lack authentic emotional subtext and interactive feedback loops.
• Aggregate EI scoring without granular deconstruction obscures specific capability gaps.

6.2 Future Directions

Recommended directions include expanding HP diversity (geographically, linguistically, clinically), adaptive topic assignment, inclusion of multimodal cues (prosody, facial signals), streamlined or semi-automated annotation for longer interactions, and rigorous stewardship of benchmark leaderboards (strict Default Mode, prohibition on test set fine-tuning).

AttuneBench provides an open dataset (CC BY 4.0), evaluation runner, and scoring scripts (MIT License), supporting transparent and diagnostic progress tracking in LLM emotional intelligence research (Lubrano et al., 20 May 2026).