---
title: 'AttuneBench: EI Benchmark for LLMs'
url: https://www.emergentmind.com/topics/attunebench
type: topic
---

# AttuneBench: EI Benchmark for LLMs

AttuneBench is a comprehensive conversation-based benchmark designed to evaluate the emotional intelligence (EI) of large language models (LLMs) through multi-turn, authentic human–model interactions. Unlike benchmarks that rely on synthetic prompts, single-turn assessments, or third-party annotation, AttuneBench assesses models’ ability to infer, track, and respond to a participant’s emotional state over the evolving course of real conversations. It provides a multi-dimensional framework for diagnosis and comparison of LLM behavior in emotionally salient contexts, incorporating granular turn-level and conversation-level metrics [2605.21739].

## 1. Dataset Construction

### 1.1 Multi-turn Human–Model Conversations

AttuneBench comprises 200 genuine dialogues, each with a minimum of five human–LLM exchanges (mean ≈ 6 turns). The participant pool—termed human participants (HPs)—includes 11 U.S.-based native English speakers, balanced by gender and mental health history (6 neurotypical, 5 with at least one clinical diagnosis). Conversation topics are assigned randomly from a 50-topic taxonomy (e.g., Money, Romantic Relationships, Hobbies), ensuring non-redundant and naturally varying emotional valence. The emotional targets are not artificially elicited; variation arises from participants’ spontaneous attitudes and mood states.

Eight original chat models serve as anonymized conversational partners (e.g., GPT-4 Turbo, Claude 3.5 Sonnet, Gemini 2.5 Flash, Inflection 3), with HPs blind to the model identity. All data are collected in single sessions to capture live emotional dynamics. Pre- and post-conversation moods are indexed using the Positive and Negative Affect Schedule (PANAS).

### 1.2 Turn-by-Turn Annotation Workflow

At every turn, HPs generate rich annotations:
- **Mood-Shift Tags:** One or more PANAS emotions, each rated on an intensity scale from 1 to 7.
- **Behavioral Judgments:** Binary responses to a selection from 36 Mayer-Salovey-Caruso Four-Branch Model–grounded questions, split into observed behavior (did the model acknowledge emotion?) and preferred behavior (would more/less acknowledgment be desired?).
- **Response Comparison:** After models adapt to HP preference, three responses (original model, preference-informed model, and a human-drafted “golden” reply) are compared in 15 pairwise judgments covering emotional understanding, guidance, safety, and adaptation.
- **Golden Response:** HPs author their own ideal reply referencing, but not copying, model outputs.

At session end, HPs provide post-conversation PANAS mood, Four-Branch evaluations, and free-response satisfaction assessments (conversation goals, emotional clarity, fit), including optional qualitative commentary on conversational “fit.”

## 2. Task Definitions and Evaluation Protocol

AttuneBench operationalizes EI along four principal axes, each capturing distinct subskills of the emotionally intelligent conversational agent:

| Task                    | Input/Output Structure                   | Metric(s)                                 |
|-------------------------|------------------------------------------|-------------------------------------------|
| Emotion Recognition     | HP's turn, model predicts mood-shift tags| Precision, Recall, F₁, Valence-Arousal VA |
| Behavioral Classification| Per turn, binary Q&A (observed/preferred)| Binary Acc. (OM & HP perspective)         |
| Preference Prediction   | Rank three response variants             | Pairwise Acc., Kendall’s τ                |
| Response Quality (Judge)| Model draft rated on 4 axes by judge LLM | Mean judge score (0–1 normalized)         |

**Formal definitions:**
- *Emotion Recognition*: For each turn $t$, ground truth $Y_t$ (subset of 20 PANAS tags), prediction $\hat{Y}_t$. $F_{1,t}$ averaged over turns yields Emotion F₁. Valence–Arousal partial credit computed via NRC VAD lexicon, capturing graded similarity between predicted and actual affect.
- *Behavioral Classification*: For question $q$ at turn $t$, ground truth binary observed $o_{tq}$ and preferred $p_{tq}$. Evaluated models (EMs) predict $\hat{o}_{tq}$, $\hat{p}_{tq}$. Accuracies reported separately for OM and HP perspectives.
- *Preference Prediction*: EM ranks the three candidate responses; correct pairwise ordering fraction and Kendall’s τ algorithmically evaluated against HP’s preference total order.
- *Response-Quality Judgment*: Judge LLM provides four 1–7 ordinal ratings, normalized and averaged.

## 3. Metrics and Composite Scoring

AttuneBench employs a rigorous, multi-axial scoring framework:

### 3.1 Primary Metrics

- **Emotion F₁**: Macro F₁ for emotion tag prediction.
- **Emotion VA**: Mean valence-arousal similarity.
- **Binary OM / HP Acc**: Accuracy for observed and preferred behavioral classification.
- **Pairwise Acc / Kendall’s τ**: Agreement in ranking response variants.
- **Draft Judge**: Judge LLM mean normalized rating.
- **Four-Branch MAE**: Mean absolute error on post-conversation Four-Branch branch scores.
- **PANAS B-Adj**: Adjusted mean absolute error for post-conversation mood, correcting for model-wide bias.
- **Composite Score**: Weighted sum for holistic evaluation:

$$
\begin{align*}
E &= 0.5(Emotion\,F_1 + Emotion\,VA) \\
V &= (Binary\,OM + Binary\,HP + Pairwise\,Acc) / 3 \\
H &= (PANAS\,B\text{-}Adj + Four\text{-}Branch + \bar{Q}) / 3 \\
Composite &= 100 \cdot (0.24 E + 0.49 V + 0.27 H)
\end{align*}
$$

The E, V, and H pillars contribute 24%, 49%, and 27%, respectively. High Composite scores indicate balanced EI: accurate perception, evaluation, and context-sensitive response production.

## 4. Model Evaluation and Empirical Findings

Eleven advanced LLMs are evaluated under uniform conditions (Default Mode, 200 conversations/model), including Anthropic Opus 4.6/4.7, Sonnet 4.6, Haiku 4.5; Google Gemini 3.1 Pro; OpenAI GPT-5.5/5.4; Mistral Large; xAI Grok 4; Alibaba Qwen 2.5 72B; and Xiaomi MiMo v2 Pro.

### 4.1 Turn- and Conversation-Level Results

- **Composite scores** range 50.1–54.3; Opus 4.6 leads (54.3), Grok 4 lags (50.1).
- **Pairwise Acc** shows highest discriminative power (η²=0.202): Opus 4.7 = 0.646, MiMo = 0.598. Five models display negative mean Kendall’s τ (preference rank anti-alignment).
- **Behavioral Classification**: Binary OM Acc uniformly ≥ 0.83 (Mistral Large = 0.862); Binary HP Acc varies 0.767–0.827, with the Opus family at the lower bound—demonstrating a “perspective gap.”
- **Emotion Recognition**: Emotion VA maximized by Gemini 3.1 (0.278), Haiku 4.5 (0.276). Emotion F₁ generally uninformative across top models (0.133–0.141); Qwen 2.5 outlier (0.106).
- **Draft Judge**: Most widely spread metric (η²=0.299), from Opus 4.6 (0.844) down to Qwen 2.5 (0.691).

### 4.2 Clustered and Conversation-Level Metrics

- **Four-Branch MAE**: Qwen 2.5, Mistral Large, and GPT-5.5 top this metric (≈81–82%); Opus 4.6/4.7 position lowest (73–76%), inverting overall Composite ranks.
- **Free-Response Q1–Q3**: Top performance (goal identification, emotion clarity, conversational fit) varies unpredictably with model and diverges from turn-level trends.

### 4.3 Decomposability of EI

Inter-task rank correlations are low:
- Emotion (F₁, VA): $r=0.62$
- Behavioral (Binary OM/HP): $r=0.60$
- Preference (Pairwise/Kendall): $r=0.89$
- Cross-cluster: near-zero or negative (e.g., Pairwise vs. Binary HP $r\approx-0.10$).

This demonstrates that EI decomposes into recognition, behavioral inference, preference alignment, and language generation—skills that existing models manifest unevenly.

## 5. Diagnostic Insights: Failure Modes, Sensitivities, and Strengths

### 5.1 Failure Modes and Sensitivities

- **Turn-drift**: Binary OM Acc declines from early to late conversation (0.866→0.832; $p<0.001$), suggesting context maintenance challenges. Emotion VA remains stable or peaks mid-conversation.
- **Participant Diagnosis**: HPs with clinical mental health history elicit sharply lower Emotion VA (0.310→0.202) but higher Binary OM Acc (0.837→0.878 for AnxDep), indicating divergent patterns between affective inference and behavior judgment.
- **Topic Difficulty**: “Romantic Relationships” emerges as the most difficult topic (mean Composite ≈ 49.7), markedly lower than the next-most challenging (“Physical Health” ≈ 51.6).
- **Mode Effects**: Omniscient mode (providing HP psychometric profiles) offers negligible Composite improvement, except for Opus 4.7’s Pairwise Acc. Verbose (reasoning-trace) mode reduces Mistral Large’s Pairwise Acc.

### 5.2 Strengths and Attribution

AttuneBench facilitates model-level capability profiling, subgroup analyses (e.g., clinical vs. neurotypical, stable vs. mood-shifting conversations), and error decomposition (misrecognition of emotion, misalignment with HP preference, poor draft quality).

## 6. Limitations and Perspectives

### 6.1 Benchmarking Limitations

- **Single-turn and synthetic benchmarks** inadequately capture conversational mood shifts, adaptive tone, and real-time user preference shifts.
- **Synthetic vignettes** lack authentic emotional subtext and interactive feedback loops.
- **Aggregate EI scoring** without granular deconstruction obscures specific capability gaps.

### 6.2 Future Directions

Recommended directions include expanding HP diversity (geographically, linguistically, clinically), adaptive topic assignment, inclusion of multimodal cues (prosody, facial signals), streamlined or semi-automated annotation for longer interactions, and rigorous stewardship of benchmark leaderboards (strict Default Mode, prohibition on test set fine-tuning).

AttuneBench provides an open dataset (CC BY 4.0), evaluation runner, and scoring scripts (MIT License), supporting transparent and diagnostic progress tracking in LLM emotional intelligence research [2605.21739].

Source: https://www.emergentmind.com/topics/attunebench