AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence

Abstract: Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others' emotional states, is central to human communication, and increasingly important to assess as LLMs assume conversational roles in everyday life. Existing EI benchmarks rely on synthetic prompts, single-turn cases, or third-party annotation. These approaches do not directly measure how models infer and respond to a participant's emotional state over the course of a real conversation. We introduce AttuneBench, a benchmark grounded in 200 genuine multi-turn human-model conversations in which participants conversed with anonymized LLMs and provided turn-by-turn annotations of their emotional state, the model's behavior, and their preferred responses. Across 11 evaluated models, we find that model rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent, indicating that emotionally intelligent behavior decomposes into separable capabilities. Preference alignment and response-quality judgments are substantially more model-discriminating than emotion-label accuracy. These results indicate that emotionally intelligent behavior requires predicting what kind of response a specific user wants in context, a distinction that aggregate scoring can obscure and that single-turn or synthetic formats cannot directly capture across turns. AttuneBench provides a framework for assessing each of these capabilities and for diagnosing model-specific strengths and failure modes in emotionally salient conversation.

• The paper introduces AttuneBench, a benchmark that quantitatively assesses LLM emotional intelligence through structured, multi-turn conversational interactions based on the Mayer-Salovey-Caruso framework.
• It employs a detailed annotation pipeline that includes PANAS ratings, turn-level mood tags, binary and pairwise evaluations to capture affective inference, behavioral classification, and preference prediction.
• Results reveal significant metric dissociations among models, with certain models excelling in preference prediction yet underperforming in first-person inference, underscoring the need for multi-metric evaluation.

AttuneBench: A Structured Benchmark for LLM Emotional Intelligence in Multi-Turn Human Conversation

Benchmark Motivation and Design

AttuneBench was conceptualized to rigorously diagnose LLMs on their operational emotional intelligence (EI) via genuine, multi-turn, participant-LMM conversations. Contemporary EI evaluation strategies rely on synthetic vignettes, single-turn prompts, or post-hoc annotation, which fail to capture the interactive, context-dependent dynamics inherent in lived emotional communication. Motivated by the increasing deployment of LLMs for personal advice, mental health reflection, and emotionally charged user engagement, AttuneBench addresses the deficiencies of existing benchmarks by introducing a dataset grounded in real conversational trajectories, annotated exhaustively by human participants (HPs).

The benchmark consists of 200 annotated conversations across 50 topic domains, with participants providing pre- and post-conversation affect ratings (PANAS), turn-level mood-shift tags, binary observed/preferred behavior labels, human-edited "golden" responses, and structured preference judgements. The annotation schema operationalizes the Mayer-Salovey-Caruso Four Branch Model: Perceiving, Understanding, Facilitating Thought, and Managing Emotion. Evaluation is performed by a secondary LLM (Evaluated Model, EM), which predicts annotation labels, generates responses, and is scored against HP ground truth via a composite metric (turn-level and conversation-level).

Data Collection Pipeline and Annotation Framework

The AttuneBench pipeline comprises three phases: (1) multi-turn data collection between HP and an anonymized Original Model (OM), (2) exhaustive annotation by HP for emotional state, model behavior, and preferences, and (3) evaluation of candidate EMs on their ability to replicate HP judgments and affective inferences.

Figure 1: Conversation interface for live multi-turn human-LLM interaction, foundational to AttuneBench's data collection protocol.

Figure 2: Assessment interface for post-conversation annotation, enabling rigorous grounding in Four Branch Model and structured conversational outcome evaluation.

HPs provide PANAS scores before and after conversations, continuous mood-shift tagging at each turn, binary and pairwise evaluation signals, and edit/model/human "golden" responses for each exchange. All annotation is immediately referenced to the conversation transcript, ensuring fidelity to live emotional context.

Figure 3: Turn-level binary judgment annotation capturing both observed and preferred model behaviors per emotional context.

Figure 4: Pairwise ranking interface for response preference comparison, fundamental to AttuneBench’s discriminative metrics.

Metrics and Scoring: Decomposition of EI Capabilities

AttuneBench's evaluation metrics span multiple axes:

• Affective Inference: PANAS-based mood tracking (Emotion F1, Emotion VA, continuous VAD embeddings).
• Behavioral Classification: Binary OM (observer-perspective), Binary HP (first-person/preference).
• Preference Prediction: Pairwise accuracy and Kendall $\tau$ for response ranking.
• Holistic Assessment: Conversation-wide alignment with participant goals, clarity, fit, and Four Branch scores.
• Composite Score: Weighted aggregation (emotion tracking 24%, evaluation quality 49%, holistic comprehension 27%) offering a normalization across operational EI dimensions.

The EM runner evaluates each model in three modes: Default (no participant profile), Verbose (with reasoning traces), and Omniscient (profile access).

Results: Discriminative Power and Capability Decoupling

Across 11 flagship and mid-tier models (Anthropic Opus, Sonnet, Haiku; OpenAI GPT-5.5/5.4; Google Gemini; Mistral Large; Grok; Qwen; Xiaomi MiMo), substantial findings are observed.

Figure 5: Per-model Composite and Pairwise Accuracy; Composite scores are compressed but preference prediction is highly discriminative across models.

Strong numerical findings:

• Composite Score range is narrow (50.1–54.3), yet 35/55 model pairs are significantly distinguished.
• Pairwise accuracy and Kendall $\tau$ expose sharp separation: Opus models lead in preference prediction ($0.646$, $+0.339$), MiMo third ($0.598$), while 5 models are anti-aligned with human preference (negative $\tau$ values).
• Binary Classification is led by Mistral Large (OM: $0.862$, HP: $0.827$); Opus models, despite topping Composite and Pairwise, are weakest on Binary HP (Opus 4.7: $0.767$).
• Emotion VA is led by Gemini (0.278) and Haiku (0.276), but model differentiation is modest for emotion tracking.

Model rankings are largely orthogonal across metrics: strong preference prediction does not imply strong behavioral classification or affect tracking. Notably, the Opus family dominates preference alignment but lags on first-person inference, and several models are systematically anti-aligned in preference ranking.

Figure 6: Grid of turn-level metrics showing operational metric separation among models.

Figure 7: Four Branch MAE per model—Understanding is uniformly most difficult, revealing a shared EI gradient.

Analysis: Structural Dissociations and Variance Sources

Statistical tests confirm metric separation (except Emotion F1), and the Composite should be interpreted as a tiered structure rather than linear rank. Metric correlation analysis shows negative correlation between pairwise preference accuracy and Binary HP accuracy ($r=-0.097$), underscoring the operational independence of preference prediction and first-person behavioral inference.

Figure 8: Pairwise win-rate matrix, highlighting Opus family’s advantage in preference prediction.

Post-conversation metrics invert Composite ranks: Qwen, Mistral, Gemini excel at holistic comprehension, while Opus models lead in turn-level but lag in conversation-wide synthesis. Ability profiles diverge: some models specialize in preference prediction, others in post-conversation goal and clarity identification.

Conversational difficulty (easy vs. hard) is a major source of variance (8–14 Composite points), but model ordering is difficulty-invariant. Models are less accurate in affect tracking for participants with mental health diagnoses; PANAS emotion prediction dropped $\tau$0 for these participants across all models.

Figure 9: Human-baseline pilot accuracy, human annotators outperform best EMs in preference prediction but bracket Binary HP accuracy.

Limitations and Practical Implications

AttuneBench's participant pool is small and demographically confined (US, native English speakers); annotation burden limited conversation length. The benchmark focuses exclusively on text-based interaction, omitting nonverbal cues critical to EI in real-world settings.

Intended for diagnostic research use, AttuneBench is not suitable as a clinical validation tool or a source for fine-tuned model training. High scores correlate with accurate in-context preference prediction, not with clinical safety or ethical deployment.

Practical deployment guidance entails no fine-tuning on AttuneBench, no test set contamination, and explicit mode reporting. The dataset, runner, and scoring pipeline are fully open-source under permissive licenses.

Implications for Future AI Research

AttuneBench's operational decomposition of EI and rigorous multi-turn conversational annotation expose critical decoupling of emotional capabilities in LLMs. Benchmarking EI cannot rely on aggregate scores or single metric proxies (e.g., emotion classification); per-metric reporting is essential to uncover model-specific strengths and failure modes.

From a theoretical standpoint, the dissociation between affective inference, behavioral classification, and preference prediction suggests that multi-component architectures or enhanced contextual modeling may be required for robust attunement in LLMs. The apparent anti-alignment of certain models in preference ranking signals a failure of current generation protocols to internalize participant-centric conversational dynamics.

Practically, as LLMs are increasingly deployed in emotionally sensitive domains (mental health, personal advice), fine-grained benchmarking as enabled by AttuneBench is vital for rigorous capability auditing and failure mode diagnosis. Extensions to multimodal data and more diverse participant pools are warranted for future versions.

Conclusion

AttuneBench rigorously resolves LLM emotional intelligence into separable operational capacities—affective tracking, behavioral inference, preference prediction, and response generation—demonstrating that these skills are largely independent in current models. Benchmarking emotional intelligence in LLMs thus requires structured, multi-modal evaluation, not aggregate scoring. AttuneBench offers a robust, extensible diagnostic platform to track the progression of EI in LLMs and to identify differential model strategies and failure points, aiding both research and practical deployment governance.