SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models

Abstract: As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human-AI communication. However, existing evaluations of machine emotional intelligence assess reasoning exclusively through isolated text or passive acoustic perception, overlooking the complex cross-modal reasoning required for active, multi-turn dialogue. We introduce \textsc{SpeechEQ}, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech-LLMs (SLMs). The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory, along with a multi-turn evaluation protocol measured by our proposed Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments show limitations in how both existing Speech Emotion Recognition and end-to-end Speech-LLMs understand and apply paralinguistic cues through speech. While end-to-end architectures outperform cascaded systems, \textsc{SpeechEQ} reveals that current multimodal models remain bottlenecked by a text-reliant modality shortcut,'' an alignment-inducedsafety trap,'' and ``contextual amnesia,'' highlighting the barriers to truly emotionally aware AI. Our benchmark can be accessed at https://huggingface.co/datasets/SpeechEQ/SpeechEQ and demo page at https://binomial14.github.io/speecheq-demo/

• The paper introduces SpeechEQ, a benchmark that evaluates voice AI's emotional intelligence by mapping EQ-i 2.0 subscales to observable acoustic behaviors.
• It employs a forced-choice, multi-turn dialogue protocol to isolate acoustic cues while assessing paralinguistic reasoning with robust accuracy metrics.
• Empirical results expose model limitations such as modality shortcut, affective flattening, and contextual amnesia, guiding future AI advancements.

Benchmarking Emotional Intelligence in Speech-LLMs: An Expert Analysis of SpeechEQ (2606.25990)

Motivation and Framework Design

SpeechEQ advances the evaluation of emotional intelligence (EQ) in voice-based conversational models, addressing the deficit of rigorous, multimodal assessments in this domain. Existing approaches predominantly measure emotional reasoning in either text-only environments or through passive emotion classification, neglecting the interactive complexity and cross-modal pragmatics inherent to real spoken dialogues. SpeechEQ operationalizes EQ evaluation using the clinically validated EQ-i 2.0 framework, decomposing emotional intelligence into fifteen subscales that are mapped onto observable acoustic behaviors.

The framework isolates paralinguistic reasoning by employing forced-choice tasks where response options share identical transcripts but vary in prosody and delivery, thus neutralizing lexical cues and enforcing reliance on acoustic characteristics. Multi-turn conversational arcs are designed to escalate emotional stakes, thereby facilitating evaluation of sustained sociolinguistic tracking. The pipeline synthesizes 2,265 dialogues (42.37 hours), systematically balanced across EQ subscales, and subjects them to stringent automated and human validation procedures. SpeechEQ formalizes its protocol with the Spoken Emotional Quotient (SEQ), a trajectory-level metric that leverages robust standardization techniques (median absolute deviation) for durable benchmarking and interpretable cohort-level ranking.

Evaluation Protocol and SEQ Metric

SpeechEQ's evaluation is structured as a two-round forced-choice task embedded within a six-turn dialogue. At critical moments (Turns 4 and 6), models must select between acoustically distinct but semantically identical responses, solely based on the contextual appropriateness of paralinguistic cues. The main metrics are:

• $Acc_1$: single-turn accuracy at the first evaluation turn.
• $Acc_2$: single-turn accuracy at the second evaluation turn.
• $Acc_{traj}$: trajectory accuracy across both turns, with random baselines of 50%, 50%, and 25%, respectively.

The SEQ score transforms $Acc_{traj}$ into a normalized, cohort-relative scale using MAD-based Z normalization and clinical psychometric scaling. This metric strictly bounds scores to $\pm 4\sigma$ within a mean of 100 and deviation of 15, consistent with standardized cognitive assessment conventions.

SEQ is empirically validated to correlate strongly with human sociolinguistic judgments (Spearman $\rho = 0.943$, $p=0.005$), outperforming traditional discrete accuracy metrics as a proxy for perceived emotional intelligence.

Empirical Findings and Model Failure Modes

Benchmarking spans cascaded pipelines (ASR + SER + text LLM) and end-to-end SLMs. End-to-end architectures, notably Qwen3-Omni-30B, attain superior SEQ scores yet exhibit core limitations:

• Modality Shortcut: Models systematically privilege text-based cues, failing when semantic information is neutralized and context is exclusively encoded in acoustics. This bias persists even in multimodal settings where identical transcripts force reliance on prosody for meaning disambiguation, revealing that acoustic reasoning remains a secondary channel.
• Affective Flattening: Alignment strategies induce a safety trap, leading to suppression of emotionally expressive tones in favor of affect-neutral or low-arousal prosody. Tasks necessitating high-arousal, boundary-setting, or assertive emotional expression are disproportionately compromised, with catastrophic failures in stress management and conflict resolution scenarios.
• Contextual Amnesia: Trajectory accuracy degrades significantly ($0.785 \rightarrow 0.708$ for Qwen3-Omni-30B) across multi-turn interactions due to inadequate affective state tracking and insufficient context retention. Direct inference with ground-truth history only partially ameliorates this, and non-uniform recovery across EQ subscales implicates complex cross-modal attention dynamics beyond mere memory leak.

Deployment analysis reveals major operational trade-offs: higher-performing models (30B scale) incur substantially increased inference latency and API/GPU costs, constraining real-time deployment of emotionally resonant voice agents.

Persona Conditioning and EQ Subscale Asymmetry

Persona conditioning exposes pronounced asymmetry in model ability to simulate distinct EQ traits. Adaptive persona prompts modestly improve performance, while deficit prompts (e.g., toxic micromanager, chronic stress) severely degrade SEQ, particularly in stress management subscales (SEQ: 74.90 vs. 147.26 baseline).

Deficits aligned with RLHF-induced behavioral priors (e.g., low assertiveness, reduced self-expression) cause minimal performance loss, reflecting congruence with default model politeness. Conversely, subscales requiring regulation of high-arousal affect or assertive/risk-intensive responses experience substantial degradation, indicating entrenched alignment-induced expressive constraints.

Implications and Future Directions

SpeechEQ reframes emotional intelligence in speech as a closed-loop, multimodal reasoning problem and sets forth a rigorous diagnostic platform for emotionally competent agentic systems. Key implications include:

• Architectural Prioritization of Acoustics: Mitigating the modality shortcut necessitates designs that elevate acoustic signals to parity with lexical content in sociolinguistic reasoning modules.
• Alignment Decoupling: Current RLHF and safety mechanisms elicit affective flattening and must be revised to dissociate harmlessness from emotional expressivity, establishing pathways for nuanced, high-arousal prosodic deployment without jeopardizing safety.
• Long-Horizon Affective Tracking: Addressing contextual amnesia requires advancements in context window architectures, tokenization strategies, and memory mechanisms attuned to dense audio stream processing under long-term conversational settings.
• Psychometric Grounding: The use of EQ-i 2.0-derived behavioral subscales consolidates benchmarking validity, enabling extrapolation to clinical, educational, and high-risk deployment settings.

Further research should integrate agentic profile measurement, long-context personalization, and robust cross-modal interaction modeling for scalable, emotionally intelligent voice agents.

Conclusion

SpeechEQ introduces a benchmark and evaluation toolkit for multimodal emotional intelligence in SLMs, grounded in the EQ-i 2.0 framework and realized via a neutralized semantic-acoustic evaluation protocol. End-to-end multimodal models demonstrate clear superiority over cascaded pipelines, yet the benchmark surfaces persistent modality, alignment, and contextual tracking bottlenecks. SpeechEQ’s design and empirical insights prompt a new research agenda centered on paralinguistic reasoning, affective alignment, and sustained conversational competence, laying the foundation for future progress in emotionally aware AI systems.