MME-Emotion: Multimodal Emotional Intelligence Benchmark

• MME-Emotion is a video-based benchmark for evaluating emotional intelligence by testing both emotion recognition and reasoning through multimodal inputs.
• It aggregates 6,500 video clips across 27 scenario types and eight emotional tasks, ensuring balanced evaluation with task-specific question-answering pairs.
• The benchmark employs a multi-agent evaluation framework that scores final predictions and stepwise reasoning using unified metrics like Rec-S, Rea-S, and CoT-S.

MME-Emotion is a video-based benchmark for evaluating emotional intelligence in multimodal LLMs (MLLMs). It is presented as the first benchmark that jointly measures whether a model can recognize emotions correctly and whether it can reason about the clues and triggering factors behind emotional states, under a design summarized as scalable capacity, diverse settings, and unified protocols (Zhang et al., 11 Aug 2025). In scope, it is described as the largest emotional intelligence benchmark for MLLMs, containing 6,500 video clips with task-specific question-answering pairs, spanning 27 distinct scenario types and eight emotional tasks (Zhang et al., 11 Aug 2025).

1. Concept and motivation

MME-Emotion was introduced to address two gaps in prior emotion benchmarks for MLLMs. First, existing benchmarks did not adequately test how well MLLMs generalize across diverse scenarios. Second, they largely ignored models’ ability to explain or reason through why an emotion is present (Zhang et al., 11 Aug 2025). The benchmark therefore targets both emotional understanding and emotional reasoning, rather than recognition accuracy alone.

The benchmark is motivated by the claim that prior emotion-related datasets and benchmarks for MLLMs were limited by inadequate scenario coverage, inconsistent evaluation protocols, and a narrow emphasis on recognition accuracy only (Zhang et al., 11 Aug 2025). This framing places MME-Emotion within a broader shift from closed, label-only affect recognition toward multimodal, explanation-sensitive evaluation. A plausible implication is that the benchmark is intended not merely to rank models, but to reveal where broad multimodal competence fails to become robust emotional intelligence.

Later work treated MME-Emotion precisely in that way. MER-R1 describes it as a holistic benchmark for evaluating emotional intelligence in multimodal LLMs, and contrasts it with MER-UniBench by emphasizing its LLM-based judging of both final predictions and reasoning processes (Han et al., 26 Jun 2026). OPPO likewise uses MME-Emotion as a held-out evaluation benchmark that reports recognition, reasoning, and CoT scores, specifically to test whether improvements in multimodal emotion reasoning translate into better judged explanation quality as well as better answers (Han et al., 24 Jun 2026).

2. Dataset composition and task structure

MME-Emotion is built by aggregating and resampling samples from multiple public datasets, including IEMOCAP, MELD, DFEW, MAFW, Context, MC-EIU, ARBEE, MER, CMU-MOSEI/MOSI-related sources, and Make (Zhang et al., 11 Aug 2025). To reduce annotation cost, the benchmark reuses videos and their associated emotion labels from those public resources. For long videos with emotional shifts, clips are segmented into shorter intervals using timestamps and intervals with consistent labels. To reduce data leakage risk, all samples are exclusively drawn from the test sets of the original sources (Zhang et al., 11 Aug 2025).

The benchmark is fundamentally video-based. It is designed around video clips rather than static images or text because emotional understanding often depends on temporal dynamics, vocal prosody, and evolving facial and body cues (Zhang et al., 11 Aug 2025). The paper states that clip durations are balanced, that all tasks contain at least 500 QA pairs, and that clips average more than 3.3 seconds (Zhang et al., 11 Aug 2025). Figure 1, as described in the paper, visualizes the eight task types, 27 distinct scenario types, and balanced distributions of question volume and video duration (Zhang et al., 11 Aug 2025).

The task construction is closed-set question answering. Because current MLLMs still lack sufficient emotional intelligence to handle open-ended tasks, candidate labels are included directly inside the prompt, and models are instructed to think step by step and place the final answer in <answer></answer> tags (Zhang et al., 11 Aug 2025). The prompts vary by task framing but share a unified protocol. For example, ER-Lab, ER-Wild, and Noise-ER ask the model to identify the most prominent emotion; FG-ER asks it to identify one or several emotions; ML-ER asks it to identify multiple emotions; SA asks it to identify the most prominent sentiment; FG-SA asks it to identify the most prominent fine-grained sentiment; and IR asks it to identify the most prominent intent (Zhang et al., 11 Aug 2025).

The benchmark taxonomy comprises eight tasks:

Task	Description
ER-Lab	emotion recognition in the lab
ER-Wild	emotion recognition in the wild
Noise-ER	emotion recognition under noise
FG-ER	fine-grained emotion recognition
ML-ER	multi-label emotion recognition
SA	sentiment analysis
FG-SA	fine-grained sentiment analysis
IR	intent recognition

This structure makes MME-Emotion broader than standard emotion recognition benchmarks. It includes not only emotion recognition, but also sentiment analysis and intent recognition, while keeping all tasks within one unified QA protocol (Zhang et al., 11 Aug 2025).

3. Evaluation framework and metrics

A major contribution of MME-Emotion is its multi-agent evaluation framework, which is designed to score not only final predictions but also reasoning quality in an annotation-free way (Zhang et al., 11 Aug 2025). The framework has at least three agent roles: the tested MLLM, a step agent that extracts concise reasoning steps from the model’s answer, and a judge agent that evaluates whether each extracted step is correct given visual clues, audio clues, labels, and predicted reasoning steps. There is also effectively an audio agent, used to extract audio clues because mainstream multimodal judge models cannot simultaneously process all modalities directly in the desired setup (Zhang et al., 11 Aug 2025).

The formal pipeline is described as

$$A = \text{MLLM}(Q,V), \qquad S = \text{Step-LLM}(P_s, A),$$

followed by

$$C_v = \text{Convert}(V), \qquad C_a = \text{Audio-LLM}(P_a, V),$$

$$\text{Rec-S}, \text{Rea-S} = \text{Judge-MLLM}(P_j, C_v, C_a, Y, S),$$

where $Q$ is the question, $V$ the video, $A$ the model answer, $S$ the extracted answer steps, $C_v$ the visual clues, $C_a$ the audio clues, and $Y$ the ground-truth emotion labels (Zhang et al., 11 Aug 2025). In implementation, the paper uses GPT-4.1 as the step agent, Qwen2-Audio as the audio agent, and GPT-4o as the judge agent (Zhang et al., 11 Aug 2025).

The benchmark defines three unified metrics. Recognition Score (Rec-S) evaluates the final prediction step. For single-label tasks, it is standard accuracy; for multi-label tasks, it is the ratio between the number of correctly predicted emotions and the total number of ground-truth emotions (Zhang et al., 11 Aug 2025). Reasoning Score (Rea-S) evaluates the non-final extracted reasoning steps as a binary classification problem at the step level and averages correctness across all reasoning steps (Zhang et al., 11 Aug 2025). Chain-of-Thought Score (CoT-S) combines recognition and reasoning: $$C_v = \text{Convert}(V), \qquad C_a = \text{Audio-LLM}(P_a, V),$$0 with default

$$C_v = \text{Convert}(V), \qquad C_a = \text{Audio-LLM}(P_a, V),$$1

Thus CoT-S is an equal-weight average of recognition and reasoning by default (Zhang et al., 11 Aug 2025).

The judge prompt instructs that each step gets 0 or 1; the last step gets 1 iff the predicted emotion matches the ground-truth label; prior reasoning steps are judged incorrect only when they clearly contradict visual or audio clues; and the number of scores must equal the number of extracted steps (Zhang et al., 11 Aug 2025). This yields a lenient but structured factual-consistency evaluation.

The automatic judging setup was validated with five human experts. They manually annotated 373 reasoning steps from 100 randomly sampled questions and answers. Agreement between GPT-based judging and experts was reported as Spearman’s rank correlation 0.9530, Cohen’s Kappa 0.8626, and ICC 0.9704 (Zhang et al., 11 Aug 2025). This suggests that the benchmark’s reasoning scores are intended as a practical surrogate for expert evaluation rather than a purely heuristic proxy.

4. Empirical findings from the benchmark

MME-Emotion evaluates 20 cutting-edge MLLMs under zero-shot conditions, including Qwen2-Audio, Audio-Reasoner, Qwen2-VL-7B, Qwen2-VL-72B, Qwen2.5-VL-7B, Qwen2.5-VL-72B, QVQ, Video-LLaVA, Video-LLaMA, Video-LLaMA2, Qwen2.5-Omni, Emotion-LLaMA, HumanOmni, R1-Omni, AffectGPT, GPT-4o, GPT-4.1, Gemini-2.0-Flash, Gemini-2.5-Flash, and Gemini-2.5-Pro (Zhang et al., 11 Aug 2025). The paper groups models conceptually into generalist versus specialist MLLMs and also distinguishes open-source from closed-source systems (Zhang et al., 11 Aug 2025).

The main overall result is that current MLLMs perform poorly on emotional intelligence as measured by the benchmark. The best model overall is Gemini-2.5-Pro with Rec-S $$C_v = \text{Convert}(V), \qquad C_a = \text{Audio-LLM}(P_a, V),$$2, Rea-S $$C_v = \text{Convert}(V), \qquad C_a = \text{Audio-LLM}(P_a, V),$$3, and CoT-S $$C_v = \text{Convert}(V), \qquad C_a = \text{Audio-LLM}(P_a, V),$$4 (Zhang et al., 11 Aug 2025). The next best CoT scores are Audio-Reasoner with 54.8 and GPT-4o with 53.8 (Zhang et al., 11 Aug 2025). The averages across all 20 models are 29.4% recognition, 49.5% reasoning, and 39.5% CoT (Zhang et al., 11 Aug 2025).

A salient pattern in the results is that many models achieve substantially higher reasoning scores than recognition scores. GPT-4o, for example, records Rec-S 27.8 and Rea-S 79.8; Qwen2.5-VL-72B records Rec-S 31.3 and Rea-S 75.7; Gemini-2.5-Pro records Rec-S 39.3 and Rea-S 72.7 (Zhang et al., 11 Aug 2025). The benchmark interprets this as evidence that models often produce plausible stepwise analyses while still failing at final emotional classification. Conversely, some specialist models such as Emotion-LLaMA and HumanOmni generate almost no explicit reasoning steps, resulting in very low Rea-S despite moderate recognition on some tasks (Zhang et al., 11 Aug 2025).

Task-level results reveal marked differences in difficulty. ER-Lab appears particularly difficult: the best CoT is Audio-Reasoner at 52.6, and the best recognition score is only 32.6 (Zhang et al., 11 Aug 2025). Noise-ER is much easier: Gemini-2.5-Pro reaches Rec-S 57.6, Rea-S 81.2, and CoT-S 69.4 (Zhang et al., 11 Aug 2025). FG-SA is described as especially challenging, with the best recognition score only 30.5 and the best CoT score 56.6 from Audio-Reasoner (Zhang et al., 11 Aug 2025). IR is also difficult, with the best recognition score only 30.7, even though reasoning can be very high, such as 93.4 for Qwen2.5-VL-72B (Zhang et al., 11 Aug 2025).

The benchmark also reports average step count and token count, observing that stronger models often generate longer reasoning chains. Gemini-2.5-Pro averages 5.1 steps and 538.6 tokens; Audio-Reasoner 5.0 steps and 356.8 tokens; GPT-4o 4.4 steps and 169.4 tokens (Zhang et al., 11 Aug 2025). The paper states that more steps correlate positively with better CoT performance (Zhang et al., 11 Aug 2025). This suggests that explicit reasoning is associated with stronger benchmark performance, although not with robust recognition in absolute terms.

5. Interpretation, failure modes, and benchmark significance

The benchmark supports several broader findings. First, both generalist and specialist routes to emotional intelligence remain viable. Generalist models such as Gemini and GPT are described as deriving emotional intelligence from generalized multimodal understanding, while specialist models such as R1-Omni and Audio-Reasoner can become competitive through domain-specific post-training adaptation (Zhang et al., 11 Aug 2025). The clearest specialist example is Audio-Reasoner, which matches or nearly matches top closed-source models despite using only audio and text (Zhang et al., 11 Aug 2025).

Second, current models do not effectively leverage multimodal signals. The paper highlights that Audio-Reasoner, using only audio and text, reaches 54.8 CoT, while several closed-source vision-LLMs that only use visual and text are also strong; meanwhile several omnimodal models underperform (Zhang et al., 11 Aug 2025). The authors suggest two reasons: emotional clues across modalities may be redundant or inconsistent, and existing omnimodal models lack robust emotional clue fusion mechanisms (Zhang et al., 11 Aug 2025).

Third, the benchmark identifies concrete failure modes. One is limited visual perception, especially for subtle facial expressions and body movements; Video-LLaMA2 and Qwen2.5-Omni are cited as examples that fail to distinguish subtle fear versus surprise or default to neutral when facial changes are slight (Zhang et al., 11 Aug 2025). Another is incomplete multimodal information use, especially missing audio: the paper describes a Gemini-2.5-Pro error case in which fear is conveyed clearly in the audio, but the model fails because it cannot jointly leverage audio and visual evidence (Zhang et al., 11 Aug 2025). This suggests that strong performance on generic multimodal tasks does not guarantee faithful integration of emotionally diagnostic cues.

These findings connect MME-Emotion to contemporaneous emotion benchmarks while preserving a distinct niche. EmoBench-M, for example, evaluates emotional intelligence through three dimensions and 13 scenarios, emphasizing foundational emotion recognition, conversational emotion understanding, and socially complex emotion analysis (Hu et al., 6 Feb 2025). MMEVerse aggregates twelve public emotion datasets into 18 evaluation benchmarks and supports both recognition and free-form multimodal emotion reasoning (Peng et al., 23 Jan 2026). By contrast, MME-Emotion is defined by its unified QA protocol, eight-task taxonomy, and explicit LLM-as-a-judge scoring of both prediction and reasoning (Zhang et al., 11 Aug 2025). A plausible implication is that MME-Emotion occupies a middle position between narrowly task-specific MER benchmarks and broader affective intelligence suites.

6. Subsequent use, reproducibility issues, and limitations

MME-Emotion became a standard held-out benchmark in later work on multimodal emotion reasoning. MER-R1 evaluates on MME-Emotion to show that slow-fast thinking synergy generalizes beyond MER-UniBench and reports mean Recognition 38.4, mean Reasoning 64.6, and mean CoT 51.5 under a replaced evaluator (Han et al., 26 Jun 2026). OPPO also evaluates on MME-Emotion and reports mean Recognition 31.0, mean Reasoning 68.1, and mean CoT 49.5, again under a replaced evaluator (Han et al., 24 Jun 2026). In both papers, the original GPT-4o evaluator used by the benchmark was unavailable, so all compared methods were re-evaluated using gemini-3.1-flash-lite-preview (Han et al., 26 Jun 2026, Han et al., 24 Jun 2026). This preserves fairness within those studies, but it also implies that MME-Emotion scores can depend materially on the judge model.

The original paper notes two explicit limitations. First, the benchmark does not classify sample difficulty levels. Second, although it uses videos from multilingual scenarios, it does not distinguish languages or analyze performance across linguistic contexts (Zhang et al., 11 Aug 2025). Additional benchmark constraints are also clear from the construction and protocol. Because the benchmark is built from existing public datasets, it inherits their scenario distributions and label conventions. Because it uses a closed-set QA format with provided candidate labels, it evaluates recognition and reasoning under constrained labeling rather than fully open-ended emotional understanding (Zhang et al., 11 Aug 2025).

The benchmark’s own empirical conclusion is deliberately conservative. It states that emotional intelligence in current MLLMs remains far from satisfactory, with the best-performing model achieving only 39.3% recognition score and 56.0% CoT score (Zhang et al., 11 Aug 2025). This conclusion is reinforced by later papers that continue to use MME-Emotion as a stress test for reasoning quality, modality utilization, and conflict robustness rather than as a solved benchmark (Han et al., 26 Jun 2026, Han et al., 24 Jun 2026). In that sense, MME-Emotion functions less as a terminal leaderboard and more as an instrument for exposing the gap between broad multimodal competence and genuine emotional understanding.