---
title: 'MME-Emotion: Multimodal Emotional Intelligence Benchmark'
url: https://www.emergentmind.com/topics/mme-emotion
type: topic
---

# MME-Emotion: Multimodal Emotional Intelligence Benchmark

MME-Emotion is a video-based benchmark for evaluating emotional intelligence in multimodal large language models (MLLMs). It is presented as the first benchmark that jointly measures whether a model can recognize emotions correctly and whether it can reason about the clues and triggering factors behind emotional states, under a design summarized as scalable capacity, diverse settings, and unified protocols [2508.09210]. In scope, it is described as the largest emotional intelligence benchmark for MLLMs, containing 6,500 video clips with task-specific question-answering pairs, spanning 27 distinct scenario types and eight emotional tasks [2508.09210].

## 1. Concept and motivation

MME-Emotion was introduced to address two gaps in prior emotion benchmarks for MLLMs. First, existing benchmarks did not adequately test how well MLLMs generalize across diverse scenarios. Second, they largely ignored models’ ability to explain or reason through why an emotion is present [2508.09210]. The benchmark therefore targets both emotional understanding and emotional reasoning, rather than recognition accuracy alone.

The benchmark is motivated by the claim that prior emotion-related datasets and benchmarks for MLLMs were limited by inadequate scenario coverage, inconsistent evaluation protocols, and a narrow emphasis on recognition accuracy only [2508.09210]. This framing places MME-Emotion within a broader shift from closed, label-only affect recognition toward multimodal, explanation-sensitive evaluation. A plausible implication is that the benchmark is intended not merely to rank models, but to reveal where broad multimodal competence fails to become robust emotional intelligence.

Later work treated MME-Emotion precisely in that way. MER-R1 describes it as a holistic benchmark for evaluating emotional intelligence in multimodal large language models, and contrasts it with MER-UniBench by emphasizing its LLM-based judging of both final predictions and reasoning processes [2606.27652]. OPPO likewise uses MME-Emotion as a held-out evaluation benchmark that reports recognition, reasoning, and CoT scores, specifically to test whether improvements in multimodal emotion reasoning translate into better judged explanation quality as well as better answers [2606.25325].

## 2. Dataset composition and task structure

MME-Emotion is built by aggregating and resampling samples from multiple public datasets, including IEMOCAP, MELD, DFEW, MAFW, Context, MC-EIU, ARBEE, MER, CMU-MOSEI/MOSI-related sources, and Make [2508.09210]. To reduce annotation cost, the benchmark reuses videos and their associated emotion labels from those public resources. For long videos with emotional shifts, clips are segmented into shorter intervals using timestamps and intervals with consistent labels. To reduce data leakage risk, all samples are exclusively drawn from the test sets of the original sources [2508.09210].

The benchmark is fundamentally video-based. It is designed around video clips rather than static images or text because emotional understanding often depends on temporal dynamics, vocal prosody, and evolving facial and body cues [2508.09210]. The paper states that clip durations are balanced, that all tasks contain at least 500 QA pairs, and that clips average more than 3.3 seconds [2508.09210]. Figure 1, as described in the paper, visualizes the eight task types, 27 distinct scenario types, and balanced distributions of question volume and video duration [2508.09210].

The task construction is closed-set question answering. Because current MLLMs still lack sufficient emotional intelligence to handle open-ended tasks, candidate labels are included directly inside the prompt, and models are instructed to think step by step and place the final answer in `<answer></answer>` tags [2508.09210]. The prompts vary by task framing but share a unified protocol. For example, ER-Lab, ER-Wild, and Noise-ER ask the model to identify the most prominent emotion; FG-ER asks it to identify one or several emotions; ML-ER asks it to identify multiple emotions; SA asks it to identify the most prominent sentiment; FG-SA asks it to identify the most prominent fine-grained sentiment; and IR asks it to identify the most prominent intent [2508.09210].

The benchmark taxonomy comprises eight tasks:

| Task | Description |
|---|---|
| ER-Lab | emotion recognition in the lab |
| ER-Wild | emotion recognition in the wild |
| Noise-ER | emotion recognition under noise |
| FG-ER | fine-grained emotion recognition |
| ML-ER | multi-label emotion recognition |
| SA | sentiment analysis |
| FG-SA | fine-grained sentiment analysis |
| IR | intent recognition |

This structure makes MME-Emotion broader than standard emotion recognition benchmarks. It includes not only emotion recognition, but also sentiment analysis and intent recognition, while keeping all tasks within one unified QA protocol [2508.09210].

## 3. Evaluation framework and metrics

A major contribution of MME-Emotion is its multi-agent evaluation framework, which is designed to score not only final predictions but also reasoning quality in an annotation-free way [2508.09210]. The framework has at least three agent roles: the tested MLLM, a step agent that extracts concise reasoning steps from the model’s answer, and a judge agent that evaluates whether each extracted step is correct given visual clues, audio clues, labels, and predicted reasoning steps. There is also effectively an audio agent, used to extract audio clues because mainstream multimodal judge models cannot simultaneously process all modalities directly in the desired setup [2508.09210].

The formal pipeline is described as
\[
A = \text{MLLM}(Q,V), \qquad S = \text{Step-LLM}(P_s, A),
\]
followed by
\[
C_v = \text{Convert}(V), \qquad C_a = \text{Audio-LLM}(P_a, V),
\]
\[
\text{Rec-S}, \text{Rea-S} = \text{Judge-MLLM}(P_j, C_v, C_a, Y, S),
\]
where \(Q\) is the question, \(V\) the video, \(A\) the model answer, \(S\) the extracted answer steps, \(C_v\) the visual clues, \(C_a\) the audio clues, and \(Y\) the ground-truth emotion labels [2508.09210]. In implementation, the paper uses GPT-4.1 as the step agent, Qwen2-Audio as the audio agent, and GPT-4o as the judge agent [2508.09210].

The benchmark defines three unified metrics. Recognition Score (Rec-S) evaluates the final prediction step. For single-label tasks, it is standard accuracy; for multi-label tasks, it is the ratio between the number of correctly predicted emotions and the total number of ground-truth emotions [2508.09210]. Reasoning Score (Rea-S) evaluates the non-final extracted reasoning steps as a binary classification problem at the step level and averages correctness across all reasoning steps [2508.09210]. Chain-of-Thought Score (CoT-S) combines recognition and reasoning:
\[
\text{CoT-S} = \alpha \times \text{Rec-S} + (1 - \alpha) \times \text{Rea-S},
\]
with default
\[
\alpha = 0.5.
\]
Thus CoT-S is an equal-weight average of recognition and reasoning by default [2508.09210].

The judge prompt instructs that each step gets 0 or 1; the last step gets 1 iff the predicted emotion matches the ground-truth label; prior reasoning steps are judged incorrect only when they clearly contradict visual or audio clues; and the number of scores must equal the number of extracted steps [2508.09210]. This yields a lenient but structured factual-consistency evaluation.

The automatic judging setup was validated with five human experts. They manually annotated 373 reasoning steps from 100 randomly sampled questions and answers. Agreement between GPT-based judging and experts was reported as Spearman’s rank correlation 0.9530, Cohen’s Kappa 0.8626, and ICC 0.9704 [2508.09210]. This suggests that the benchmark’s reasoning scores are intended as a practical surrogate for expert evaluation rather than a purely heuristic proxy.

## 4. Empirical findings from the benchmark

MME-Emotion evaluates 20 cutting-edge MLLMs under zero-shot conditions, including Qwen2-Audio, Audio-Reasoner, Qwen2-VL-7B, Qwen2-VL-72B, Qwen2.5-VL-7B, Qwen2.5-VL-72B, QVQ, Video-LLaVA, Video-LLaMA, Video-LLaMA2, Qwen2.5-Omni, Emotion-LLaMA, HumanOmni, R1-Omni, AffectGPT, GPT-4o, GPT-4.1, Gemini-2.0-Flash, Gemini-2.5-Flash, and Gemini-2.5-Pro [2508.09210]. The paper groups models conceptually into generalist versus specialist MLLMs and also distinguishes open-source from closed-source systems [2508.09210].

The main overall result is that current MLLMs perform poorly on emotional intelligence as measured by the benchmark. The best model overall is Gemini-2.5-Pro with Rec-S \(= 39.3\), Rea-S \(= 72.7\), and CoT-S \(= 56.0\) [2508.09210]. The next best CoT scores are Audio-Reasoner with 54.8 and GPT-4o with 53.8 [2508.09210]. The averages across all 20 models are 29.4% recognition, 49.5% reasoning, and 39.5% CoT [2508.09210].

A salient pattern in the results is that many models achieve substantially higher reasoning scores than recognition scores. GPT-4o, for example, records Rec-S 27.8 and Rea-S 79.8; Qwen2.5-VL-72B records Rec-S 31.3 and Rea-S 75.7; Gemini-2.5-Pro records Rec-S 39.3 and Rea-S 72.7 [2508.09210]. The benchmark interprets this as evidence that models often produce plausible stepwise analyses while still failing at final emotional classification. Conversely, some specialist models such as Emotion-LLaMA and HumanOmni generate almost no explicit reasoning steps, resulting in very low Rea-S despite moderate recognition on some tasks [2508.09210].

Task-level results reveal marked differences in difficulty. ER-Lab appears particularly difficult: the best CoT is Audio-Reasoner at 52.6, and the best recognition score is only 32.6 [2508.09210]. Noise-ER is much easier: Gemini-2.5-Pro reaches Rec-S 57.6, Rea-S 81.2, and CoT-S 69.4 [2508.09210]. FG-SA is described as especially challenging, with the best recognition score only 30.5 and the best CoT score 56.6 from Audio-Reasoner [2508.09210]. IR is also difficult, with the best recognition score only 30.7, even though reasoning can be very high, such as 93.4 for Qwen2.5-VL-72B [2508.09210].

The benchmark also reports average step count and token count, observing that stronger models often generate longer reasoning chains. Gemini-2.5-Pro averages 5.1 steps and 538.6 tokens; Audio-Reasoner 5.0 steps and 356.8 tokens; GPT-4o 4.4 steps and 169.4 tokens [2508.09210]. The paper states that more steps correlate positively with better CoT performance [2508.09210]. This suggests that explicit reasoning is associated with stronger benchmark performance, although not with robust recognition in absolute terms.

## 5. Interpretation, failure modes, and benchmark significance

The benchmark supports several broader findings. First, both generalist and specialist routes to emotional intelligence remain viable. Generalist models such as Gemini and GPT are described as deriving emotional intelligence from generalized multimodal understanding, while specialist models such as R1-Omni and Audio-Reasoner can become competitive through domain-specific post-training adaptation [2508.09210]. The clearest specialist example is Audio-Reasoner, which matches or nearly matches top closed-source models despite using only audio and text [2508.09210].

Second, current models do not effectively leverage multimodal signals. The paper highlights that Audio-Reasoner, using only audio and text, reaches 54.8 CoT, while several closed-source vision-language models that only use visual and text are also strong; meanwhile several omnimodal models underperform [2508.09210]. The authors suggest two reasons: emotional clues across modalities may be redundant or inconsistent, and existing omnimodal models lack robust emotional clue fusion mechanisms [2508.09210].

Third, the benchmark identifies concrete failure modes. One is limited visual perception, especially for subtle facial expressions and body movements; Video-LLaMA2 and Qwen2.5-Omni are cited as examples that fail to distinguish subtle fear versus surprise or default to neutral when facial changes are slight [2508.09210]. Another is incomplete multimodal information use, especially missing audio: the paper describes a Gemini-2.5-Pro error case in which fear is conveyed clearly in the audio, but the model fails because it cannot jointly leverage audio and visual evidence [2508.09210]. This suggests that strong performance on generic multimodal tasks does not guarantee faithful integration of emotionally diagnostic cues.

These findings connect MME-Emotion to contemporaneous emotion benchmarks while preserving a distinct niche. EmoBench-M, for example, evaluates emotional intelligence through three dimensions and 13 scenarios, emphasizing foundational emotion recognition, conversational emotion understanding, and socially complex emotion analysis [2502.04424]. MMEVerse aggregates twelve public emotion datasets into 18 evaluation benchmarks and supports both recognition and free-form multimodal emotion reasoning [2601.16449]. By contrast, MME-Emotion is defined by its unified QA protocol, eight-task taxonomy, and explicit LLM-as-a-judge scoring of both prediction and reasoning [2508.09210]. A plausible implication is that MME-Emotion occupies a middle position between narrowly task-specific MER benchmarks and broader affective intelligence suites.

## 6. Subsequent use, reproducibility issues, and limitations

MME-Emotion became a standard held-out benchmark in later work on multimodal emotion reasoning. MER-R1 evaluates on MME-Emotion to show that slow-fast thinking synergy generalizes beyond MER-UniBench and reports mean Recognition 38.4, mean Reasoning 64.6, and mean CoT 51.5 under a replaced evaluator [2606.27652]. OPPO also evaluates on MME-Emotion and reports mean Recognition 31.0, mean Reasoning 68.1, and mean CoT 49.5, again under a replaced evaluator [2606.25325]. In both papers, the original GPT-4o evaluator used by the benchmark was unavailable, so all compared methods were re-evaluated using gemini-3.1-flash-lite-preview [2606.27652][2606.25325]. This preserves fairness within those studies, but it also implies that MME-Emotion scores can depend materially on the judge model.

The original paper notes two explicit limitations. First, the benchmark does not classify sample difficulty levels. Second, although it uses videos from multilingual scenarios, it does not distinguish languages or analyze performance across linguistic contexts [2508.09210]. Additional benchmark constraints are also clear from the construction and protocol. Because the benchmark is built from existing public datasets, it inherits their scenario distributions and label conventions. Because it uses a closed-set QA format with provided candidate labels, it evaluates recognition and reasoning under constrained labeling rather than fully open-ended emotional understanding [2508.09210].

The benchmark’s own empirical conclusion is deliberately conservative. It states that emotional intelligence in current MLLMs remains far from satisfactory, with the best-performing model achieving only 39.3% recognition score and 56.0% CoT score [2508.09210]. This conclusion is reinforced by later papers that continue to use MME-Emotion as a stress test for reasoning quality, modality utilization, and conflict robustness rather than as a solved benchmark [2606.27652][2606.25325]. In that sense, MME-Emotion functions less as a terminal leaderboard and more as an instrument for exposing the gap between broad multimodal competence and genuine emotional understanding.

Source: https://www.emergentmind.com/topics/mme-emotion