EmoBench-Reddit: Hierarchical Multimodal Benchmark

• The paper introduces a hierarchical benchmark that advances multimodal emotion understanding by testing models from low-level perception to high-level cognition.
• The dataset consists of 350 curated Reddit samples with images, texts, and emotion labels (sad, humor, sarcasm, happy), ensuring quality via Reddit flair and manual verification.
• Evaluation highlights a clear gap between strong perceptual accuracy and poor high-level emotional inference, especially for sarcasm and deep cognitive tasks.

Searching arXiv for the benchmark paper and closely related Reddit emotion/evaluation work to ground the article in current literature. EmoBench-Reddit is a hierarchical benchmark for evaluating the emotional intelligence of multimodal LLMs on real Reddit posts that combine image and text. Introduced as a benchmark for multimodal emotion understanding, it is designed around a specific diagnostic question: whether a model that can recognize visual facts can also infer what a post means emotionally, why it was posted, and which contextual cues give it its emotional force. Its central contribution is a staged evaluation design that moves from low-level perception to high-level cognition over 350 curated Reddit samples labeled as sad, humor, sarcasm, or happy (Li et al., 14 Sep 2025).

1. Definition and conceptual scope

EmoBench-Reddit is framed around a perceived deficiency in existing multimodal evaluation practice. The benchmark argues that prevailing benchmarks for multimodal LLMs predominantly test objective capabilities such as OCR, object recognition, spatial reasoning, or captioning, while providing inadequate coverage of subjective emotion understanding. The benchmark therefore targets “emotional intelligence” in a multimodal setting, with particular emphasis on posts whose meaning depends on the interaction between image content, user-provided text, and social context (Li et al., 14 Sep 2025).

The benchmark’s organizing concept is the “perception-cognition gap.” In this formulation, a model may correctly identify colors, objects, and basic scene elements, yet still fail to infer sadness, humor, or sarcasm in context. This gap is especially salient for Reddit-derived material, where emotional meaning is often implicit, culturally loaded, and dependent on intent rather than literal semantics. Sarcasm is presented as the clearest case: it requires recognition of contradiction between surface meaning and intended meaning, together with background knowledge and inferred communicative purpose (Li et al., 14 Sep 2025).

A plausible implication is that EmoBench-Reddit is not primarily a sentiment benchmark in the narrow classification sense. It is instead a benchmark for staged multimodal interpretation, where emotion recognition is embedded within broader visual grounding, scene reasoning, and context integration.

2. Dataset composition and curation pipeline

The dataset contains 350 carefully curated samples from Reddit. Each sample consists of an image, the associated user-provided text, and one of four emotion categories: sad, humor, sarcasm, and happy. The source communities are subreddits strongly associated with those emotions, including r/funny, r/happy, r/sadness, and r/sarcasm (Li et al., 14 Sep 2025).

The label source is the subreddit flair system rather than arbitrary ex post annotation. The construction procedure retains posts whose author explicitly used a flair consistent with the emotion or theme of the subreddit, removes violent, pornographic, hateful, or otherwise inappropriate content, manually verifies that image and text are tightly related and jointly express the intended emotion, and discards posts where the image is mostly text so that the visual modality remains central. The abstract states that annotation quality was ensured through “AI assistance (Claude 4) and manual verification,” and the detailed construction process specifies that open-ended answers were first generated by Claude 4 and then reviewed and revised by at least two human annotators. For multiple-choice items, annotators created the questions, and a separate cross-validation round had other annotators answer them independently to remove ambiguity and reach consensus (Li et al., 14 Sep 2025).

The benchmark is therefore curated around multimodal coherence rather than raw platform harvesting. This matters because the intended evaluation target is not only emotion labeling, but also whether a model can recover the emotionally relevant relation between what is shown and what is said.

Component	Description	Function
Sample unit	Image + user-provided text + emotion label	Multimodal evaluation item
Emotion set	sad, humor, sarcasm, happy	Target emotional categories
Label source	Reddit flairs with manual validation	Weak supervision plus confirmation
Quality control	Claude 4 assistance + manual verification	Annotation refinement

This design distinguishes the benchmark from text-only Reddit emotion datasets. It also suggests a preference for high-precision benchmark items over large-scale coverage.

3. Hierarchical task architecture

Each sample contains seven questions in total: six multiple-choice questions and one open-ended question. These are arranged as a hierarchy from perception to cognition. The benchmark describes three perception-related tasks and three cognition-related tasks, with the open-ended task used for the deepest perception-level description (Li et al., 14 Sep 2025).

The perception side is defined as follows. Perception 1 is color recognition, such as identifying the primary color of a prominent object. Perception 2 has two subparts: P2(a), object presence, asks whether a specific object is in the image; and P2(b), object localization, asks where that object is located if it is present. Perception 3 is an open-ended image description task requiring an objective, comprehensive description of visible content. This is treated as a foundational stage that checks whether the model can perceive accurately and richly before deeper interpretation (Li et al., 14 Sep 2025).

The cognition side progressively increases inferential load. Cognition 1 is simple inference, requiring a basic logical inference from the scene. Cognition 2 is intent recognition, where the model must infer the emotion or intent the author is trying to convey and connect multimodal evidence to the intended emotional label. Cognition 3 is deep reasoning, requiring integration of image and text in a context-dependent way, often with background knowledge, empathy, or irony understanding. The benchmark’s sarcasm example—an image of a candle inside a box labeled as an “energy-saving lightbulb” with the text “My new energy-saving bulb arrived”—is meant to force recognition of mismatch and intended sarcasm rather than literal meaning (Li et al., 14 Sep 2025).

This hierarchy operationalizes an important methodological claim: higher-level emotional understanding should be evaluated only after lower-level perceptual competence is established. That claim is made explicit in the evaluation protocol.

4. Evaluation protocol and scoring methodology

For the multiple-choice questions, the evaluation metric is plain accuracy, defined as the proportion of correctly answered questions. For the open-ended Perception 3 task, exact-match evaluation is not used. Instead, the benchmark adopts a hybrid scoring protocol: the generated description and the ground-truth description are converted into embeddings and compared by cosine similarity; Claude 4 is then used as an LLM-as-a-judge to score semantic and content similarity on a scale from 0 to 1; and the two scores are combined into a weighted average. The benchmark states that if this composite score is above 0.75, the answer is considered correct, and the final metric for the task is the pass rate (Li et al., 14 Sep 2025).

Two additional evaluation strategies are defined. In gated evaluation, perception is scored first, and cognition is evaluated for a sample only if the model’s average accuracy on the perception tasks for that item is above 75%. In contextual evaluation, the model receives each new question together with the previous question-answer history, simulating a conversation and testing whether context improves performance. The reported results are mainly based on gated evaluation, because the paper treats it as the clearest test of the dependence of cognition on perception (Li et al., 14 Sep 2025).

Nine multimodal LLMs are evaluated in zero-shot mode with a uniform concise instruction format: GPT-5, GPT-4o, GPT-4V, Gemini-2.5-pro, Gemini-2.0-flash, GLM-4v-plus, Qwen-vl-max, Qwen-vl-7b, and Deepseek-vl-7b. Closed-source models are accessed through their official APIs (Li et al., 14 Sep 2025).

The evaluation design is notable because it mixes conventional exact-answer scoring with a thresholded semantic evaluation for open-ended output, while also using gated execution to probe task dependency. This suggests that the benchmark is intended as a diagnostic instrument rather than a single aggregate leaderboard.

5. Empirical findings and diagnostic interpretation

The reported results show a consistent performance hierarchy. GPT-5, Gemini-2.5-pro, and GPT-4o form the top tier and lead across most emotions and task levels, especially on basic perception and simple cognition. The paper gives task-specific examples: GPT-5 reaches 0.97 on Perception 1 for sad, while GLM-4v-plus obtains the highest score on Perception 1 for humor at 0.95 (Li et al., 14 Sep 2025).

The dominant empirical pattern, however, is not the ranking among top models but the gap between task families. Perception tasks are generally strong across the board, whereas cognition tasks exhibit a marked performance drop, especially Cognition 3. The authors repeatedly interpret this decline as the benchmark’s main finding: current multimodal systems are substantially better at seeing than at understanding emotion in context (Li et al., 14 Sep 2025).

Emotion category strongly conditions difficulty. Sarcasm is identified as the hardest category for all models, particularly at the deep reasoning level. On Cognition 3 for sarcasm, the best score is only 0.63, achieved by GPT-5, and some models score as low as 0.27. Sad is also difficult, plausibly because it often requires empathy and narrative inference. Humor and happy are comparatively easier, which the paper attributes to more explicit visual or textual cues and stronger representation in training data (Li et al., 14 Sep 2025).

The benchmark therefore functions diagnostically in two senses. First, it isolates where multimodal systems fail: not in low-level recognition, but in intent, empathy, and irony-sensitive integration. Second, it reveals model-capacity effects. Smaller open-source models such as Qwen-vl-7b and Deepseek-vl-7b show clear bottlenecks on complex cognitive tasks, which the authors interpret as evidence that advanced emotion understanding depends on both model capacity and training scale (Li et al., 14 Sep 2025).

A plausible implication is that benchmark success on visual question answering or captioning should not be treated as evidence of emotionally competent multimodal reasoning.

6. Relation to adjacent Reddit emotion and affect benchmarks

EmoBench-Reddit belongs to a broader family of Reddit-based affective evaluation resources, but its task formulation is distinct. The closest contrast is the text-only study “Emotion Detection in Reddit: Comparative Study of Machine Learning and Deep Learning Techniques,” which uses GoEmotions Reddit comments, maps 27 emotions into Ekman’s six basic emotions plus neutral, and compares six machine learning models, three ensemble methods, an LSTM, and EmoBERTa. In that setting, the strongest result comes from a Stacking Classifier over TF-IDF features rather than from a pretrained transformer baseline (Alaeddini, 2024). That work addresses single-label text classification on Reddit comments; EmoBench-Reddit instead evaluates multimodal staged reasoning over image-text posts.

It is also related to the broader “EmoBench” benchmark for LLMs, which is theory-driven, bilingual, and organized around Emotional Understanding and Emotional Application rather than multimodal Reddit interpretation. That benchmark is hand-crafted and aims to move beyond emotion recognition toward implicit cause reasoning, perspective-taking, and socially appropriate action selection (Sabour et al., 2024). EmoBench-Reddit can be read as extending part of that agenda into a Reddit-native multimodal setting.

Other Reddit resources address neighboring constructs rather than the same benchmark target. APPReddit is a corpus of 500 Reddit posts segmented into 1,091 events and annotated for five appraisal dimensions derived from Roseman’s theory: Unexpectedness, Consistency, Certainty, Control, and Responsibility (Stranisci et al., 2022). EnDex, built from the Reddit Engagement Dataset, evaluates dialogue engagingness through real human reactions and four dimensions of engagement: Emotional Engagement, Attentional Engagement, Behavioral Engagement, and Reply Engagement (Xu et al., 2022). MiroBench evaluates realism in simulated Reddit discussions through thread-level distributions over repetition, narrative content, toxicity and aggression, and structural complexity (Yu et al., 10 May 2026). Together, these resources indicate that Reddit has become a common substrate for benchmarking emotion classification, appraisal, engagingness, and social simulation, but EmoBench-Reddit is specifically a hierarchical benchmark for multimodal emotion understanding (Li et al., 14 Sep 2025).

7. Limitations, scope conditions, and future development

The benchmark is explicit about its limitations. The dataset is small, with 350 samples; it includes only four emotions; and because Reddit is an English-speaking platform with its own cultural norms, the benchmark may carry Western cultural bias and may not generalize to other cultural contexts. The paper therefore presents it as a diagnostic benchmark rather than a comprehensive emotional taxonomy (Li et al., 14 Sep 2025).

The authors identify several future directions. These include expanding the dataset, adding more emotions such as surprise, anger, and jealousy, incorporating other modalities such as video and audio, improving automated open-ended evaluation to reduce dependence on LLM-as-a-judge, and designing more interactive evaluation scenarios (Li et al., 14 Sep 2025).

These limitations are consequential for interpretation. The benchmark offers high-value evidence about the current weakness of multimodal models on emotion-laden Reddit posts, especially sarcasm and deep contextual inference, but it does not establish broad coverage of emotional expression across platforms, languages, or cultural settings. Its present contribution is narrower and more diagnostic: it exposes the difference between multimodal perception and emotionally grounded multimodal cognition.