---
title: 'OV‑MERD: Open‑Vocabulary Multimodal Emotion Dataset'
url: https://www.emergentmind.com/topics/open-vocabulary-multimodal-emotion-recognition-dataset-ov-merd
type: topic
---

# OV‑MERD: Open‑Vocabulary Multimodal Emotion Dataset

OV‑MERD is the official open‑vocabulary multimodal emotion dataset from the MER 2024 challenge and is used as the core benchmark for Open‑Vocabulary Multimodal Emotion Recognition (MER‑OV), a setting in which models are not restricted to a closed set of labels such as happiness, sadness, or anger, but instead generate flexible natural-language emotion descriptors and often multiple labels per clip [2404.17113; 2512.20938]. In the later OV‑MER formulation, it is the dataset explicitly built for the open‑vocabulary MER task, with labels drawn from an unrestricted emotion lexicon and evaluated by set-based metrics that collapse synonyms and hierarchical variants rather than requiring exact token identity [2410.01495].

## 1. Definition and historical placement

OV‑MERD emerged from a broader dissatisfaction with fixed-taxonomy multimodal emotion recognition. Earlier MER benchmarks typically used one-hot or small multi-label spaces, whereas OV‑MER was proposed as a paradigm that “enables emotion prediction without being confined to predefined spaces” and allows “arbitrary emotion descriptors” and “any number of such labels per sample” [2410.01495]. In the MER2024 challenge, this appeared as the MER‑OV track, which encouraged participants to “generate any number of labels in any category, trying to describe emotions accurately” rather than choosing from a fixed candidate set [2404.17113].

Within that trajectory, OV‑MERD occupies a specific role. One paper describes it as “the only publicly available benchmark for MER‑OV” and repurposes the MER 2024 challenge release as the standard dataset for evaluating 19 mainstream MLLMs under open-vocabulary multimodal emotion recognition [2512.20938]. Another paper explicitly names it “OV‑MERD (Open‑Vocabulary Multimodal Emotion Recognition Dataset)” and positions it as the dataset created for the OV‑MER task, extending MER from one-hot or fixed multi-label prediction to open-vocabulary multi-label emotion description [2410.01495].

A closely related precursor is EMER, “Explainable Multimodal Emotion Recognition,” which demonstrated that multimodal explanations could be used to extract richer emotions in an open-vocabulary manner from audio, video, and text. EMER is not OV‑MERD itself, but it established the explanation-first logic that later OV‑MERD work operationalized at the dataset and benchmark level [2306.15401].

## 2. Corpus composition, modalities, and annotation workflows

The dataset is consistently described as trimodal. Its samples are short clips with synchronized **video**, **audio**, and **text**. In the large-scale benchmark study, the dataset contains “332 multimodal clips” from “Chinese television dramas, movies, and interviews,” with durations from \(0.2\,\text{s}\) to \(22.1\,\text{s}\), mean duration \(3.9\,\text{s}\), average frame rate \(24.9\) FPS, “248 unique emotion terms,” and an average of “3.34 labels” per sample; 20 samples are “video-only ‘due to their extremely short durations’” [2512.20938]. The MER2024 challenge paper, however, reports that the open-vocabulary track selects “332 samples” from the larger corpus and later lists MER‑OV as “322 labeled samples,” with “2.92 labels per sample” [2404.17113]. This suggests that the literature is referring to different filtering stages or benchmark variants of the same resource.

The underlying corpus construction in MER2024 is also documented. The broader video corpus is built from movies and TV series; long scripts are segmented using subtitle timestamps when available, or with Silero VAD and Deep Speaker when subtitles are absent. YuNet is used so that most frames contain only one face, and face.evoLVe is used so that faces within a clip belong to the same person. The challenge corpus enforces clip lengths between 2 and 16 seconds and focuses on a single main character [2404.17113].

Two annotation workflows are described in the literature. The MER2024 challenge track uses an LLM-based pipeline: multimodal large language models first generate rich emotion-related descriptions from audio, video, and subtitles; GPT‑3.5 then extracts open-vocabulary labels from those descriptions; finally, labels are merged with participant submissions and manually checked to obtain final ground truth [2404.17113]. The later OV‑MER paper describes a more explicitly human–LLM collaborative procedure: GPT‑4V produces initial visual clues from sampled frames, SALMONN produces acoustic clues, human annotators conduct two rounds of manual checking, and GPT‑3.5 merges the checked visual clues, checked acoustic clues, and transcripts into a multimodal description called **CLUE‑Multi**. Final open-vocabulary labels are then extracted in both English and Chinese, translated cross-lingually, merged, and manually filtered to obtain consensus labels [2410.01495].

These workflows are important because OV‑MERD is not merely a list of tags attached to clips. It is a dataset in which the labels are derived from multimodal descriptions and thus inherit explicit evidence from facial expressions, body movements, environment, acoustic information, and subtitle content [2404.17113; 2410.01495].

## 3. Emotion representation and evaluation formalism

OV‑MERD is defined by its label space as much as by its raw media. Labels are free-text emotion words or phrases rather than members of a predefined taxonomy. In the benchmark paper, each sample stores its emotions as a list of strings such as \([\text{suspicious}, \text{angry}, \text{dissatisfied}, \text{questioning}]\) [2512.20938]. In the OV‑MER paper, samples can carry between 1 and 9 labels, “most samples have 2 to 4 labels,” and the dataset exhibits a long-tail distribution over 248 emotion categories [2410.01495].

Because exact string match is too brittle for open-vocabulary emotion recognition, evaluation is semantic and set-based. The MER2024 challenge first groups labels by meaning using ChatGPT and defines
\[
Y = \{ G(x) \mid x \in \{y_i\}_{i=1}^{M} \}, \qquad
\hat{Y} = \{ G(x) \mid x \in \{\hat{y}_i\}_{i=1}^{N} \},
\]
followed by
\[
\mbox{Accuracy}_{s} = \frac{|Y \cap \hat{Y}|}{|\hat{Y}|}, \qquad
\mbox{Recall}_{s} = \frac{|Y \cap \hat{Y}|}{|Y|}, \qquad
\mbox{Avg} = \frac{\mbox{Accuracy}_{s} + \mbox{Recall}_{s}}{2}.
\]
This makes the track intrinsically multi-label and synonym-tolerant [2404.17113].

The OV‑MER paper generalizes this idea with hierarchical normalization and emotion-wheel grouping. Morphological normalization \(F_{l_1}\), synonym normalization \(F_{l_2}\), and wheel-specific hierarchical mapping \(F_{l_3}^{w_k}\) are combined into
\[
G_{w_k}(\cdot) = F_{l_3}^{w_k}\Big(F_{l_2}(F_{l_1}(\cdot))\Big),
\]
and set-level precision, recall, and F-score are then averaged across five emotion wheels [2410.01495]. Later OV‑MER work adopts the same logic and refers to this as the **Emotion Wheel-based (EW) metric**, with \(S_1\) as coarse-grained F1 and \(S_2\) as fine-grained F1 in OV‑FG settings [2603.16463].

This evaluation design is not incidental. The OV‑MER paper shows directly why fixed-label MER is incomplete in the open-vocabulary regime: when MER2023 one-hot labels are evaluated against OV labels, English performance is reported as \(\mathrm{F}_{\mathrm{s}} = 65.71\), \(\mathrm{Precision}_{\mathrm{s}} = 92.17\), and \(\mathrm{Recall}_{\mathrm{s}} = 51.05\). The closed labels are therefore often not incorrect, but they are incomplete relative to the richer label sets in OV‑MERD [2410.01495].

## 4. Benchmarking resource for multimodal large language models

OV‑MERD quickly became the central benchmark for MLLM evaluation in open-vocabulary emotion recognition. The MER2024 challenge paper provides zero-shot baseline results for a range of pretrained multimodal LLMs. In that setting, GPT‑4V achieves the best average score at **56.0%**, Qwen‑Audio achieves **46.0%**, and SALMONN achieves **36.0%** on the MER‑OV metric \(\mbox{Avg}\), showing that open-vocabulary emotion reasoning is difficult even for strong proprietary systems [2404.17113].

A later large-scale study evaluates **19 mainstream MLLMs**, including general-purpose, modality-specialized, and reasoning-enhanced architectures, and concludes that “a two-stage, trimodal (audio, video, and text) fusion approach achieves optimal performance in MER‑OV, with video emerging as the most critical modality” [2512.20938]. Within the emotional clue-based two-stage framework, the reported semantic F-scores are:

| Modality setting | \( \text{F}_\text{s} \) |
|---|---:|
| Text only | 55.0 |
| Video only | 57.6 |
| Audio only | 47.2 |
| Text + Video + Audio | **61.0** |

The same study also reports that removing video from trimodal fusion drops performance from 61.0 to 55.5, whereas video-only already exceeds text-only and audio-only, leading to the conclusion that video is “the most critical contributor” on OV‑MERD [2512.20938].

Methodologically, the dominant benchmark pattern is a two-stage pipeline. Stage 1 converts video and audio into textual clues using a Video‑LLM and an Audio‑LLM; Stage 2 feeds those clues together with subtitles into a text LLM that outputs an open-vocabulary emotion set. The paper compares this to objective-description pipelines and video-only one-stage inference, and finds the emotional clue-based two-stage method best overall [2512.20938].

A persistent empirical theme is that open-vocabulary MER on OV‑MERD is not merely a scaling exercise. The same benchmark finds a “surprisingly narrow gap between open- and closed-source LLMs,” but also shows that reasoning-enhanced models do not necessarily dominate: prompt design, multimodal clue extraction, and fusion strategy remain decisive [2512.20938].

## 5. OV‑MERD as supervision for reasoning architectures

Later work treats OV‑MERD not only as an evaluation set but as supervision for explicit affective reasoning. In HyDRA, OV‑MERD and MERCaption+ are the key resources for open-vocabulary emotion reasoning; OV‑MERD is used in the “cold-start SFT subset,” and AffectGPT trained on OV‑MERD is treated as a strong open-vocabulary baseline [2603.16463]. HyDRA frames emotion recognition as reconstructing a latent situational state from multimodal observations, textualizes multimodal evidence into an **Observation Graph (ObsG)**, and learns a **Propose–Verify–Decide** protocol with GRPO and hierarchical reward shaping. In that design, OV‑MERD matters because it provides “free-text descriptors,” “multiple labels per sample,” and “cue-level textual annotations” that can support semantic grounding and intra-trace evidence consistency [2603.16463].

The RL line of work uses OV‑MERD+—an extended evaluation set derived from OV‑MERD—as the standard open-vocabulary benchmark. AffectGPT‑R1 reports **62.52** on OV‑MERD+ for AffectGPT and **66.35** for AffectGPT‑R1 after Group Relative Policy Optimization with EW-based rewards, directly targeting the non-differentiable emotion-wheel metric rather than token-level cross-entropy alone [2508.01318]. AffectGPT‑RL pushes this further, reporting **68.05** with penalties and **68.39** without penalty on OV‑MERD+, and showing that the improvement comes from RL aligned with EW metrics rather than from simply adding MER2025‑OV data under supervised training [2605.06126].

The dataset’s influence also extends into unified benchmarks. Emotion‑LLaMAv2’s MMEVerse includes OV‑MERD+ as the dedicated **OV‑Emotion** benchmark within an 18-benchmark suite, which places OV‑MERD in a broader ecosystem spanning basic emotions, sentiment, multi-label affect, VAD, and intention recognition [2601.16449]. In that sense, OV‑MERD became the canonical fine-grained open-vocabulary evaluation slice within larger multimodal emotion-understanding frameworks.

## 6. Limitations, reporting discrepancies, and legacy

Several limitations are repeatedly emphasized. First, the dataset is small. One paper calls it “the only publicly available benchmark for MER‑OV,” but at the same time reports only 332 clips, all from Chinese media sources, with 20 video-only samples and a label space that is highly long-tailed [2512.20938]. The MER2024 challenge paper similarly operates on 322 or 332 clips, depending on where in the annotation pipeline the description is taken from [2404.17113]. This suggests that OV‑MERD is a foundational benchmark rather than a large-scale training corpus.

Second, the literature records nontrivial reporting discrepancies. Sample count appears as both **332** and **322**; average labels per sample appear as **3.34** and **2.92**; duration is described both as \(0.2\,\text{s}\)–\(22.1\,\text{s}\) and, via the broader corpus construction rules, as 2–16 seconds [2512.20938; 2404.17113]. These discrepancies suggest versioning, filtering, or benchmark-subset differences rather than a single immutable release.

Third, the evaluation pipeline depends on semantic grouping mechanisms that are themselves model-mediated. The MER2024 challenge uses ChatGPT grouping for synonym normalization, and the later benchmarking study uses GPT‑3.5 clustering to map strings into cluster IDs [2404.17113; 2512.20938]. Later OV‑MER work partially addresses reproducibility with multi-wheel EW mappings, but the problem of ontology dependence is not fully removed [2410.01495].

Fourth, cultural breadth remains limited. HyDRA explicitly notes that “current datasets are limited in cultural breadth” and states that OV‑MERD “should eventually include cross-cultural annotations and ontologies” [2603.16463]. The 2025 benchmarking study similarly calls for “new, dedicated datasets” and “multilingual and multicultural evaluations” [2512.20938].

Despite those limitations, OV‑MERD has had a disproportionate influence on the field. It established open-vocabulary MER as a concrete benchmark; motivated larger descriptive datasets such as MER‑Caption and MER‑Caption+ with **115,595** samples and **2,932** or **1,972** fine-grained emotion categories depending on the filtered subset; and supplied the evaluation substrate for RL-aligned and reasoning-centric emotion models [2501.16566]. A plausible implication is that OV‑MERD’s lasting significance lies less in scale than in formalizing a new target: multimodal emotion recognition as open-vocabulary, multi-label, semantically normalized generation rather than fixed-set classification.

Source: https://www.emergentmind.com/topics/open-vocabulary-multimodal-emotion-recognition-dataset-ov-merd