MER2025-SEMI: Multimodal Emotion Recognition Data

• MER2025-SEMI is a multimodal dataset that integrates video, audio, and text modalities for fixed-taxonomy emotion recognition under semi-supervised conditions.
• It features high-quality human annotations with rigorous quality control and label refinement, supporting both supervised and semi-supervised learning frameworks.
• Baseline systems demonstrate strong performance, promoting research in multimodal fusion, cross-modal attention, and label denoising techniques.

The MER2025-SEMI dataset is a large-scale, multimodal resource curated for research in categorical emotion recognition under semi-supervised learning conditions. Developed as part of the 2025 Multimodal Emotion Recognition (MER2025) challenge, it targets fixed-taxonomy affect recognition, emphasizing the integration of unlabeled, in-domain data to enhance model generalization in a low-label regime. The dataset offers multi-granular video, audio, and textual modalities with high-quality human annotations, standardized preprocessing protocols, and supports both supervised and semi-supervised algorithmic frameworks (Lian et al., 28 Apr 2025, Hu et al., 8 Aug 2025).

1. Dataset Scope and Modalities

MER2025-SEMI is constructed from a diverse set of conversational corpora (MC-EIU, M3ED) and 19 Chinese TV dramas, yielding heterogeneous, in-the-wild content representative of spontaneous spoken emotion. The full corpus comprises 132,171 utterances, partitioned into three primary segments:

• Labeled training set: 7,369 video clips, each annotated with a single basic emotion label (“neutral”, “happy”, “angry”, “sad”, “surprise”, “worry”).
• Unlabeled pool: 124,802 utterances provided without label for semi-supervised learning.
• Test set: 2,026 pseudo-labeled, high-agreement samples, curated from the unlabeled pool.

Each utterance is represented across three synchronized modalities:

• Visual: MP4 video segments (1–5 s), single-speaker, front-facing.
• Audio: mono-channel WAV files at 16 kHz, time-aligned to video.
• Text: Chinese transcript as UTF-8 subtitle.

Emotion category selection follows appraisal theory constructs: "neutral" (no affective coloring), "happy" (positive valence), "angry" (negative valence, high activation), "sad" (negative valence, low activation), "surprise" (unexpected event), and "worry" (future-oriented negative anticipation) (Lian et al., 28 Apr 2025).

2. Annotation Protocol and Quality Control

The annotation pipeline employed nine initial annotators, filtered with a gold-standard pretest (two eliminated), to ensure domain proficiency. Each candidate video was labeled for a single emotion (from eight possible labels) through a forced-choice interface. Test set curation prioritized label reliability, admitting only those samples with ≥ 80% inter-annotator agreement (at least 4 of 5 assigned the same primary emotion, excluding “others” and “unknown”).

To address observed label noise—manifest as discrepancies between assigned and perceived emotion—a multi-source relabeling system was developed in follow-up research (Hu et al., 8 Aug 2025). This method combines predictions from three unimodal "weak" classifiers (visual, audio, text) and a LLM (Qwen-Omni) using majority voting. Samples where the refined label disagrees with all sources are manually reviewed.

3. Data Splits, Access, and Preprocessing

The data are organized for both standard training/validation/test protocols and flexible experimentation:

• Training: 7,369 labeled samples; customizable via cross-validation or validation splits.
• Unlabeled: 124,802 (official release) or 20,000 (as referenced in some challenge documentation) samples for semi-supervised tasks.
• Test: 2,026 high-agreement pseudo-labeled clips; a hidden partition is reserved for leaderboard evaluation.

Access requires agreeing to the End-User License Agreement (EULA) for academic, non-commercial use. Retrieval and loading utilize the Hugging Face platform:

pip install datasets from datasets import load_dataset ds = load_dataset("MERChallenge/MER2025", "MER-SEMI")

No pre-extracted features are provided. Standard preprocessing includes: visual cropping and resizing to $224 \times 224$, amplitude normalization and 16 kHz framing for audio, and Chinese tokenization (e.g., WordPiece) for text. The challenge baseline uses feature embeddings from pretrained models such as CLIP (visual), HuBERT (audio), and MacBERT (text) (Lian et al., 28 Apr 2025, Hu et al., 8 Aug 2025).

4. Baseline Systems and Semi-Supervised Objectives

The official MER-SEMI baseline is a supervised multimodal classifier:

• Each modality’s embedding is projected into a 256-dimensional space via a multilayer perceptron (MLP).
• Modalities are fused using a simple attention mechanism.
• Classification uses a final softmax over six emotion classes.

The baseline encourages, but does not mandate, semi-supervised techniques. An example loss:

• Supervised cross-entropy on labeled data:

$$\mathcal{L}_{\mathrm{sup}} = -\frac{1}{N_\ell}\sum_{i=1}^{N_\ell}\sum_{c=1}^{6} y_{i,c}\,\log p_{i,c}$$

• Consistency regularization on unlabeled data ($\tilde x$ with dropout or noise):

$$\mathcal{L}_{\mathrm{cons}} = \frac{1}{N_u}\sum_{j=1}^{N_u}\bigl\|p(\tilde x_j;\theta)-p^{\prime}(\tilde x_j;\theta)\bigr\|_2^2$$

• Full loss:

$$\mathcal{L} = \mathcal{L}_{\mathrm{sup}} + \lambda\,\mathcal{L}_{\mathrm{cons}},$$

where $\lambda$ linearly ramps up.

Supervised results (baseline, test set):

• Best single modality (HuBERT-large): weighted F1 (WAF) ~72.27%, Accuracy ~72.90%
• Trimodal fusion: WAF ~78.63%, Accuracy ~78.77%

Label refinement with multi-source voting yields further improvements (WAF, test):

• No refinement: 76.80%
• With refinement: 78.67%

Advanced fusion methods (ECMF) achieve WAF 87.49%, demonstrating the utility of cross-modal attention and label denoising (Hu et al., 8 Aug 2025).

5. Usage Recommendations and Best Practices

• Primary metric: Weighted F1 (WAF) addresses strong class imbalance.
• Hyperparameter tuning: Random/grid search recommended, with stability ensured by averaging performance across seeds (the baseline used 50 seeds and 6 repeats).
• Cross-validation: Five-fold CV on the training set is a reliable proxy for leaderboard evaluations.
• Semi-supervised strategies: Incorporation of the large unlabeled pool via consistency or contrastive learning is strongly encouraged.
• Label quality: Multi-source refinement plus manual audit is necessary to mitigate human annotation noise.
• Licensing: Academic use only, no redistribution or reannotation.

6. Unique Properties, Challenges, and Research Impact

MER2025-SEMI presents a confluence of challenges:

• Severe label scarcity (7,369 labeled samples) amid six-class categorization.
• Substantial human annotation noise, necessitating systematic relabeling.
• Significant modality heterogeneity, with frequent cross-modal disagreement.
• Large-scale unlabeled pool, supporting exploration of semi-supervised, pseudo-labeling, and self-training paradigms.

The dataset has catalyzed work in multimodal representation learning, cross-modal fusion, and label denoising, providing a standardized benchmark for evaluating semi-supervised emotion recognition algorithms. Performance close to 80% WAF with simple supervised fusion sets a strong baseline; further gains hinge on exploiting unlabeled data and robust multimodal learning strategies (Lian et al., 28 Apr 2025, Hu et al., 8 Aug 2025).

7. Citation and Access

Researchers obtaining or publishing on the MER2025-SEMI dataset are required to cite:

• ACM MM ’25 proceedings for the challenge description (Lian et al., 28 Apr 2025).
• Associated methods papers for algorithmic contributions (Hu et al., 8 Aug 2025).

The dataset is available via Hugging Face (subject to EULA):

• https://huggingface.co/datasets/MERChallenge/MER2025

No manual reannotation or public reposting is permitted. For additional implementation details or baseline code, refer to the official challenge GitHub repositories and MERTools documentation.