MER-SEMI: Multimodal Emotion Recognition

• MER-SEMI Challenge is a multimodal semi-supervised emotion recognition task integrating visual, acoustic, and lexical data to classify six universal emotions.
• It employs large-scale unlabeled datasets alongside a modest labeled set using teacher–student pseudo-labeling, attention fusion, and contrastive learning methods.
• Recent innovations like enhanced cross-modal fusion and MoE frameworks have pushed the weighted average F1 score close to 89%, setting new benchmarks in affective computing.

The MER-SEMI Challenge is a foundational sub-track within the Multimodal Emotion Recognition (MER) competitive series, focused on advancing categorical emotion recognition from video, audio, and text under semi-supervised conditions. Its central aim is to catalyze methodological innovation for leveraging extensive unlabeled corpora to improve discrete emotion classification, addressing the annotation bottleneck that limits generalizability and scalability of affective computing systems. Over successive iterations (MER 2023, MER 2024, and MER 2025), MER-SEMI has become a principal international testbed for algorithmic developments in multimodal semi-supervised learning, advanced fusion architectures, and robust evaluation protocols (Lian et al., 28 Apr 2025, Lian et al., 2023, Lian et al., 2024).

1. Task Formulation and Problem Definition

MER-SEMI is defined as a single-label, fixed-taxonomy categorical emotion recognition task. Each video clip is assigned one label from a set of six universal emotion categories:

• neutral
• happy
• angry
• sad
• surprise
• worry

Auxiliary categories such as “others” or “unknown” may appear in preprocessing stages, but only these six are retained for evaluation and scoring (Lian et al., 28 Apr 2025). The challenge is strictly multimodal, requiring integration of visual (video, face), acoustic (speech), and lexical (transcribed text) modalities. Participants are expressly encouraged to utilize large-scale unlabeled video data, in addition to a modest labeled set, to train models under a semi-supervised paradigm. The primary evaluation metric is the weighted average F1 score (WAF), which addresses class imbalance inherent in spontaneous emotion corpora (Lian et al., 2023, Lian et al., 2024).

2. Datasets: Curation, Composition, and Annotation

The evolution of the MER-SEMI datasets reflects ongoing efforts to combine high annotation fidelity with realistic data scale and diversity. Datasets are primarily sourced from MC-EIU (multimodal conversational emotion), M3ED (multi-scene emotion dialogues), and large collections from Chinese TV series (Lian et al., 28 Apr 2025, Lian et al., 2024). A typical dataset structure for MER2025 is as follows:

Split	Size	Annotation Protocol
Labeled Training	7,369	Manual, majority-vote (≥5/7 annotators, ≥80% agreement)
Labeled Validation/Test	2,026	High-agreement, randomly sampled from the unlabeled pool
Unlabeled	124,802	No manual annotation; strictly for SSL
Total	132,171

Data access is regulated via academic-use EULA terms; any manual relabeling of unlabeled samples is expressly prohibited (Lian et al., 28 Apr 2025). Datasets are hosted on Hugging Face, with previous years' splits released in open-source repositories (Lian et al., 2024, Lian et al., 2023).

3. Semi-Supervised Learning Methodologies

The core methodological emphasis in MER-SEMI is on semi-supervised learning (SSL), where models learn from both labeled and unlabeled data. The general supervised plus unsupervised loss is given by

$$L_{\text{total}} = L_{\text{supervised}} + \lambda \cdot L_{\text{unsupervised}},$$

where

• $L_{\text{supervised}}$ is typically cross-entropy or focal loss on labeled examples,
• $L_{\text{unsupervised}}$ may be any regularizer for unlabeled samples (contrastive, consistency, masked autoencoding, etc.),
• and $\lambda$ is a hyperparameter set via validation (Lian et al., 28 Apr 2025, Lian et al., 2023, Lian et al., 2024).

Common SSL instantiations include:

• Masked autoencoders: $$L_{\text{unsup}} = \|x_{\text{masked}} - \text{Dec}(\text{Enc}(x_{\text{masked}}))\|^2$$
• Contrastive learning: $$L_{\text{unsup}} = -\sum_{\text{pos}} \log(\frac{\mathrm{sim}(z_i, z_j)}{\sum_{\text{all}} \mathrm{sim}(z_i, z_k)})$$

Leading strategies include teacher–student pseudo-labeling with data augmentation, consistency regularization (e.g., across perturbed views/modalities), momentum-teacher models, graph-based label propagation, masked autoencoding, and curriculum/self-training (Hu et al., 8 Aug 2025, Zhao et al., 2024, Fan et al., 2024, Xie et al., 8 Aug 2025). Confidence-thresholded pseudo-labeling and careful balance of SSL weight $\lambda$ are critical to prevent confirmation bias and noisy label propagation (Lian et al., 2023, Lian et al., 2024).

4. Baseline Systems and Fusion Architectures

The official MER-SEMI baseline for MER2025 employs a lightweight attention-based fusion of modality-specific encoders. Each modality (visual, acoustic, lexical) passes through a pretrained backbone (e.g., CLIP, HuBERT, RoBERTa) and projects to a joint space before being fused via learned attention weights:

1. Map feature $f^m$ to hidden $h^m$ via linear transformation.
2. Compute attention: $\alpha^m = \text{Softmax}(W[h^1; h^2; h^3] + b)$.
3. Fuse: $L_{\text{supervised}}$0.
4. Classify: $L_{\text{supervised}}$1.

This baseline provides a reference for model selection and ablation (Lian et al., 28 Apr 2025). Performance of single modalities varies—acoustic (HuBERT-large) typically yields WAF $L_{\text{supervised}}$272%, visual (CLIP-large) $L_{\text{supervised}}$360%, lexical (RoBERTa-large) $L_{\text{supervised}}$454%. Attention-based fusion of all three modalities yields up to 78.6% WAF on the official test set, setting a supervised baseline (Lian et al., 28 Apr 2025, Lian et al., 2024).

5. State-of-the-Art Methods and Empirical Results

Recent advances have consistently exceeded baseline performance via increasingly sophisticated multimodal architectures and SSL techniques:

• ECMF (Enhanced Cross-Modal Fusion) (Hu et al., 8 Aug 2025): Introduces a dual-branch visual encoder (global + face streams via CLIP), LLM-enriched contextual encoding (GPT-4, Qwen-Omni), gated self-attention fusion, and multi-source consensus label refinement. Achieves 87.49% WAF on MER2025-SEMI—an increase of +8.86 points over baseline.
• MoE-Based Frameworks (Xie et al., 8 Aug 2025): Employs a diverse mixture of six expert pathways—including action unit streams and vision-LLM pseudo-knowledge—multi-stage consensus pseudo-labeling (baseline–Gemini agreement), and voting ensembles with human-inspired re-ranking. Achieves 87.72% WAF, with explicit distribution calibration that better aligns predictions with annotator tendencies.
• Acoustic Adaptation and Visual Alignment (Zhao et al., 2024): Leverages parameter-efficient fine-tuning (PEFT) on HuBERT layers 16–21, unsupervised video–audio contrastive alignment, and an attention fusion classifier. Yields 88.90% WAF in MER2024, top quartile among participants.
• MR-CCL (Modality Representation Combinatorial Contrastive Learning) (Fan et al., 2024): Utilizes intra- and inter-modality contrastive pre-training, oversampling for class balance, high-confidence self-training, and a weighted soft-voting ensemble. Records 88.25% WAF on MER2024-SEMI.
• MER2023 Top Entries (Lian et al., 2023): Achieved 89.1% WAF via aggressive teacher–student pseudo-labeling, strong augmentation, and curriculum-momentum learning, outperforming a 86.8% supervised baseline.

Aggregate results confirm that fusion strategies exploiting nonlinear, attention-based or MoE aggregators, combined with SSL techniques that rigorously filter or calibrate pseudo-labels, are crucial to robust performance. Incremental gains are most pronounced when explicitly addressing label noise, class imbalance, and domain adaptation to the in-domain unlabeled pool (Hu et al., 8 Aug 2025, Xie et al., 8 Aug 2025, Fan et al., 2024).

6. Analysis, Challenges, and Future Directions

MER-SEMI has catalyzed identification of key bottlenecks and methodological priorities for multimodal emotion recognition:

• Modal complementarity: Acoustic embeddings dominate in unimodal settings; however, maximal performance requires leveraging complementary visual and lexical cues (Lian et al., 28 Apr 2025, Lian et al., 2024).
• Label noise and agreement: Reliable annotation and consensus-based label refinement measurably improve model gradients in the low-label regime (Hu et al., 8 Aug 2025).
• Class imbalance: Oversampling and hybrid loss (e.g., LMF) regularization reduce performance gaps between majority and minority emotion classes.
• Pseudo-label confirmation bias: Mitigated via dual-model or oracle consensus, curriculum self-training, and dynamic thresholding (Lian et al., 2023, Fan et al., 2024).
• Fairness and robustness: Quantitative analysis of model error across speaker, context, and speaking style remains limited; robust SSL objectives (e.g., noise-robust loss functions) are highlighted as open problems (Lian et al., 28 Apr 2025).

Future research directions identified by organizers and leading teams include:

• Semi-supervised domain-adaptive pre-training (e.g., VideoMAE, audio contrastive predictive coding).
• Graph neural fusion frameworks for deep cross-modal interaction.
• LLM-driven prompt engineering for lexical streams.
• Active learning selection for annotation-efficient label expansion.
• Granular benchmarking of fairness/demographic bias and noise robustness (Lian et al., 28 Apr 2025, Hu et al., 8 Aug 2025, Xie et al., 8 Aug 2025).

7. Impact and Ongoing Significance

The MER-SEMI track, through its open data protocols, reproducible baselines, and systematic benchmarking, has become a de facto standard for multimodal SSL in affective computing, particularly in Chinese-language media contexts. The challenge is directly shaping the direction and evaluation of semi-supervised learning, label noise handling, cross-modal fusion, and LLM integration in emotion AI. By lowering the reliance on scarce labeled data while exploiting abundant domain-matched unlabeled resources, MER-SEMI advances not only algorithmic state-of-the-art (WAF $L_{\text{supervised}}$5 89%) but also practical deployment scalability (Lian et al., 28 Apr 2025, Lian et al., 2024, Lian et al., 2023, Hu et al., 8 Aug 2025, Xie et al., 8 Aug 2025, Zhao et al., 2024, Fan et al., 2024). These benchmarks continue to inform the next generation of robust, fair, and context-aware emotion recognition systems in naturalistic, multimodal environments.