---
title: 'MER-SEMI: Multimodal Emotion Recognition'
url: https://www.emergentmind.com/topics/mer-semi-challenge
type: topic
---

# MER-SEMI: Multimodal Emotion Recognition

The MER-SEMI Challenge is a foundational sub-track within the Multimodal Emotion Recognition (MER) competitive series, focused on advancing categorical emotion recognition from video, audio, and text under semi-supervised conditions. Its central aim is to catalyze methodological innovation for leveraging extensive unlabeled corpora to improve discrete emotion classification, addressing the annotation bottleneck that limits generalizability and scalability of affective computing systems. Over successive iterations (MER 2023, MER 2024, and MER 2025), MER-SEMI has become a principal international testbed for algorithmic developments in multimodal semi-supervised learning, advanced fusion architectures, and robust evaluation protocols [2504.19423][2304.08981][2404.17113].

## 1. Task Formulation and Problem Definition

MER-SEMI is defined as a single-label, fixed-taxonomy categorical emotion recognition task. Each video clip is assigned one label from a set of six universal emotion categories:

- neutral
- happy
- angry
- sad
- surprise
- worry

Auxiliary categories such as “others” or “unknown” may appear in preprocessing stages, but only these six are retained for evaluation and scoring [2504.19423]. The challenge is strictly multimodal, requiring integration of visual (video, face), acoustic (speech), and lexical (transcribed text) modalities. Participants are expressly encouraged to utilize large-scale unlabeled video data, in addition to a modest labeled set, to train models under a semi-supervised paradigm. The primary evaluation metric is the weighted average F1 score (WAF), which addresses class imbalance inherent in spontaneous emotion corpora [2304.08981][2404.17113].

## 2. Datasets: Curation, Composition, and Annotation

The evolution of the MER-SEMI datasets reflects ongoing efforts to combine high annotation fidelity with realistic data scale and diversity. Datasets are primarily sourced from MC-EIU (multimodal conversational emotion), M3ED (multi-scene emotion dialogues), and large collections from Chinese TV series [2504.19423][2404.17113]. A typical dataset structure for MER2025 is as follows:

| Split                     | Size    | Annotation Protocol                      |
|---------------------------|---------|------------------------------------------|
| Labeled Training          | 7,369   | Manual, majority-vote (≥5/7 annotators, ≥80% agreement) |
| Labeled Validation/Test   | 2,026   | High-agreement, randomly sampled from the unlabeled pool |
| Unlabeled                 | 124,802 | No manual annotation; strictly for SSL   |
| **Total**                 | 132,171 |                                          |

Data access is regulated via academic-use EULA terms; any manual relabeling of unlabeled samples is expressly prohibited [2504.19423]. Datasets are hosted on Hugging Face, with previous years' splits released in open-source repositories [2404.17113][2304.08981].

## 3. Semi-Supervised Learning Methodologies

The core methodological emphasis in MER-SEMI is on semi-supervised learning (SSL), where models learn from both labeled and unlabeled data. The general supervised plus unsupervised loss is given by

$$
L_{\text{total}} = L_{\text{supervised}} + \lambda \cdot L_{\text{unsupervised}},
$$

where 
- $L_{\text{supervised}}$ is typically cross-entropy or focal loss on labeled examples,
- $L_{\text{unsupervised}}$ may be any regularizer for unlabeled samples (contrastive, consistency, masked autoencoding, etc.),
- and $\lambda$ is a hyperparameter set via validation [2504.19423][2304.08981][2404.17113].

Common SSL instantiations include:
- **Masked autoencoders:** $L_{\text{unsup}} = \|x_{\text{masked}} - \text{Dec}(\text{Enc}(x_{\text{masked}}))\|^2$
- **Contrastive learning:** $L_{\text{unsup}} = -\sum_{\text{pos}} \log(\frac{\mathrm{sim}(z_i, z_j)}{\sum_{\text{all}} \mathrm{sim}(z_i, z_k)})$

Leading strategies include teacher–student pseudo-labeling with data augmentation, consistency regularization (e.g., across perturbed views/modalities), momentum-teacher models, graph-based label propagation, masked autoencoding, and curriculum/self-training [2508.05991][2409.05015][2409.04447][2508.06036]. Confidence-thresholded pseudo-labeling and careful balance of SSL weight $\lambda$ are critical to prevent confirmation bias and noisy label propagation [2304.08981][2404.17113].

## 4. Baseline Systems and Fusion Architectures

The official MER-SEMI baseline for MER2025 employs a lightweight attention-based fusion of modality-specific encoders. Each modality (visual, acoustic, lexical) passes through a pretrained backbone (e.g., CLIP, HuBERT, RoBERTa) and projects to a joint space before being fused via learned attention weights:

1. Map feature $f^m$ to hidden $h^m$ via linear transformation.
2. Compute attention: $\alpha^m = \text{Softmax}(W[h^1; h^2; h^3] + b)$.
3. Fuse: $H = \sum_m \alpha^m h^m$.
4. Classify: $\hat y = \text{Softmax}(H W_{\text{out}} + b_{\text{out}})$.

This baseline provides a reference for model selection and ablation [2504.19423]. Performance of single modalities varies—acoustic (HuBERT-large) typically yields WAF $\sim$72%, visual (CLIP-large) $\sim$60%, lexical (RoBERTa-large) $\sim$54%. Attention-based fusion of all three modalities yields up to 78.6% WAF on the official test set, setting a supervised baseline [2504.19423][2404.17113].

## 5. State-of-the-Art Methods and Empirical Results

Recent advances have consistently exceeded baseline performance via increasingly sophisticated multimodal architectures and SSL techniques:

- **ECMF (Enhanced Cross-Modal Fusion)** [2508.05991]: Introduces a dual-branch visual encoder (global + face streams via CLIP), large language model-enriched contextual encoding (GPT-4, Qwen-Omni), gated self-attention fusion, and multi-source consensus label refinement. Achieves 87.49% WAF on MER2025-SEMI—an increase of +8.86 points over baseline.
  
- **MoE-Based Frameworks** [2508.06036]: Employs a diverse mixture of six expert pathways—including action unit streams and vision-language model pseudo-knowledge—multi-stage consensus pseudo-labeling (baseline–Gemini agreement), and voting ensembles with human-inspired re-ranking. Achieves 87.72% WAF, with explicit distribution calibration that better aligns predictions with annotator tendencies.

- **Acoustic Adaptation and Visual Alignment** [2409.05015]: Leverages parameter-efficient fine-tuning (PEFT) on HuBERT layers 16–21, unsupervised video–audio contrastive alignment, and an attention fusion classifier. Yields 88.90% WAF in MER2024, top quartile among participants.

- **MR-CCL (Modality Representation Combinatorial Contrastive Learning)** [2409.04447]: Utilizes intra- and inter-modality contrastive pre-training, oversampling for class balance, high-confidence self-training, and a weighted soft-voting ensemble. Records 88.25% WAF on MER2024-SEMI.

- **MER2023 Top Entries** [2304.08981]: Achieved 89.1% WAF via aggressive teacher–student pseudo-labeling, strong augmentation, and curriculum-momentum learning, outperforming a 86.8% supervised baseline.

Aggregate results confirm that fusion strategies exploiting nonlinear, attention-based or MoE aggregators, combined with SSL techniques that rigorously filter or calibrate pseudo-labels, are crucial to robust performance. Incremental gains are most pronounced when explicitly addressing label noise, class imbalance, and domain adaptation to the in-domain unlabeled pool [2508.05991][2508.06036][2409.04447].

## 6. Analysis, Challenges, and Future Directions

MER-SEMI has catalyzed identification of key bottlenecks and methodological priorities for multimodal emotion recognition:

- **Modal complementarity**: Acoustic embeddings dominate in unimodal settings; however, maximal performance requires leveraging complementary visual and lexical cues [2504.19423][2404.17113].

- **Label noise and agreement**: Reliable annotation and consensus-based label refinement measurably improve model gradients in the low-label regime [2508.05991].

- **Class imbalance**: Oversampling and hybrid loss (e.g., LMF) regularization reduce performance gaps between majority and minority emotion classes.

- **Pseudo-label confirmation bias**: Mitigated via dual-model or oracle consensus, curriculum self-training, and dynamic thresholding [2304.08981][2409.04447].

- **Fairness and robustness**: Quantitative analysis of model error across speaker, context, and speaking style remains limited; robust SSL objectives (e.g., noise-robust loss functions) are highlighted as open problems [2504.19423].

**Future research directions** identified by organizers and leading teams include:
- Semi-supervised domain-adaptive pre-training (e.g., VideoMAE, audio contrastive predictive coding).
- Graph neural fusion frameworks for deep cross-modal interaction.
- Large language model-driven prompt engineering for lexical streams.
- Active learning selection for annotation-efficient label expansion.
- Granular benchmarking of fairness/demographic bias and noise robustness [2504.19423][2508.05991][2508.06036].

## 7. Impact and Ongoing Significance

The MER-SEMI track, through its open data protocols, reproducible baselines, and systematic benchmarking, has become a de facto standard for multimodal SSL in affective computing, particularly in Chinese-language media contexts. The challenge is directly shaping the direction and evaluation of semi-supervised learning, label noise handling, cross-modal fusion, and LLM integration in emotion AI. By lowering the reliance on scarce labeled data while exploiting abundant domain-matched unlabeled resources, MER-SEMI advances not only algorithmic state-of-the-art (WAF $\to$ 89%) but also practical deployment scalability [2504.19423][2404.17113][2304.08981][2508.05991][2508.06036][2409.05015][2409.04447]. These benchmarks continue to inform the next generation of robust, fair, and context-aware emotion recognition systems in naturalistic, multimodal environments.

Source: https://www.emergentmind.com/topics/mer-semi-challenge