---
title: Open-Vocabulary Multimodal Emotion Recognition
url: https://www.emergentmind.com/topics/open-vocabulary-multimodal-emotion-recognition-mer-ov
type: topic
---

# Open-Vocabulary Multimodal Emotion Recognition

Open-Vocabulary Multimodal Emotion Recognition (MER-OV) refers to the computational task of inferring human emotion states from multimodal data—commonly visual (video/image), auditory (speech, environmental sounds), and textual (transcript)—and expressing predictions as arbitrary emotion descriptors taken from an open, essentially unlimited vocabulary, rather than a fixed set of emotion categories such as Ekman's six classes. This paradigm is motivated by the recognition that fixed-label sets cannot accommodate the complexity, subtlety, and multi-appraisal nature of actual human emotional experience, as revealed in both affective computing and contemporary cognitive science [2410.01495, 2512.20938]. Modern MER-OV frameworks leverage multimodal large language models (MLLMs), cross-modal fusion networks, or embedding-based architectures to achieve fine-grained, context-rich, and scalable emotion recognition, often incorporating explanation or justification for predictions.

## 1. Motivation and Formal Problem Definition

Traditional multimodal emotion recognition (MER) systems operate over predefined label sets (e.g., "happy", "sad", "angry") and are constrained by taxonomic rigidity, annotation bottlenecks, and poor coverage of minority or nuanced emotional states [2410.01495, 2404.17113]. The MER-OV paradigm formalizes the prediction task as learning a function

\[
f : (A, V, T) \rightarrow \mathcal{Y}_{open}
\]

where $(A, V, T)$ denote the audio, visual, and text streams of a sample, and $\mathcal{Y}_{open}$ is the space of all possible emotion terms (words, phrases, sentences). Unlike one-hot or multi-label schemes, MER-OV requires generation (and justification) of any number and type of emotion descriptors, accommodating compound, subtle, and dynamic affect [2512.20938, 2306.15401].

Key drivers include: (1) psychological theories estimating tens of thousands of distinct human emotions [2410.01495]; (2) the need for models that adapt to cultural, situational, and subjective variance; (3) limitations of annotation protocols reliant on majority voting or closed taxonomies [2404.17113].

## 2. Datasets and Annotation Methodologies

MER-OV requires datasets annotated with open-vocabulary emotion labels, often relying on hybrid human–LLM procedures due to the infeasibility of exhaustive manual labeling. Notable datasets include OV-MERD [2410.01495, 2512.20938], MER2024-OV [2404.17113, 2408.11286], and benchmarks derived from EMER [2306.15401]. Annotation typically involves:

1. **Multi-stage LLM–human pipelines**: Visual and acoustic clues are extracted by dedicated LLMs (e.g., GPT-4V for video, SALMONN for audio), validated or augmented by experts, and merged into unified, free-form multimodal descriptions [2410.01495].
2. **Label extraction**: Final emotion sets $\mathcal{Y}$ are mined by LLMs from these descriptions, sometimes translated and synonym-grouped to ensure consistency [2512.20938].
3. **Taxonomic mapping and grouping**: Labels are grouped into equivalence classes using taxonomy-driven (e.g., Parrott's tree) or embedding-based clustering, permitting semantic evaluation [2509.21950].

Dataset statistics:

| Dataset         | Samples | Modalities   | Unique Labels | Labels/sample |
|-----------------|---------|-------------|--------------|---------------|
| OV-MERD         | 332     | A,V,T       | 248          | ~3.3          |
| MER2024-OV      | 332     | A,V,T       | 301          | ~2.92         |
| EMER            | 332     | A,V,T       | 301          | ~3            |

Expert annotation, LLM verification, and synonym expansion procedures are standard; this enables robust, reproducible open-vocabulary label sets for benchmarking [2306.15401, 2410.01495].

## 3. Model Architectures and Fusion Strategies

MER-OV models predominantly employ multimodal deep neural architectures, often leveraging LLMs for decoding free-form emotion descriptions or label sets. Prominent frameworks:

1. **MLLM Fusion**: Modern benchmarks demonstrate the best results using a two-stage fusion methodology, extracting modality-specific "emotion clues" (visual gestures, audio pitch, textual keywords) via dedicated MLLMs (e.g., InternVL2.5, Qwen2-Audio) before projection and final label-generation by a strong LLM [2512.20938, 2408.11286]. Trimodal approaches (A,V,T) yield Fₛ up to 61.0%, with video found to be the dominant modality.
2. **Label Encoder-Guided CLIP-based Models**: MER-CLIP uses frozen CLIP text encoders as label embedding generators, supporting arbitrary textual descriptions as labels and enabling open-vocabulary cosine-similarity classification [2506.00903].
3. **Explainable EMER Models**: EMER and variants generate evidence-based explanations from multimodal features, then mine open-vocabulary labels directly from these natural-language summaries [2306.15401, 2407.16552].
4. **Continuous-Valence Models**: Some frameworks embed emotion states in continuous Valence–Arousal–Dominance (VAD) spaces, with nearest-neighbor retrieval for open-vocab label generation [2409.07901].
5. **Semi-supervised Fusion Backbones**: Methods such as Conv-Attention combine convolutional and attention-based fusion branches, using pseudo-labeled data to increase coverage and robustness [2408.10500].

## 4. Evaluation Metrics and Benchmarking

MER-OV evaluation requires metrics sensitive to variable-length open sets and semantic similarity between predictions and ground truth. Core metrics include:

- **Set-level Precision and Recall**: Grouping labels via synonym expansion or taxonomy, precision and recall are defined as

\[
\mathrm{Precision}_s = \frac{|\mathcal{Y} \cap \hat{\mathcal{Y}}|}{|\hat{\mathcal{Y}}|},\quad
\mathrm{Recall}_s    = \frac{|\mathcal{Y} \cap \hat{\mathcal{Y}}|}{|\mathcal{Y}|},\quad
\mathrm{Avg}         = \frac{\mathrm{Precision}_s + \mathrm{Recall}_s}{2}
\]

where $\mathcal{Y}$ and $\hat{\mathcal{Y}}$ are sets of grouped ground-truth and predicted labels [2306.15401, 2410.01495].

- **Emotion-Wheel (EW) Proximity**: Some models (e.g., AffectGPT-R1) optimize and evaluate directly on EW-based semantic distance metrics, scoring predicted terms by their proximity in a continuous affective space [2508.01318].
- **Open-vocabulary accuracy and recall**: Exact match and recall of ground-truth descriptors (points, phrases, sentences), sometimes as P@k or mAP for ranked predictions [2408.11286, 2408.10500].
- **Qualitative judgment tasks**: Custom emotion statement judgment protocols (e.g., ESJ) evaluate interpretation, context, and subjectivity via human-guided correctness labels [2509.21950].

Representative benchmark results:

| Model                     | Precisionₛ | Recallₛ | Fₛ / Avg |
|---------------------------|------------|---------|----------|
| GPT-4V (multimodal)       | 48.5       | 64.9    | 55.5     |
| Conv-Attention/Emotion-LLaMA | 69.61      | 62.59   | 66.10    |
| AffectGPT-R1 (EW metric)  | —          | —       | 66.35    |
| EMER(Multi) (upper bound) | 80.05      | —       | 80.05    |

State-of-the-art models consistently outperform randomly-selected baselines and closed-set classifiers, achieving robust performance on both primary and nuanced emotional states [2410.01495, 2512.20938].

## 5. Key Innovations and Qualitative Advances

MER-OV architectures exhibit several core advances:

1. **Open-vocabulary generalization**: Label encoders (e.g., CLIP-based, LLM-driven) enable prediction on arbitrary descriptors (words, phrases, sentences), eliminating dependence on fixed output heads [2506.00903, 2410.01495].
2. **Context and subjectivity modeling**: Automated pipelines (e.g. INSETS) generate statements factoring context, roles/personas, and perception subjectivity, increasing the fidelity of emotion classification [2509.21950].
3. **Fine-grained multimodal cue extraction**: Models like MicroEmo integrate global-local attention over facial regions and utterance-aware Q-Formers, improving temporal and contextual granularity in prediction [2407.16552].
4. **Semi-supervised and reinforcement learning for metric optimization**: Some frameworks (e.g., AffectGPT-R1) employ reinforcement learning to maximize non-differentiable EW metrics, while others use pseudo-labeling and sample weighting to expand annotation coverage [2508.01318, 2408.10500].

Qualitative observations consistently report correct retrieval of minority or composite labels ("frustrated", "resigned", "exuberant", etc.) and appropriate alignment between model explanations and multimodal evidence [2306.15401, 2408.11286].

## 6. Limitations, Open Problems, and Future Directions

While MER-OV frameworks surpass closed-set models in label diversity and expressiveness, several limitations persist:

- **Dataset scale**: Most benchmarks (e.g., OV-MERD, MER2024-OV) remain much smaller than required for comprehensive training and cross-cultural generalization [2410.01495, 2306.15401].
- **Reliance on proprietary LLM APIs**: Annotator pipelines depend extensively on GPT-series or similar models, challenging reproducibility [2410.01495].
- **Subjectivity and reliability**: Current annotation and evaluation protocols inadequately capture inter-annotator consistency, personalized emotion profiles, and explanation faithfulness [2509.21950].
- **Fusion and prompt bottlenecks**: Optimal fusion strategies (e.g., two-stage, Conv-Attention) and prompt designs (e.g., self-consistency, few-shot) remain highly empirical and model-dependent [2512.20938, 2408.10500].
- **Metric alignment**: Absence of differentiable metrics for open-vocabulary set similarity restricts end-to-end optimization, though RL-based approaches (AffectGPT-R1) begin to address this gap [2508.01318].

Future research will likely focus on: (1) scalable, multilingual, and cross-domain dataset expansion; (2) unsupervised or active annotation strategies; (3) integrated architectures enabling continual, open-set generalization; (4) advanced metric-learning for fine-grained label alignment; and (5) human-in-the-loop evaluation to optimize subjectivity and explanation quality [2410.01495, 2509.21950].

## 7. Historical Trajectory and Impact

The MER-OV paradigm emerged in response to the limitations of majority-vote, fixed-class emotion recognition in audio-visual-textual corpora, bolstered by psychological evidence for emotion complexity, and the rise of generative MLLMs capable of unrestricted, contextually-justified label production [2410.01495, 2306.15401]. Early benchmarks formalized set-based precision/recall metrics and user-centered annotation pipelines; subsequent innovations generalized to metric-learning, RL optimization, and multimodal fusion [2506.00903, 2508.01318, 2409.07901]. MER-OV now serves as a critical foundation for affective computing, empathetic human-AI interfaces, and personalized emotion-based applications, demanding continual advances in scale, reliability, and interpretability.

---

MER-OV research is now at the forefront of multimodal understanding, establishing benchmarks and architectures for fine-grained, semantically rich, and context-sensitive emotion AI systems, with broad cross-disciplinary implications for cognitive science, HCI, and affective computing [2512.20938, 2410.01495, 2306.15401].

Source: https://www.emergentmind.com/topics/open-vocabulary-multimodal-emotion-recognition-mer-ov