Open-Vocabulary Multimodal Emotion Recognition

• OV-MER is a paradigm that enables emotion recognition beyond fixed taxonomies by predicting variable natural language labels from multimodal evidence.
• It integrates continuous affective representations like VAD with lexicon-based decoding and synonym-aware evaluation to capture fine-grained emotional nuances.
• The approach supports generative emotion understanding and drives innovations such as reinforcement learning, explainable models, and advanced multimodal processing.

Open-Vocabulary Multimodal Emotion Recognition (OV-MER) denotes multimodal emotion recognition in which a system receives multimodal evidence—typically audio, video, and text—and predicts emotion descriptors without being confined to a predefined label space. In the MER2024 formulation, the system may output any number of natural-language emotion labels from any category; in the subsequent MER challenge trajectory, the same shift is described as a transition from discriminative emotion recognition toward generative emotion understanding, with fine-grained and descriptive emotion representations as the target (Lian et al., 2024, Lian et al., 21 Apr 2026).

1. Emergence of the open-vocabulary paradigm

OV-MER emerged as a response to the well-known limitations of fixed-taxonomy multimodal emotion recognition. Earlier MER pipelines generally assumed one-hot or small multi-label outputs over basic categories, but this design discards non-majority and non-candidate labels and compresses mixed or context-dependent affect into a few coarse classes. The OV-MER formulation therefore opens both the label set and the label cardinality: a model may produce a variable-size set of emotion words rather than a single class from a closed inventory (Lian et al., 2024, Lian et al., 2024).

An important precursor was Explainable Multimodal Emotion Recognition (EMER), which made the explanation rather than the class label the primary supervised object. In EMER, multimodal explanations are used to derive discrete labels, valence, and open-vocabulary emotion labels; the resulting dataset contains 301 distinct emotion labels and about 3 labels per sample, explicitly demonstrating that richer affective descriptions can be extracted from multimodal evidence than standard majority-vote annotations preserve (Lian et al., 2023).

The challenge infrastructure subsequently stabilized the paradigm. MER2024 introduced an explicit MER-OV track for open-word emotion recognition (Lian et al., 2024). MER2026 then framed the broader series as a shift “from discriminative emotion recognition to generative emotion understanding,” with MER-FG allowing participants to predict any number of emotion labels across diverse categories and MER-Prefer evaluating natural-language emotion descriptions by human preference (Lian et al., 21 Apr 2026). This suggests that OV-MER is no longer treated as an isolated benchmark variant, but as a central formulation for fine-grained affective computing.

2. Emotion representation and evaluation

A defining property of OV-MER is that the output space is linguistic rather than categorical. In the MER2024 and OV-MER formulations, the prediction can be written as

$$\hat{Y} = \{\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_N\}, \quad \hat{y}_i \in \mathcal{V},$$

where $\mathcal{V}$ is an open vocabulary of emotion descriptors (Lian et al., 2024, Lian et al., 2024). Because exact string matching is inadequate, evaluation is performed after semantic normalization. A common set-level protocol maps both predictions and references into grouped label sets and computes

$$\text{Accuracy}_{s} = \frac{|Y \cap \hat{Y}|}{|\hat{Y}|}, \qquad \text{Recall}_{s} = \frac{|Y \cap \hat{Y}|}{|Y|}, \qquad \text{Avg} = \frac{\text{Accuracy}_{s} + \text{Recall}_{s}}{2}.$$

This makes synonym-aware multi-label comparison the basic unit of evaluation rather than token identity (Lian et al., 2024).

Later work replaced ad hoc grouping with emotion-wheel-based normalization. MER2026 formalized a hierarchical mapping

$$\mathcal{F}_i(\cdot) = C_i(B(A(\cdot))), \quad i \in [1,5],$$

where $A$ performs lemmatization, $B$ synonym mapping, and $C_i$ maps fine-grained labels to wheel-specific basic emotions. Precision, recall, and $F$-score are then computed on grouped sets and averaged across wheels (Lian et al., 21 Apr 2026). AffectGPT-RL adopts the same basic idea, using five emotion wheels to define the non-differentiable objective optimized by reinforcement learning (Lian et al., 7 May 2026).

A second representation family replaces direct text generation with a continuous semantic bridge. In the VAD-based multimodal strategy, a model predicts a point in three-dimensional Valence-Arousal-Dominance space and then decodes that point into nearby lexicon entries. The system uses the NRC-VAD lexicon, extracts 195 emotion vocabularies, places them in $[-1,1]^3$, and retrieves open-vocabulary labels within an $L_2$ radius $\mathcal{V}$0: $\mathcal{V}$1 This yields a lexicon-grounded notion of openness: any emotion word with a VAD triple can be considered at inference time, even if it was not used as a training label (Jia et al., 2024).

3. Datasets, tasks, and benchmark trajectory

The first dedicated OV-MER datasets are small but lexically dense. OV-MERD contains 332 multimodal clips from Chinese TV dramas, movies, and interviews, annotated with 248 unique emotion terms and an average of 3.34 labels per clip (Han et al., 24 Dec 2025). The MER2024 OV track used a closely related subset of about 322–332 samples and reported an average of 2.92 labels per sample after LLM-assisted clue extraction and label consolidation (Lian et al., 2024). These statistics contrast sharply with conventional MER corpora, whose label spaces typically contain 4–10 categories and one label per sample.

The benchmark ecosystem broadened quickly. The original OV-MER paper introduced OV-MERD as a curated database together with evaluation metrics and a preliminary benchmark (Lian et al., 2024). MER2026 extended the landscape with Human-OV, containing 1,532 manually labeled samples, MER-Caption+ with 31,327 automatically labeled samples, and a 1,000-sample manually labeled MER-FG test set for fine-grained open-vocabulary recognition (Lian et al., 21 Apr 2026). MER-Prefer added pairwise human preference labels over alternative emotion descriptions, yielding 574 majority-voted training samples in EmoPrefer-Data, 2,096 single-annotator samples in EmoPrefer-Data-V2, and a 379-sample majority-voted test set (Lian et al., 21 Apr 2026).

This benchmark trajectory has two consequences. First, OV-MER supervision is now distributed across label-set prediction, descriptive generation, and preference modeling rather than a single annotation format. Second, the field increasingly distinguishes among open-vocabulary recognition, descriptive emotion analysis, and explanation-oriented emotion reasoning, even though the same multimodal-to-language backbone may support all three (Lian et al., 21 Apr 2026, Lian et al., 2023).

4. Methodological families

Current OV-MER systems fall into several distinct but partially overlapping design families.

Family	Core mechanism	Representative papers
VAD-bridged models	Regress continuous VAD, then decode by clustering or lexicon retrieval	(Jia et al., 2024)
Explanation-first MLLMs	Generate multimodal clues or descriptions, then extract emotion words	(Lian et al., 2023, Ge et al., 2024, Cheng et al., 2024)
Time-sensitive MLLMs	Use utterance-aware video/audio tokenization for local temporal dynamics	(Zhang, 2024, Zhang et al., 21 Mar 2026)
Evidential reasoning models	Generate competing hypotheses, verify them, then decide	(Liu et al., 17 Mar 2026)

The VAD-bridged line is significant because it offers a non-generative route to openness. The multimodal backbone remains conventional—vision, audio, and text encoders with weighted fusion—but the output head predicts $\mathcal{V}$2 rather than a six-way class, and K-means or radius-based lexicon lookup converts that continuous point into both discrete MER labels and open-vocabulary words (Jia et al., 2024). This family treats OV-MER as inference in an affective semantic space rather than pure language generation.

The explanation-first line instead makes free-text generation primary. EMER constructs multimodal textual clues and then extracts labels from them (Lian et al., 2023). MER2024 challenge systems based on InternVL and AffectGPT similarly generate detailed descriptions and then summarize them into emotion word lists; one such system created 26,000 high-quality image-text emotion pairs for fine-tuning and reported an AffectGPT-SFT score of 0.7429 on MER2024-OV (Ge et al., 2024). Emotion-LLaMA-based systems follow the same general pattern but specialize the teacher model for affective annotation and then use a second LLM to extract keywords; the description-based variant reported $\mathcal{V}$3 on MER-OV, compared with $\mathcal{V}$4 for GPT-4V (Cheng et al., 2024).

A third line emphasizes temporal granularity. MicroEmo introduces a global-local attention visual encoder and an utterance-aware video Q-Former to preserve local facial micro-expression dynamics and utterance-level temporal structure in EMER-style open-vocabulary generation (Zhang, 2024). AcoustEmo applies the same principle to audio: a timestamp-synchronized sliding window extracts utterance-aligned acoustic segments, an Utterance-Aware Acoustic Q-Former converts them to local acoustic tokens, and the full multimodal sequence $\mathcal{V}$5 is passed to a LLaMA-2 backbone (Zhang et al., 21 Mar 2026). In a separate large-scale benchmark of 19 mainstream MLLMs on OV-MERD, the best overall pattern was a two-stage, trimodal emotional-clue-based pipeline, and video emerged as the most critical modality (Han et al., 24 Dec 2025).

5. Optimization, reasoning, and reinforcement learning

A central methodological problem in OV-MER is objective mismatch. AffectGPT-R1 states the issue directly: token-level language modeling loss is misaligned with emotion-wheel-based evaluation, and the latter cannot be directly optimized by gradient backpropagation because the path from generated tokens to wheel score is discrete (Lian, 2 Aug 2025). AffectGPT-RL makes the same point and shows that adding more OV-MER data without RL produces only a minimal improvement relative to direct reward optimization (Lian et al., 7 May 2026).

This has made reinforcement learning a major thread in OV-MER. AffectGPT-R1 uses Group Relative Policy Optimization (GRPO) and defines a final reward as the sum of an EW-based accuracy reward and a format reward, improving OV-MERD+ from 62.52 to 66.35 (Lian, 2 Aug 2025). AffectGPT-RL expands this analysis with additional alignment, dual, and perception rewards, introduces explicit length penalties to prevent reward hacking through overlong emotion lists, and reports an increase from 62.52 to 68.39 on OV-MERD+ under its controlled setting (Lian et al., 7 May 2026).

A second line of work asks whether explicit reasoning actually helps recognition. MER-R1 reports that explicit reasoning does not necessarily translate into better MER accuracy: fast thinking tends to improve recall with broader predictions, whereas slow thinking favors precision through conservative filtering (Han et al., 26 Jun 2026). Its proposed remedy is a dual-objective RL formulation that disentangles recall and precision and a slow-fast confidence calibration mechanism that aligns the final slow-thinking answer with fast-thinking intuition (Han et al., 26 Jun 2026). A different reasoning-oriented architecture, HyDRA, formalizes inference as a Propose-Verify-Decide protocol, uses $\mathcal{V}$6 competing hypotheses over latent situational dynamics, and applies hierarchical reward shaping so that evidence-grounded reasoning trajectories are favored over prior-driven shortcuts (Liu et al., 17 Mar 2026). This suggests that reasoning becomes useful in OV-MER when it is explicitly evidence-constrained and reward-aligned, rather than merely prompted.

6. Limitations, misconceptions, and research directions

A common misconception is that “open vocabulary” implies evaluation without structure. In practice, many OV-MER benchmarks map free-form predictions back into grouped sets through synonym clustering or emotion wheels before scoring (Lian et al., 2024, Lian et al., 21 Apr 2026). Another misconception is that adding chain-of-thought automatically improves emotion recognition; MER-R1 shows that slow reasoning can reduce recall unless the optimization procedure explicitly balances recall and precision (Han et al., 26 Jun 2026).

Several limitations recur across the literature. Lexicon-grounded approaches inherit linguistic and cultural bias: the NRC-VAD lexicon is English-centric, while MER2024 is built from culturally consistent Chinese movie and TV clips (Jia et al., 2024). Evaluation pipelines often depend on GPT-based grouping or proprietary APIs, which raises reproducibility and bias concerns (Lian et al., 2024, Lian et al., 2024). MLLM systems remain vulnerable to low-SNR audio, overlapping speakers, modality conflict, and prompt sensitivity; AcoustEmo explicitly identifies noise robustness and computational cost as unresolved issues (Zhang et al., 21 Mar 2026). Dataset scale is also still modest for fully open supervision, and the lexical distribution is strongly long-tailed (Lian et al., 2024, Han et al., 24 Dec 2025).

The forward agenda is correspondingly broad. AcoustEmo points to continuous emotion tracking, cross-lingual emotion recognition, and low-latency deployment as near-term extensions (Zhang et al., 21 Mar 2026). MER2026 adds preference supervision and physiological signals, making it possible to train reward models for emotional language and to couple open-vocabulary generation with continuous valence-arousal trajectories or EEG-fNIRS encoders (Lian et al., 21 Apr 2026). The large-scale OV-MER benchmark argues for larger, multilingual, and multicultural datasets and for stronger video-LLMs specialized for fine-grained affect (Han et al., 24 Dec 2025). The VAD-bridge literature suggests another direction: jointly modeling discrete labels, continuous affective coordinates, and free-text descriptors in a shared latent space, so that OV-MER can switch among symbolic, continuous, and generative outputs without changing task definition (Jia et al., 2024).

In that sense, OV-MER is best understood not as a single architecture but as a research program. Its core claim is that multimodal emotion recognition should move from fixed-label discrimination toward semantically grounded, explanation-capable, and lexically flexible emotion understanding.