Descriptive Multimodal Emotion Recognition

• DMER is a paradigm that infers and articulates emotions using multimodal inputs in free-form natural language, capturing intensity, uncertainty, and context.
• It shifts from fixed categorical labels to expressive, fine-grained emotion understanding through joint optimization of speech, text, image, and video cues.
• Evaluation methods and architectures emphasize modality alignment, interpretability, and robustness under missing inputs to enhance descriptive accuracy.

Descriptive Multimodal Emotion Recognition (DMER) is a multimodal emotion understanding paradigm in which a model infers a person’s emotional state from modality-compound inputs and expresses that state in richer forms than a fixed label set, up to and including free-form natural language descriptions. In its contemporary formulation, DMER is explicitly defined as recognizing emotion from multimodal inputs and expressing it in free-form natural language rather than in a predefined taxonomy, with descriptions that may capture intensity, uncertainty, temporal change, and multimodal evidence (Lian et al., 6 Jul 2025). Across recent research, DMER also denotes a broader transition from closed-set multimodal emotion recognition toward fine-grained, descriptive, and explainable emotion understanding, including open-vocabulary labels, continuous affective representations, preference modeling over descriptions, and instruction-following generative systems (Lian et al., 21 Apr 2026, Zhang et al., 20 May 2026).

1. Conceptual scope and relation to conventional multimodal emotion recognition

Classical multimodal emotion recognition systems typically map speech, text, image, or video inputs to a small number of discrete emotion classes. DMER departs from that formulation by treating emotion as something that may be mixed, weak, context dependent, temporally evolving, and only partially captured by a taxonomy. In the explicit task definition of DMER, the model produces a natural-language description that can mention multiple emotional cues at once, including the likely emotion, its intensity, uncertainty, temporal change, and the multimodal evidence behind the judgment (Lian et al., 6 Jul 2025).

This shift is aligned with a broader recasting of multimodal emotion recognition as multimodal emotion understanding. In the MER-with-LLMs paradigm, a model receives multimodal input $X$ together with an instruction $Q$ and generates a textual response $R=[r_1,r_2,\dots,r_n]$, with autoregressive generation formalized as

$$r_i = \arg\max_{r \in \mathcal{V}} p(r \mid X, Q, R_{<i}),$$

so that outputs may be labels, explanations, or open-ended descriptions (Zhang et al., 20 May 2026). The same survey states that this paradigm “deepens the MER task itself, extending classification toward explanation,” which places descriptive emotion output at the center rather than the periphery of the task (Zhang et al., 20 May 2026).

The challenge literature makes the same trajectory explicit. MER2023 concentrated on discriminative emotion recognition with fixed basic labels, whereas MER2024 and MER2025 transitioned to generative emotion understanding and introduced fine-grained emotion recognition and descriptive emotion analysis; MER2026 continues this direction with tracks including MER-FG for fine-grained emotion recognition and MER-Prefer for human preference over emotion descriptions (Lian et al., 21 Apr 2026). This suggests that DMER is not only a modeling choice but also an evaluation and benchmark design problem.

2. Representation learning foundations in early and intermediate DMER-related systems

A substantial part of DMER-related research emerged from multimodal classification systems that already modeled complementary emotional evidence across modalities and, in some cases, exposed part of that evidence through interpretable mechanisms. A representative early example is the hybrid deep multimodal framework for sentence-level spoken language emotion recognition, which uses paired spoken utterances and transcripts rather than raw speech or text alone (Gu et al., 2018). Its multimodal inputs include pre-trained word2vec vectors of dimension 300, part-of-speech tags encoded with learned embeddings, Mel-frequency spectral coefficients with delta and double-delta features, and a 6382-dimensional handcrafted acoustic vector extracted by OpenSMILE containing pitch, energy, jitter, shimmer, MFCC-related descriptors, zero-crossing rate, and multiple functionals (Gu et al., 2018). The architecture combines a text ConvNet, a POS ConvNet, a CNN-LSTM MFSC branch, and a DNN that transforms handcrafted acoustic descriptors into high-level acoustic associations; fusion is performed by concatenation followed by a three-layer deep neural network with a final softmax classifier (Gu et al., 2018). On IEMOCAP, the full model achieves 60.4% weighted accuracy over five emotion classes, and joint training outperforms separate training by 2.7% weighted accuracy (Gu et al., 2018). Although this system remains classification-oriented, it establishes several ingredients later central to DMER: joint multimodal optimization, modeling of temporal dynamics, and explicit use of complementary low-level and high-level affective cues.

A second line of work introduces interpretability more directly. The hybrid-fusion system "ParallelNet" for paired speech utterances and corresponding images uses a log-mel spectrogram of size $128 \times 128 \times 1$ and an image of size $128 \times 128 \times 3$, combines intermediate and late fusion, and supplements prediction with a divide-and-conquer Shapley-value explanation method for both speech and image features (Kumar et al., 2022). The model’s final multimodal accuracy on the IIT-R SIER dataset is 83.29%, and the explanation procedure identifies important speech spectrogram regions and image regions for a particular emotion class (Kumar et al., 2022). In DMER terms, this is still not free-form description, but it moves beyond bare label assignment by exposing which multimodal evidence drove the decision.

Context-rich visual systems extend this pattern by incorporating scene and semantic information around the target person. On EMOTIC, a model that combines face, body, pose, scene, semantic, and depth/proximity features uses a ResNet-18 scene encoder pretrained on Places365, a Vision Transformer-style semantic encoder, a modified EmbraceNet for body-pose fusion, and a depth CNN, reaching 40.39% mAP across 26 emotion categories (Wang et al., 2023). Its ablations show that local personal features alone yield 33.52 mAP, while adding scene and semantic features raises performance to 39.00 mAP, and the full combination reaches 40.39 mAP (Wang et al., 2023). This demonstrates that descriptive multimodal emotion understanding is not limited to face or voice: contextual objects, environments, and proximity cues can materially alter the inferred affective state.

3. Fine-grained multimodal structure: disentanglement, alignment, semantic fusion, and missing modalities

A major technical theme in DMER-related research is that descriptive emotion understanding depends on how multimodal affective structure is organized internally. Fine-grained disentangled representation learning (FDRL) addresses modality heterogeneity, distribution gaps, and information redundancy by projecting each modality into modality-shared and modality-private subspaces through a shared encoder $\mathbb E_s(\cdot;\theta_s)$ and private encoders $\mathbb E_i(\cdot;\theta_i)$, producing shared representations $S_i$ and private representations $P_i$ (Sun et al., 2023). It then uses a global domain discriminator, a local subdomain discriminator, a dynamic factor $Q$0, a modality discriminator, an orthogonality penalty, and a fine-grained predictor that preserves emotion-label consistency. After stacking $Q$1, the model applies multi-head self-attention for cross-modal fusion and achieves 78.34% WAR and 79.44% UAR on IEMOCAP, outperforming compared baselines including MISA and SDTF-Net (Sun et al., 2023). The design is directly relevant to DMER because it explicitly separates modal consistency from modality-specific nuance.

Decoupled Hierarchical Multimodal Distillation (DHMD) advances the same logic by decoupling each modality into modality-irrelevant homogeneous features and modality-exclusive heterogeneous features, then applying two-stage knowledge distillation: coarse-grained graph distillation through GD-Units and fine-grained distillation through cross-modal dictionary matching (Li et al., 4 Feb 2026). Its total objective

$Q$2

balances task prediction, decoupling, graph distillation, and dictionary alignment (Li et al., 4 Feb 2026). The paper reports relative improvements on CMU-MOSI and CMU-MOSEI and provides visual evidence that graph edges and dictionary activations exhibit meaningful distribution patterns across homogeneous and heterogeneous spaces (Li et al., 4 Feb 2026). This is significant for DMER because descriptive outputs require multimodal evidence that is aligned without collapsing modality-specific detail.

A related approach, DeepMSI-MER, uses BERT for text, Wav2Vec 2.0 for audio, Swin Transformer and TCN for video, and a three-stage pipeline of modality-specific feature extraction, early feature fusion, and late fusion with contrastive learning (Dai et al., 12 Feb 2025). Its distinctive mechanism is visual sequence compression guided by a high-level semantic representation $Q$3 fused from text and audio, so that emotionally irrelevant visual frames can be filtered while preserving useful information through selective remapping (Dai et al., 12 Feb 2025). The final loss combines cross-entropy and contrastive learning,

$Q$4

and the model reports average accuracy/F1 of 84.7/84.7 on IEMOCAP and 69.4/67.9 on MELD (Dai et al., 12 Feb 2025). The paper is not framed as DMER in the strict free-form sense, but its emphasis on high-level semantic guidance and cross-modal alignment is closely aligned with descriptive emotion modeling.

Robustness under missing modalities is another structurally important issue. In audio-visual dimensional emotion recognition on RECOLA, dynamic ensemble selection methods such as DS, DW, DWS, and Meta-DW are evaluated against a cross-attention baseline when one modality is replaced by zeros or mean feature values (Menon et al., 2024). The study reports that audio is particularly strong for arousal and video is more useful for valence; for example, the best individual arousal feature is Acoustic at 0.69 ± 0.06 CCC, while the best valence feature is Geometric at 0.56 ± 0.14 CCC (Menon et al., 2024). Across most missing-modality cases, all dynamic selection methods outperform the mean baseline, and cross-attention is often weaker, especially on arousal (Menon et al., 2024). For DMER, this is a reminder that descriptive output quality depends not only on semantic expressiveness but also on whether the model can remain stable when one evidential channel disappears.

4. From discrete labels to continuous and open-vocabulary emotion description

One direct bridge from conventional MER to DMER is the replacement of fixed labels by continuous affective geometry. A multimodal strategy for complex emotion detection maps facial expressions, voice tones, and transcripts into a three-dimensional Valence-Arousal-Dominance (VAD) space and then uses K-means to transit emotions from traditional discrete categorization to a continuous labeling system (Jia et al., 2024). The framework keeps the ME2E Lite multimodal backbone—MTCNN-based facial extraction, CNN layers for spectrogram-like audio features, ALBERT for transcripts, Transformer-based temporal contextualization, and weighted fusion—but replaces the final discrete classification head with a continuous regression head predicting valence, arousal, and dominance (Jia et al., 2024).

The central mechanism is to use the NRC-VAD lexicon to assign VAD values to emotion words, extract 195 emotion words, choose six anchors aligned with the dataset—happy, sad, worried, surprised, angry, and neutral, with neutral manually assigned $Q$5—and run K-means with $Q$6 initialized by those anchors (Jia et al., 2024). The clustering groups emotion words by proximity in VAD space, enabling mapping from continuous predictions back to discrete categories and, more importantly for DMER, generation of nearby descriptive emotion terms such as Alert, Excited, Confused, Curious, Shocked, Caring, Relaxed, Calm, and Humorous (Jia et al., 2024). On MER2024, the proposed VAD model improves over a raw VAD baseline on the continuous task, reducing L2 from 0.96 to 0.64, MSE from 0.38 to 0.19, and MAE from 0.48 to 0.36 while increasing PCC from -0.01 to 0.47; on the discrete task it matches ME2E Lite in F1 at 0.42 while yielding higher precision and recall (Jia et al., 2024). The reported mean similarity between MER2024 discrete ground truth and MER2024 open-set labels is about 0.879, the proposed VAD open set reaches about 0.85 mean similarity, and ME2E Lite reaches about 0.86 (Jia et al., 2024). The paper is explicit that the objective is not absolute state-of-the-art classification accuracy but preservation of competitive recognition quality while expanding expressiveness.

The survey of MER-with-LLMs organizes the descriptive turn more systematically. It divides the literature into Affective Data Augmentation, Multimodal Affective Representation, and Multimodal Affective Reasoning, and emphasizes datasets and benchmarks that evaluate perception, description, ranking, assessment, forecasting, multi-turn understanding, open-vocabulary recognition, emotion conflicts, and hallucination (Zhang et al., 20 May 2026). Among the most DMER-relevant resources it names are OV-MERD for multi-label open-vocabulary emotion descriptions, EEmo-Bench for unified evaluation of perception, description, ranking, and assessment, MVEI for emotion statement judgment, MTMEUR for multi-turn multimodal emotion understanding and reasoning, CA-MER for cross-modal emotion conflicts, and EmotionHallucer for emotion hallucinations in multimodal LLMs (Zhang et al., 20 May 2026). The same survey highlights explainability-focused methods such as ExpLLM, PEP-MEK, Affective-CoT, FIRES, and MultiMood, as well as subjective-reasoning systems such as Agent-MER, AffectGPT-R1, and EmoCaliber that address open-vocabulary emotion recognition, multiple plausible descriptions, and confidence verbalization (Zhang et al., 20 May 2026). In DMER, these lines of work are not auxiliary; they define how descriptive outputs become measurable, interpretable, and compatible with the subjectivity of emotion perception.

5. Evaluation: preference, ranking, and descriptive metrics without exhaustive references

DMER introduces an evaluation problem that differs qualitatively from conventional classification. If model outputs are free-form natural-language descriptions, reference-based comparison is expensive because comprehensive emotion descriptions are labor-intensive to annotate, while label-only scoring ignores temporal dynamics, intensity, uncertainty, cue-based reasoning, and the quality of the generated text (Lian et al., 6 Jul 2025). DMER-Ranker addresses this by replacing “prediction-ground truth” comparison with “prediction-prediction” comparison: for the same video, two model-generated descriptions are compared by human annotators, and the majority-voted preference is aggregated into a pairwise preference matrix $Q$7 (Lian et al., 6 Jul 2025). Global model rankings are then obtained using the Bradley–Terry model,

$Q$8

with parameters estimated by maximizing the likelihood of observed pairwise outcomes (Lian et al., 6 Jul 2025).

To support this protocol, the paper introduces DMER-Preference, described as the first preference dataset specifically designed for human emotions (Lian et al., 6 Jul 2025). It is built by collecting 1,368 video samples, providing two emotion descriptions per video, recruiting three annotators who pass a preliminary emotion-recognition test, and keeping only samples with unanimous agreement, yielding 574 final samples (Lian et al., 6 Jul 2025). Automatic preference prediction is studied with multimodal LLMs under four prompting strategies, and evaluation uses WAF and ACC in two-class and three-class settings together with flip consistency, which measures robustness to reversing the order of the two descriptions (Lian et al., 6 Jul 2025). The paper reports that AffectGPT performs best overall among tested models, with 68.90% two-class WAF and 85.45% flip consistency, but also states that automatic preference prediction is not reliable enough to replace humans and should be treated as a reference rather than a definitive measure (Lian et al., 6 Jul 2025).

The MER challenge series institutionalizes related evaluation ideas. MER-FG evaluates fine-grained open-vocabulary emotion recognition using five emotion wheels and a three-level grouping strategy that normalizes word forms, maps synonyms, and maps outer-wheel labels to innermost basic emotions through

$Q$9

with the final score defined as the average F-score across five wheels (Lian et al., 21 Apr 2026). MER-Prefer formulates human preference toward different emotion descriptions as a two-class preference classification task and uses weighted F1-score as the primary metric and accuracy as the secondary metric; its reported zero-shot baselines include Qwen2.5-VL at 76.77/77.84 and Qwen2.5-Omni at 78.74/78.89 for WAF/ACC (Lian et al., 21 Apr 2026). Together, these evaluation designs show that DMER requires metrics sensitive to semantic proximity, descriptive adequacy, and human preference rather than only category correctness.

6. Deployment, limitations, and research directions

DMER-related systems also appear in deployment-oriented settings. A practical end-to-end video multimodal emotion recognition system processes raw videos of arbitrary length by extracting audio with librosa, segmenting utterances with Silero VAD, detecting and cropping faces from five sampled frames with MTCNN, transcribing speech with Silero Models, and then feeding five face images, 10 seconds of audio, and 100 tokens into a multimodal model with InceptionResNet, DistilHuBERT, and ALBERT backbones followed by 1D convolution, averaging, concatenation, and two linear layers (Lee et al., 2023). The system treats emotion recognition as multi-label classification, displays utterance-level emotion probabilities and thresholded bars through an interface, and summarizes the whole video by averaging utterance-level probabilities; on CMU-MOSEI it reports F1 score 48.3 and accuracy 72.7 (Lee et al., 2023). The paper is explicit that it is only weakly descriptive: it offers probabilities and utterance-wise transparency rather than verbal explanations.

Several limitations recur across the literature. Some systems are evaluated on a single dataset or on relatively small datasets, which constrains generalization; this is stated explicitly for the VAD-based complex emotion detection framework and is also implicit in studies on RECOLA and IEMOCAP (Jia et al., 2024, Menon et al., 2024, Gu et al., 2018). Cross-cultural transfer remains uncertain when the affective lexicon is English-centric, even if the dataset itself is culturally consistent, as in MER2024 (Jia et al., 2024). Missing modalities remain a practical problem, and robustness results indicate that modality contributions are asymmetric across affective dimensions, with audio more informative for arousal and video more informative for valence on RECOLA (Menon et al., 2024). DHMD notes that it is not integrated with multimodal foundation models, suggesting another gap between strong specialized architectures and the emerging large-model paradigm (Li et al., 4 Feb 2026).

The survey literature identifies the field’s main bottlenecks as sparse emotional labels, cross-modal affective gaps, and interpretability (Zhang et al., 20 May 2026). Its future directions—Subjectivity-embraced Framework, Unified and Generalized MER, Mechanism-level Exploration for MER, Agentic Emotion Understanding, and Safety, Bias, and Cultural Adaptation—indicate where DMER is likely to develop next (Zhang et al., 20 May 2026). The challenge roadmap is consistent with that forecast: MER-Cross moves from individual to dyadic interaction scenarios, MER-FG expands open-vocabulary recognition, MER-Prefer formalizes human preference over emotion descriptions, and MER-PS extends emotion recognition to physiological signals (Lian et al., 21 Apr 2026). A plausible implication is that DMER will continue to broaden from descriptive labeling toward fully generative, evidence-grounded, preference-aware emotion understanding, but the current literature also indicates that descriptive richness, faithfulness, cultural robustness, and automatic evaluation remain unresolved technical problems rather than settled capabilities.