DMER-Ranker: Multimodal Emotion Evaluation

• The paper introduces DMER-Ranker, an evaluation framework that replaces traditional reference matching with direct pairwise comparison of model-generated emotion descriptions.
• It employs the Bradley–Terry algorithm to aggregate human preference judgments, transforming noisy pairwise outcomes into reliable model-level rankings.
• The approach is further enhanced by the DMER-Preference dataset, which benchmarks automatic preference prediction in multimodal emotion recognition scenarios.

DMER-Ranker is an evaluation strategy for Descriptive Multimodal Emotion Recognition (DMER), a task in which a model generates a free-form natural-language description of a person’s emotional state from multimodal input rather than assigning a predefined emotion label. Its central reformulation is to replace traditional “prediction-ground truth” comparison with “prediction-prediction” comparison, so that two model-generated descriptions for the same video are judged directly against each other, without requiring ground-truth descriptions. Model-level rankings are then derived from pairwise outcomes using the Bradley–Terry algorithm, and the framework is further extended with DMER-Preference, a preference dataset for human-emotion judgments, together with experiments on automatic preference prediction by multimodal LLMs (MLLMs) (Lian et al., 6 Jul 2025).

1. Task setting and evaluation motivation

DMER is defined as a setting that goes beyond assigning a discrete emotion label and instead asks a system to generate a free-form natural-language description of emotion from multimodal evidence. In this formulation, a model may describe not only an emotion category, but also emotion category, intensity, uncertainty, temporal change, and multimodal clues such as facial expression, gesture, vocal tone, or head movement. The underlying premise is that emotion is not always well captured by a single taxonomy-bound label, especially when the affective state is ambiguous, mixed, temporally dynamic, or only partially observable (Lian et al., 6 Jul 2025).

The evaluation problem follows directly from that formulation. A single ground-truth description is expensive to annotate and may still be incomplete, because different annotators may reasonably emphasize different cues or different aspects of the same emotional episode. The paper argues that this makes conventional text-to-reference evaluation both costly and brittle. It also argues that label-based evaluation is too narrow, because it overlooks temporal dynamics, intensity, uncertainty, and whether the generated description actually reflects multimodal evidence. In that sense, DMER-Ranker is designed to evaluate the quality of the description as an explanation of emotion in context, rather than the correctness of a coarse label alone.

A frequent misconception is that the absence of ground-truth descriptions implies a relaxation of evaluation rigor. DMER-Ranker adopts the opposite stance: the judged object is still the model output, but the comparison is shifted from reference matching to relative human preference conditioned on the original video. This preserves the multimodal grounding requirement while avoiding reliance on a single textual target.

2. Pairwise preference protocol

The core protocol compares two models on the same sample. For models $m_i$ and $m_j$, and a sample $x^k$, the models generate descriptions $d_i^k$ and $d_j^k$. Human annotators then inspect the video and decide which description better captures the character’s emotional state. The allowable outcomes are description 1, description 2, or tie. The paper states that multiple annotators are used for reliability, and the majority-voted preference for sample $k$ is denoted $p^k_{ij}$ (Lian et al., 6 Jul 2025).

What is compared in this protocol is not lexical overlap. The judgment target includes whether the description is emotionally appropriate in context, whether it reflects multimodal clues, whether it handles uncertainty or intensity sensibly, and whether it is grounded in the video rather than guessed. This is important because, in DMER, there may be no uniquely correct wording. Two descriptions may both be partially valid while differing in emphasis or interpretive framing. Pairwise comparison is therefore used as a relative assessment of which description is better, not which one matches a canonical reference.

This protocol changes the semantics of evaluation in a consequential way. Instead of asking whether a prediction is close to a gold string, it asks whether one prediction is preferable to another when both are presented with the original multimodal evidence. The resulting supervision is ordinal rather than referential, which is particularly suitable for subjective, open-ended affective description.

3. Bradley–Terry aggregation and model-level ranking

After pairwise outcomes are collected, DMER-Ranker uses the Bradley–Terry model to derive model-level rankings. Each model $m_i$ is assigned a positive latent strength parameter $\theta_i > 0$, and the probability that $m_i$ beats $m_j$0 is

$m_j$1

The pairwise results are summarized in a preference matrix $m_j$2, where

$m_j$3

and the parameters are estimated by maximum likelihood. The paper gives the likelihood as

$m_j$4

and optimizes the equivalent log-likelihood objective

$m_j$5

The paper states that gradient descent is used to estimate $m_j$6, after which models are ranked by descending $m_j$7 (Lian et al., 6 Jul 2025).

This aggregation step is what converts local, noisy, and potentially inconsistent pairwise judgments into a global ranking. A concrete scale example given in the paper uses $m_j$8 models and $m_j$9 samples, for a total of

$x^k$0

pairwise comparisons. The Bradley–Terry formulation provides a principled way to compress such comparisons into a single ranking over models without requiring direct score calibration across all systems.

4. DMER-Preference dataset and automatic preference prediction

To study whether pairwise emotional preference judgments can be automated, the paper introduces DMER-Preference, described as the first preference dataset focused specifically on human emotions. The dataset construction starts from 1,368 video samples; for each video, annotators are presented with two emotion descriptions and asked which one better matches the character’s emotional state. The annotation protocol uses three annotators, and only samples with unanimous agreement are kept, yielding a final dataset of 574 samples (Lian et al., 6 Jul 2025).

Each sample contains a video and two competing descriptions. The label space is three-way: description 1 preferred, description 2 preferred, or tie. Because ties are reported to be ambiguous and difficult to agree on, the paper evaluates both two-class settings, excluding ties, and three-class settings, including them. The dataset integrates text, image/video, and audio modalities, which is a notable distinction from preference datasets that target only generic text or generic vision-language tasks.

The paper also reports inter-annotator consistency. It is about 70% when ties are excluded, and about 10% lower when ties are included. This is significant because it places an empirical ceiling on purely automatic judging: the task itself contains a nontrivial degree of ambiguity, especially around tie cases.

Automatic preference prediction is studied through four prompting strategies for MLLMs:

Strategy	Procedure
S1	Input the video plus two candidate descriptions directly to an MLLM and ask it to choose the better one
S2	First ask the MLLM to generate a detailed description of the video, then use that generated description as a reference to compare the two candidates
S3	Same as S2, but the final comparison is done by an external LLM instead of the MLLM itself
S4	Further split the second step into explicit reasoning plus preference decision

These strategies are used to test whether direct comparison, generated-reference comparison, external-LLM comparison, or explicit reasoning yields more reliable preference judgments (Lian et al., 6 Jul 2025).

5. Metrics, systems, and empirical findings

The paper evaluates automatic preference prediction with two groups of metrics. The first is recognition performance, measured by WAF and ACC, reported in both two-class and three-class settings. The second is flip consistency, which measures whether the model gives the same preference when the order of the two descriptions is reversed; this is intended to detect position bias. The paper also studies multi-run consistency, i.e., stability across repeated runs (Lian et al., 6 Jul 2025).

The experimental suite includes a broad set of open-source and closed-source MLLMs that support at least audio or video input. Examples listed in the paper include AffectGPT, Qwen2.5-VL, Qwen2.5-Omni, GPT-4o, Gemini-2.0-Flash, Gemini-2.5-Flash, GPT-4.1, Gemini-1.5-Pro, and Gemini-1.5-Flash. The reported results show that current MLLMs are only moderately good judges of emotion-description preference, and many are near chance or only modestly above it.

Among the reported systems, the best open-source overall is AffectGPT, with two-class WAF = 68.90 and flip consistency = 85.45. Strong closed-source results include Gemini-1.5-Flash, with two-class WAF = 64.64 and flip consistency = 72.04, and GPT-4.1, with two-class WAF = 60.75 and flip consistency = 80.84. The paper explicitly notes that some models exhibit high flip consistency but low recognition performance, indicating that robustness to input order and correctness of preference prediction are distinct properties.

The strategy comparison is not uniform across models. The paper states that S3/S4 often outperform S1/S2 for many systems, which suggests that decomposing the decision through a generated reference or an external LLM can improve preference prediction when direct comparison is weak. However, it also reports that this is not universal: for some stronger language-centric models, longer reasoning chains can introduce error accumulation and degrade performance. The paper further studies model-based crowdsourcing by filtering models whose recognition and flip consistency both exceed 60% and then applying majority voting among the top-$x^k$1 models. Performance improves with a small ensemble and then deteriorates as noisier judges are added, with the reported sweet spot around $x^k$2.

6. Interpretation, limitations, and relation to adjacent ranking literature

DMER-Ranker is best understood as an evaluation framework for a setting in which free-form emotion description lacks a unique, comprehensive textual ground truth. Its main claim is not that pairwise ranking eliminates subjectivity, but that it handles subjectivity more appropriately than direct reference matching. The human-agreement figures in DMER-Preference show why: even human annotators achieve only about 70% agreement when ties are excluded, and agreement drops when ties are included. This indicates that ambiguity is intrinsic to the task rather than merely a failure of annotation (Lian et al., 6 Jul 2025).

The framework also has explicit limitations. The paper states that automatic preference prediction is not yet sufficiently accurate to replace humans. It identifies only moderate accuracy for MLLM judges, sensitivity to input order, multi-run instability, and model dependence, since the best prompting strategy varies across systems. For that reason, automatic evaluation is presented as a reference, not a definitive judge, and the paper suggests practical future workflows such as fewer comparison samples, hierarchical pairwise ranking, and model-assisted but human-verified evaluation.

Within the broader arXiv literature, DMER-Ranker should be distinguished from other methods that also involve “ranking” or “rankers” but solve different problems. In recommendation, DRM addresses differentiable optimization of top-$x^k$3 ranking metrics for recommender systems (Lee et al., 2020). In generative commonsense reasoning, DKMR$x^k$4 uses metric-guided distillation to train a cross-encoder ranker and then distill it into a retriever (He et al., 2022). In two-stage recommendation, CoRR studies cooperative retriever–ranker training with KL-based distillation (Huang et al., 2022). In image captioning, a belief-revision re-ranker uses visual-semantic relatedness to reorder beam candidates (Sabir et al., 2022). In online information retrieval evaluation, Multi-Dueling Bandits and MergeDTS study online ranker comparison under click feedback rather than evaluation of free-form emotion descriptions (Brost et al., 2016, Li et al., 2018). These adjacent lines of work are methodologically relevant because they also operate on pairwise preference, ranking, or ranker selection, but their evaluation target is distinct.

The distinctive contribution of DMER-Ranker is therefore specific: it reframes evaluation for multimodal emotion description generation as a relative preference problem over model outputs, uses Bradley–Terry aggregation to derive global model rankings, introduces DMER-Preference as a benchmark for human-emotion preference prediction, and empirically shows that current automatic judges remain useful but unreliable substitutes for human evaluation. In the context of affective computing and human-computer interaction, this formulation supports systems aimed at emotion-aware assistants, conversational agents, social robots, affective tutoring systems, and emotionally intelligent interfaces, where the quality of a description may depend as much on interpretive adequacy and multimodal grounding as on categorical correctness (Lian et al., 6 Jul 2025).