---
title: 'DMER-Ranker: Multimodal Emotion Evaluation'
url: https://www.emergentmind.com/topics/dmer-ranker
type: topic
---

# DMER-Ranker: Multimodal Emotion Evaluation

DMER-Ranker is an evaluation strategy for **Descriptive Multimodal Emotion Recognition (DMER)**, a task in which a model generates a **free-form natural-language description** of a person’s emotional state from multimodal input rather than assigning a predefined emotion label. Its central reformulation is to replace traditional **“prediction-ground truth” comparison** with **“prediction-prediction” comparison**, so that two model-generated descriptions for the same video are judged directly against each other, without requiring ground-truth descriptions. Model-level rankings are then derived from pairwise outcomes using the **Bradley–Terry** algorithm, and the framework is further extended with **DMER-Preference**, a preference dataset for human-emotion judgments, together with experiments on automatic preference prediction by multimodal large language models (MLLMs) [2507.04278].

## 1. Task setting and evaluation motivation

DMER is defined as a setting that goes beyond assigning a discrete emotion label and instead asks a system to generate a **free-form natural-language description** of emotion from multimodal evidence. In this formulation, a model may describe not only an emotion category, but also **emotion category, intensity, uncertainty, temporal change, and multimodal clues** such as **facial expression, gesture, vocal tone, or head movement**. The underlying premise is that emotion is not always well captured by a single taxonomy-bound label, especially when the affective state is ambiguous, mixed, temporally dynamic, or only partially observable [2507.04278].

The evaluation problem follows directly from that formulation. A single ground-truth description is expensive to annotate and may still be incomplete, because different annotators may reasonably emphasize different cues or different aspects of the same emotional episode. The paper argues that this makes conventional text-to-reference evaluation both costly and brittle. It also argues that **label-based evaluation is too narrow**, because it overlooks **temporal dynamics, intensity, uncertainty**, and whether the generated description actually reflects multimodal evidence. In that sense, DMER-Ranker is designed to evaluate the **quality of the description as an explanation of emotion in context**, rather than the correctness of a coarse label alone.

A frequent misconception is that the absence of ground-truth descriptions implies a relaxation of evaluation rigor. DMER-Ranker adopts the opposite stance: the judged object is still the model output, but the comparison is shifted from reference matching to **relative human preference conditioned on the original video**. This preserves the multimodal grounding requirement while avoiding reliance on a single textual target.

## 2. Pairwise preference protocol

The core protocol compares two models on the same sample. For models \(m_i\) and \(m_j\), and a sample \(x^k\), the models generate descriptions \(d_i^k\) and \(d_j^k\). Human annotators then inspect the video and decide which description better captures the character’s emotional state. The allowable outcomes are **description 1**, **description 2**, or **tie**. The paper states that **multiple annotators** are used for reliability, and the majority-voted preference for sample \(k\) is denoted \(p^k_{ij}\) [2507.04278].

What is compared in this protocol is not lexical overlap. The judgment target includes whether the description is emotionally appropriate in context, whether it reflects multimodal clues, whether it handles uncertainty or intensity sensibly, and whether it is grounded in the video rather than guessed. This is important because, in DMER, there may be no uniquely correct wording. Two descriptions may both be partially valid while differing in emphasis or interpretive framing. Pairwise comparison is therefore used as a relative assessment of which description is **better**, not which one matches a canonical reference.

This protocol changes the semantics of evaluation in a consequential way. Instead of asking whether a prediction is close to a gold string, it asks whether one prediction is preferable to another when both are presented with the original multimodal evidence. The resulting supervision is ordinal rather than referential, which is particularly suitable for subjective, open-ended affective description.

## 3. Bradley–Terry aggregation and model-level ranking

After pairwise outcomes are collected, DMER-Ranker uses the **Bradley–Terry model** to derive model-level rankings. Each model \(m_i\) is assigned a positive latent strength parameter \(\theta_i > 0\), and the probability that \(m_i\) beats \(m_j\) is

\[
P(m_i \; \text{beats} \; m_j) = \frac{\theta_i}{\theta_i+\theta_j}.
\]

The pairwise results are summarized in a preference matrix \(\mathbf{W}\), where

\[
\mathbf{W}_{ij} = \begin{cases} 1, & m_i \text{ beats } m_j \\
0, & m_i \text{ loses to } m_j \end{cases}
\]

and the parameters are estimated by maximum likelihood. The paper gives the likelihood as

\[
\max L(\theta) = \prod_{i<j}\left(\frac{\theta_i}{\theta_i + \theta_j}\right)^{\mathbf{W}_{ij}}\left(\frac{\theta_j}{\theta_i + \theta_j}\right)^{1-\mathbf{W}_{ij}}
\]

and optimizes the equivalent log-likelihood objective

\[
\min - \sum_{i<j}\left[\mathbf{W}_{ij} \ln{\theta_i} + (1-\mathbf{W}_{ij})\ln{\theta_j}-\ln{(\theta_i + \theta_j)}\right].
\]

The paper states that **gradient descent** is used to estimate \(\theta\), after which models are ranked by descending \(\theta\) [2507.04278].

This aggregation step is what converts local, noisy, and potentially inconsistent pairwise judgments into a global ranking. A concrete scale example given in the paper uses **\(M=10\) models** and **\(N=332\) samples**, for a total of

\[
\binom{M}{2}\times N = 14{,}940
\]

pairwise comparisons. The Bradley–Terry formulation provides a principled way to compress such comparisons into a single ranking over models without requiring direct score calibration across all systems.

## 4. DMER-Preference dataset and automatic preference prediction

To study whether pairwise emotional preference judgments can be automated, the paper introduces **DMER-Preference**, described as the first preference dataset focused specifically on human emotions. The dataset construction starts from **1,368 video samples**; for each video, annotators are presented with **two emotion descriptions** and asked which one better matches the character’s emotional state. The annotation protocol uses **three annotators**, and **only samples with unanimous agreement are kept**, yielding a final dataset of **574 samples** [2507.04278].

Each sample contains a video and two competing descriptions. The label space is three-way: **description 1 preferred**, **description 2 preferred**, or **tie**. Because ties are reported to be ambiguous and difficult to agree on, the paper evaluates both **two-class** settings, excluding ties, and **three-class** settings, including them. The dataset integrates **text, image/video, and audio** modalities, which is a notable distinction from preference datasets that target only generic text or generic vision-language tasks.

The paper also reports inter-annotator consistency. It is **about 70% when ties are excluded**, and **about 10% lower when ties are included**. This is significant because it places an empirical ceiling on purely automatic judging: the task itself contains a nontrivial degree of ambiguity, especially around tie cases.

Automatic preference prediction is studied through four prompting strategies for MLLMs:

| Strategy | Procedure |
|---|---|
| **S1** | Input the video plus two candidate descriptions directly to an MLLM and ask it to choose the better one |
| **S2** | First ask the MLLM to generate a detailed description of the video, then use that generated description as a reference to compare the two candidates |
| **S3** | Same as S2, but the final comparison is done by an external LLM instead of the MLLM itself |
| **S4** | Further split the second step into explicit reasoning plus preference decision |

These strategies are used to test whether direct comparison, generated-reference comparison, external-LLM comparison, or explicit reasoning yields more reliable preference judgments [2507.04278].

## 5. Metrics, systems, and empirical findings

The paper evaluates automatic preference prediction with two groups of metrics. The first is **recognition performance**, measured by **WAF** and **ACC**, reported in both **two-class** and **three-class** settings. The second is **flip consistency**, which measures whether the model gives the same preference when the order of the two descriptions is reversed; this is intended to detect **position bias**. The paper also studies **multi-run consistency**, i.e., stability across repeated runs [2507.04278].

The experimental suite includes a broad set of **open-source and closed-source MLLMs** that support at least audio or video input. Examples listed in the paper include **AffectGPT**, **Qwen2.5-VL**, **Qwen2.5-Omni**, **GPT-4o**, **Gemini-2.0-Flash**, **Gemini-2.5-Flash**, **GPT-4.1**, **Gemini-1.5-Pro**, and **Gemini-1.5-Flash**. The reported results show that current MLLMs are **only moderately good judges** of emotion-description preference, and many are near chance or only modestly above it.

Among the reported systems, the **best open-source overall** is **AffectGPT**, with **two-class WAF = 68.90** and **flip consistency = 85.45**. Strong closed-source results include **Gemini-1.5-Flash**, with **two-class WAF = 64.64** and **flip consistency = 72.04**, and **GPT-4.1**, with **two-class WAF = 60.75** and **flip consistency = 80.84**. The paper explicitly notes that some models exhibit **high flip consistency but low recognition performance**, indicating that robustness to input order and correctness of preference prediction are distinct properties.

The strategy comparison is not uniform across models. The paper states that **S3/S4 often outperform S1/S2** for many systems, which suggests that decomposing the decision through a generated reference or an external LLM can improve preference prediction when direct comparison is weak. However, it also reports that this is **not universal**: for some stronger language-centric models, longer reasoning chains can introduce error accumulation and degrade performance. The paper further studies **model-based crowdsourcing** by filtering models whose recognition and flip consistency both exceed 60% and then applying majority voting among the top-\(N\) models. Performance improves with a small ensemble and then deteriorates as noisier judges are added, with the reported sweet spot around **\(N=4\)**.

## 6. Interpretation, limitations, and relation to adjacent ranking literature

DMER-Ranker is best understood as an evaluation framework for a setting in which **free-form emotion description lacks a unique, comprehensive textual ground truth**. Its main claim is not that pairwise ranking eliminates subjectivity, but that it handles subjectivity more appropriately than direct reference matching. The human-agreement figures in DMER-Preference show why: even human annotators achieve only **about 70%** agreement when ties are excluded, and agreement drops when ties are included. This indicates that ambiguity is intrinsic to the task rather than merely a failure of annotation [2507.04278].

The framework also has explicit limitations. The paper states that automatic preference prediction is **not yet sufficiently accurate to replace humans**. It identifies **only moderate accuracy** for MLLM judges, **sensitivity to input order**, **multi-run instability**, and **model dependence**, since the best prompting strategy varies across systems. For that reason, automatic evaluation is presented as **a reference, not a definitive judge**, and the paper suggests practical future workflows such as **fewer comparison samples**, **hierarchical pairwise ranking**, and **model-assisted but human-verified evaluation**.

Within the broader arXiv literature, DMER-Ranker should be distinguished from other methods that also involve “ranking” or “rankers” but solve different problems. In recommendation, **DRM** addresses differentiable optimization of top-\(K\) ranking metrics for recommender systems [2008.13141]. In generative commonsense reasoning, **DKMR\(^2\)** uses metric-guided distillation to train a cross-encoder ranker and then distill it into a retriever [2210.11708]. In two-stage recommendation, **CoRR** studies cooperative retriever–ranker training with KL-based distillation [2206.14649]. In image captioning, a belief-revision re-ranker uses visual-semantic relatedness to reorder beam candidates [2209.08163]. In online information retrieval evaluation, **Multi-Dueling Bandits** and **MergeDTS** study online ranker comparison under click feedback rather than evaluation of free-form emotion descriptions [1608.06253; 1812.04412]. These adjacent lines of work are methodologically relevant because they also operate on pairwise preference, ranking, or ranker selection, but their evaluation target is distinct.

The distinctive contribution of DMER-Ranker is therefore specific: it reframes evaluation for multimodal emotion description generation as a **relative preference problem over model outputs**, uses **Bradley–Terry** aggregation to derive global model rankings, introduces **DMER-Preference** as a benchmark for human-emotion preference prediction, and empirically shows that current automatic judges remain useful but unreliable substitutes for human evaluation. In the context of affective computing and human-computer interaction, this formulation supports systems aimed at **emotion-aware assistants**, **conversational agents**, **social robots**, **affective tutoring systems**, and **emotionally intelligent interfaces**, where the quality of a description may depend as much on interpretive adequacy and multimodal grounding as on categorical correctness [2507.04278].

Source: https://www.emergentmind.com/topics/dmer-ranker