DMER-Ranker: Learning to Rank Emotion Descriptions in the Absence of Ground Truth

Abstract: Descriptive Multimodal Emotion Recognition (DMER) is a newly proposed task that aims to describe a person's emotional state using free-form natural language. Unlike traditional discriminative methods that rely on predefined emotion taxonomies, DMER provides greater flexibility in emotional expression, enabling fine-grained and interpretable emotion representations. However, this free-form prediction paradigm introduces significant challenges in evaluation. Existing methods either depend on ground-truth descriptions that require substantial manual effort or simplify the task by shifting the focus from evaluating descriptions to evaluating emotion labels. However, the former suffers from the labor-intensive collection of comprehensive descriptions, while the latter overlooks critical aspects such as emotional temporal dynamics, intensity, and uncertainty. To address these limitations, we propose DMER-Ranker, a novel evaluation strategy that reformulates the traditional prediction-ground truth'' comparison into theprediction-prediction'' comparison, eliminating the need for ground-truth descriptions. We then employ the Bradley-Terry algorithm to convert pairwise comparison results into model-level rankings. Additionally, we explore the possibility of automatic preference prediction and introduce DMER-Preference, the first preference dataset specifically designed for human emotions. Our work advances the field of DMER and lays the foundation for more intelligent human-computer interaction systems.