ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

Abstract: In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.

• The paper introduces ReactMotionNet, a 151,328-pair dataset with Gold, Silver, and Negative preference tiers that captures multiple valid listener reactions across 8,298 utterances and 47 emotions.
• The ReactMotion model combines multimodal tokenization, a T5-based sequence-to-sequence generator, Gold-likelihood training, and group-wise preference learning to improve reactive appropriateness while controlling repetitive motion.
• Under full conditioning, ReactMotion reaches 0.797 Win(g>S), 0.960 Gen@3, and 4.760 FID, substantially outperforming cascaded and retrieval baselines, although synthetic labels and limited real-world validation remain important limitations.

Task and Motivation

This paper introduces Reactive Listener Motion Generation from Speaker Utterance, a new task in which a model generates the 3D body motion of a listener that appropriately responds to a speaker's utterance, conditioned on any subset of speaker transcript ($T$), audio ($A$), and emotion label ($E$) (2603.15083). Unlike text-to-motion or co-speech gesture generation, where the condition explicitly specifies or is temporally synchronized with the output, the mapping from speaker utterance to listener reaction is inherently one-to-many: the same utterance can elicit multiple valid reactions. The authors identify three obstacles—non-deterministic listener behavior, absence of a large-scale dataset with multiple listener motions per utterance, and the inadequacy of single-reference metrics for evaluating reactive appropriateness—and address each with a dataset, an evaluation protocol, and a generative model.

ReactMotionNet Dataset

ReactMotionNet contains 151,328 labeled (speaker utterance, listener reactive motion) pairs spanning 8,298 unique utterances and 2,029 distinct listener motions, with an average of 18.24 candidate motions per utterance. Each candidate is assigned to one of three preference tiers—Gold (9,307 pairs), Silver (34,196), and Negative (107,825)—capturing graded appropriateness rather than a single ground truth. The dataset covers 47 emotion categories and is split by disjoint speaker utterances at an 8:1:1 ratio.

A notable methodological choice is that the data are not collected via paired speaker–listener motion capture. Instead, a four-step automated pipeline repurposes HumanML3D: LLM-based verifiers filter conversation-relevant motions; OpenAI o3-mini infers plausible speaker transcripts and emotion labels for each reaction; GPT-4o mini TTS synthesizes audio, verified by a speech emotion recognizer (Hume AI); and multiple LLM agents plus a zero-shot NLI model score dyadic appropriateness to assign tier labels. This avoids expensive capture but means both conditions and appropriateness labels derive from synthetic pipelines, so label quality ultimately depends on LLM judgment—a dependency the paper does not quantify against human annotation.

Evaluation Protocol

Because single-reference similarity cannot capture one-to-many reactivity, the authors train a multimodal judge network scoring compatibility between speaker utterances and candidate motion token sequences, evaluated under a Strict-L2 missing-modality protocol where disabled modalities receive information-free inputs. Judge reliability on tiered annotations is strong: Win(G>N) ≈ 0.99, Win(S>N) ≈ 0.98, Win(G>S) ≈ 0.87–0.88 across all six input modes, with MRR(G) ≈ 0.82–0.84 and nDCG@10 ≈ 0.93. Generation quality is then measured by win rates of generated samples against each reference tier (Win(g>G/S/N)), Gen@3, FID, and Diversity. The judge itself is trained contrastively with group-wise InfoNCE objectives over Gold/Silver/Negative sets, a memory bank of generic negatives, and random modality dropout.

ReactMotion Framework

ReactMotion uses modality-specific tokenizers—MiMi audio codec codes and a T2M-GPT VQ-VAE for motion—and a T5-base Seq2Seq backbone whose vocabulary is extended with audio, motion, and emotion tokens (235.9M trainable parameters). Training combines maximum likelihood on Gold reactions with group-wise preference learning: length-normalized log-likelihoods are aggregated per tier via log-mean-exp, and soft-margin ranking losses enforce $$\ell_{\mathcal{G}} > \ell_{\mathcal{S}} > \ell_{\mathcal{N}}$$, with inverse-frequency weighting over motion sequence IDs to prevent dominance of frequent generic motions.

The full sweep over margin $m$, $\lambda_{\text{rank}}$, and $\lambda_{\text{gn}}$ shows moderate settings ($m{=}0.5$, both weights $=0.25$) are optimal; excessive ranking pressure degrades fidelity, indicating the objective sits at a genuine likelihood–preference trade-off.

Results

Under full conditioning ($T{+}A{+}E$), ReactMotion achieves Win(g>N)=1.000, Win(g>S)=0.797, Gen@3=0.960, and FID=4.760 (GT FID is 0.278), versus the strongest cascaded baseline LLM→MG-MotionLLM* at Win(g>S)=0.274 and FID=25.723. Retrieval attains higher diversity (8.207 vs. 4.804) but far lower appropriateness (Win(g>S)=0.252). Text alone is the strongest single modality (Win(g>S)=0.774, FID=4.706); adding emotion to audio yields the best Win(g>G)=0.323. Ablations attribute gains to inverse-frequency reweighting (its removal drops Win(g>G) from 0.266 to 0.220), the ranking loss (removal worsens FID from 4.760 to 5.950 while inflating diversity), and Gold likelihood supervision.

A 59-participant user study corroborates these findings: ReactMotion wins 67.8% against a cross-entropy variant, 72.0% against LLM→MG-MotionLLM*, and receives 44.1% of votes against Silver references—substantially above baselines (~31%) though still below parity, confirming a remaining perceptual gap. Inference runs at roughly 0.60 s per motion sequence end-to-end on a single A100, approaching interactive latency.

Two claims deserve emphasis as potentially contradictory to expectations. First, despite using a powerful fine-tuned motion generator, the cascaded LLM→T2M pipeline performs poorly on strict appropriateness, which the authors attribute to noisy out-of-domain captions and repetitive outputs—suggesting caption-mediated generation is a poor proxy for implicit interaction intent. Second, the CE-trained variant achieves competitive Win rates (Win(g>S)=0.741) yet worse FID and qualitative behavior, implying that appropriateness metrics alone can mask regression toward averaged behaviors.

Limitations and Open Questions

The paper concedes several limitations directly. The architecture is deliberately simple (a T5-base backbone) to ensure training stability, leaving architectural headroom unexplored. Failure cases arise in ambiguous or long-tail situations requiring deeper conversational-intent understanding. The dataset's reliance on synthetic speaker utterances (TTS) and LLM-assigned tier labels introduces potential distribution shift relative to spontaneous human conversation, and the judge-based evaluation inherits whatever biases the contrastively trained judge encodes—the paper validates judge consistency with its own annotations but not against independent human preference judgments at scale. Whether preference-tier supervision transfers to real captured dyadic interactions remains an open question this work leaves unanswered.

Conclusion

This work formalizes reactive listener body-motion generation as a task, supplies a large-scale one-to-many dataset with graded preference tiers, proposes a tier-aware evaluation protocol built on a validated multimodal judge, and demonstrates that group-wise preference learning on a unified multimodal Seq2Seq model substantially outperforms retrieval and cascaded LLM→T2M alternatives in appropriateness, realism, and human preference. Its main open problems are robustness to ambiguous conversational intent and validation beyond synthetically constructed dyadic data.