Chronologically Accurate Retrieval for Temporal Grounding of Motion-Language Models (2407.15408v1)

Abstract: With the release of large-scale motion datasets with textual annotations, the task of establishing a robust latent space for language and 3D human motion has recently witnessed a surge of interest. Methods have been proposed to convert human motion and texts into features to achieve accurate correspondence between them. Despite these efforts to align language and motion representations, we claim that the temporal element is often overlooked, especially for compound actions, resulting in chronological inaccuracies. To shed light on the temporal alignment in motion-language latent spaces, we propose Chronologically Accurate Retrieval (CAR) to evaluate the chronological understanding of the models. We decompose textual descriptions into events, and prepare negative text samples by shuffling the order of events in compound action descriptions. We then design a simple task for motion-LLMs to retrieve the more likely text from the ground truth and its chronologically shuffled version. CAR reveals many cases where current motion-LLMs fail to distinguish the event chronology of human motion, despite their impressive performance in terms of conventional evaluation metrics. To achieve better temporal alignment between text and motion, we further propose to use these texts with shuffled sequence of events as negative samples during training to reinforce the motion-LLMs. We conduct experiments on text-motion retrieval and text-to-motion generation using the reinforced motion-LLMs, which demonstrate improved performance over conventional approaches, indicating the necessity to consider temporal elements in motion-language alignment.