Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance (2403.18036v1)

Abstract: Despite significant advancements in text-to-motion synthesis, generating language-guided human motion within 3D environments poses substantial challenges. These challenges stem primarily from (i) the absence of powerful generative models capable of jointly modeling natural language, 3D scenes, and human motion, and (ii) the generative models' intensive data requirements contrasted with the scarcity of comprehensive, high-quality, language-scene-motion datasets. To tackle these issues, we introduce a novel two-stage framework that employs scene affordance as an intermediate representation, effectively linking 3D scene grounding and conditional motion generation. Our framework comprises an Affordance Diffusion Model (ADM) for predicting explicit affordance map and an Affordance-to-Motion Diffusion Model (AMDM) for generating plausible human motions. By leveraging scene affordance maps, our method overcomes the difficulty in generating human motion under multimodal condition signals, especially when training with limited data lacking extensive language-scene-motion pairs. Our extensive experiments demonstrate that our approach consistently outperforms all baselines on established benchmarks, including HumanML3D and HUMANISE. Additionally, we validate our model's exceptional generalization capabilities on a specially curated evaluation set featuring previously unseen descriptions and scenes.

Language-guided Human Motion Generation with Scene Affordance

Introduction

The challenge of generating language-guided human motion within 3D environments remains substantial due to the complexity of jointly modeling natural language, 3D scenes, and human motion. This complexity is compounded by the scarcity of comprehensive, high-quality language-scene-motion datasets. A novel two-stage framework that leverages scene affordance as an intermediate representation aims to address these challenges. The proposed framework consists of an Affordance Diffusion Model (ADM) for predicting explicit affordance maps and an Affordance-to-Motion Diffusion Model (AMDM) to generate plausible human motions. This approach evidentially surpasses traditional models on benchmarks, including HumanML3D and HUMANISE, and demonstrates superior capabilities in generalizing to unseen scenarios.

Related Work

The integration of language, human motion, and 3D scenes for guided motion generation has seen significant advancements, primarily focusing on combining two of these elements at a time. This includes work on 3D Vision-Language (3D-VL) tasks and conditional human motion generation based on past motions, audio, action labels, natural language descriptions, and 3D scenes. However, these approaches often stumble when it comes to generating semantically driven, scene-aware motions due to challenges like multimodal alignment and data scarcity. The proposed method differentiates itself by utilizing scene affordance maps to bridge the gap between 3D scene understanding and conditioned motion generation, hence addressing the shortcomings of current methodologies.

Methodology

The proposed method employs scene affordance as an intermediary for enhanced 3D scene grounding and enriched motion generation under multimodal condition signals. The first stage, ADM, predicts an affordance map given a 3D scene and language description by employing the Perceiver architecture. The second stage, AMDM, synthesizes human motions by integrating both the language descriptions and the affordance maps generated in the first stage, using a structure that includes an affordance encoder and a Transformer backbone.

Experiments

Extensive experiments demonstrate the proposed approach's effectiveness over existing baselines on established benchmarks, including HumanML3D and HUMANISE. Numerical results indicate a superior performance in text-to-motion generation tasks, underlining the model's innovative capacity to generalize across previously unseen language-scene pairs. This showcases not only the method's practical applicability but also its theoretical contributions to understanding the geometric interplay between scenes and human motions.

Contributions

This work introduces a practical framework for language-guided human motion generation in 3D environments that effectively leverages scene affordance for improved performance and generalization. The benefits of using an affordance map as an intermediate representation are demonstrated through quantitative evaluations, showcasing advancements over existing motion generation models. Additionally, the research illuminates pathways for future developments in AI, especially in enhancing model generalization capabilities in the face of limited training data.

Implications

The theoretical implications of this research are profound. This work advances the understanding of how scene affordances can serve as a practical tool for improving the generalization of generative models, especially in areas characterized by data scarcity. Practically, the proposed approach can significantly benefit applications that require the generation of human motions within 3D environments based on natural language input, such as virtual reality, animation, and interactive AI systems.

Future Directions

Despite its successes, the method is not without limitations, such as the dependence on diffusion models which results in slower inference times. Addressing these challenges, alongside the persistent issue of dataset scarcity, constitutes a crucial direction for future research. Advancements in these areas could further unlock the potential of generative AI in simulating complex human-environment interactions.