Long-Term Human Trajectory Prediction using 3D Dynamic Scene Graphs (2405.00552v4)

Abstract: We present a novel approach for long-term human trajectory prediction in indoor human-centric environments, which is essential for long-horizon robot planning in these environments. State-of-the-art human trajectory prediction methods are limited by their focus on collision avoidance and short-term planning, and their inability to model complex interactions of humans with the environment. In contrast, our approach overcomes these limitations by predicting sequences of human interactions with the environment and using this information to guide trajectory predictions over a horizon of up to 60s. We leverage LLMs to predict interactions with the environment by conditioning the LLM prediction on rich contextual information about the scene. This information is given as a 3D Dynamic Scene Graph that encodes the geometry, semantics, and traversability of the environment into a hierarchical representation. We then ground these interaction sequences into multi-modal spatio-temporal distributions over human positions using a probabilistic approach based on continuous-time Markov Chains. To evaluate our approach, we introduce a new semi-synthetic dataset of long-term human trajectories in complex indoor environments, which also includes annotations of human-object interactions. We show in thorough experimental evaluations that our approach achieves a 54% lower average negative log-likelihood and a 26.5% lower Best-of-20 displacement error compared to the best non-privileged (i.e., evaluated in a zero-shot fashion on the dataset) baselines for a time horizon of 60s.

• The paper introduces a language-driven probabilistic modeling approach that uses 3D dynamic scene graphs to extend prediction horizons to 60 seconds.
• It leverages large language models to forecast human-object interactions and encode detailed environmental context for improved trajectory prediction.
• Extensive validation on a novel semi-synthetic dataset demonstrates reduced errors, advancing proactive human-robot collaboration in dynamic spaces.

Exploring Long-term Human Trajectory Prediction with 3D Dynamic Scene Graphs

Introduction to Human Trajectory Prediction

Long-term Scenario: Understanding human motion in environments bustling with activity, like offices or homes, is crucial for robots designed to interact and coexist with humans. Current trajectory prediction methods focus mainly on short spans (up to 10 seconds), largely for collision avoidance. These methods generally falter in complex, interaction-dense settings typical of human-centered spaces.

Breaking New Ground: This setup extends the prediction horizon notably up to 60 seconds, handling the nuances of human-object interactions and their impacts on movement, a significant leap from standard short-term forecasting methods. The tool used for this broadened perspective? A blend of LLMs for predicting interactions and 3D dynamic scene graphs (DSGs) for rich, contextual environment modeling.

Key Components and Methods

1. Utilizing 3D Dynamic Scene Graphs: The research leverages DSGs to encode detailed information about the environment, such as geometry and traversability, into a structured, hierarchical format that machines can understand and process. This complex representation allows for more nuanced reasoning about the space humans are navigating.
2. Predicting Human-Environment Interactions: At the heart of their approach is the predictive power of LLMs, which estimate potential human-object interactions within the environment. This prediction isn't just about where a human might move but involves understanding potential interactions with objects in their path, adding a layer of depth to trajectory forecasting.
3. Probabilistic Trajectory Modeling: Once potential interactions are identified, the system employs probabilistic models (specifically, continuous-time Markov Chains) to map these interactions into likely pathways and positions in a given timeframe. Referred to in the paper as the Language-driven Probabilistic Long-term Prediction (LP^2), this method focuses on where humans might end up, factoring in their interactions with their surroundings.
4. Extensive Validation with a Novel Dataset: Recognizing the lack of suitable datasets for training and testing their model, the researchers developed a new semi-synthetic dataset featuring complex indoor environments and detailed annotations of human-object interactions. Their method excelled in experimental benchmarks, notably reducing errors in trajectory prediction over a 60-second horizon.

Practical Implications and Theoretical Contributions

Beyond Collision Avoidance: By enhancing trajectory prediction from mere collision avoidance to include proactive interaction understanding, this research paves the way for robots that can more effectively assist, collaborate with, and adapt to humans in shared spaces.

Open-Source Resources: The commitment to open-source sharing of their method and dataset encourages further development and application of this advanced prediction framework, potentially accelerating advancements in robotic systems designed for complex human environments.

Foundation for Future Innovation: The incorporation of LLMs into trajectory prediction models opens intriguing possibilities for even more dynamic interaction models, potentially extending to predicting interactions among multiple agents or more complicated human behaviors.

Future Directions in AI and Robotics

The integration of LLMs for interaction prediction and dynamic scene graphs for environmental modeling sets a promising direction for the development of intelligent systems that understand and anticipate human needs and actions. Future enhancements could include refining these models for greater accuracy over longer periods or expanding them to predict interactions in even more complex or unpredictable environments. As AI continues to evolve, the intersection with robotics in shared human spaces remains a fertile ground for groundbreaking research and transformative applications.