MoST: Multi-modality Scene Tokenization for Motion Prediction (2404.19531v1)

Abstract: Many existing motion prediction approaches rely on symbolic perception outputs to generate agent trajectories, such as bounding boxes, road graph information and traffic lights. This symbolic representation is a high-level abstraction of the real world, which may render the motion prediction model vulnerable to perception errors (e.g., failures in detecting open-vocabulary obstacles) while missing salient information from the scene context (e.g., poor road conditions). An alternative paradigm is end-to-end learning from raw sensors. However, this approach suffers from the lack of interpretability and requires significantly more training resources. In this work, we propose tokenizing the visual world into a compact set of scene elements and then leveraging pre-trained image foundation models and LiDAR neural networks to encode all the scene elements in an open-vocabulary manner. The image foundation model enables our scene tokens to encode the general knowledge of the open world while the LiDAR neural network encodes geometry information. Our proposed representation can efficiently encode the multi-frame multi-modality observations with a few hundred tokens and is compatible with most transformer-based architectures. To evaluate our method, we have augmented Waymo Open Motion Dataset with camera embeddings. Experiments over Waymo Open Motion Dataset show that our approach leads to significant performance improvements over the state-of-the-art.

• The paper introduces scene tokenization by fusing multi-modal sensor data, enabling context-aware motion prediction.
• It employs image foundation models and LiDAR networks to generate compact scene tokens for transformer-based motion forecasting.
• Experiments on the Waymo Open Motion Dataset show a 10.3% soft-mAP boost and a 6.6% reduction in minADE over baseline approaches.

An Expert Overview of MoST: Multi-modality Scene Tokenization for Motion Prediction

The paper "MoST: Multi-modality Scene Tokenization for Motion Prediction" proposes a novel approach to enhancing motion prediction for autonomous systems through advanced sensor data representation. The authors, from Waymo LLC, present a methodology that tokenizes the visual scene elements from raw sensor inputs, combining insights from pre-trained image models and LiDAR data to improve motion prediction accuracy and robustness.

Methodological Innovation

Traditionally, motion prediction models in autonomous systems rely on symbolic perception outputs such as bounding boxes and road graphs. However, these symbolic outputs present challenges including a lack of context sensitivity and vulnerability to perception errors. As an alternative, the authors introduce the concept of scene tokenization that efficiently combines symbolic representations with multi-modality data from raw sensors.

The MoST approach employs image foundation models to extract generalized visual world knowledge and LiDAR neural networks to capture geometric representations. These collective insights are encoded into a set of scene tokens—compact representations of multi-modality observations suitable for transformer-based architectures. This methodology facilitates an open-vocabulary scene understanding, allowing for the interpretation of previously unrecognized objects and conditions (such as open-vocabulary obstacles or poor road conditions).

Experimental Foundation

The authors base their evaluations on the Waymo Open Motion Dataset (WOMD) augmented with camera embeddings. The dataset incorporates LiDAR and image data, establishing a robust benchmark for one second of historical data leveraged for predicting future trajectories.

Experimental results demonstrate notable improvements over state-of-the-art baseline approaches. The MoST method enhances prediction metrics such as soft mean Average Precision (soft-mAP) by 10.3% and minimum Average Displacement Error (minADE) by 6.6%. These findings underscore the importance of leveraging multi-modality data fused into enriched scene tokens.

Implications and Future Directions

The practical implications of MoST are significant in the field of autonomous driving and robotics. By bridging the gap between raw sensor data and symbolic perception, MoST offers a refined framework that aligns well with the real-world complexities encountered by autonomous systems.

In terms of future directions, the use of large pre-trained models for scene tokenization presents potential expansion across various autonomous system applications, from robotics to advanced driver-assistance systems (ADAS). The adaptive and versatile nature of MoST might also extend to enhancing interpretability and robustness in other domains reliant on complex scene understanding.

Overall, the MoST method serves as a compelling alternative to current motion prediction standards, offering a structured yet multifaceted approach to improving the interaction of autonomous systems with their environments. As transformer-based frameworks continue to evolve, integrating such multi-modality scene tokenization strategies could become a key component in advancing the efficacy and safety of autonomous technologies.