Scene Transformer: A unified architecture for predicting multiple agent trajectories (2106.08417v3)

Abstract: Predicting the motion of multiple agents is necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g. vehicles and pedestrians) and their associated behaviors may be diverse and influence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent predictions. However, planning against independent predictions can make it challenging to represent the future interaction possibilities between different agents, leading to sub-optimal planning. In this work, we formulate a model for predicting the behavior of all agents jointly, producing consistent futures that account for interactions between agents. Inspired by recent LLMing approaches, we use a masking strategy as the query to our model, enabling one to invoke a single model to predict agent behavior in many ways, such as potentially conditioned on the goal or full future trajectory of the autonomous vehicle or the behavior of other agents in the environment. Our model architecture employs attention to combine features across road elements, agent interactions, and time steps. We evaluate our approach on autonomous driving datasets for both marginal and joint motion prediction, and achieve state of the art performance across two popular datasets. Through combining a scene-centric approach, agent permutation equivariant model, and a sequence masking strategy, we show that our model can unify a variety of motion prediction tasks from joint motion predictions to conditioned prediction.

Scene Transformer: A Unified Architecture for Predicting Multiple Agent Trajectories

The paper presents the Scene Transformer, a novel architectural model developed to forecast the trajectories of multiple agents in dynamic environments, particularly addressing the complexities inherent in autonomous driving scenarios. This task is essential for the navigation and planning required in densely populated urban environments where vehicles, pedestrians, and other objects exhibit diverse behaviors and interactions.

Key Contributions

The Scene Transformer model introduces several noteworthy advancements in trajectory prediction:

1. Unified Framework: Unlike traditional models that predict trajectories independently for each agent, the Scene Transformer predicts the behavior of all agents collectively. This joint prediction approach results in trajectories that are inherently consistent with the interactions of agents, providing a more cohesive and reliable forecast.
2. Scene-Centric Representation: The model uses a scene-centric approach, leveraging a permutation equivariant Transformer-based architecture. This setup efficiently encodes and processes scenes by focusing on agents, time, and road graph elements concurrently, leveraging self-attention mechanisms. The model maintains permutation equivariance, meaning it does not depend on the order of input, thereby managing complex agent interactions more effectively.
3. Masking Strategy: Drawing inspiration from LLMing, it employs a masking strategy akin to BERT’s to condition the predictions on various factors like the autonomous vehicle’s (AV) future trajectory or goals. This allows the model to adapt to different types of prediction tasks, including motion prediction, conditional motion prediction, and goal-conditioned prediction, using a single architecture.
4. Efficient Loss Formulations: The paper discusses a nuanced loss formulation switch between marginal (individual) predictions and joint (collective) predictions, enabling flexibility in optimizing for either task without altering the model's fundamental structure.

Experimental Evaluation

The model was rigorously evaluated on standard datasets like Argoverse and the Waymo Open Motion Dataset (WOMD), demonstrating state-of-the-art performance in both joint and marginal motion prediction tasks:

• For Argoverse, the Scene Transformer exhibits superior performance on metrics such as minADE and minFDE, emphasizing the precision and accuracy of its predictions.
• On the WOMD, which presents both marginal and joint interaction prediction challenges, the model excels by generating predictions that respect interactions between agents, notably achieving lower inter-agent prediction overlap rates when compared to traditional marginal models.

Implications and Future Directions

The Scene Transfomer’s ability to monitor and predict multiple agents enables practical applications in developing robust autonomous navigation systems. By facilitating consistent multi-agent trajectory forecasts, this model addresses key challenges in planning and decision-making processes for AVs, potentially enhancing safety and efficiency in autonomous systems.

The implications for future development include deeper integrations of motion prediction and planning into cohesive, end-to-end autonomous driving models. Additionally, extending this approach to other domains requiring collaborative multi-agent prediction, such as robotics and traffic management systems, remains a promising avenue.

Conclusion

The Scene Transformer represents a significant step forward in multi-agent trajectory prediction by fostering joint consistency and adaptability through its core architectural innovations. Its application to autonomous driving demonstrates important capabilities that enhance current methodologies, setting a strong foundation for future advancements in AI-driven mobility and beyond.