Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation (2311.16091v1)

Abstract: Deep reinforcement learning (DRL) provides a promising way for intelligent agents (e.g., autonomous vehicles) to learn to navigate complex scenarios. However, DRL with neural networks as function approximators is typically considered a black box with little explainability and often suffers from suboptimal performance, especially for autonomous navigation in highly interactive multi-agent environments. To address these issues, we propose three auxiliary tasks with spatio-temporal relational reasoning and integrate them into the standard DRL framework, which improves the decision making performance and provides explainable intermediate indicators. We propose to explicitly infer the internal states (i.e., traits and intentions) of surrounding agents (e.g., human drivers) as well as to predict their future trajectories in the situations with and without the ego agent through counterfactual reasoning. These auxiliary tasks provide additional supervision signals to infer the behavior patterns of other interactive agents. Multiple variants of framework integration strategies are compared. We also employ a spatio-temporal graph neural network to encode relations between dynamic entities, which enhances both internal state inference and decision making of the ego agent. Moreover, we propose an interactivity estimation mechanism based on the difference between predicted trajectories in these two situations, which indicates the degree of influence of the ego agent on other agents. To validate the proposed method, we design an intersection driving simulator based on the Intelligent Intersection Driver Model (IIDM) that simulates vehicles and pedestrians. Our approach achieves robust and state-of-the-art performance in terms of standard evaluation metrics and provides explainable intermediate indicators (i.e., internal states, and interactivity scores) for decision making.

• The paper introduces a DRL method enhanced with internal state inference to predict and adapt to the intentions and actions of surrounding agents.
• It employs counterfactual reasoning and graph neural networks to model dynamic interactions in complex, multi-agent environments.
• Experimental results in a simulated intersection demonstrate superior navigation success, collision avoidance, and driving efficiency compared to baselines.

Enhancing Autonomous Navigation via Intrinsic Behavior Modeling

Introduction to Interactive Autonomous Navigation

Autonomous vehicles (AVs) navigating in urban environments must engage with other road users, such as vehicles and pedestrians, making decisions complex. Traditional Deep Reinforcement Learning (DRL) approaches face challenges in these multi-agent settings, particularly due to their black-box nature and suboptimal performance when interactions are dense and highly dynamic. To improve upon this, researchers have proposed a novel DRL framework that introduces mechanisms for reasoning about the transient intentions and persistent traits of surrounding agents.

Understanding Internal States and Agent Interactions

AVs must be adept at understanding other road users' behaviors which are influenced by latent characteristics – aggressive drivers might take fewer precautions than conservative ones, affecting how AVs should respond. The proposed method infers these internal states and integrates them with DRL. This inference offers a clearer decision-making indicator, as the AV learns to anticipate actions like yielding or acceleration by observing others' behaviors.

To achieve this, the framework employs counterfactual reasoning to consider scenarios both with and without the AV present, thereby understanding how its presence changes others' actions. Additionally, a graph neural network captures the interactions and relationships between agents, enhancing the predictive accuracy and strategic choices of the AV.

Estimating Interactivity Between Agents

Not all agents in the environment equally impact the AV's decision-making process. Recognizing this, interactivity estimation is introduced, measuring how the expected trajectories of other agents shift in the presence of the AV. This metric quantifies the AV's influence over other agents, aiding it in determining when it is crucial to negotiate or when it can safely proceed with less caution.

Experimental Validation and Practical Contributions

Researchers developed an intersection driving simulator based on the Intelligent Intersection Driver Model (IIDM) to validate the approach. It simulates a mix of aggressive and conservative drivers as well as pedestrians, reflecting real-world complexity. The approach outperforms baselines in various metrics including successful navigation completion, collision avoidance, and driving efficiency.

The researchers present three primary contributions:

1. A DRL method enhanced with internal state inference of surrounding drivers, trajectory predictions, and interactivity estimation for more refined autonomous navigation and richer interpretability of the AV's decision-making process.
2. A sophisticated simulation environment for testing and benchmarking across different traffic scenarios, focusing particularly on a challenging partially controlled intersection.
3. Demonstrated robustness and performance; the method shows superior decision-making over current models, providing a strong foundation for future advancements in AV systems.

These advancements signify a step forward in creating AVs that can seamlessly blend into traffic by understanding the nuanced behaviors of human drivers and dynamically adapting to the environment. As AVs become more widespread, the need for such sophisticated, understandable, and reliable decision-making frameworks will become increasingly critical.