Dynamic Input for Deep Reinforcement Learning in Autonomous Driving

Abstract: In many real-world decision making problems, reaching an optimal decision requires taking into account a variable number of objects around the agent. Autonomous driving is a domain in which this is especially relevant, since the number of cars surrounding the agent varies considerably over time and affects the optimal action to be taken. Classical methods that process object lists can deal with this requirement. However, to take advantage of recent high-performing methods based on deep reinforcement learning in modular pipelines, special architectures are necessary. For these, a number of options exist, but a thorough comparison of the different possibilities is missing. In this paper, we elaborate limitations of fully-connected neural networks and other established approaches like convolutional and recurrent neural networks in the context of reinforcement learning problems that have to deal with variable sized inputs. We employ the structure of Deep Sets in off-policy reinforcement learning for high-level decision making, highlight their capabilities to alleviate these limitations, and show that Deep Sets not only yield the best overall performance but also offer better generalization to unseen situations than the other approaches.

Dynamic Input for Deep Reinforcement Learning in Autonomous Driving

The paper presents an approach to optimizing decision-making processes in autonomous driving utilizing Deep Reinforcement Learning (DRL). Within the autonomous driving context, making optimal decisions is complex due to the dynamic nature of variables, notably the number of surrounding vehicles which can affect driving strategies substantially. Traditional methods have limitations when applied in these scenarios due to their static input requirements. This research introduces and evaluates novel DRL architectures specifically designed to process inputs of varying sizes, aiming to improve upon traditional approaches such as fully-connected, convolutional, and recurrent neural networks.

Overview of Methods

The core contribution of the paper is the deployment of Deep Sets and Set2Set architectures for handling inputs with dynamic sizes in reinforcement learning frameworks. These architectures offer flexibility and permutation invariance, making them suitable for autonomous driving wherein the number of surrounding objects, such as vehicles, can vary considerably. The paper outlines two main algorithms: DeepSet-Q and Set2Set-Q, each leveraging unique neural network structures to manage variable input lengths effectively.

DeepSet-Q employs a set-based architecture that ensures permutation invariance using a sum pooling function with modules ($\phi$, $\rho$, and Q) designed to handle dynamic inputs of surrounding objects. Conversely, Set2Set-Q uses an LSTM-based recurrent architecture with attention mechanisms to obtain order-agnostic sequence embeddings. Both approaches are implemented within the DRL framework to enhance high-level decision-making processes, specifically focusing on lane-change maneuvers in autonomous driving settings.

Numerical Results

The empirical findings demonstrate that Deep Sets outperform other architectures in terms of overall driving policy effectiveness and generalization capabilities in unseen situations. The research evaluates these models in a simulated environment using SUMO, where agents tackled scenarios with varying numbers of surrounding vehicles. When trained on diverse datasets encompassing a broad range of vehicle configurations, Deep Sets exhibited superior performance with less variance compared to CNNs and Set2Set architectures, especially in dense traffic situations.

In generalization tests, involving both altered numbers of lanes and increased vehicle traffic, Deep Sets showed robustness by maintaining higher performance levels, contrasting the performance dip observed in CNN-based models. The paper suggests that this ability to generalize is fundamental for real-world applications, where vehicle configurations and traffic conditions can fluctuate unpredictably.

Implications and Future Directions

While the paper does not claim revolutionary breakthroughs, the implications of scalable, flexible neural architectures in deep reinforcement learning for autonomous driving are significant. The effectiveness of Deep Sets in dynamic real-world inputs presents an opportunity to enhance driving algorithms beyond predefined static models, paving the way for more responsive and adaptive autonomous systems.

Future developments can build upon this architectural framework to incorporate multiple object types and temporal context consideration. Expanding the application of Deep Sets to real-world systems is an integral next step, requiring further validation with physical vehicles. Moreover, the integration of temporal dependencies, possibly through recurrent features, could potentially elevate the current models' responsiveness.

By introducing and evaluating permutations invariant architectures in DRL, this paper contributes to the ongoing discourse on adaptive AI systems. It offers insightful directions into improving AI strategies in autonomous contexts, emphasizing the importance of handling real-world dynamic inputs efficiently.