Deep Imitative Models for Flexible Inference, Planning, and Control

Abstract: Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.

• The paper introduces a deep imitative model merging imitation learning with goal-driven planning for flexible, autonomous control.
• It leverages probabilistic predictive models to produce interpretable expert-like trajectories and adapts to unanticipated goals.
• Experiments in CARLA demonstrate state-of-the-art performance with improved collision avoidance and lane adherence over baselines.

Deep Imitative Models for Flexible Inference, Planning, and Control: A Synopsis

The paper "Deep Imitative Models for Flexible Inference, Planning, and Control" presents an innovative approach to autonomous behavior learning by integrating the strengths of Imitation Learning (IL) and goal-directed planning. It introduces "Imitative Models," which are probabilistic predictive models that facilitate the planning of interpretable, expert-like trajectories to meet specified goals.

Summary of Contributions

The authors propose a novel framework that merges the flexibility of model-based reinforcement learning (MBRL) with the efficiency of IL, aiming to provide a robust model capable of pursuing new tasks at test time without the need for explicit reward engineering. The key contributions of this work can be summarized as follows:

• Interpretable Expert-like Plans: The methodology developed produces multi-step expert-like trajectories, enhancing interpretability compared to traditional IL approaches.
• Goal Flexibility: Unlike conventional IL, the proposed method can achieve new, unanticipated goals at test time through a set of derived flexible goal objectives.
• Robustness: The proposed models demonstrate resilience to poorly specified goals, maintaining performance even when faced with suboptimal goal input.
• State-of-the-Art Performance: The approach significantly outperformed six IL methods and one MBRL method in simulations involving dynamic autonomous driving tasks, evidencing its practical efficacy.

Methodological Insights

The authors formalize their approach in the context of continuous-state, discrete-time, partially-observed Markov processes. They focus on constructing a model that learns expert behavior dynamics. Using deep neural architectures, they fit an Imitative Model to forecast expert trajectories, powered by a probabilistic interpretation that captures the inherent stochasticity of expert behavior.

The core innovation lies in their categorical decomposition of goal objectives into constrained and unconstrained categories, involving:

1. Constraint-Based Planning: Utilizing fixed-destination scenarios (e.g., waypoint paths).
2. Unconstrained Planning: Employing likelihood-driven variable goals (e.g., Gaussian mixtures over potential final states).
3. Costed Planning: Incorporating test-time-specific costs, such as avoiding obstacles not encountered during training (e.g., unseen potholes).

These planning objectives allow for flexible incorporation of novel tasks without additional model training, using the learned imitative prior as a behavioral guide.

Practical Implications

Applying their approach to the CARLA driving simulator, the authors trained their models using simulation data of autonomous driving to replicate expert-like driving behavior. The robustness of their models was demonstrated through metrics such as success rate, collision avoidance, and lane adherence.

The models demonstrated scalable applicability for changing situations and scenarios not explicitly covered during training, making them highly suitable for dynamic, real-world environments. Moreover, the method alleviates the need for extensive hyper-parameter tuning typically associated with reward-based coaching in reinforcement learning.

Future Directions

This work opens several potential avenues for future exploration:

• Real-world Deployment: Extending the approach to real-world settings and collecting data from physical sensors could validate and refine model robustness.
• Complex Task Integration: Incorporating more complex multi-agent interactions and dynamic environments to simulate urban driving conditions.
• Interactive Model Feedback: Developing systems where model feedback and human interaction mutually enhance learning processes.

In conclusion, the presented work lays a foundation for developing highly adaptable autonomous systems that leverage offline learning to deliver goal-directed behavior in versatile, real-time environments. The ability to plan effectively without intensive reward engineering is a notable advantage that could reshape how autonomous systems are trained and deployed.