Training Efficient Controllers via Analytic Policy Gradient (2209.13052v3)

Abstract: Control design for robotic systems is complex and often requires solving an optimization to follow a trajectory accurately. Online optimization approaches like Model Predictive Control (MPC) have been shown to achieve great tracking performance, but require high computing power. Conversely, learning-based offline optimization approaches, such as Reinforcement Learning (RL), allow fast and efficient execution on the robot but hardly match the accuracy of MPC in trajectory tracking tasks. In systems with limited compute, such as aerial vehicles, an accurate controller that is efficient at execution time is imperative. We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. We address training instabilities that frequently occur with APG through curriculum learning and experiment on a widely used controls benchmark, the CartPole, and two common aerial robots, a quadrotor and a fixed-wing drone. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics. To facilitate the exploration of APG, we open-source our code and make it available at https://github.com/lis-epfl/apg_trajectory_tracking.

• The paper introduces a novel APG methodology that uses differentiable simulators to train controllers through backpropagation for precise trajectory tracking.
• It addresses training instabilities with curriculum learning, achieving robust performance on benchmarks like CartPole and aerial robotic tasks.
• Results demonstrate superior tracking accuracy and sample efficiency, offering significant computational savings compared to traditional MPC.

Analytic Policy Gradient for Training Efficient Controllers

The paper "Training Efficient Controllers via Analytic Policy Gradient" investigates the application of Analytic Policy Gradient (APG) methods to the control of robotic systems, specifically focusing on tracking complex trajectories with limited computational resources. The authors address the challenge of achieving Model Predictive Control (MPC) level accuracy with the computational efficiency typically associated with Reinforcement Learning (RL), by leveraging gradient-based training of controllers, facilitated by differentiable simulators.

Overview

The control of robotic systems, particularly aerial vehicles, involves sophisticated optimization to precisely adhere to desired trajectories. Classical approaches such as MPC are capable of excellent tracking performance but are computation-intensive, demanding real-time optimization which is infeasible on systems with limited on-board processing capabilities. RL, while computationally efficient at runtime due to offline-trained policies, often falls short in achieving the precision of MPC.

In response to this dichotomy, the authors propose APG as an intermediate approach, exploiting differentiable simulators to enable offline training of controllers through direct minimization of tracking error gradients. This methodology promises the dual benefits of precision in trajectory tracking and reduced computational load at execution.

Key Contributions

1. APG Methodology: The introduction of a novel control strategy that uses differentiable simulators to compute analytic gradients of reward functions, enabling direct training of control policies through backpropagation. This approach addresses the fundamental compromise between computational demand and control accuracy.
2. Addressing Training Instabilities: The authors tackle the common issues of instability in training, such as vanishing and exploding gradients, by employing curriculum learning strategies. This stabilizes the learning process, enabling reliable control policy learning.
3. Comprehensive Evaluation: The APG method was tested on standard benchmarks, including CartPole and aerial vehicular tasks with quadrotors and fixed-wing drones. The results demonstrate APG's superior performance in tracking error compared to both model-free and model-based RL techniques while showcasing a significant reduction in computation time compared to MPC.
4. Sample Efficiency and Flexibility: Highlighted is the sample efficiency of the APG method, which reduces training data requirements by several orders of magnitude compared to conventional RL. Additionally, its ability to handle high-dimensional inputs and adapt to new environmental dynamics makes it a robust framework for real-world applications.

Practical and Theoretical Implications

The implications of this work are twofold:

• Practical: The presented APG framework advances the field of robotics by offering a scalable control strategy that achieves high precision without the resource burden of traditional MPC. This makes it particularly advantageous for systems with stringent real-time and computational constraints, such as autonomous drones and other agile robotic platforms.
• Theoretical: The integration of differentiable programming within robotics signifies a shift towards more analytically tractable control policy learning. This sets a precedent for future research focusing on leveraging differentiable environments to solve complex control problems with gradient-based methods.

Future Developments

APG represents a significant step towards integrating model-based control paradigms with data-driven techniques. Future research directions may involve enhancing the scalability of APG for more complex, long-horizon tasks and real-world applications. Moreover, exploring robustness against uncertainties in dynamical models and addressing sim-to-real transfer issues will be critical for deploying these methods in diverse environments.

In conclusion, the paper underscores the promise of APG methods in robotics, providing a compelling case for their adoption in scenarios requiring high accuracy and efficiency. This approach marks a stride towards bridging the gap between the computational demands of classical optimization-based control and the flexibility of learning-based paradigms.