Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification (2502.02133v1)

Abstract: The fields of MPC and RL consider two successful control techniques for Markov decision processes. Both approaches are derived from similar fundamental principles, and both are widely used in practical applications, including robotics, process control, energy systems, and autonomous driving. Despite their similarities, MPC and RL follow distinct paradigms that emerged from diverse communities and different requirements. Various technical discrepancies, particularly the role of an environment model as part of the algorithm, lead to methodologies with nearly complementary advantages. Due to their orthogonal benefits, research interest in combination methods has recently increased significantly, leading to a large and growing set of complex ideas leveraging MPC and RL. This work illuminates the differences, similarities, and fundamentals that allow for different combination algorithms and categorizes existing work accordingly. Particularly, we focus on the versatile actor-critic RL approach as a basis for our categorization and examine how the online optimization approach of MPC can be used to improve the overall closed-loop performance of a policy.

• The paper surveys and classifies methods for synthesizing Model Predictive Control (MPC) and Reinforcement Learning (RL), identifying three key strategies: MPC as an expert actor, MPC within the deployed policy, and MPC as a critic.
• Synthesizing MPC and RL leverages MPC's strengths in handling constraints and stability with RL's ability to learn optimal policies from data, enhancing decision-making in stochastic environments.
• Challenges in integrating MPC and RL include computational complexity, especially in high-dimensional spaces, requiring advancements in real-time computation and scalable software frameworks.

Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification

The paper "Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification" systematically explores the complementary paradigms of Model Predictive Control (MPC) and Reinforcement Learning (RL) and their potential for synthesis to enhance decision-making in Markov Decision Processes (MDPs).

Both MPC and RL are cornerstone methodologies in designing control systems for stochastic environments. MPC has traditionally been rooted in optimization-based control, known for handling constraints and system stability rigorously. It solves a finite horizon problem and applies the first action in a receding horizon strategy. RL, on the other hand, emphasizes maximizing expected returns through interaction with the environment, typically without requiring a model of the environment.

The paper categorizes hybrid approaches into three high-level strategies:

1. MPC as an Expert Actor: MPC can serve as an expert for RL by generating high-quality trajectories that RL can mimic through imitation learning. This not only initializes RL with good policies but also provides a structured way to explore the action space.
2. MPC within the Deployed Policy: In this setup, MPC is used not just as an expert but as a real-time component of the policy that RL optimizes. This involves parameterized MPCs where parameters (potentially including the model, cost weights, or constraints) are learned to improve overall closed-loop performance.
3. MPC as a Critic: Here, MPC provides approximations of value functions (critic) for policies that RL optimizes. This role capitalizes on MPC's ability to provide structured, optimization-based feedback on policies, thereby potentially enhancing the sample efficiency and stability of the learning process.

The survey highlights how these combinations can lead to improved solutions in various applications, such as robotics and autonomous systems. Specifically, hybrid approaches harness MPC's strength in dealing with constraints and guarantees of stability while utilizing RL's strengths in learning optimal policies from data.

A significant challenge in joining MPC with RL lies in the computational complexity and real-time application, especially under derivative-based MPC constraints when dealing with high-dimensional state-action spaces. The integration of MPC into RL frameworks requires careful consideration of how both paradigms handle uncertainty, constraints, and model fidelity.

Future work suggested in the paper includes enhancing capabilities in real-time computation, addressing the scalability of solutions, and developing software frameworks that seamlessly integrate these approaches. Moreover, there is a call for extending theoretical work, particularly in understanding stability and robustness guarantees when integrating these methods.

By addressing these aspects, the synthesis of MPC and RL is poised to advance the state-of-the-art in designing intelligent, autonomous systems capable of robustly managing real-world dynamics and uncertainties.