Integrated MPC-RL Framework

• Integrated MPC-RL Framework is a hybrid approach that combines MPC’s real-time optimization and constraint satisfaction with RL’s adaptive policy improvement.
• It employs neural residuals to adjust dynamics and cost parameters on-the-fly, ensuring safety while adapting to unpredictable, complex environments.
• Empirical results in bipedal locomotion tasks show significant improvements in success rates and tracking error reductions across challenging terrains.

An integrated Model Predictive Control–Reinforcement Learning (MPC-RL) framework refers to approaches that fuse the real-time optimization and constraint satisfaction of MPC with the adaptivity and policy improvement capabilities of RL. These architectures are especially prominent in settings that demand both safety-critical control and on-the-fly adaptation to complex, uncertain, or unmodeled dynamics, such as agile bipedal locomotion on challenging terrain. The unification leverages the ability of RL to optimize policy components and adapt to model mismatch, while retaining the recursive feasibility and robustness properties that are central to MPC.

1. Problem Statement and Motivation

MPC provides tractable online optimization subject to dynamics, constraints, and objectives, but accuracy depends critically on the system model, which often limits robustness and adaptivity—particularly in environments with nonstationary or hard-to-model disturbances (e.g., variable friction, uncertain contact, terrain geometry). RL, by contrast, directly adapts to environmental feedback and can, in principle, discover robust control strategies; however, RL alone does not guarantee satisfaction of hard constraints and often struggles to generalize or ensure safety during learning.

The motivation for integrating MPC and RL is to attain the robustness and constraint-satisfaction of MPC while introducing adaptability and sample efficiency gains from RL components, particularly through online learning of dynamics residuals, trajectory parameters, or control schedules. This hybridization is especially relevant in highly dynamic, contact-rich robotics, exemplified by bipedal locomotion tasks requiring rapid online adaptation for stability and terrain negotiation (Kamohara et al., 22 Sep 2025).

2. Mathematical Framework

In a prototypical integrated MPC-RL architecture for bipedal locomotion (Kamohara et al., 22 Sep 2025), the MPC backbone employs an SRB (Single-Rigid-Body) model with a 13-dimensional state vector: $$x \in \mathbb{R}^{13} = [p \in \mathbb{R}^3; \ \phi \in\mathbb{R}^3; \ v \in\mathbb{R}^3; \ \omega \in\mathbb{R}^3; \ 1]$$ where $p$ is the CoM position, $\phi$ is orientation, $v$ is linear velocity, $\omega$ is angular velocity, and the scalar $1$ ensures gravity is consistent in state-space form.

The control $u \in \mathbb{R}^{12}$ comprises the wrenches at two feet, and the MPC optimization is a standard QP: $$\min_{x_{0:H},u_{0:H-1}} \sum_{k=0}^{H-1} (x_{k+1} - x_{k+1}^{ref})^T Q (x_{k+1} - x_{k+1}^{ref}) + u_k^T R u_k$$ subject to dynamics, contact mode, and friction constraints: $x_{k+1} = A x_k + B u_k,\quad g(x_k, u_k) \le 0$ The QP is constructed such that contact and frictional force constraints are strictly enforced, and the optimization horizon typically covers 0.25 s (H = 10, dt = 0.025 s).

RL augments the MPC by predicting parameter residuals that directly adapt:

• Linear and angular accelerations in the SRB dynamics
• Diagonal entries of the actuation mapping
• Swing-leg Bezier trajectory parameters
• Gait frequency (sampling time adjustment)

Actions $a_t\in\mathbb{R}^{15}$ are output by an MLP policy trained via Soft Actor-Critic (SAC), and modify the MPC cost and dynamic matrices on-the-fly by applying neural residuals: $$A_{lin} \leftarrow A_{lin} + \delta A(a_t),\quad B_{lin} \leftarrow B_{lin} + \delta B(a_t)$$ Swing-leg apex and duration are tuned via RL outputs, and the MPC QP is solved with these adapted parameters at every control cycle. This results in an online, hierarchical closed loop as follows:

1. Observe current state $o_t$
2. Sample action $a_t\sim\pi(\cdot|o_t)$
3. Update MPC parameters using $a_t$
4. Solve QP for optimal wrenches $u_t$, then apply low-level PD control to map $u_t$ to joint torques
5. Observe next state and reward, store for RL update
6. Update policy every $N_{rl}$ steps with SAC gradient steps

3. Integration Strategies and Theoretical Properties

The key integration mechanism is hierarchical: RL produces neural residuals that modify only the parameters of the linearized MPC problem, not the problem's fundamental structure or constraints. This guarantees that the resulting QP remains convex and constraint satisfaction is never lost—even as the neural policy adapts the cost, swing trajectory, and timing in real-time (Kamohara et al., 22 Sep 2025).

Crucially, strict constraint satisfaction is enabled for all training and deployment phases: no RL action can induce infeasibility, as all actions map to parametric modifications of feasible QPs. The architecture ensures that, as the neural network learns to compensate for unmodeled effects (e.g., terrain irregularities, slippage, foot contact errors), robustness is achieved without sacrificing safety.

Empirically, ablation studies confirm that removing any of the major RL-augmented parameters (dynamics, swing, frequency) results in a significant drop in success rate, particularly in the most difficult environments (e.g., pyramid stairs), indicating tight coupling is necessary for high performance.

4. Empirical Validation and Quantitative Impact

The RL-augmented MPC framework has been extensively validated on simulated bipedal locomotion with the HECTOR humanoid across varied terrain classes (Kamohara et al., 22 Sep 2025):

• Pyramid stairs (step up to 10 cm, 1% SR for baseline MPC, 86% SR for RL-MPC)
• Random stairs (15% vs. 85% SR)
• Stepping stones (68% vs. 89% SR)
• Slippery surfaces ($\mu=0.05$, 5% vs. 74% SR)

Velocity-tracking error is reduced by 10–40% compared with classical MPC. RL-augmented MPC yields a 20–85 percentage point improvement in task success rate, depending on the terrain. Tracking error of the CoM velocity also decreases substantially, reflecting enhanced robustness under model uncertainty.

5. Representative Architectures in Literature

A variety of integrated MPC-RL frameworks have been proposed in different contexts, reflecting the diversity in how the RL and MPC layers are coupled:

• Direct parameter augmentation: RL predicts residuals that augment MPC cost and dynamics matrices, as in bipedal locomotion (Kamohara et al., 22 Sep 2025).
• Value blending: RL value function blends MPC local Q-approximations with learned value functions for robust long-horizon planning, leveraging TD($\lambda$)-style interpolation (Bhardwaj et al., 2020).
• Terminal cost transfer: RL is used off-line to train a value function, which then forms the terminal cost in a receding-horizon MPC (Msaad et al., 16 Jun 2025).
• Safety filters: MPC-derived safe sets constrain RL policies during training; at deployment, lightweight Lipschitz filters ensure that RL actions remain within MPC-certified bounds (Kostelac et al., 14 Dec 2025).
• Hierarchical architectures: RL selects high-level actions (e.g., footstep locations or tactical goals) that are tracked via low-level constrained MPC (Studt et al., 19 Sep 2025, Bang et al., 25 Jul 2024).
• Differentiable MPC: Full end-to-end learning is achieved by embedding a KKT-differentiable MPC problem as a policy or value function module, propagating RL gradients through the optimization layer (Romero et al., 2023, Amos et al., 2018, Lawrence et al., 1 Apr 2025).

6. Advantages, Limitations, and Extensions

The hierarchical (parameter-residual) integration strategy offers several decisive benefits:

• Safety and feasibility: Convexity and constraint satisfaction are preserved under all neural parameterizations.
• Adaptivity: Neural residuals rapidly adapt to unmodeled disturbances, irregularities, or parameter drift without retraining the entire MPC structure.
• Sample efficiency: By restricting RL search to residual, (near-)optimal policy spaces, sample complexity and training time are minimized.
• Real-time performance: The architecture incurs negligible additional online computation, as only parametric modifications precede the QP solve at each cycle.

Limitations are tied to the parameterization: RL must be restricted to perturbative residuals to prevent loss of convexity and feasibility. Moreover, while empirical robustness is strong, stability proofs in the presence of arbitrary time-varying neural residuals remain challenging.

Extensions include learning time-varying or state-dependent QP weights and constraints, integration of exteroceptive feedback for terrain adaptation, and generalization to manipulation and multi-agent domains. The modular architecture—neural augmentation of MPC structures—has proved broadly applicable across legged locomotion, manipulation, energy systems, and aerospace guidance (Reiter et al., 4 Feb 2025, Romero et al., 2023, Chen et al., 2023).

7. Outlook and Significance

The integrated MPC-RL framework represents a paradigm shift for safety-critical, high-performance control in high-dimensional, underactuated, and variable environments. The architecture effectively resolves the tension between flexibility (RL) and safety (MPC) by architecturally constraining neural policy learning to the parametric modification of a convex, constraint-enforcing controller. The approach is empirically validated with substantial performance gains in domains where previous methods failed to generalize, and forms a canonical example for modern adaptive control in robotics and autonomous systems (Kamohara et al., 22 Sep 2025, Chen et al., 2023, Kostelac et al., 14 Dec 2025).

---

References

• "RL-augmented Adaptive Model Predictive Control for Bipedal Locomotion over Challenging Terrain" (Kamohara et al., 22 Sep 2025)
• "MPC-Guided Safe Reinforcement Learning and Lipschitz-Based Filtering for Structured Nonlinear Systems" (Kostelac et al., 14 Dec 2025)
• "Learning Agile Locomotion and Adaptive Behaviors via RL-augmented MPC" (Chen et al., 2023)
• "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning" (Romero et al., 2023)
• "MPCritic: A plug-and-play MPC architecture for reinforcement learning" (Lawrence et al., 1 Apr 2025)
• "Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification" (Reiter et al., 4 Feb 2025)
• "Blending MPC & Value Function Approximation for Efficient Reinforcement Learning" (Bhardwaj et al., 2020)
• "RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control" (Bang et al., 25 Jul 2024)
• "Safe Reinforcement Learning Using Robust MPC" (Zanon et al., 2019)