RL-Augmented MPC: Safe, Data-Driven Control

• RL-Augmented MPC Framework is a hybrid approach that integrates reinforcement learning with model predictive control to adaptively tune parameters while ensuring stability and constraint adherence.
• It embeds MPC as a parameterized Q-function within the RL loop, enabling robust exploration, guaranteed constraint handling, and improved economic performance in safety-critical systems.
• The framework leverages structured, model-informed updates to deliver interpretable and reliable policies, making it ideal for applications like process control and energy systems.

A reinforcement learning (RL)-augmented model predictive control (MPC) framework combines learning-based adaptation with the established constrained optimization and stability guarantees of MPC. In this paradigm, MPC is not treated merely as an open-loop optimizer but as a flexible, parameterized function approximator (often for Q-functions or policies) whose key structural ingredients—cost weights, constraint sets, models—are adapted via interaction-driven RL algorithms. By embedding the MPC optimization inside the RL loop, and permitting the RL agent to tune its critical parameters, the resulting controller enjoys both the constraint-handling and stability typical of MPC and the model-free, data-driven optimality of RL. Notably, algorithmic adaptations are required to ensure well-posedness and robust convergence under this highly nonlinear, nonconvex parameterization.

1. Embedding MPC as a Function Approximator in RL

Instead of using generic neural networks or tabular lookups for the Q-function, the RL-augmented MPC scheme parameterizes the Q-function as the value of an MPC problem:

$$Q^N_\theta(s,a) = \min_z \Big\{ \lambda_\theta(x_0) + \gamma^N V^f_\theta(x^N) + \sum_{k=0}^{N-1} \gamma^k \ell_\theta(x_k,u_k) + \sum_{k=0}^N \gamma^k (s_k^\top W_s s_k + w_s^\top s_k) \Big\}$$

subject to dynamic, input, and state constraints:

$$\begin{align*} x_0 &= s \ u_0 &= a \ x_{k+1} &= f_\theta(x_k,u_k) \ g_\theta(u_k) &\leq 0 \ h_\theta(x_k,u_k) &\leq s_k \ h^f_\theta(x^N) &\leq s^N \end{align*}$$

Here, the parameter vector $\theta$ includes weights and possibly parameterizations of the model dynamics themselves. The feedback policy is generated as $\pi_\theta(s) = \arg \min_a Q^N_\theta(s,a)$, and the closed-loop value function is defined by this constrained optimization.

This structural integration enables the RL agent to adjust features with physical meaning (cost weights, constraint priorities, nominal models), ensuring every Q-evaluation is feasible with respect to the system's physical and safety constraints—highly significant in safety-critical domains.

2. Advantages and Theoretical Guarantees

Combining RL with MPC delivers several notable benefits:

• Constraint Enforcement: The embedded MPC structure ensures that action selection inherently respects state, input, and safety constraints.
• Closed-Loop Stability: By enforcing positive definiteness on the stage and terminal cost Hessians ($$\nabla^2 \ell_\theta \succ 0, \nabla^2 V^f_\theta \succ 0$$), the learned policy retains the stability and recursive feasibility guarantees of traditional MPC.
• Interpretability: Tunable parameters have a direct mapping to system and cost features, contrasting with black-box neural policy approximators.
• Robust and Data-Efficient Learning: Initial guesses informed by prior model knowledge serve as warm starts, accelerating and regularizing learning.
• Avoidance of Divergence: Standard RL parameter update (stochastic gradient) steps can be inefficient or divergent due to the complex, nonconvex dependency of $Q_\theta$ on $\theta$; the proposed method addresses this by explicitly enforcing feasibility and stability via constrained second-order updates.

3. RL Algorithmic Adaptation for Stability and Convergence

The RL update is not a naive gradient step, but a constrained quadratic minimization over the TD errors:

$$\min_\theta \sum_{k=0}^n \Big[ \ell(s_k,a_k) + \gamma \min_{a'} Q^N_{\tilde{\theta}}(s_{k+1},a') - Q^N_\theta(s_k,a_k) \Big]^2$$

$$\text{subject to} \;\; \nabla^2 \ell_\theta \succ 0, \;\; \nabla^2 V^f_\theta \succ 0$$

A best-fit update is computed to (locally) minimize the batch TD error, ensuring stability and positive definiteness of cost Hessians at each iteration. This addresses parameter scaling issues (which can destabilize gradient-based RL), and provides robust, descent-guaranteed updates insensitive to scaling—rendering the scheme both data- and computationally efficient.

By contrast, unconstrained or single-step Newton updates reduce to standard Q-learning, which is observed to either fail to converge or require impractically small learning rates in this context.

4. Simulation Example: RL-Augmented MPC in Batch Chemical Evaporation

The framework was validated in a closed-loop simulation of an evaporation process characterized by nonlinear dynamics, bounded state (concentration, pressure) and input (flow rate, control signal) constraints. Salient points include:

• NMPC formulation: Horizon $N=10$, quadratic cost structure, nominal model plus constant offset.
• RL Implementation: Parameters updated via batch policy updates, $\alpha=10^{-2}$, with $\epsilon$-greedy exploration (90% greedy, 10% random bounded perturbation).
• Performance: Strong convergence of tuned parameters, steady drop in average TD error, and superior economic performance—~14% gain over naive tuning, 12% over nominal economic-tuning.
• Constraint Handling: The RL-tuned policy occasionally violates state constraints slightly, intentionally trading off operation cost and constraint penalty for improved mean performance under stochastic perturbations, as captured by the MPC-embedded Q-function's structure.

A summary table of benchmark results:

Controller	Economic Gain (%)	Avg. Constraint Violation	Mean TD Error
RL-tuned NMPC	14 (vs naive)	Minor, when advantageous	Steady decay
Nominal economic	2 (vs naive)	None	Higher, flat
Naive	Baseline	None	Highest

In practice, standard update rules failed to converge or required tiny steps, whereas the proposed batch-constrained fit yielded robust improvements.

5. Numerical and Practical Implications

Key empirical observations:

• Parameter Convergence: RL-tuned MPC parameters stabilize rapidly when updated using the full-convergence, constraint-enforced rule.
• Robustness to Perturbations: The RL-MPC controller tolerates model/plant noise and time-varying perturbations, adaptively violating (soft) state constraints when optimal with respect to cumulative cost and penalty.
• Systematic Improvement Over Ad-Hoc Approaches: Incorporating model knowledge directly in the parametric form outperforms both naive and nominally-tuned controllers.
• Interpretability and Safety: Every learned policy is interpretable in terms of standard control system weighting and constraint parameters, facilitating certification and acceptance in regulated industries.

6. Implementation Considerations

Implementing this RL-augmented MPC framework involves the following key elements:

1. MPC Formulation: Encode the Q-function as a parametric MPC problem with all relevant system, cost, and constraint structures.
2. RL Loop: At each policy update, solve the batch-constrained TD error minimization problem for $\theta$, enforcing positive definiteness of cost Hessians.
3. Exploration: Adopt $\epsilon$-greedy exploration with bounded perturbations to encourage sufficient system excitation under operational constraints.
4. Policy Extraction: Use the (parametric) MPC Q-function to solve for the greedy action via the inner minimization.

Resource requirements depend heavily on the underlying MPC optimization (nonlinear, constrained). Scalability relies on solver capabilities—matrix factorization and constraint Jacobian/Hessian evaluation dominate computation. The approach is well-suited to applications where the number of tunable parameters is modest and precise constraint handling and interpretability are essential.

Observations in the presented industrial example suggest that the approach is particularly well-matched to process industries, energy systems, and other domains in which models can be partially specified and constraint adherence is non-negotiable.

7. Position within RL-MPC Research

The presented approach stands out by embedding the MPC optimization as the RL value approximator, in contrast to approaches using neural or table-based critics. This method leverages both prior control theory (warm-start, structured stability, constraint handling) and empirical data, converging to policies that are both meaningfully interpretable and robustly feasible by construction. The learning paradigm naturally extends to nonlinear, constraint-rich settings.

A plausible implication is that this hybridization of RL and MPC provides a template for self-tuning controllers in safety-critical applications, where policy verification and explicit constraint satisfaction cannot be sacrificed for raw performance alone (Zanon et al., 2019).