Integrated MPC-RL Framework

• The paper presents an integrated control framework that fuses MPC's constraint handling with RL's adaptive policy search for event-triggered decision making.
• It employs a hierarchical policy to jointly tune meta-parameters such as prediction horizon, cost weights, and event triggers, balancing performance and computation.
• Empirical results report a 36% reduction in MPC computation time and an 18.4% improvement in control performance, underscoring its practical efficacy.

An integrated Model Predictive Control–Reinforcement Learning (MPC–RL) framework is an advanced closed-loop control architecture that combines classic MPC—known for constraint handling and model-based prediction—with reinforcement learning, which enables adaptive policy search, end-to-end performance optimization, and tuning of parameters that influence control structure, operation, and computational efficiency. Such hybrid approaches aim to simultaneously optimize control performance, resource utilization, and computational cost by leveraging both model-based and data-driven learning principles in a unified framework (Bøhn et al., 2021).

1. Architecture and Control Workflow

The integrated MPC–RL framework exposes meta-parameters of the MPC controller (such as prediction horizon, cost-function weights, and event-triggering thresholds) as decision variables for the RL agent. At each plant step $t$, the RL-based control loop operates as a hierarchical, event-triggered decision process:

1. The agent observes an augmented state $$s_t = [\bar x_i, \hat p_i, N_i, \bar x_t, \hat p_t, t-i]^\top$$, encoding the last OCP solve, current state and input, and the “age” since the last solve.
2. It samples a binary decision $c_t \sim \textrm{Bernoulli}(w_t)$ (“recompute MPC?”), where $w_t=\sigma\bigl(\pi^{c}_\theta(s_t)\bigr)$ and $\sigma$ is the logistic sigmoid.
3. If $c_t=1$, it samples a prediction horizon $N_t\in\{N_{\min},\ldots,N_{\max}\}$ (from policy $\pi^{N}_\theta$), solves the MPC open-loop optimal control problem (OCP), and applies the first MPC input $u_t = u_0^\textrm{MPC}$, updating all predicted trajectories and LQR gains.
4. If $c_t=0$, it applies $$u_t = u_{t-i}^\textrm{MPC} + K_{t-i}(\hat x_{t-i} - \bar x_t)$$, using previously planned MPC actions with corrective LQR gains computed by linearizing around the last MPC trajectory.
5. All four policy components $\{\pi^c, \pi^N, \pi^M, \pi^{ML}\}$ share parameters $\theta$ and are jointly optimized via PPO.

This structure enables the RL policy to govern both discrete (event-trigger and horizon) and continuous (input selection) meta-parameters, with the event-trigger logic adaptively trading off re-optimization frequency and control performance (Bøhn et al., 2021).

2. Meta-Parameter Optimization and Policy Representation

Key meta-parameters treated as RL actions include:

• Prediction Horizon $N_t\in\{N_{\min},...,N_{\max}\}$, controlling the look-ahead window and impacting both MPC performance and computational load.
• MPC Cost-function Weights $(Q,R)$, shaping stage cost and control priorities—these also define the LQR feedback law between solves.
• Event-trigger Threshold $\delta$, mapped to the Bernoulli parameter of $\pi^c$.

A hierarchical mixture-distribution policy factorizes as

$$\pi_\theta(a|s) = P^c(c|s) \times P^N(N|s) \times P^\mu(u|s,N)$$

where $c$ and $N$ select the structure of the OCP, and the continuous control $u$ is sampled from a Gaussian whose mean is determined by either the new MPC solution (if $c=1$) or the composite MPC-LQR feedback (if $c=0$). This policy parametrization ensures efficient exploration in both discrete and continuous spaces (Bøhn et al., 2021).

3. Optimal Control Problem and Dual-Mode Operation

At each event-triggered recomputation ($c_t=1$), the MPC OCP has the generic form:

$$\begin{aligned} &\min_{x_{0:N_t},\,u_{0:N_t-1}} \sum_{k=0}^{N_t-1}\rho^k\,\ell_{\theta^M}(x_k,u_k,\hat p_{t+k}) + \rho^{N_t}\,m_{\theta^M}(x_{N_t}) \ &\text{s.t.} \quad x_0 = \bar x_t,\quad x_{k+1} = \hat f_{\theta^M}(x_k, u_k, \hat p_{t+k}), \ &\qquad\qquad h_{\theta^M}(x_k, u_k) \le 0, \quad u_k \in \mathcal U,\, x_k \in \mathcal X \end{aligned}$$

For linear-quadratic cases, stage and terminal costs reduce to $\ell(x,u)=x^\top Qx + u^\top Ru$, $m(x)=x^\top P x$.

In between OCP solutions, the control law switches to a dual-mode architecture, applying $$u_t = u_{t-i}^\mathrm{MPC} + K_{t-i}(\hat x_{t-i} - \bar x_t)$$ where $K_{t-i}$ is computed by linearizing the MPC dynamics along the predicted trajectory (Bøhn et al., 2021).

4. Reinforcement Learning Formulation

The RL Markov Decision Process is defined as:

• State: $s_t$ includes all plant states and OCP “history” variables.
• Action: $a_t=(c_t, N_t, u_t^\mathrm{MPC}, u_t^\mathrm{ML})$ draws both discrete recomputation/horizon and continuous MPC or LQR perturbations.
• Reward:

$$r_t = -\ell(\bar x_{t+1}, u_t) - \lambda_h[\textrm{\# constraint violations at } t+1] - \lambda_c c_t N_t$$

with $\lambda_h < 0$ imposing episode-ending penalties for constraints and $\lambda_c > 0$ penalizing time and effort spent in MPC computation.

End-to-end training is performed with Proximal Policy Optimization (PPO), updating all policy parameters through advantage-weighted policy gradients (Bøhn et al., 2021).

5. Event-Triggered Computation and Computational Efficiency

The event-triggered mechanism, governed by the Bernoulli policy’s logit, determines when to recompute the MPC solution. When $c_t=1$, the OCP is re-solved at a chosen horizon; when $c_t=0$, the controller relies on the shifted MPC plan and associated LQR gain. The RL agent thus learns to:

• Invoke long horizons when necessary (e.g., instability, risky regions).
• Avoid unnecessary computation when the system is close to nominal or can be stabilized cheaply by LQR.

Empirical evidence on the inverted pendulum task showed a $36\%$ reduction in total MPC computation time (fewer OCP solves, longer intervals between solves) and an $18.4\%$ improvement in control performance compared to the best fixed-horizon, always-recompute MPC baseline, validating the computational and performance synergy of the approach (Bøhn et al., 2021).

6. Training Procedure and Practical Implementation

The control framework is implemented with multiple parallel actors operating under PPO, with episodes terminated on constraint violation or after $T$ steps. Training involves the following loop:

• Form state $s_t$, sample decision $c_t$.
• If $c_t=1$, sample $N_t$, solve MPC, apply $u_t = u_0^\mathrm{MPC}$.
• If $c_t=0$, apply the next input from the last MPC plan plus LQR correction.
• Store transitions $(s_t, c_t, N_t, u_t, r_t, s_{t+1})$.
• After $Z$ steps, batch PPO updates are performed on the collected trajectories.

Key hyperparameters: $\gamma=0.99$, learning rate $3\times 10^{-4}$, minibatch size $256$, PPO clip $\epsilon=0.25$, value-loss coefficient $0.5$, no entropy bonus.

Efficient real-time application is ensured by embedding the RL policy evaluation (for the event trigger and horizon) immediately before each control/optimization step, leveraging fast warm-started quadratic/convex solvers whose computational cost scales linearly in $N_t$ (Bøhn et al., 2021).

7. Significance, Generality, and Empirical Results

The integrated MPC–RL scheme establishes a flexible and computationally scalable paradigm for algorithmic tuning of predictive controllers. It:

• Automatically selects the sequence of OCP solves (frequency, horizon, and associated cost weights) for each plant state, trading off performance and computation.
• Recovers significant gains over both naive trial-and-error MPC tuning and fixed-parameter deployments, as shown by substantial cost reduction and compute savings.
• Is readily extensible: the mixture-policy and meta-parameter RL setup apply to any prediction-based controller with tunable horizon, weights, and event logic.

The approach has been demonstrated to reduce total closed-loop cost by $21.5\%$ and MPC computation time by $36\%$ on well-established control benchmarks, offering a blueprint for future adaptive and resource-aware controllers in embedded and fast real-time environments (Bøhn et al., 2021).

---

References:

• “Optimization of the Model Predictive Control Meta-Parameters Through Reinforcement Learning” (Bøhn et al., 2021)