Soft MPCritic: RL-MPC Integration

• Soft MPCritic is a reinforcement-learning–model-predictive-control framework that leverages MPPI for online control and learns a terminal soft Q-function via fitted value iteration.
• It integrates short-horizon planning with long-horizon value estimates through warm-started MPPI and ensemble-based dynamics models to reduce computational cost and mitigate model bias.
• Empirical studies on benchmarks like Hopper-v5 and double inverted pendulum show that Soft MPCritic achieves faster learning and higher cumulative rewards compared to standard deep-RL methods.

Soft MPCritic is a reinforcement-learning–model-predictive-control framework that uses model predictive path integral control (MPPI) for online receding-horizon control and for generating value-function targets, while learning a terminal soft Q-function via fitted value iteration. Its defining claim is that it “lives entirely in value space”: rather than extracting or training a parametric actor for deployment, it performs control with MPPI at runtime and trains $Q_\phi(s,a)$ to be consistent with the planner’s own soft value estimates, thereby implicitly extending the effective planning horizon of short-horizon planning (Banker et al., 1 Apr 2026).

1. Definition and the meaning of “soft”

In Soft MPCritic, “soft” refers to a stochastic-control and soft-value-iteration formulation in which value is expressed through free-energy-like quantities rather than a hard $\arg\min$ or $\arg\max$. The state-dependent soft value function is written as

$V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$

and the associated soft Q-function satisfies

$$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$

This formulation replaces hard optimality with a log-sum-exp or free-energy structure, so MPPI acts as a “soft optimizer” of the stage cost and terminal value (Banker et al., 1 Apr 2026).

A common misunderstanding is to equate the term “soft” here with Soft Actor-Critic in the narrow sense of an entropy-regularized actor network. Soft MPCritic explicitly does not learn a parametric policy for control. Control is always done by MPPI, and the learned object is a terminal Q-function that is consumed by the planner. The softness therefore resides in the value representation and sampling-based optimization, not in a separate deployed actor (Banker et al., 1 Apr 2026).

This also distinguishes Soft MPCritic from methods that use “soft” to describe training-time relaxations of MPC optimization. A plausible implication is that the name marks a shift from softening the optimization problem itself toward softening the value operator with which planning and learning are coupled.

2. Genealogy within RL–MPC research

Soft MPCritic emerged after a closely related architecture, MPCritic, which defined a critic directly from an MPC formulation and introduced a fictitious controller $\mu$ so that the critic could be evaluated by deterministic forward rollout during reinforcement-learning updates, without solving an MPC optimization problem at training time (Lawrence et al., 1 Apr 2025). In that earlier design, the MPC objective was treated as a differentiable loss landscape, the exact MPC structure was preserved, and deployment could discard $\mu$ and revert to a standard online MPC solve (Lawrence et al., 1 Apr 2025).

Soft MPCritic retains the general objective of synthesizing RL and MPC, but its mechanism is different. Instead of replacing the optimization with a fictitious closed-loop rollout, it uses MPPI itself for both online control and value-target generation. Instead of a plug-and-play critic architecture built out of $f,\ell,V,h,g,\mu$, it learns a terminal soft Q-function and aligns that function with the MPPI planner through fitted value iteration (Banker et al., 1 Apr 2026). This makes the planner and the critic mutually dependent: MPPI uses $Q_\phi$ at the horizon endpoint, and $Q_\phi$ is trained from MPPI-derived targets.

The contrast can be summarized as follows.

Aspect	MPCritic	Soft MPCritic
Training-time mechanism	No MPC optimization problem is solved; critic is a rollout of $\arg\min$0 and $\arg\min$1	MPPI is used for value target generation
Runtime control	Standard online MPC after discarding $\arg\min$2	MPPI receding-horizon control with terminal $\arg\min$3
Learned object	MPC parameters and fictitious controller	Terminal soft Q-function aligned with MPPI

This lineage matters because it clarifies that Soft MPCritic is not merely MPCritic with entropy regularization added. It is a different RL–MPC synthesis strategy centered on amortized model predictive value iteration.

3. Algorithmic architecture

The architecture has three coupled components: an MPPI planner, a terminal Q-network $\arg\min$4, and an ensemble of learned dynamics models $\arg\min$5. Online interaction proceeds under receding-horizon control. At state $\arg\min$6, MPPI is initialized with an open-loop control sequence $\arg\min$7, samples noise sequences, propagates trajectories under sampled dynamics models, evaluates a trajectory cost functional with terminal $\arg\min$8, updates the control sequence through the path-integral rule, and applies the first action $\arg\min$9 to the environment (Banker et al., 1 Apr 2026).

The trajectory cost used by the planner is

$\arg\max$0

The MPPI value estimate at state $\arg\max$1 is

$\arg\max$2

This value estimate is not an auxiliary diagnostic. It is the core training target for the Q-function and the soft analogue of a Bellman backup (Banker et al., 1 Apr 2026).

The Q-network is trained by fitted value iteration. After executing an action and observing $\arg\max$3, the algorithm stores the transition together with the shifted open-loop control sequence in a replay buffer. During updates, MPPI is rerun from $\arg\max$4, the resulting soft value estimate $\arg\max$5 is computed, and the Q-function is trained toward

$\arg\max$6

The training objective is presented as

$\arg\max$7

The architectural consequence is central: short-horizon planning and long-horizon learning are fused through the terminal cost. If MPPI used only stage costs over a short horizon $\arg\max$8, it would remain myopic. With $\arg\max$9 at the horizon endpoint, it sees an approximation to the infinite-horizon soft return. This is the sense in which Soft MPCritic “implicitly extends the effective planning horizon” while keeping the planner itself short-horizon (Banker et al., 1 Apr 2026).

4. MPPI, amortized warm starts, and computational practicality

The main computational innovation is the amortized warm-start strategy. When online MPPI optimizes an open-loop action sequence, that sequence is not discarded after the first control is executed. The shifted sequence $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$0 is stored with the transition. Later, when the same transition is sampled for Q-learning, MPPI is warm-started from that stored sequence rather than initialized from scratch (Banker et al., 1 Apr 2026).

Algorithmically, the replay item is removed, MPPI is rerun from the next state $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$1 using the stored $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$2, a refined sequence $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$3 and an MPPI value estimate are obtained, $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$4 is updated with the resulting target, the model ensemble is updated with next-step prediction loss, and the tuple is reinserted with $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$5. This means that target-generation trajectories are gradually refined as $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$6 and $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$7 evolve, rather than repeatedly replanned from a cold start (Banker et al., 1 Apr 2026).

The computational effect is explicit in the reported double inverted pendulum study. Warm-starting MPPI for targets with 1 iteration and $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$8 rollouts yields environment steps per second similar to no target refinement, whereas cold-starting with multiple iterations is significantly slower. With $V(s) = -\lambda \log \left( \mathbb{E}_{\tau \sim p^{\tilde{\beta}} \left[\exp\left( - \frac{1}{\lambda} \sum_{t=0}^\infty \gamma^t \ell(s_t, a_t) \right) \mid s_0 = s \right] \right),$9, warm start with 1 iteration gives approximately $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$0 SPS, while cold start with 5 iterations gives approximately $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$1 SPS and cold start with 10 iterations gives approximately $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$2 SPS. Cold starting needs many iterations to match performance, causing at least a $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$3 slowdown relative to warm starting (Banker et al., 1 Apr 2026).

The paper explains why small refinements can suffice by the exponential weighting structure of MPPI. The ratio of two sample weights is approximated as

$$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$4

This suggests that even modest cost differences can produce large changes in the effective contribution of sampled trajectories, so a good warm start can be improved with comparatively few additional samples.

5. Model ensembles, scenario-based planning, and empirical behavior

Soft MPCritic is explicitly model-based. Dynamics are learned online with a neural network $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$5 trained by next-step mean-squared error,

$$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$6

To mitigate model bias, the method uses an ensemble of dynamics models. MPPI samples $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$7 during rollouts, so each rollout is a scenario under a sampled dynamics hypothesis. Planning is therefore scenario-based: the control sequence is selected to perform well across the ensemble rather than a single nominal model (Banker et al., 1 Apr 2026).

This ensemble-based construction is not peripheral. In Hopper-v5, the ablation without terminal $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$8 collapses because a short horizon $$Q(s,a) = \ell(s,a) + \gamma \mathbb{E}_{s' \sim p(\cdot \mid s,a)} \left[ V(s') \right].$$9 is insufficient. Adding terminal Q but using only a single model degrades performance because long-horizon value estimates derived from an inaccurate model amplify model bias. The combination of terminal Q and ensemble dynamics gives strong and robust performance (Banker et al., 1 Apr 2026).

A second Hopper-v5 ablation examines where MPPI is used. Using MPPI only for control produces similar early learning, but long-run performance degrades because of misalignment between a parametric policy and the MPPI-induced optimal policy. Using MPPI only for targets limits performance because control no longer benefits from replanning. Using MPPI for both control and targets gives the best performance, because the controller and critic remain aligned and online replanning avoids dependence on a learned actor (Banker et al., 1 Apr 2026).

Against standard deep-RL baselines, the reported comparison is direct. With a uniform prior, Soft MPCritic achieves higher cumulative reward than SAC and DDPG on Hopper-v5, and reaches the asymptotic performance of those baselines, measured at $\mu$0 steps, within the first $\mu$1 steps (Banker et al., 1 Apr 2026). On double inverted pendulum, warm-started MPPI target computation improves learning speed and stability relative to cold starts, and an ensemble improves robustness and tightens the reward distribution across seeds (Banker et al., 1 Apr 2026).

A plausible implication is that Soft MPCritic is especially suited to regimes in which direct long-horizon planning is unreliable because of model error, yet direct policy extraction is also undesirable. Its reported success depends on keeping planning short-horizon, learning in value space, and preserving alignment between the planner that acts and the value function that bootstraps.

6. Interpretation, limitations, and open directions

The conceptual position of Soft MPCritic is unusually specific. It is neither standard MPC with a long online horizon nor a conventional actor–critic method with a deployed policy network. It uses sample-based planning at runtime, but avoids relying on long-horizon planning by attaching a learned terminal Q-function. It uses bootstrapped value learning, but avoids policy extraction by allowing MPPI to remain the controller (Banker et al., 1 Apr 2026).

Several limitations are explicit. The method relies on dynamics models that are reasonably accurate at least over short horizons; severe model bias can degrade both MPPI control and Q-learning. MPPI’s sampling cost scales with action dimension and horizon, so very high-dimensional problems remain difficult. Performance depends on tuning $\mu$2, $\mu$3, the rollout count $\mu$4, and the covariance $\mu$5. MPPI is not the exact solution to a deterministic optimal-control problem, particularly under learned models and short horizons. The present formulation focuses on unconstrained control and does not address hard safety constraints of the kind traditionally associated with MPC (Banker et al., 1 Apr 2026).

These caveats are useful for clarifying what Soft MPCritic is not. It is not a generic replacement for constrained nonlinear MPC, and it does not inherit the direct constraint-satisfaction framing emphasized by the earlier MPCritic line (Lawrence et al., 1 Apr 2025). Its contribution is instead a practical blueprint for synthesizing MPC policies when policy extraction and direct, long-horizon planning may fail (Banker et al., 1 Apr 2026).

The forward-looking agenda is correspondingly precise. Proposed extensions include advanced model-learning objectives beyond next-step MSE, integration of control-theoretic tools such as IQCs and Lyapunov-based analyses for stability and performance guarantees, richer MPC-enabled RL designs with constraints and more complex cost structures, more sophisticated MPPI variants, and adaptation of the framework to traditional Bellman optimality equations rather than soft versions (Banker et al., 1 Apr 2026). This suggests that Soft MPCritic should be understood less as a finalized algorithmic endpoint than as a concrete design pattern: short-horizon stochastic planning, planner-aligned soft value iteration, and amortized target computation within a unified RL–MPC loop.