Dream-MPC: Latent Gradient Planning

• Dream-MPC is a reinforcement learning framework that uses a learned latent space and gradient ascent to plan efficiently for complex, high-dimensional continuous control.
• It integrates world models, policy priors, and uncertainty regularization to ensure robust performance across both visual and proprioceptive tasks.
• Action reuse amortization and gradient optimization reduce computational overhead, enabling scalable and real-time inference in diverse control settings.

Dream-MPC is a model-based reinforcement learning (RL) framework that enables scalable, gradient-based model predictive control (MPC) in complex, high-dimensional continuous control settings by combining latent-space imagination, a learned policy prior, uncertainty regularization, and action-reuse amortization. Distinct from prior MPC approaches that rely on computationally intensive population-based (gradient-free) planners, Dream-MPC exploits efficient gradient ascent in the latent space of a learned differentiable world model, achieving competitive or superior performance with substantially reduced model calls and real-time inference overhead. The framework generalizes to visual and proprioceptive tasks, and extends to long-horizon planning via integration with recurrent world models and ensemble-based uncertainty calibration (Spieler et al., 6 May 2026, Du et al., 6 May 2026).

1. Latent World Modeling and Policy Priors

Dream-MPC relies on a learned, differentiable world model that compactly encodes environment states into a latent space, supports forward dynamics, and facilitates planning over imagined rollouts. The standard blueprint incorporates the following components:

• Encoder $h$: Processes raw states $s_t$ into latent representations $z_t$ via a multi-layer perceptron (MLP), typically with 2–5 layers and 512-dimensional output.
• Latent Dynamics $d$: Models forward progression in latent space as $z_{t+1} = d(z_t, a_t)$, commonly realized by a 3-layer MLP with 512 hidden units.
• Reward and Q-Value Predictors: MLPs for both reward prediction $\hat{r} = R(z_t, a_t)$ and value estimation $\hat{q} = Q(z_t, a_t)$, with $Q$ implemented as an ensemble of $M=5$ heads—providing uncertainty estimates.
• Policy Prior $\pi_\theta$: An MLP outputting a diagonal Gaussian over actions given latent input $s_t$0, serving two purposes: (i) warm-starting candidate trajectories for planning, and (ii) generating fresh action proposals.

Latent imagination is realized by rolling out candidate action sequences under the world model in latent space, predicting future latents, rewards, and Q-values: $s_t$1 (Spieler et al., 6 May 2026).

When extended to visual domains and long-horizon planning, Dream-MPC uses a Dreamer-style Recurrent State-Space Model (RSSM) that augments deterministic hidden states $s_t$2 with stochastic latents $s_t$3 to model aleatoric uncertainty, learned via variational objectives (Du et al., 6 May 2026).

2. Gradient-Based Planning and Amortization

Unlike traditional population-based planners (e.g., MPPI), Dream-MPC generates a small set of candidate action sequences via policy rollout, then optimizes them in the latent space using a few steps of gradient ascent with respect to a regularized trajectory return objective. For candidate $s_t$4, the planning objective is: $s_t$5 where $s_t$6 denotes an epistemic-uncertainty term (computed as the standard deviation across Q-value ensemble heads, modulated by their mean), and $s_t$7 (typically $s_t$8) trades off return and uncertainty (Spieler et al., 6 May 2026).

Each action sequence is updated for $s_t$9 gradient ascent steps with stepsize $z_t$0 (commonly $z_t$1): $z_t$2

To optimize further, Dream-MPC amortizes planning by reusing previously optimized action tails as initialization for the next MPC step: $z_t$3 This “action reuse” mechanism distributes computation across time and enables single-step (per-control step) optimization, retaining high sample efficiency (Spieler et al., 6 May 2026).

3. Integration with Uncertainty Regularization

Compounding model errors and epistemic uncertainty present a major challenge in latent-based planning. Dream-MPC incorporates epistemic-uncertainty regularization directly into the planning objective ($z_t$4), using the ensemble variance of Q-values as a proxy. This regularization penalizes trajectories that pass through high-uncertainty states, effectively mitigating over-optimism during open-loop rollouts and improving reliability, particularly in high-dimensional and visual tasks (Spieler et al., 6 May 2026).

Ablation studies confirm that policy warm-start, uncertainty regularization, and action-reuse all contribute significantly to Dream-MPC’s empirical performance; omission of any component leads to degradation in task return and stability (Spieler et al., 6 May 2026).

4. Algorithmic Structure and Implementation

Dream-MPC’s online planning algorithm proceeds as follows for each time step:

1. Initialization: Encode current state $z_t$5. Roll out $z_t$6 action sequences of length $z_t$7 from the policy prior $z_t$8.
2. Amortization: Blend in previous optimized actions with policy proposals via action reuse.
3. Optimization: For each candidate, perform one (or a few) gradient ascent steps to maximize $z_t$9.
4. Execution: Deploy the first action of the candidate with maximal $d$0.

Typical hyperparameters are $d$1, $d$2, $d$3, $d$4, $d$5. The underlying world model is based on TD-MPC2, with $d$65M parameters, replay buffer of $d$7 samples, Q-ensemble of $d$8, and separate learning rates for model/encoder (Adam optimizer) (Spieler et al., 6 May 2026).

When integrated with visual models and long-horizon planning (e.g., in ELVIS), the approach leverages a Dreamer-style RSSM with a Gaussian-mixture MPPI planner and an ensemble-calibrated, time-varying $d$9-return, further enhancing sample efficiency and robustness (Du et al., 6 May 2026).

5. Empirical Evaluation

Dream-MPC was benchmarked on 24 continuous control environments, including:

• DeepMind Control Suite (with both proprioceptive and image-based tasks)
• Meta-World (manipulation)
• HumanoidBench (high-dimensional locomotion)

Baselines include state-of-the-art gradient-free MPC (TD-MPC2 with policy-guided MPPI), pure policy-only methods, Dreamer-v3, Soft Actor-Critic (SAC), and BMPC (policy imitation of MPPI).

Key empirical findings:

• Dream-MPC combined with BMPC achieved the best overall interquartile mean (IQM, ≈+26.7%) and mean return (≈+20.5%) over BMPC.
• Dream-MPC with TD-MPC2 improved TD-MPC2 policy performance by +144.7% (IQM) and +43.4% (mean), although it lagged MPPI in the highest-dimensional tasks.
• On visual DMControl environments, Dream-MPC+BMPC matched or exceeded MPPI performance on 5/6 benchmarks (Spieler et al., 6 May 2026).

Representative normalized returns (DMControl, state-only) are summarized below:

Algorithm	Mean ± SD
TD-MPC2 (MPPI)	657 ± 225
TD-MPC2 (policy only)	367 ± 247
Dream-MPC (TD-MPC2)	433 ± 259
BMPC (MPPI)	711 ± 181
Dream-MPC (BMPC)	725 ± 181

In long-horizon visual domains, Dream-MPC (as part of ELVIS) demonstrated improved sample efficiency and robustness, attaining expert-level performance with fewer environment frames, and successful zero-shot sim-to-real transfer on industrial visual manipulation tasks (Du et al., 6 May 2026).

6. Computational Efficiency and Scalability

Dream-MPC exhibits a dramatic reduction in model calls compared to gradient-free planners. For a typical configuration,

• MPPI model calls/step: $z_{t+1} = d(z_t, a_t)$0
• Dream-MPC model calls/step: $z_{t+1} = d(z_t, a_t)$1

Inference timings (Acrobot Swingup, RTX4090):

• TD-MPC2 (MPPI): 20.8 ms
• Dream-MPC (BMPC): 18.2 ms
• Grad-MPC (alternate implementation): $z_{t+1} = d(z_t, a_t)$2195 ms

On high-dimensional tasks (e.g., HumanoidBench, state dim 151, action dim 61), Dream-MPC achieves real-time execution ($z_{t+1} = d(z_t, a_t)$350 ms/step), scaling gracefully with problem dimensionality (Spieler et al., 6 May 2026).

7. Limitations and Future Directions

Dream-MPC's efficacy depends on several factors:

• Hyperparameter Selection: Fixed planning horizon and optimization hyperparameters may not be optimal for all tasks; dynamic or adaptive settings are a suggested avenue.
• Policy Prior Quality: A strong policy prior is essential; poor priors can cause suboptimal performance or early convergence.
• Single-task Focus: Multi-task and lifelong learning extensions remain unexplored.
• Uncertainty Modeling: Further improvements could be made by adopting more sophisticated forms of uncertainty quantification, e.g., Bayesian ensembles.

Future research directions include adaptive planning and optimization parameters, integrated end-to-end training of world models and planners, extending to multi-task scenarios, and enhanced uncertainty calibration (Spieler et al., 6 May 2026).

Dream-MPC demonstrates that gradient-based planning in latent space, synergized with a robust policy prior, explicit uncertainty regularization, and action reuse, can match or exceed traditional sampling-based planners in sample and computational efficiency, with broad applicability to high-dimensional continuous control and visual RL settings (Spieler et al., 6 May 2026, Du et al., 6 May 2026).