---
title: Value Diffusion World Models (Valdi)
url: https://www.emergentmind.com/topics/value-diffusion-world-models-valdi
type: topic
---

# Value Diffusion World Models (Valdi)

Searching arXiv for the named Valdi-related papers and closely related diffusion world model work to ground the article.
Tool call: arxiv_search{"query":"\"Value Diffusion World Models\" OR \"Advantage-Guided Diffusion for Model-Based Reinforcement Learning\" OR \"Value function estimation using conditional diffusion models for control\"","max_results":10,"sort_by":"submittedDate"}
Value Diffusion World Models, often abbreviated as “Valdi,” designate a class of diffusion-based world-model formulations in which generative modeling of future states or trajectory segments is coupled to value estimation, policy evaluation, or planning. The name is not limited to a single algorithmic instantiation. In the 2023 Diffused Value Function (DVF) work, value estimation is recast as a conditional diffusion-based world model over future states [2306.07290]. In the 2026 technical report on Advantage-Guided Diffusion for Model-Based Reinforcement Learning, “Valdi” refers to advantage-guided steering of a diffusion world model during reverse denoising [2604.09035]. In the 2026 paper "Valdi: Value Diffusion World Models," the term denotes a latent diffusion dynamics model trained end-to-end for Model Predictive Control (MPC), with reward and value heads used in planning [2607.00917]. Across these usages, the common theme is that diffusion models are used not only for generative dynamics modeling, but also as substrates for value-aware control.

## 1. Terminological scope and research lineage

The term covers at least three closely related but technically distinct formulations.

| Formulation | Core mechanism | Primary setting |
|---|---|---|
| DVF | Conditional diffusion model for \( \rho_\theta(s_{t+\Delta t}\mid s_t,\phi(\pi),\Delta t) \) | Value estimation from state sequences |
| AGD-MBRL / Valdi | Reverse-diffusion guidance using \(A_\pi(s,a)=Q_\pi(s,a)-V_\pi(s)\) | Model-based RL with synthetic rollouts |
| Valdi | Latent diffusion dynamics with learned reward and value for MPC | Online planning in CarRacing |

DVF is explicitly described as a *conditional diffusion–based world model* that directly generates future states, and the accompanying description states that, in the language of Value Diffusion World Models, DVF is precisely a *Value Diffusion World Model* [2306.07290]. This usage foregrounds value estimation through occupancy-measure modeling rather than rollout autoregression or temporal-difference backups.

The AGD-MBRL technical report uses “Valdi” as a name for a framework that steers reverse-diffusion denoising with the agent’s estimated advantage function. Here the defining innovation is not merely diffusion-based trajectory generation, but guidance of that generation by \(A_\pi\) so that sampling concentrates on trajectories expected to yield higher long-term return beyond the generated window [2604.09035].

The 2026 paper titled "Valdi: Value Diffusion World Models" shifts the emphasis again. It treats Valdi as a latent world model for MPC, combining end-to-end online training with a latent diffusion dynamics model and using a single diffusion step at both training and inference to satisfy low-latency planning constraints [2607.00917]. A plausible implication is that “Valdi” now functions both as a descriptive category and as the title of a specific latent-planning architecture.

## 2. DVF: value estimation as conditional diffusion over future states

DVF begins from the observation that the \(Q\)-function can be written via the discounted occupancy measure:
\[
Q^\pi(s_t,a_t)=r(s_t,a_t)+\gamma\cdot \mathbb{E}_{s'\sim \rho^\pi(\cdot\mid s_t,a_t)}[V^\pi(s')].
\]
It then learns a conditional generative model
\[
\rho_\theta(s_{t+\Delta t}\mid s_t,\phi(\pi),\Delta t)
\]
by treating \(s_{t+\Delta t}\) as the data to be generated from noise, conditioned on \((s_t,\Delta t,\phi(\pi))\) [2306.07290]. In this formulation, the diffusion model is the world model: it represents future-state occupancy directly rather than generating one step at a time.

The forward process is a standard Gaussian diffusion with \(T\) timesteps. Writing \(x_0\equiv s_{t+\Delta t}\), the noising chain is
\[
q(x_{t_d}\mid x_{t_d-1})=\mathcal{N}(\sqrt{1-\beta_{t_d}}\cdot x_{t_d-1},\beta_{t_d}\cdot I),
\]
with \(x_T\approx \mathcal{N}(0,I)\) after \(T\) steps. The learned reverse chain is
\[
p_\theta(x_{t_d-1}\mid x_{t_d},s_t,\Delta t,\phi(\pi))
=
\mathcal{N}\!\bigl(\mu_\theta(x_{t_d},t_d;s_t,\Delta t,\phi(\pi)),\Sigma_\theta(x_{t_d},t_d)\bigr).
\]
The description specifies that \(\mu_\theta\) is implemented by a Perceiver I/O network that ingests the noisy future state, the conditioning tuple, and the diffusion time index \(t_d\), while \(\Sigma_\theta\) is either fixed or learned.

Under the common DDPM reparameterization,
\[
\bar\alpha_{t_d}=\prod_{i=1}^{t_d}(1-\beta_i),
\]
and training samples
\[
x_{t_d}=\sqrt{\bar\alpha_{t_d}}\cdot x_0+\sqrt{1-\bar\alpha_{t_d}}\cdot \epsilon,\qquad \epsilon\sim \mathcal{N}(0,I).
\]
The diffusion loss is the standard DDPM noise-prediction objective, the reward loss is a regression objective
\[
\ell_{\text{reward}}(\psi)=\mathbb{E}_{(s,a,r)\sim D_{\text{labeled}}}\|r_\psi(s,a)-r\|_2^2,
\]
and the policy loss is a soft actor–critic projection
\[
\ell_{\text{policy}}(\phi)=\mathbb{E}_{s\sim D}\!\left[
KL\!\bigl(\pi_\phi(\cdot\mid s)\,\|\,\mathrm{Softmax}_a\,\hat Q(s,a)\bigr)
\right].
\]

Policy conditioning \(\phi(\pi)\) can be either a scalar index, such as a training-step identifier, or a sequential embedding of rollout states, with actions included if available. At inference, \(V^\pi(s_t)\) is estimated by drawing \(n\) samples from the conditional diffusion model, evaluating a trained reward predictor \(r_\psi(s_{t+\Delta t,i},\pi(s_{t+\Delta t,i}))\), and averaging them with a discount-dependent prefactor. \(Q(s_t,a_t)\) is then obtained by a one-step backup,
\[
\hat Q(s_t,a_t)=r_\psi(s_t,a_t)+\gamma\cdot \hat V(s_{t+1}).
\]
Because \(\nabla_a Q=\nabla_a r\) in this one-step backup, the diffusion model need not be differentiated through for policy improvement. This separation of dynamics, reward regression, and policy projection is one of the defining structural features of DVF.

## 3. Advantage-guided Valdi: steering diffusion with \(A_\pi\)

In AGD-MBRL, Valdi is formulated on an MDP
\[
M=\bigl(S,A,P,r,\gamma,\rho\bigr),
\]
with a diffusion world model \(p_\theta(\tau)\) that learns to sample trajectory segments
\[
\tau=(s_1,a_1,\dots,s_H,a_H)
\]
under the current policy \(\pi_\phi\). The extension is to steer reverse-diffusion denoising steps using the estimated advantage
\[
A_\pi(s,a)=Q_\pi(s,a)-V_\pi(s),
\]
so that synthetic trajectories are biased toward high-advantage state–action pairs [2604.09035].

The guided reverse step is written as
\[
p_\theta(\tau_{i-1}\mid \tau_i,y)
=
\mathcal{N}\bigl(\mu_\theta(\tau_i,i)+\alpha\,\Sigma_i\,g_i,\Sigma_i\bigr),
\]
where
\[
g_i=\nabla_\tau \log p(y\mid\tau)\big|_{\tau=\mu_\theta(\tau_i,i)},
\]
and \(\alpha>0\) is a guide scale. Two guidance choices are defined.

For **Sigmoid Advantage Guidance (SAG)**, a binary optimality variable \(O_t\) is introduced with
\[
p(O_t=1\mid s_t,a_t)=\sigma(A_\pi(s_t,a_t)).
\]
Assuming independence over \(t\),
\[
p(O_{1:H}=1\mid\tau)=\prod_{t=1}^H \sigma(A_\pi(s_t,a_t)),
\]
so the guided model becomes
\[
p_\theta(\tau\mid O_{1:H}=1)\propto p_\theta(\tau)\prod_{t=1}^H \sigma(A_\pi(s_t,a_t)).
\]
The corresponding guide gradient is
\[
g_i=\sum_{t=1}^H \frac{1}{1+\exp(A_\pi(s_t,a_t))}\,\nabla_{(s_t,a_t)}A_\pi(s_t,a_t).
\]

For **Exponential Advantage Guidance (EAG)**, the cumulative-advantage energy is
\[
E(\tau)=\sum_{t=1}^H A_\pi(s_t,a_t),
\]
and the reweighted model is
\[
p_\theta(\tau\mid E(\tau))\propto p_\theta(\tau)\exp(E(\tau)).
\]
The guide gradient simplifies to
\[
g_i=\nabla_\tau E(\tau)=\sum_{t=1}^H \nabla_{(s_t,a_t)}A_\pi(s_t,a_t).
\]

The report characterizes SAG as more conservative because the sigmoid saturates at \(\{0,1\}\), whereas EAG aggressively reweights by \(\exp(A)\). It further sketches a policy-improvement argument under the assumption \(p_\theta(\tau)\approx p^{\pi_\phi}(\tau)\). For SAG, the reweighted density induces the policy
\[
\pi^\sigma(a\mid s)
=
\frac{\pi_\phi(a\mid s)\sigma(A_\pi(s,a))}
{\int \pi_\phi(a'\mid s)\sigma(A_\pi(s,a'))\,da'},
\]
and the standard Policy-Improvement Theorem is then invoked to conclude \(J(\pi^\sigma)\ge J(\pi_\phi)\). In the report’s phrasing, guided sampling is sampling under an improved policy up to reweighting, guaranteeing policy improvement in expectation.

Operationally, Valdi interleaves real rollouts under \(\pi_\phi\), score-matching updates to \(p_\theta\), synthetic trajectory sampling with SAG or EAG, and policy plus advantage-estimator updates, for example via Advantage Actor-Critic. The result is a diffusion world model whose imaginary data are explicitly value-biased rather than merely policy-conditioned.

## 4. Latent Valdi for end-to-end MPC

The 2026 paper "Valdi: Value Diffusion World Models" defines a latent world model for a POMDP or MDP in which observations \(s_t\in\mathcal S\) are encoded as latent states
\[
z_t=E_\theta(s_t)\in \mathbb{R}^d,
\]
with \(d=64\). The objective is to predict future latent sequences \(z_{t+1:t+H}\) from \((z_t,a_{t:t+H-1})\) in order to evaluate candidate action sequences by
\[
a^*_{0:H-1}
=
\arg\max_{a_{0:H-1}}
\left[
\sum_{h=0}^{H-1}\gamma^h R_\theta(\hat z_h,a_h)+\gamma^H V_\theta(\hat z_H)
\right],
\]
subject to learned latent dynamics [2607.00917].

Instead of a deterministic MLP transition, Valdi models
\[
p(z_{t+1:t+H}\mid z_t,a_{t:t+H-1})
\]
with a latent-space diffusion model using the velocity-parameterization of Salimans et al. (2022). A target encoder \(E_{\bar\theta}\), maintained as an EMA of \(\theta\), produces clean latent targets \(\bar z_{t+1:t+H}\). Given a diffusion index \(\tau\in\{1,\dots,T\}\), these are noised by
\[
z^\tau_{t+1:t+H}
=
\sqrt{\alpha^\tau}\cdot \bar z_{t+1:t+H}
+
\sqrt{1-\alpha^\tau}\cdot \epsilon,
\qquad
\epsilon\sim\mathcal N(0,I),
\]
with \(\alpha^\tau=\prod_{i=1}^\tau(1-\beta_i)\) and schedule \(\beta_1=1e{-}4\dots \beta_T=2e{-}2\).

The diffusion network jointly predicts the velocity
\[
\hat u^\tau_{t+1:t+H}
=
D_\theta(z_t,z^\tau_{t+1:t+H},a_{t:t+H-1},\tau),
\]
and reconstructs denoised latents
\[
\hat z_{t+1:t+H}
=
\sqrt{\alpha^\tau}\cdot z^\tau_{t+1:t+H}
-
\sqrt{1-\alpha^\tau}\cdot \hat u^\tau_{t+1:t+H}.
\]
Training combines four losses:
\[
L(\theta)
=
(1-\lambda_{\mathrm{reg}})
\bigl[
\lambda_{\mathrm{diff}}L_{\mathrm{diff}}
+
\lambda_{\mathrm{rew}}L_{\mathrm{rew}}
+
\lambda_{\mathrm{val}}L_{\mathrm{val}}
\bigr]
+
\lambda_{\mathrm{reg}}L_{\mathrm{reg}}.
\]
Here \(L_{\mathrm{diff}}\) is the velocity-form denoising loss, \(L_{\mathrm{rew}}\) reconstructs \(r_t\) from \(\hat z_{t+1}\) and \(a_t\), \(L_{\mathrm{val}}\) applies a temporal-difference loss on denoised latents, and the SIGReg regularizer \(L_{\mathrm{reg}}\) encourages isotropic latent covariance.

A crucial clarification is given explicitly: the diffusion network \(D_\theta\) is **not directly weighted by value**. Instead, value enters through the TD loss on denoised latents during training and through action-sequence scoring at inference. This sharply distinguishes latent Valdi from advantage-guided denoising, even though both are “value diffusion” formulations.

To meet MPC’s latency requirements, the method uses exactly one denoising step at both training and inference. Inference treats the model as a deterministic 1-step DDIM sampler with \(K=1\). The online loop rolls out episodes using MPC with CEM, stores transitions in a replay buffer, and performs repeated gradient updates. The key hyperparameters reported are \(\gamma=0.99\), horizon \(H=5\) world steps, action-chunking of \(3\) for \(15\) environment steps, CEM population \(=512\), elites \(=64\), iterations \(=10\), loss weights \(\lambda_{\mathrm{diff}}=1\), \(\lambda_{\mathrm{rew}}=0.01\), \(\lambda_{\mathrm{val}}=0.01\), \(\lambda_{\mathrm{reg}}=0.05\), replay size \(=1000\) trajectories, batch \(=256\), \(U=60\) updates per episode, and EMA \(\beta=0.005\).

## 5. Experimental evidence across control settings

The empirical record attached to Value Diffusion World Models spans value estimation, offline RL, online MBRL, and MPC.

In **Mountain Car**, DVF was trained for \(500\) steps and evaluated by Pearson correlation between ground-truth returns, \(\hat Q\), and single-step reward predictions. All pairwise correlations were reported as \(>0.9\), supporting the claim that the diffusion-based estimate closely matches empirical returns without temporal-difference learning [2306.07290].

In **Maze2D (D4RL offline)**, DVF was tested on two mazes, U-maze and Large maze, using three waypoint planners that induced three data-collection policies. With scalar policy conditioning \(\phi(\pi)=\) policy index \((1,2,3)\), DVF correctly disentangled three distinct conditional future-state distributions. The report also states that varying \(\gamma\) naturally samples further along the trajectory, without compounding errors.

In **PyBullet Offline RL**, DVF was evaluated on four continuous-control tasks with offline data from random, medium-quality, mixed, and SAC-collected datasets. The baselines were Behavior Cloning (BC) and Conservative Q-Learning (CQL), and the metric was normalized return. The reported result is that DVF matches or outperforms BC and CQL, especially on low-quality data labeled “random,” while an ablation found the sequential policy embedding more robust than the scalar embedding when the policy index is unknown or unbounded.

For **MuJoCo** in AGD-MBRL, the benchmark tasks were Hopper, HalfCheetah, Walker2D, and Reacher, with \(1.5\) million environment steps and diffusion horizon \(H=32\). The architecture used a 4-layer MLP with \(256\) units for diffusion, \(1{,}000\) reverse steps, and an actor–critic MLP of \(2\times 256\) for policy and value. Baselines were PolyGRAD, Online Diffuser, PPO, and TRPO [2604.09035].

| Task | AGD-EAG | AGD-SAG |
|---|---:|---:|
| Hopper | \(3328 \pm 39\) | \(3268 \pm 25\) |
| HalfCheetah | **\(4865 \pm 157\)** | \(4642 \pm 163\) |
| Walker2D | \(3794 \pm 165\) | **\(3845 \pm 362\)** |
| Reacher | **\(-3.90 \pm 0.13\)** | \(-3.87 \pm 0.37\) |

The same report states that training curves show faster convergence and fewer regressions with AGD than with unguided or reward-guided diffusion.

In **CarRacing**, the latent Valdi paper compares a one-step diffusion dynamics model against a deterministic MLP baseline identical in all respects except the dynamics map \(z_{t+1}=\mathrm{MLP}(z_t,a_t)\). The reported outcome is that both match within run-to-run variance in control performance [2607.00917]. However, increasing the number of inference diffusion steps beyond one makes rollouts visually diverse, with increased LPIPS, while control degrades slightly. The paper also reports value-function diagnostics: at short rollout depths the MLP is slightly more accurate, but near \(h\approx H\) Valdi has smaller \(|\delta|\), which the paper states suggests better long-horizon value consistency.

## 6. Strengths, limitations, and recurrent points of clarification

Several strengths recur across the literature. DVF emphasizes **\(O(1)\) long-horizon sampling**, avoidance of autoregressive compounding, pre-training on **state-only** data with no actions or rewards, zero-shot policy evaluation by policy conditioning \(\phi(\pi)\), and efficient policy gradients through a one-step \(Q\)-backup [2306.07290]. AGD-MBRL emphasizes that advantage guidance remedies **short-horizon myopia** by injecting downstream value information into guided sampling, while requiring no change to the diffusion training objective [2604.09035]. The latent Valdi formulation emphasizes that a **one-step latent diffusion model** can match a deterministic MLP in control performance while retaining the ability to generate multimodal futures [2607.00917].

The limitations are equally explicit. DVF works in **observation space**, which requires careful noise-schedule tuning per domain; it also requires explicit \(\phi(\pi)\) conditioning, and in online settings non-stationarity requires re-training or continual adaptation of \(\rho_\theta\). AGD-MBRL identifies sensitivity to the guidance scale \(\alpha\), critic overestimation, and the trade-off between conservative SAG and more aggressive but critic-sensitive EAG. The latent Valdi paper identifies the possibility that the one-step diffusion approximation may not suffice for more complex dynamics, together with a training–inference schedule mismatch when varying the number of diffusion steps \(K\).

A common misconception is that “value diffusion” always means directly weighting the diffusion model by value. The latent Valdi paper explicitly rejects that interpretation: the diffusion network is not directly weighted by value, and value enters through a TD loss on denoised latents plus planner scoring. Another common misconception is that increased multimodality necessarily improves control. The CarRacing experiments instead expose a trade-off: richer multimodality can coexist with worse MPC return.

The proposed extensions also indicate the field’s open directions. DVF suggests latent diffusion, action-conditioned diffusion \( \rho(s_{t+\Delta t}\mid s_t,a_t,\phi(\pi)) \), contrastive or score-matching objectives to improve sample efficiency, and hierarchical policy embeddings for transfer across controllers. AGD-MBRL suggests practical stabilizers such as annealing \(\alpha\) or clipping \(A_\omega\). The latent Valdi paper suggests multi-step diffusion with distillation to a single-step student and training procedures that permit flexible inference-time diffusion schedules without degrading planning. Taken together, these proposals suggest an emerging research program in which diffusion world models are not merely generative simulators, but value-structured models of controllable uncertainty.

Source: https://www.emergentmind.com/topics/value-diffusion-world-models-valdi