---
title: Action-Only Diffusion Policies
url: https://www.emergentmind.com/topics/action-only-diffusion-policies
type: topic
---

# Action-Only Diffusion Policies

Action-only diffusion policies are conditional generative policies that denoise actions, or short action chunks, directly in action space given the current observation or state, rather than diffusing over pixels or full state-action trajectories. In the visuomotor formulation introduced by Diffusion Policy, the policy models a future action sequence $A$ conditioned on observations $O$ and executes only the first part of that sequence in a receding-horizon, closed-loop manner [2303.04137]. In offline reinforcement learning, the same term also denotes a conditional diffusion model for a single action $a$ given a state $s$, with the terminal sample of the reverse chain serving as the action [2208.06193]. Across imitation learning, offline RL, online RL, constraint-guided control, and cross-embodiment transfer, the common premise is that a diffusion model can represent complex, multimodal action distributions while retaining stepwise conditioning on the current control context.

## 1. Formalization and scope

The standard action-only formulation uses a forward Gaussian noising process on actions and a reverse denoising model conditioned on the current observation or state. A representative DDPM-style forward kernel is
$$
q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),
$$
with closed-form corruption
$$
a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),
$$
and a conditional reverse model that predicts either $\epsilon_\theta$ or scores [2511.04671]. In Diffusion-QL, the policy is the marginal $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$ obtained by running the reverse chain from Gaussian noise to a single action conditioned on the current state, with the action returned at diffusion timestep $i=0$ [2208.06193].

Action-only does not imply a single scalar control output or a strictly one-step temporal horizon. In Diffusion Policy, the generative variable is a future action sequence $A$ of length $T_p$, while observations are conditioning inputs only; the model then executes the first $T_a$ actions and replans [2303.04137]. In X-Diffusion, the policy predicts an action sequence $A_t = a_{t:t+S}$ of length $S$ conditioned on current observation/state $s_t$, with $S=8$ and observation horizon $1$ [2511.04671]. A recurring distinction in the literature is therefore between action-only diffusion over current or short-horizon controls and trajectory diffusion over full future state-action trajectories.

The training objective is usually the conditional denoising loss. One common form is
$$
L_{\text{simple}} = \mathbb{E}_{k,A_t^0,\epsilon}\left[\lVert \epsilon - \epsilon_\theta(A_t^k,k,s_t)\rVert^2\right],
$$
which is the form used in X-Diffusion for action chunks [2511.04671]. Diffusion-QL uses the analogous action-level objective
$$
L_{\text{diffusion}}(\theta)=\mathbb{E}\left[\left\|\epsilon-\epsilon_\theta\!\left(\sqrt{\bar{\alpha}_i}a+\sqrt{1-\bar{\alpha}_i}\epsilon,s,i\right)\right\|^2\right],
$$
thereby treating denoising score matching as a direct conditional model of the behavior action distribution [2208.06193].

A separate branch formulates action-only diffusion through score-based SDEs or ODEs. Contractive Diffusion Policies describe the policy as a conditional action generator $\pi_\theta(a \mid s)$ whose reverse dynamics are parameterized by a score function $s_\theta(a,s,t)=\nabla_a \log p_t(a\mid s)$ and sampled by solving a reverse-time SDE or its probability flow ODE [2601.01003]. Q-score matching takes a continuous-time view in which the denoising vector field $\xi(s,a)$ defines the action flow, and training aligns that field with $\nabla_a Q(s,a)$ rather than with a behavior-cloning target [2312.11752]. This suggests that “action-only diffusion policy” is best understood as a family of conditional action generators sharing the same object of generation—actions rather than states or trajectories—even when their stochastic-process formalism differs.

## 2. Control semantics and architectural patterns

The original visuomotor recipe couples action diffusion with receding-horizon control. At each control step, Diffusion Policy encodes the recent observation context once, initializes the action sequence from noise, iteratively denoises it, returns $A_t^0$, executes only the first $T_a$ actions, and then replans with new observations [2303.04137]. This design combines sequence-level temporal consistency with closed-loop feedback. The paper reports that typical values were $T_o \approx 2$, $T_p \approx 16$, and $T_a \approx 6$–$8$, and that DDIM with about $10$–$16$ inference steps yielded about $0.1\,\text{s}$ latency on a single GPU [2303.04137].

Conditioning pipelines typically fuse visual and low-dimensional information. Diffusion Policy uses a ResNet-18 visual encoder trained end-to-end, spatial softmax pooling, GroupNorm, and either FiLM or cross-attention depending on whether the denoiser is a temporal CNN or a time-series diffusion transformer [2303.04137]. X-Diffusion adopts a visual encoder over masked RGB frames at $96\times96$ with end-effector or human-keypoint overlays, concatenates proprioceptive features $q_t=\{p_t,r_t,g_t\}$, and predicts actions in end-effector position, orientation, and gripper space using a Diffusion Policy UNet for actions [2511.04671]. In offline RL, much smaller state-conditioned MLP denoisers are common; Diffusion-QL uses a $3$-layer MLP with Mish activations and timestep embeddings, while Q-score matching uses a two-hidden-layer MLP taking $[s;a]$ as input [2208.06193].

The architectural spectrum is therefore broad, but the conditioning logic is stable: the current state or observation is encoded once, diffusion time is embedded, and only the action variable is iteratively updated. Temporal Logic Guidance for Action-Only Diffusion Policies with World Models makes this separation explicit by emphasizing that “states are not generated by the diffusion process itself—only action sequences are denoised,” while future states needed for constraint evaluation are supplied by an external world model [2606.22729]. A common misconception is that diffusion control necessarily implies joint action-state generation; action-only policies are specifically defined by not doing so.

## 3. Learning regimes: imitation, offline RL, on-policy RL, and MARL

The earliest large-scale successes of action-only diffusion were in imitation learning and offline RL. Diffusion Policy benchmarked action diffusion across $12$ tasks from $4$ robot manipulation benchmarks and reported an average improvement of $46.9\%$ over existing state-of-the-art methods, attributing much of that gain to multimodal action modeling, training stability, and receding-horizon control [2303.04137]. Diffusion-QL then established action-only diffusion as a strong policy class for offline RL by combining the diffusion cloning loss with a Q-guided term,
$$
L(\theta)=L_{\text{diffusion}}(\theta)-\alpha\,\mathbb{E}[Q_\phi(s,a^0)],
$$
and reported normalized D4RL averages of $88.0$ on Gym, $69.6$ on AntMaze, $65.1$ on Adroit, and $69.0$ on Kitchen [2208.06193].

Subsequent offline RL work diversified the policy-improvement mechanism. PAO-DP keeps an action-only conditional diffusion model for the behavior distribution, samples multiple in-distribution candidate actions from it, uses the critic to select preferred actions, and optimizes a surrogate diffusion policy with a Bradley–Terry-style preference loss plus an anti-noise mixture term [2405.18729]. Q-score matching instead regresses the denoising field directly toward $\alpha \nabla_a Q_\phi(s,a)$,
$$
\mathcal{L}_\pi(\theta)=\mathbb{E}\big[\|\xi_\theta(s,a)-\alpha\nabla_a Q_\phi(s,a)\|_2^2\big],
$$
so that actor improvement differentiates only through the denoiser rather than through the entire diffusion trajectory [2312.11752]. CausalGDP adds intervention-aware gradients from a learned causal dynamical model to the denoising process and reports, for example, Maze2D-large-v1 performance of $266.2\pm4.6$ versus $252.9$ for Diffusion-QL and an AntMaze average of $70.8$ versus $69.6$ [2602.09207].

On-policy learning required a different solution because PPO-style updates depend on tractable likelihood ratios. Diffusion Policy through Conditional Proximal Policy Optimization addresses this by aligning policy iteration with a conditional Gaussian residual kernel
$$
p_{\epsilon,\theta}(a\mid a_0,s)=\mathcal{N}(a; a_0+\mu_\theta(a_0,s), \Sigma_\theta(a_0,s)),
$$
and using PPO only on that conditional Gaussian, so that the ratio
$$
r_t(\theta)=\frac{p_{\epsilon,\theta}(a_t\mid a_{0,t},s_t)}{p_{\epsilon,\theta_{\text{old}}}(a_t\mid a_{0,t},s_t)}
$$
requires evaluating only a single Gaussian probability [2603.04790]. After the CPPO step, a flow-matching model is fit to the improved action distribution, preserving multimodality while avoiding diffusion-chain likelihood computation.

The same action-only perspective has also been extended to online multi-agent RL. OMAD factorizes the joint policy as $\pi_\theta(a\mid s)=\prod_i \pi_{\theta_i}(a_i\mid s)$, treats each agent’s action as generated by an action diffusion process, and combines decentralized diffusion actors with a centralized joint distributional critic and a tractable entropy lower bound based on per-agent ELBO surrogates [2602.18291]. Across $10$ MPE and MAMuJoCo tasks, OMAD is reported to achieve a $2.5\times$ to $5\times$ improvement in sample efficiency [2602.18291].

| Method | Setting | Distinctive mechanism |
|---|---|---|
| Diffusion-QL | Offline RL | Q-guided diffusion loss |
| PAO-DP | Offline RL | Preferred-action optimization with anti-noise preference loss |
| Q-score matching | Off-policy RL | Regress denoiser toward $\nabla_a Q$ |
| CausalGDP | RL | Intervention-aware causal guidance during denoising |
| DP-CPPO | On-policy RL | PPO on a conditional Gaussian residual kernel |
| OMAD | Online MARL | Factorized diffusion actors with joint distributional critic |

These variants preserve the action-only target of generation while changing the learning signal: demonstrations, Q-gradients, preference pairs, causal gradients, PPO surrogates, or entropy-augmented CTDE objectives.

## 4. Guidance, constraints, and cross-embodiment supervision

A major development after the initial action-only formulations was the use of auxiliary structure to guide denoising beyond vanilla conditional generation. X-Diffusion addresses cross-embodiment transfer from human videos by exploiting the forward diffusion process itself. The method trains a classifier $c_\theta(k,A_t^k,s_t)$ to distinguish noisy human actions from noisy robot actions, defines the earliest indistinguishability step
$$
k^*(A_t)=\min\{k:c_\theta(k,A_t^k,s_t)\ge 0.5\},
$$
and then trains the diffusion policy on all robot actions but only on human actions for $k\ge k^*(A_t)$ [2511.04671]. The resulting gated objective preserves fine-grained low-noise supervision from robot data and only coarse high-noise supervision from human data. On five real-world Franka tasks, X-Diffusion achieves a $16\%$ higher average success rate than the best baseline, and on specific tasks reports Mug On Rack $10/10$, Serve Egg $9/10$, and Push Plate $8/10$ [2511.04671].

Constraint guidance at inference time is another prominent branch. Temporal Logic Guidance for Action-Only Diffusion Policies with World Models introduces a separate differentiable world model $x_{t+1}=F_\theta(x_t,a_t)$, uses it to roll out candidate action sequences, computes STL robustness $\rho_\phi(\hat{x}_{1:H})$, and injects the gradient into the reverse update,
$$
a^{k-1}=\mu_\theta(a^k,s_t,k)+\lambda\nabla_{a^k}\rho_\phi(\hat{x}_{1:H})+\sigma^k\epsilon.
$$
On Robomimic Can Transport, with the constraint $\mathbf{G}(R_{zz}>\cos(5^\circ))$, the method maintains $100\%$ task success while reducing violations from $84.0\%$ for the base policy and $82.0\%$ for sample-and-rank to $4.0\%$ [2606.22729].

Self-guided action diffusion modifies inference more locally. Rather than relying on an external value or logic signal, it biases each denoising step toward the prior decision through an overlap loss on the chunk overlap,
$$
L=\sum_{i=t+h}^{t+l} w_i \|\hat{a}_i-a_i^{\text{prior}}\|_2^2,
$$
followed by a gradient update inside the denoising loop [2508.12189]. The paper reports that under a tight sampling budget the method achieves up to $70\%$ higher success rates than existing counterparts on challenging dynamic tasks, and that in the single-sample regime it attains an average success rate $71.4\%$ higher across all Robomimic benchmarks than random sampling [2508.12189]. This suggests that cross-chunk consistency can be internalized as self-guidance rather than as expensive multi-sample search.

CausalGDP uses yet another guidance signal. It learns a structural causal model over $(S_t,A_t,S_{t+1},R_t)$ and adds
$$
\nabla_{a_t^k}\log p(S_{t+1},R_t \mid S_t,\mathrm{do}(a_t^k))
$$
to the denoising guidance, weighting transition and reward terms separately [2602.09207]. Relative to conventional association-based guidance, this explicitly privileges action components judged to have causal leverage over future states and rewards. A plausible implication is that action-only diffusion has become a general substrate for modular guidance: the base denoiser models the action manifold, while world models, logic, causal models, or prior decisions steer which mode is realized.

## 5. Efficiency, robustness, and alternatives to repeated denoising

The principal practical criticism of action-only diffusion policies is inference cost. Diffusion Policy itself noted that iterative sampling increases latency relative to one-shot regressors, even though action-only conditioning is lighter than trajectory diffusion [2303.04137]. Several later works therefore target the sampling loop directly.

One-Step Diffusion Policy distills a pre-trained iterative diffusion policy into a one-step generator by minimizing an expected reverse KL along the diffusion chain. The distilled policy requires only $2\%$–$10\%$ additional pre-training cost for convergence, boosts action prediction frequency from $1.5\,\text{Hz}$ to $62\,\text{Hz}$, and on six simulation tasks reports average success $0.843$ for OneDP-S versus $0.829$ for the DDPM teacher while using a single step [2410.21257]. In four Franka tasks, OneDP-S reports average success $0.98$ versus $0.83$ for the $10$-step DDIM teacher [2410.21257].

D3P keeps the iterative denoiser but varies how much computation is spent on each control step. A lightweight state-aware adaptor chooses the stride in the DDIM chain so that crucial actions receive more denoising steps and routine actions fewer. The method reports an averaged $2.2\times$ inference speed-up on simulated tasks without degrading success and a $1.9\times$ acceleration on a physical robot [2508.06804]. Two-Steps Diffusion Policy via Genetic Denoising pushes the same logic further: by starting later in the diffusion schedule, reducing injected noise, and using a small population with OoD-aware selection, it reports that diffusion policies can operate effectively with as few as $5$ neural function evaluations and can solve challenging tasks with only $2$ NFE, achieving up to $20\%$ performance gains with significantly fewer inference steps [2510.21991].

Robustness-oriented work addresses a different consequence of iterative denoising: solver and score errors. Contractive Diffusion Policies regularize the score Jacobian so that the reverse ODE is contractive, bounding seed sensitivity and stabilizing action generation [2601.01003]. Across benchmarks, CDP reports an overall average reward of $65.7\pm2.2$ on D4RL RL compared with $61.2\pm2.6$ for EDP and shows pronounced benefits under data scarcity [2601.01003]. This does not remove iterative sampling, but it changes its numerical behavior.

A nearby but conceptually distinct response is to move diffusion out of action space entirely. Latent Weight Diffusion uses diffusion to generate the weights of a closed-loop runtime policy rather than actions or trajectories, claiming longer action horizons, robustness to perturbations, and about $1/45$ of the inference-time FLOPS relative to Diffusion Policy while keeping multitask performance comparable [2410.14040]. Because it samples a reactive controller rather than denoising each action, it is not itself an action-only diffusion policy. It is better understood as an adjacent design motivated by the same deployment bottleneck.

## 6. Empirical landscape, misconceptions, and open problems

Empirically, action-only diffusion policies now span robot imitation, offline RL, online RL, multi-agent coordination, cross-embodiment transfer, constraint-guided control, and non-stationary visual RL. Diffusion Policy established the paradigm in visuomotor imitation with an average improvement of $46.9\%$ across $15$ tasks [2303.04137]. Diffusion-QL showed that action-only diffusion can be competitive in offline RL across Gym, AntMaze, Adroit, and Kitchen [2208.06193]. X-Diffusion demonstrated gains from large human-video datasets even under severe embodiment mismatch [2511.04671]. OMAD extended the same object of generation to decentralized agents in CTDE [2602.18291].

The literature also corrects several recurring misconceptions. First, action-only is not synonymous with unimodal or myopic control: the motivation for diffusion in actions is precisely its ability to capture multimodal action distributions, whether the policy outputs a single action or a short horizon [2303.04137]. Second, action-only is not automatically fast. Without distillation, schedule adaptation, or dynamic denoising, inference remains an iterative reverse process and can be the dominant latency source [2410.21257]. Third, action-only does not necessarily outperform simpler baselines in every regime. In the non-stationary vision-based study on Procgen and PointMaze, the table at $500$k steps reports Diffuser as better than PPO and DQN on CoinRun, competitive on PointMaze, but substantially worse on Maze, while the paper also highlights higher compute requirements and inference latency [2504.00280]. This indicates that the benefits are task-dependent and sensitive to environment structure.

Open problems recur across papers. Critic quality remains a bottleneck in RL-oriented methods: PAO-DP explicitly introduces anti-noise preference optimization because weighted-regression-style methods are sensitive to Q-values, and Q-score matching notes that poor Q-gradients can misguide the actor [2405.18729]. Guidance modules introduce their own calibration risks: X-Diffusion depends on correct $k^*$ estimation by the embodiment classifier, STL guidance depends on world-model accuracy, and CausalGDP depends on the fidelity of the learned structural causal model [2511.04671]. More generally, action-only conditioning can limit explicit long-horizon planning because future states are not modeled by the policy itself; several works therefore attach auxiliary world models, causal dynamics, or trajectory-level coherence mechanisms rather than abandoning action-only generation outright [2606.22729].

Taken together, the field has shifted from treating action-only diffusion as a direct transfer of image-generation machinery to treating it as a specialized control primitive. The most successful variants modify the loss, the guidance signal, the sampler, or the deployment pathway to match the structured, low-dimensional, closed-loop character of robot action distributions. That evolution suggests that the central question is no longer whether actions can be diffused, but how the denoising process should be constrained, accelerated, or guided for the control regime at hand.

Source: https://www.emergentmind.com/topics/action-only-diffusion-policies