Action-Only Diffusion Policies

• Action-only diffusion policies are conditional generative models that denoise short action sequences based solely on the current observation or state, capturing multimodal action distributions.
• They utilize a forward Gaussian noising process with iterative reverse denoising—via score matching or SDE/ODE formulations—to generate effective receding-horizon control strategies.
• Recent advancements enhance efficiency and robustness through techniques like one-step policy distillation, constraint-guided updates, and fusion of visual and proprioceptive conditioning.

Action-only diffusion policies are conditional generative policies that denoise actions, or short action chunks, directly in action space given the current observation or state, rather than diffusing over pixels or full state-action trajectories. In the visuomotor formulation introduced by Diffusion Policy, the policy models a future action sequence $A$ conditioned on observations $O$ and executes only the first part of that sequence in a receding-horizon, closed-loop manner (Chi et al., 2023). In offline reinforcement learning, the same term also denotes a conditional diffusion model for a single action $a$ given a state $s$, with the terminal sample of the reverse chain serving as the action (Wang et al., 2022). Across imitation learning, offline RL, online RL, constraint-guided control, and cross-embodiment transfer, the common premise is that a diffusion model can represent complex, multimodal action distributions while retaining stepwise conditioning on the current control context.

1. Formalization and scope

The standard action-only formulation uses a forward Gaussian noising process on actions and a reverse denoising model conditioned on the current observation or state. A representative DDPM-style forward kernel is

$$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$

with closed-form corruption

$$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$

and a conditional reverse model that predicts either $\epsilon_\theta$ or scores (Pace et al., 6 Nov 2025). In Diffusion-QL, the policy is the marginal $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$ obtained by running the reverse chain from Gaussian noise to a single action conditioned on the current state, with the action returned at diffusion timestep $i=0$ (Wang et al., 2022).

Action-only does not imply a single scalar control output or a strictly one-step temporal horizon. In Diffusion Policy, the generative variable is a future action sequence $A$ of length $O$0, while observations are conditioning inputs only; the model then executes the first $O$1 actions and replans (Chi et al., 2023). In X-Diffusion, the policy predicts an action sequence $O$2 of length $O$3 conditioned on current observation/state $O$4, with $O$5 and observation horizon $O$6 (Pace et al., 6 Nov 2025). A recurring distinction in the literature is therefore between action-only diffusion over current or short-horizon controls and trajectory diffusion over full future state-action trajectories.

The training objective is usually the conditional denoising loss. One common form is

$O$7

which is the form used in X-Diffusion for action chunks (Pace et al., 6 Nov 2025). Diffusion-QL uses the analogous action-level objective

$O$8

thereby treating denoising score matching as a direct conditional model of the behavior action distribution (Wang et al., 2022).

A separate branch formulates action-only diffusion through score-based SDEs or ODEs. Contractive Diffusion Policies describe the policy as a conditional action generator $O$9 whose reverse dynamics are parameterized by a score function $a$0 and sampled by solving a reverse-time SDE or its probability flow ODE (Abyaneh et al., 2 Jan 2026). Q-score matching takes a continuous-time view in which the denoising vector field $a$1 defines the action flow, and training aligns that field with $a$2 rather than with a behavior-cloning target (Psenka et al., 2023). This suggests that “action-only diffusion policy” is best understood as a family of conditional action generators sharing the same object of generation—actions rather than states or trajectories—even when their stochastic-process formalism differs.

2. Control semantics and architectural patterns

The original visuomotor recipe couples action diffusion with receding-horizon control. At each control step, Diffusion Policy encodes the recent observation context once, initializes the action sequence from noise, iteratively denoises it, returns $a$3, executes only the first $a$4 actions, and then replans with new observations (Chi et al., 2023). This design combines sequence-level temporal consistency with closed-loop feedback. The paper reports that typical values were $a$5, $a$6, and $a$7–$a$8, and that DDIM with about $a$9–$s$0 inference steps yielded about $s$1 latency on a single GPU (Chi et al., 2023).

Conditioning pipelines typically fuse visual and low-dimensional information. Diffusion Policy uses a ResNet-18 visual encoder trained end-to-end, spatial softmax pooling, GroupNorm, and either FiLM or cross-attention depending on whether the denoiser is a temporal CNN or a time-series diffusion transformer (Chi et al., 2023). X-Diffusion adopts a visual encoder over masked RGB frames at $s$2 with end-effector or human-keypoint overlays, concatenates proprioceptive features $s$3, and predicts actions in end-effector position, orientation, and gripper space using a Diffusion Policy UNet for actions (Pace et al., 6 Nov 2025). In offline RL, much smaller state-conditioned MLP denoisers are common; Diffusion-QL uses a $s$4-layer MLP with Mish activations and timestep embeddings, while Q-score matching uses a two-hidden-layer MLP taking $s$5 as input (Wang et al., 2022).

The architectural spectrum is therefore broad, but the conditioning logic is stable: the current state or observation is encoded once, diffusion time is embedded, and only the action variable is iteratively updated. Temporal Logic Guidance for Action-Only Diffusion Policies with World Models makes this separation explicit by emphasizing that “states are not generated by the diffusion process itself—only action sequences are denoised,” while future states needed for constraint evaluation are supplied by an external world model (Zoellner et al., 22 Jun 2026). A common misconception is that diffusion control necessarily implies joint action-state generation; action-only policies are specifically defined by not doing so.

3. Learning regimes: imitation, offline RL, on-policy RL, and MARL

The earliest large-scale successes of action-only diffusion were in imitation learning and offline RL. Diffusion Policy benchmarked action diffusion across $s$6 tasks from $s$7 robot manipulation benchmarks and reported an average improvement of $s$8 over existing state-of-the-art methods, attributing much of that gain to multimodal action modeling, training stability, and receding-horizon control (Chi et al., 2023). Diffusion-QL then established action-only diffusion as a strong policy class for offline RL by combining the diffusion cloning loss with a Q-guided term,

$s$9

and reported normalized D4RL averages of $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$0 on Gym, $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$1 on AntMaze, $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$2 on Adroit, and $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$3 on Kitchen (Wang et al., 2022).

Subsequent offline RL work diversified the policy-improvement mechanism. PAO-DP keeps an action-only conditional diffusion model for the behavior distribution, samples multiple in-distribution candidate actions from it, uses the critic to select preferred actions, and optimizes a surrogate diffusion policy with a Bradley–Terry-style preference loss plus an anti-noise mixture term (Zhang et al., 2024). Q-score matching instead regresses the denoising field directly toward $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$4,

$$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$5

so that actor improvement differentiates only through the denoiser rather than through the entire diffusion trajectory (Psenka et al., 2023). CausalGDP adds intervention-aware gradients from a learned causal dynamical model to the denoising process and reports, for example, Maze2D-large-v1 performance of $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$6 versus $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$7 for Diffusion-QL and an AntMaze average of $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$8 versus $$q(a_k \mid a_{k-1}) = \mathcal{N}(a_k; \sqrt{1-\beta_k}\, a_{k-1}, \beta_k I),$$9 (Xiao et al., 9 Feb 2026).

On-policy learning required a different solution because PPO-style updates depend on tractable likelihood ratios. Diffusion Policy through Conditional Proximal Policy Optimization addresses this by aligning policy iteration with a conditional Gaussian residual kernel

$$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$0

and using PPO only on that conditional Gaussian, so that the ratio

$$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$1

requires evaluating only a single Gaussian probability (Liu et al., 5 Mar 2026). After the CPPO step, a flow-matching model is fit to the improved action distribution, preserving multimodality while avoiding diffusion-chain likelihood computation.

The same action-only perspective has also been extended to online multi-agent RL. OMAD factorizes the joint policy as $$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$2, treats each agent’s action as generated by an action diffusion process, and combines decentralized diffusion actors with a centralized joint distributional critic and a tractable entropy lower bound based on per-agent ELBO surrogates (Li et al., 20 Feb 2026). Across $$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$3 MPE and MAMuJoCo tasks, OMAD is reported to achieve a $$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$4 to $$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$5 improvement in sample efficiency (Li et al., 20 Feb 2026).

Method	Setting	Distinctive mechanism
Diffusion-QL	Offline RL	Q-guided diffusion loss
PAO-DP	Offline RL	Preferred-action optimization with anti-noise preference loss
Q-score matching	Off-policy RL	Regress denoiser toward $$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$6
CausalGDP	RL	Intervention-aware causal guidance during denoising
DP-CPPO	On-policy RL	PPO on a conditional Gaussian residual kernel
OMAD	Online MARL	Factorized diffusion actors with joint distributional critic

These variants preserve the action-only target of generation while changing the learning signal: demonstrations, Q-gradients, preference pairs, causal gradients, PPO surrogates, or entropy-augmented CTDE objectives.

4. Guidance, constraints, and cross-embodiment supervision

A major development after the initial action-only formulations was the use of auxiliary structure to guide denoising beyond vanilla conditional generation. X-Diffusion addresses cross-embodiment transfer from human videos by exploiting the forward diffusion process itself. The method trains a classifier $$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$7 to distinguish noisy human actions from noisy robot actions, defines the earliest indistinguishability step

$$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$8

and then trains the diffusion policy on all robot actions but only on human actions for $$a_k = \sqrt{\bar{\alpha}_k}\, a_0 + \sqrt{1-\bar{\alpha}_k}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$9 (Pace et al., 6 Nov 2025). The resulting gated objective preserves fine-grained low-noise supervision from robot data and only coarse high-noise supervision from human data. On five real-world Franka tasks, X-Diffusion achieves a $\epsilon_\theta$0 higher average success rate than the best baseline, and on specific tasks reports Mug On Rack $\epsilon_\theta$1, Serve Egg $\epsilon_\theta$2, and Push Plate $\epsilon_\theta$3 (Pace et al., 6 Nov 2025).

Constraint guidance at inference time is another prominent branch. Temporal Logic Guidance for Action-Only Diffusion Policies with World Models introduces a separate differentiable world model $\epsilon_\theta$4, uses it to roll out candidate action sequences, computes STL robustness $\epsilon_\theta$5, and injects the gradient into the reverse update,

$\epsilon_\theta$6

On Robomimic Can Transport, with the constraint $\epsilon_\theta$7, the method maintains $\epsilon_\theta$8 task success while reducing violations from $\epsilon_\theta$9 for the base policy and $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$0 for sample-and-rank to $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$1 (Zoellner et al., 22 Jun 2026).

Self-guided action diffusion modifies inference more locally. Rather than relying on an external value or logic signal, it biases each denoising step toward the prior decision through an overlap loss on the chunk overlap,

$p_\theta(a \mid s) = p_\theta(a^0 \mid s)$2

followed by a gradient update inside the denoising loop (Malhotra et al., 17 Aug 2025). The paper reports that under a tight sampling budget the method achieves up to $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$3 higher success rates than existing counterparts on challenging dynamic tasks, and that in the single-sample regime it attains an average success rate $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$4 higher across all Robomimic benchmarks than random sampling (Malhotra et al., 17 Aug 2025). This suggests that cross-chunk consistency can be internalized as self-guidance rather than as expensive multi-sample search.

CausalGDP uses yet another guidance signal. It learns a structural causal model over $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$5 and adds

$p_\theta(a \mid s) = p_\theta(a^0 \mid s)$6

to the denoising guidance, weighting transition and reward terms separately (Xiao et al., 9 Feb 2026). Relative to conventional association-based guidance, this explicitly privileges action components judged to have causal leverage over future states and rewards. A plausible implication is that action-only diffusion has become a general substrate for modular guidance: the base denoiser models the action manifold, while world models, logic, causal models, or prior decisions steer which mode is realized.

5. Efficiency, robustness, and alternatives to repeated denoising

The principal practical criticism of action-only diffusion policies is inference cost. Diffusion Policy itself noted that iterative sampling increases latency relative to one-shot regressors, even though action-only conditioning is lighter than trajectory diffusion (Chi et al., 2023). Several later works therefore target the sampling loop directly.

One-Step Diffusion Policy distills a pre-trained iterative diffusion policy into a one-step generator by minimizing an expected reverse KL along the diffusion chain. The distilled policy requires only $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$7–$p_\theta(a \mid s) = p_\theta(a^0 \mid s)$8 additional pre-training cost for convergence, boosts action prediction frequency from $p_\theta(a \mid s) = p_\theta(a^0 \mid s)$9 to $i=0$0, and on six simulation tasks reports average success $i=0$1 for OneDP-S versus $i=0$2 for the DDPM teacher while using a single step (Wang et al., 2024). In four Franka tasks, OneDP-S reports average success $i=0$3 versus $i=0$4 for the $i=0$5-step DDIM teacher (Wang et al., 2024).

D3P keeps the iterative denoiser but varies how much computation is spent on each control step. A lightweight state-aware adaptor chooses the stride in the DDIM chain so that crucial actions receive more denoising steps and routine actions fewer. The method reports an averaged $i=0$6 inference speed-up on simulated tasks without degrading success and a $i=0$7 acceleration on a physical robot (Yu et al., 9 Aug 2025). Two-Steps Diffusion Policy via Genetic Denoising pushes the same logic further: by starting later in the diffusion schedule, reducing injected noise, and using a small population with OoD-aware selection, it reports that diffusion policies can operate effectively with as few as $i=0$8 neural function evaluations and can solve challenging tasks with only $i=0$9 NFE, achieving up to $A$0 performance gains with significantly fewer inference steps (Clemente et al., 24 Oct 2025).

Robustness-oriented work addresses a different consequence of iterative denoising: solver and score errors. Contractive Diffusion Policies regularize the score Jacobian so that the reverse ODE is contractive, bounding seed sensitivity and stabilizing action generation (Abyaneh et al., 2 Jan 2026). Across benchmarks, CDP reports an overall average reward of $A$1 on D4RL RL compared with $A$2 for EDP and shows pronounced benefits under data scarcity (Abyaneh et al., 2 Jan 2026). This does not remove iterative sampling, but it changes its numerical behavior.

A nearby but conceptually distinct response is to move diffusion out of action space entirely. Latent Weight Diffusion uses diffusion to generate the weights of a closed-loop runtime policy rather than actions or trajectories, claiming longer action horizons, robustness to perturbations, and about $A$3 of the inference-time FLOPS relative to Diffusion Policy while keeping multitask performance comparable (Hegde et al., 2024). Because it samples a reactive controller rather than denoising each action, it is not itself an action-only diffusion policy. It is better understood as an adjacent design motivated by the same deployment bottleneck.

6. Empirical landscape, misconceptions, and open problems

Empirically, action-only diffusion policies now span robot imitation, offline RL, online RL, multi-agent coordination, cross-embodiment transfer, constraint-guided control, and non-stationary visual RL. Diffusion Policy established the paradigm in visuomotor imitation with an average improvement of $A$4 across $A$5 tasks (Chi et al., 2023). Diffusion-QL showed that action-only diffusion can be competitive in offline RL across Gym, AntMaze, Adroit, and Kitchen (Wang et al., 2022). X-Diffusion demonstrated gains from large human-video datasets even under severe embodiment mismatch (Pace et al., 6 Nov 2025). OMAD extended the same object of generation to decentralized agents in CTDE (Li et al., 20 Feb 2026).

The literature also corrects several recurring misconceptions. First, action-only is not synonymous with unimodal or myopic control: the motivation for diffusion in actions is precisely its ability to capture multimodal action distributions, whether the policy outputs a single action or a short horizon (Chi et al., 2023). Second, action-only is not automatically fast. Without distillation, schedule adaptation, or dynamic denoising, inference remains an iterative reverse process and can be the dominant latency source (Wang et al., 2024). Third, action-only does not necessarily outperform simpler baselines in every regime. In the non-stationary vision-based study on Procgen and PointMaze, the table at $A$6k steps reports Diffuser as better than PPO and DQN on CoinRun, competitive on PointMaze, but substantially worse on Maze, while the paper also highlights higher compute requirements and inference latency (Baveja, 31 Mar 2025). This indicates that the benefits are task-dependent and sensitive to environment structure.

Open problems recur across papers. Critic quality remains a bottleneck in RL-oriented methods: PAO-DP explicitly introduces anti-noise preference optimization because weighted-regression-style methods are sensitive to Q-values, and Q-score matching notes that poor Q-gradients can misguide the actor (Zhang et al., 2024). Guidance modules introduce their own calibration risks: X-Diffusion depends on correct $A$7 estimation by the embodiment classifier, STL guidance depends on world-model accuracy, and CausalGDP depends on the fidelity of the learned structural causal model (Pace et al., 6 Nov 2025). More generally, action-only conditioning can limit explicit long-horizon planning because future states are not modeled by the policy itself; several works therefore attach auxiliary world models, causal dynamics, or trajectory-level coherence mechanisms rather than abandoning action-only generation outright (Zoellner et al., 22 Jun 2026).

Taken together, the field has shifted from treating action-only diffusion as a direct transfer of image-generation machinery to treating it as a specialized control primitive. The most successful variants modify the loss, the guidance signal, the sampler, or the deployment pathway to match the structured, low-dimensional, closed-loop character of robot action distributions. That evolution suggests that the central question is no longer whether actions can be diffused, but how the denoising process should be constrained, accelerated, or guided for the control regime at hand.