DynaGuide: Steering Diffusion Policies

• The paper presents DynaGuide, a steering method that decouples the base diffusion policy from outcome objectives using an external dynamics model.
• It employs a log-sum-exp-based metric to fuse multiple positive and negative guidance conditions during the diffusion denoising process.
• Experimental results demonstrate significant success improvements in complex manipulation tasks and improved access to rare behaviors.

Searching arXiv for the specified paper and closely related uses of the name "DynaGuide". arxiv_search(query="(Du et al., 16 Jun 2025) DynaGuide Steering Diffusion Policies with Active Dynamic Guidance", max_results=5) Searching arXiv now. {"query":"(Du et al., 16 Jun 2025) DynaGuide Steering Diffusion Policies with Active Dynamic Guidance","max_results":5} DynaGuide is a steering method for diffusion policies that uses guidance from an external dynamics model during the diffusion denoising process, leaving the base diffusion policy unchanged and introducing a decoupling between the policy’s action prior and the steering objective (Du et al., 16 Jun 2025). In the formulation reported for robot manipulation, DynaGuide is designed for settings in which large visuomotor policies have already acquired rich skills from human or self-supervised play data, but deployment requires inference-time control over outcomes such as opening rather than closing a drawer, or preferring one object over another, without retraining or fine-tuning the entire policy. The method is presented as a modular approach that can steer off-the-shelf pretrained diffusion policies, support arbitrarily many positive and negative objectives, enhance underrepresented behaviors, and remain robust when objectives are novel, noisy, or low-quality (Du et al., 16 Jun 2025).

1. Problem setting and conceptual basis

The immediate problem addressed by DynaGuide is the mismatch between the breadth of behaviors represented in a pretrained diffusion policy and the narrower, situational objectives that arise at deployment time. Common steering approaches, such as goal-conditioning, require the robot policy to be trained with a distribution of test-time objectives in mind, and they typically admit only a single goal vector or image as input (Du et al., 16 Jun 2025). In the reported motivation, this makes such approaches brittle when inference-time goals are new, novel, or low-quality.

DynaGuide takes a different approach: it leaves the base diffusion policy unchanged and injects a gradient “push” from a separately trained latent dynamics model during each denoising step (Du et al., 16 Jun 2025). The dynamics model encodes long-horizon visual outcome objectives rather than the action prior itself. The paper characterizes this as a decoupling between “what the robot can do” and “what we want it to do,” with the former residing in the base policy and the latter in the external guidance model (Du et al., 16 Jun 2025).

This separation is central to the method’s stated properties. Because the guidance signal is external, DynaGuide can be applied to off-the-shelf pretrained diffusion policies, can fuse multiple desired and undesired outcomes, and can amplify behaviors that are rare in the base policy’s training distribution (Du et al., 16 Jun 2025). A plausible implication is that the method treats steering as an inference-time control layer rather than as a property that must be embedded into the policy during training.

2. Inference algorithm and mathematical formulation

DynaGuide operates during diffusion-policy inference. Its inputs are the current observation $o_t$, a positive guidance set $\mathcal G^+$ of desired outcome images, a negative guidance set $\mathcal G^-$ of undesired outcome images, a pretrained diffusion policy $\epsilon_p(a^k,o_t)$, a latent dynamics model $h_\theta(\phi(o_t),a)$, an image embedder $\phi$, a guidance strength $s$, a variance $\sigma$, and a number of stochastic refinements $M$ (Du et al., 16 Jun 2025).

The procedure initializes $a^K \sim \mathcal N(0,I)$ and iterates from diffusion step $\mathcal G^+$0 down to $\mathcal G^+$1. At each step, the policy predicts noise, the guidance metric is evaluated from the dynamics model’s far-future latent prediction, the gradient of that metric with respect to the current noisy action is backpropagated, and the resulting gradient is combined with the policy’s denoising signal under DDIM guidance (Du et al., 16 Jun 2025). The denoising update is

$\mathcal G^+$2

where $\mathcal G^+$3 is the DDIM noise schedule (Du et al., 16 Jun 2025).

The guidance metric itself is defined through latent distances between predicted future outcomes and the embeddings of positive and negative guidance images. Let $\mathcal G^+$4 and let $\mathcal G^+$5 for each guidance image $\mathcal G^+$6. Then

$\mathcal G^+$7

This log-sum-exp construction is the basis for multi-objective steering, because it naturally fuses any number of positive and negative conditions (Du et al., 16 Jun 2025).

The dynamics model is trained separately with the objective

$\mathcal G^+$8

The reported interpretation is that DynaGuide leverages classifier-style diffusion guidance, but replaces the classifier with a latent dynamics model that predicts far-future outcomes (Du et al., 16 Jun 2025).

3. Decoupled guidance and steering capabilities

The decoupling of the dynamics model from the base policy is the defining design choice of DynaGuide. The paper identifies four consequences of this separation: modular steering of any pretrained diffusion policy at inference time, multi-objective steering through the log-sum-exp metric, robustness to mismatched or noisy objectives, and improved access to rare behaviors (Du et al., 16 Jun 2025).

For multi-objective steering, DynaGuide can mix several desired $\mathcal G^+$9 and undesired $\mathcal G^-$0 conditions, such as “press button OR open drawer AND avoid closing” (Du et al., 16 Jun 2025). Because the objectives are represented as outcome images in the latent space used by the dynamics model, the system is not restricted to a single goal vector or single goal image.

For robustness, the paper argues that guidance conditions $\mathcal G^-$1 and the policy inputs remain in-distribution even if $\mathcal G^-$2 mismatches the real scene, so latent errors average out (Du et al., 16 Jun 2025). This is used to explain why DynaGuide remains effective when guidance images are underspecified or low-quality.

For rare or underrepresented behaviors, the paper contrasts diffusion guidance with sampling-based rankers. Diffusion guidance is described as sculpting the denoising path directly into low-probability regions, whereas sampling-based rankers need many draws from $\mathcal G^-$3 to hit rare modes (Du et al., 16 Jun 2025). This suggests that DynaGuide is not merely selecting among candidate actions already favored by the policy, but modifying the trajectory of denoising itself.

4. Experimental evaluation

The experimental program spans simulated CALVIN tasks, multi-objective steering, underrepresented-behavior tests, and real-robot experiments on an ARX5 platform with a UMI policy (Du et al., 16 Jun 2025). In simulation, success rate is defined as the fraction of rollouts in which the target behavior occurs within 400 steps.

Setting	Main result	Context
ArticulatedParts	DynaGuide $\mathcal G^-$4 success vs. base $\mathcal G^-$5	Outperforms sampling by $\mathcal G^-$6 and ITPS by $\mathcal G^-$7
MovableObjects	DynaGuide $\mathcal G^-$8 vs. sampling $\mathcal G^-$9	Goal-conditioning collapses to $\epsilon_p(a^k,o_t)$0 on OOD goals
Underspecified guidance	DynaGuide retains $\epsilon_p(a^k,o_t)$1	Goal-conditioning $\epsilon_p(a^k,o_t)$2, a $\epsilon_p(a^k,o_t)$3 improvement
Multi-objective steering	DynaGuide obtains $\epsilon_p(a^k,o_t)$4 joint success	Nearly zero unwanted behaviors
Rare behavior regime	DynaGuide reaches $\epsilon_p(a^k,o_t)$5	Base policy does Switch-On $\epsilon_p(a^k,o_t)$6, sampling reaches $\epsilon_p(a^k,o_t)$7

The simulated 3D CALVIN tasks are divided into ArticulatedParts and MovableObjects. ArticulatedParts includes opening or closing a drawer, flipping a switch, and pressing a button; MovableObjects includes pushing red or blue cubes that are randomly placed (Du et al., 16 Jun 2025). Baselines are the base policy without steering, a goal-conditioned policy trained on final-image goals, a sampling-based ranking method denoted GPC that samples $\epsilon_p(a^k,o_t)$8 five times and picks the action maximizing $\epsilon_p(a^k,o_t)$9, and position guidance denoted ITPS for fixed objects (Du et al., 16 Jun 2025).

On ArticulatedParts, DynaGuide reaches approximately $h_\theta(\phi(o_t),a)$0 success, compared with approximately $h_\theta(\phi(o_t),a)$1 for the base policy, while goal-conditioning reaches approximately $h_\theta(\phi(o_t),a)$2 in-distribution (Du et al., 16 Jun 2025). On MovableObjects, goal-conditioning collapses to below $h_\theta(\phi(o_t),a)$3 because of out-of-distribution goals, while DynaGuide stays at approximately $h_\theta(\phi(o_t),a)$4 and sampling remains at approximately $h_\theta(\phi(o_t),a)$5 (Du et al., 16 Jun 2025). When guidance images are underspecified by a random robot pose in $h_\theta(\phi(o_t),a)$6, goal-conditioning remains below $h_\theta(\phi(o_t),a)$7 while DynaGuide retains approximately $h_\theta(\phi(o_t),a)$8, yielding the reported $h_\theta(\phi(o_t),a)$9 improvement (Du et al., 16 Jun 2025).

In multi-objective steering, DynaGuide obtains approximately $\phi$0 joint success with nearly zero unwanted behaviors, whereas sampling loses approximately $\phi$1 percentage points (Du et al., 16 Jun 2025). In the underrepresented-behavior experiment, $\phi$2 of Switch-On examples are removed from the base policy’s training data. The base policy then performs Switch-On in less than $\phi$3 of cases, sampling guidance rises to approximately $\phi$4, and DynaGuide rises to approximately $\phi$5 even in the $\phi$6 data regime (Du et al., 16 Jun 2025).

The real-robot experiments use an off-the-shelf cup-to-saucer diffusion policy. The dynamics model is trained on public UMI cup data together with custom two-cup demos and mouse demos (Du et al., 16 Jun 2025). In CupPreference, two cups are equidistant and the system is steered to choose red rather than grey over 20 trials. In HiddenCup, the red cup is occluded behind the grey cup, and DynaGuide picks the red cup $\phi$7 of the time versus $\phi$8 for the base policy. In NovelBehavior, steering toward touching a mouse, a behavior never seen by the base policy, doubles mouse contacts (Du et al., 16 Jun 2025).

5. Ablations, sensitivity, and operational constraints

The paper reports several sensitivity analyses. Guidance strength $\phi$9 that is too small produces no steering, whereas guidance that is too large produces incoherent trajectories; the reported optimal interval is $s$0 (Du et al., 16 Jun 2025). For the variance $s$1 in the guidance metric, small $s$2 sharpens one mode and overfits, while large $s$3 blunts the signal; the typical range is $s$4 (Du et al., 16 Jun 2025).

The number of stochastic repeats also matters. The reported choice $s$5 stabilizes high-$s$6 guidance through MCMC-style resampling (Du et al., 16 Jun 2025). With respect to the number of conditions $s$7, the method is reported to remain robust even down to $s$8–$s$9 conditions, although more conditions produce a smoother log-sum-exp objective (Du et al., 16 Jun 2025).

The most important ablation concerns the dynamics model’s training distribution. The paper states that the dynamics model must be pretrained with noised actions; otherwise, $\sigma$0 is meaningless on out-of-distribution $\sigma$1 and steering fails (Du et al., 16 Jun 2025). This constraint follows directly from the fact that DynaGuide evaluates and differentiates the guidance metric on noisy action samples encountered during denoising rather than on clean actions alone.

6. Naming, scope, and related uses of “DynaGuide”

The name “DynaGuide” is not unique to diffusion-policy steering. In robotics motion planning, “DynaGuide” denotes a guided sampling-based kinodynamic planner for navigation in unknown or partially observed environments (Khanal et al., 2023). In reinforcement learning, Guided Dyna-Q is also referred to as DynaGuide and integrates automated planning into Dyna-Q to focus exploration on plannable transitions (Hayamizu et al., 2020). In text-to-image diffusion, “DynaGuide” denotes a framework for dynamic classifier-free diffusion guidance via online feedback (Papalampidi et al., 19 Sep 2025). In computer vision, “DynaGuide” names a dual-guidance framework for unsupervised semantic segmentation that combines global pseudo-labels with local boundary refinement (Guermazi et al., 13 Feb 2026).

These works are unrelated in method and application domain. A plausible implication is that citation by arXiv identifier is necessary for disambiguation whenever the term appears in bibliographies, search systems, or cross-domain discussion.

Within the diffusion-policy setting, DynaGuide is defined by three interlocking features: classifier-style guidance during denoising, a separately trained latent dynamics model that predicts far-future outcomes, and inference-time steering without fine-tuning the base policy (Du et al., 16 Jun 2025). In that sense, its contribution is not only a steering heuristic but a reorganization of policy control into a modular relation between action prior and outcome preference.