---
title: Goal-Conditioned Diffusion Policy
url: https://www.emergentmind.com/topics/goal-conditioned-diffusion-policy-dp
type: topic
---

# Goal-Conditioned Diffusion Policy

A goal-conditioned diffusion policy (DP) is a generative policy architecture that leverages denoising diffusion models to learn and sample multi-step sequences of actions or sub-trajectories conditioned explicitly on a goal specification. This framework has become central in both goal-conditioned imitation learning (GCIL) and offline/online goal-conditioned reinforcement learning (GCRL), due to its capacity to model multi-modal trajectory distributions, capture long-range dependencies, and generalize to diverse goal specifications across high-dimensional robotic and control domains. The formalism is rooted in Denoising Diffusion Probabilistic Models (DDPMs), with conditioning mechanisms tailored to inject observations and goals into both the forward (noising) and reverse (denoising) processes. Goal-conditioned DPs have achieved state-of-the-art results across a variety of domains, including whole-body humanoid loco-manipulation, multi-stage robotic manipulation, offline GCRL benchmarks, and vision-driven deformable object manipulation [2603.13707][2606.08775][2402.07226][2403.10401].

## 1. Mathematical Formalism and Learning Objectives

At the core of a goal-conditioned DP lies the parameterization of the conditional policy as the reverse process of a Markovian forward noising chain. Let $A_t^0$ or $a^0$ denote the ground-truth action sequence (or chunk), $S_t$ the observation, and $g$ the explicit task goal.

The forward process typically follows
\[
q(A_t^k | A_t^{k-1}) = \mathcal{N}\Bigl(\sqrt{\alpha_k}\,A_t^{k-1}, (1-\alpha_k)I\Bigr),
\]
or, equivalently,
\[
A_t^k = \sqrt{\bar\alpha_k}\,A_t^0 + \sqrt{1-\bar\alpha_k}\,\epsilon,\qquad \epsilon\sim\mathcal{N}(0, I),
\]
where $\bar{\alpha}_k = \prod_{i=1}^k \alpha_i$. The reverse (denoising) process learns $p_\theta(A_t^{k-1} | A_t^{k}, S_t, g)$, parameterized as a time- and goal-conditioned Gaussian.

Training is performed via denoising score matching or maximum likelihood, using objectives such as
\[
L(\theta) = \mathbb{E}\left[ \|\epsilon - \epsilon_\theta(A_t^k, S_t, g, k)\|^2 \right],
\]
where $\epsilon_\theta$ is a neural network regressor predicting the noise at each diffusion step.

Some variants (e.g., Merlin) perform the diffusion in the state space rather than the action space, employing a single backward (denoising) step per environment step and reducing inference complexity [2310.02505]. There exist both discrete-time (DDPM-based) and continuous-time (score-based SDE/ODE) instantiations [2304.02532].

## 2. Goal Conditioning, Architectural Mechanisms, and Sampling

Goal conditioning is handled via multiple mechanisms:

- **Direct input concatenation**: The goal $g$ (and possibly other contextual embeddings) is concatenated with the observation and/or injected as additional tokens for Transformer-based policies [2603.13707][2402.07226][2606.08775].
- **Prompt tokens**: Some architectures prepend learned goal embedding vectors as dedicated tokens (e.g., Sub-trajectory Stitching with Diffusion, SSD) [2402.07226].
- **Feature-wise modulation** (FiLM): The conditioning vector modulates the intermediate activations in each layer [2403.10401].

The neural backbone often comprises Transformer encoders or U-Net+Transformer hybrids to facilitate long-range temporal dependencies and cross-attention from actions to the encoded observation/goal [2603.13707][2606.08775]. Some architectures use 1D CNNs (e.g. SculptDiff) for short horizon action chunks within a goal-conditioned diffusion policy [2403.10401].

The sampling procedure typically follows the standard reverse-diffusion chain, repeatedly denoising a Gaussian-initialized action sequence with goal and state conditioning. Acceleration techniques such as DDIM and deterministic ODE sampling have been adopted for efficient inference, yielding action generation in far fewer denoising steps (e.g., BESO’s 3-step DDIM) compared to naive DDPM sampling [2304.02532].

## 3. Hierarchical and Modular Integration

Goal-conditioned DPs are deployed in both flat and hierarchical policy architectures:

- **Hierarchical Control**: DP generates high-level action commands (e.g., base velocity and Cartesian hand poses), which are executed by a separately parameterized low-level controller (typically an MLP-based or RL-trained policy) [2603.13707].
- **Model-predictive and subgoal-chaining**: DPs are nested within higher-level world models or subgoal planners, receiving short-horizon subgoals as conditioning inputs and producing action sequences for each episode section [2606.08775][2402.07226].

In hierarchical setups such as REFINE-DP, both high-level diffusion planner and low-level controller are jointly updated via alternating optimization—DP parameters are fine-tuned using RL (e.g., PPO gradients through diffusion steps), while the controller is trained to accurately track the evolving distribution of high-level commands, thus minimizing distributional mismatch [2603.13707].

## 4. Application Domains and Empirical Evaluation

Goal-conditioned DPs have demonstrated high performance across a variety of robotic and control domains:

- **Humanoid Loco-Manipulation**: REFINE-DP yields over $90\%$ success rates in complex, long-horizon, out-of-distribution simulation and real-world tasks, substantially outperforming pre-trained baselines and flattened RL-only fine-tuning [2603.13707].
- **Multi-stage Visual Manipulation**: In WorldDP, object-centric DPs track subgoals from a learned high-level world model, with horizon and conditioning tailored to the stage granularity. Empirical ablations indicate superior performance for horizon $H=40$ and criticality of subgoal-conditioning [2606.08775].
- **Offline Goal-Conditioned RL Benchmarks**: SSD attains state-of-the-art scores on D4RL GCRL tasks—Maze2D (Umaze: 144.6, Large: 183.5), Fetch Reach/Push/PickAndPlace—with robust sub-trajectory stitching and long-horizon planning [2402.07226].
- **Real-World Deformable Object Manipulation**: SculptDiff (point-cloud-conditioned DP) achieves best-in-class geometric proximity to target shapes in real-world clay sculpting, outstripping both model-based and imitation learning baselines [2403.10401].
- **General GCIL Benchmarks**: BESO achieves highest rewards/success rates on Block-Push and Relay-Kitchen (Block-Push C-BESO: $0.96 \pm 0.02$, CALVIN vision goal chaining) in only 3 inference steps [2304.02532]. Merlin, with its single-step denoising per environment step, is both computationally efficient and empirically superior or competitive with BC or RL baselines across 10 tasks [2310.02505].

## 5. Extensions, Ablations, and Practical Considerations

Key ablation findings and deployment observations include:

- **Network Backbone**: Transformer-based diffusion backbones outperform MLPs both pre-trained and with fine-tuning [2603.13707][2304.02532].
- **Conditioning scheme**: Absence of goal-conditioned inputs to the diffusion process severely impairs planning (WorldDP ablations) [2606.08775].
- **Horizon trade-off**: Too short a horizon impedes subgoal reachability, while too long leads to drifting from precise subgoal tracking; optimal $H$ is domain- and hierarchy-dependent [2606.08775].
- **Sample efficiency and speed**: Score-based DPs with efficient ODE solvers (BESO) or single-step denoising (Merlin) yield 10–15$\times$ inference speed-ups over full DDPM policies, while retaining success rates [2304.02532][2310.02505].
- **Multi-modality**: DP backbones capture multiple diverse valid behaviors per goal, essential for tasks with many viable solution strategies [2603.13707][2304.02532].

## 6. Limitations and Future Directions

Current limitations of goal-conditioned DPs include:

- **Perception and embodiment**: Most published DPs are conditioned on state-based (pose or MoCap) observations; direct vision-conditioned DPs or policies grounded in raw sensory input remain an active area [2603.13707][2403.10401].
- **Action/goal representation**: Incorporation of latent skills or hierarchical abstractions holds promise for improving out-of-distribution generalization [2603.13707][2402.07226].
- **Temporal scaling**: While sub-trajectory stitching (SSD) flexibly scales to very long-horizon tasks, slow sampling and over-reliance on the value estimator remain bottlenecks [2402.07226].
- **Force/interaction modeling**: For deformable or contact-rich manipulation, lack of tactile/force feedback and non-physical gripper geometries limit expressivity [2403.10401].

Planned directions include fully vision-conditioned goal-embedding, online fine-tuning via limited (on-policy) rollouts, hierarchical DP stacks for compositional planning, and integration of uncertainty-aware value conditioning for robust sampling [2603.13707][2402.07226][2606.08775].

---

**References**:  
REFINE-DP: [2603.13707]  
WorldDP: [2606.08775]  
SSD: [2402.07226]  
SculptDiff: [2403.10401]  
Merlin: [2310.02505]  
BESO: [2304.02532]

Source: https://www.emergentmind.com/topics/goal-conditioned-diffusion-policy-dp