---
title: VLM-guided Trajectory-conditioned Diffusion Policy
url: https://www.emergentmind.com/topics/vlm-guided-trajectory-conditioned-diffusion-policy-vlm-tdp
type: topic
---

# VLM-guided Trajectory-conditioned Diffusion Policy

to=functions.arxiv_search  ฝ่ายขายออนไลน์ 无码不卡高清免费  天天送钱json
{"query":"arXiv:2507.04524 OR \"VLM-TDP\" OR \"VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation\"","max_results":5}
to=functions.arxiv_search 娱乐总代  彩票天天乐json
{"query":"\"VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation\"","max_results":10}
to=functions.arxiv_search  彩神争霸邀请码 早点加盟json
{"query":"trajectory-conditioned diffusion policy manipulation diffusion policy long-horizon VLM 2025","max_results":10}
to=functions.arxiv_search 夜夜啪  ฝ่ายขายข่าวেলjson
{"query":"Diffusion Policy robotic manipulation arXiv long-horizon robustness vision-language model trajectory-conditioned","max_results":10}
VLM-guided Trajectory-conditioned Diffusion Policy (VLM-TDP) is a robotic manipulation framework in which a vision-language model (VLM) decomposes a long-horizon instruction into concise sub-tasks and generates a voxel-based end-effector trajectory for each sub-task, while a trajectory-conditioned diffusion policy (TDP) uses that trajectory as an additional conditioning signal for action generation. The method is designed to address two limitations identified for standard diffusion policies in manipulation: degradation on long-horizon tasks and fragility under visual perturbations. In simulation, the reported gains include an average 44% increase in success rate, over 100% improvement in long-horizon tasks, and a 20% reduction in performance degradation in challenging conditions such as noisy images or altered environments; the paper further reports that the performance gap becomes more pronounced in real-world long-horizon tasks [2507.04524].

## 1. Problem Setting and Core Idea

The framework is motivated by two failure modes of standard diffusion policies for manipulation. First, long-horizon execution is difficult because multi-stage tasks require the policy to decide which object to interact with next, where to move the gripper, and how to transition between stages. The paper illustrates this with tasks such as grasping a handle, pulling a drawer, grasping an item, and placing the item into the drawer. Second, RGB-conditioned policies are sensitive to image noise, altered backgrounds, object texture changes, object size changes, clutter, and occlusion [2507.04524].

VLM-TDP addresses these issues by separating high-level task structuring from low-level visuomotor control. The VLM decomposes a high-level instruction into manageable sub-tasks and generates a voxel-based trajectory for each sub-task. The diffusion policy then predicts actions conditioned not only on the observation but also on that trajectory prior. The paper’s central claim is that a policy guided by a VLM-supplied trajectory prior can follow a more structured plan and become more robust.

A common misunderstanding is to treat the VLM as the direct action generator. In VLM-TDP, the VLM does not replace the control policy. Instead, it supplies sub-task descriptions and spatial trajectory guidance, while the diffusion model remains responsible for action generation. The method therefore implements a division between high-level decomposition and low-level execution rather than a direct language-to-action pipeline.

## 2. Task Decomposition and Voxel Trajectory Generation

The pipeline begins with a high-level instruction $\mathcal{I}$ and an initial observation $O_0$. The VLM $\mathcal{M}$ decomposes the task into a sequence of sub-tasks,
$$
\mathcal{M}(O_0, \mathcal{I}) = \{\mathcal{S}_i\}_{i=1}^n .
$$
Each sub-task $\mathcal{S}_i$ is defined as a manipulation phase that begins and ends around a gripper open/close event. This definition is operational: it partitions long-horizon behavior into discrete interaction phases. For the task **Put Item in Drawer**, the paper gives the sequence “grasp the bottom drawer handle,” “pull the drawer out,” “grasp the item,” and “put the item into the bottom drawer” [2507.04524].

For each sub-task, the system asks the VLM to generate a trajectory,
$$
\mathcal{M}(O_{i,0}, \mathcal{S}_i) = \mathcal{T}_i .
$$
The visual prompting process uses the front RGB image, projects it into a top-down view, fills shadow regions with table texture for consistency, and creates a height map to encode elevation. The image is divided into an $M \times N$ grid, vertical space is divided into $K$ levels, and the paper sets
$$
M = N = K = 6 .
$$
The VLM outputs a sequence of region indices or waypoints in this $M \times N \times K$ voxel grid.

The resulting trajectory is encoded as a sparse voxel tensor of size $M \times N \times K$. Voxels on the trajectory are labeled according to their order in the path, while non-trajectory voxels are set to $0$. This representation preserves planar position, height information, and ordering along the path. The paper emphasizes that it is more compact and easier to represent than continuous 2D or 2.5D sketch trajectories. Features are extracted from this trajectory representation using three layers of 3D CNNs, and those features are passed to the policy as an additional conditioning modality [2507.04524].

## 3. Trajectory-Conditioned Diffusion Policy

The policy backbone is the Trajectory-conditioned Diffusion Policy (TDP), which models
$$
p(A_t \mid O_t, \mathcal{T}),
$$
where $A_t$ is the action at time $t$, $O_t$ is the observation, and $\mathcal{T}$ is the trajectory condition. The implementation uses a denoising diffusion probabilistic model (DDPM). At inference time, the model starts from Gaussian noise,
$$
A_t^K \sim \mathcal{N}(0, I),
$$
and iteratively denoises it until it obtains the final action $A_t^0$. The denoising update is written as
$$
A^{k-1}_{t} = \alpha \left(A^k_{t} - \gamma \epsilon_\theta(O_{t}, \mathcal{T}, A^k_t, k) + \mathcal{N}(0, \sigma ^ 2 I)\right).
$$
Here, $\epsilon_\theta$ conditions jointly on the observation $O_t$, the trajectory $\mathcal{T}$, the noisy action $A_t^k$, and the diffusion step $k$ [2507.04524].

Training uses demonstration data. For each demonstration, the method takes the clean action $A_t^0$, samples a random diffusion step $k$, adds random noise $\epsilon^k$, and trains the network to predict that noise through the loss
$$
\mathcal{L} = MSE\big(\epsilon^k,\ \epsilon_\theta(O_t, \mathcal{T}, A^0_t + \epsilon^k, k)\big).
$$
The encoded trajectory is concatenated with the encoded image-observation history and the flattened robot state. The paper specifies that trajectory information is included only once and is assumed constant within a sub-task.

The distinction between TDP and VLM-TDP is methodological. In TDP, the trajectory condition comes from ground-truth demonstrations. In VLM-TDP, the trajectory is generated by the VLM from the current observation and the task or sub-task description. The training workflow therefore uses demonstration trajectories to teach the policy how to follow trajectories, whereas deployment uses VLM-generated trajectories to guide behavior. This suggests that VLM-TDP is not merely a new diffusion objective; it is a deployment architecture in which a policy trained with trajectory supervision is driven at test time by language-conditioned trajectory synthesis.

## 4. Evaluation Protocol and Reported Performance

The reported experiments cover simulation on RLBench, generalization and variation tests on Colosseum, and real-world trials on a Franka Emika Panda robot with a parallel gripper and an Intel RealSense D435i camera. The RLBench benchmark uses 7 tasks, 40 demonstrations per task, 20 test episodes per task, a single-view setup for the main benchmark, a multi-view setup for long-horizon tasks, training for 500 epochs, evaluation every 50 epochs, and averaging over the five highest success-rate checkpoints. The baselines are Diffusion Policy (DP), 3D Diffusion Policy (DP3), and Language-guided object-centric Diffusion Policy (Lang-o3dp) [2507.04524].

The main simulation result on the 7-task RLBench benchmark reports average success rates of **0.49** for Diffusion Policy, **0.71** for TDP, **0.69** for VLM-TDP, **0.50** for 3D Diffusion Policy, and **0.69** for Lang-o3dp. The long-horizon stacking study evaluates 1-block, 2-block, and 4-block settings, with repeated **Pick** and **Place** sub-tasks. For 1 block, the combined success rates are **0.26** for DP, **0.48** for TDP, and **0.52** for VLM-TDP. For 2 blocks, the combined rates are **0.02**, **0.17**, and **0.15**, respectively. For 4 blocks, they are **0.00**, **0.05**, and **0.04**. The paper interprets this as evidence that trajectory conditioning reduces the burden of the difficult **Pick** phase and that sub-task decomposition makes each stage easier to solve, even though performance still declines as task complexity increases.

In the real-world study, the tasks are **Pick One Orange**, **Pick Two Bananas**, and **Pick Three Bananas**. Object positions are randomized, banana orientations are also randomized, and the dataset contains 40 successful teleoperated demonstrations per task collected with a 6-DOF SpaceMouse. Evaluation uses 5 randomized setups, 4 trials per setup, and 20 evaluations per task. The reported success rates are **0.70** versus **0.85** for DP and TDP on **One Orange**, **0.70** versus **0.95** on **Two Bananas**, and **0.20** versus **0.70** on **Three Bananas** [2507.04524].

| Setting | Systems | Reported result |
|---|---|---|
| RLBench, 7-task average | DP / TDP / VLM-TDP / DP3 / Lang-o3dp | 0.49 / 0.71 / 0.69 / 0.50 / 0.69 |
| Stack 1 Block, combined | DP / TDP / VLM-TDP | 0.26 / 0.48 / 0.52 |
| Stack 2 Blocks, combined | DP / TDP / VLM-TDP | 0.02 / 0.17 / 0.15 |
| Stack 4 Blocks, combined | DP / TDP / VLM-TDP | 0.00 / 0.05 / 0.04 |
| Real world, One Orange | DP / TDP | 0.70 / 0.85 |
| Real world, Two Bananas | DP / TDP | 0.70 / 0.95 |
| Real world, Three Bananas | DP / TDP | 0.20 / 0.70 |

These results show an asymmetry between TDP and VLM-TDP. On the main 7-task RLBench average, TDP slightly exceeds VLM-TDP, whereas on Stack 1 Block the VLM-guided system is higher. This pattern is consistent with the paper’s distinction between ground-truth trajectory conditioning during TDP analysis and VLM-generated trajectory conditioning during deployment: VLM guidance is effective, but its quality is bounded by the quality of the generated spatial plan.

## 5. Robustness, Mechanism, and Limitations

The robustness analysis addresses both noisy images and altered environments. In the noisy-image test, Gaussian noise is added to image inputs without additional training, using **Stack 1 Block** as the evaluation task. The paper reports that standard Diffusion Policy degrades sharply under noise, especially in **Pick** sub-tasks where object localization is crucial, whereas TDP degrades more slowly because the trajectory provides a strong geometric cue. At sufficiently high noise, the baseline can fall to **0% success** in **Pick**, while TDP retains meaningful performance. Specifically, TDP keeps about **17.0% of its original performance** in **Pick** at $\sigma=0.64$, and about **10.4% of its original complete-task performance** [2507.04524].

The Colosseum experiments test background texture changes, object texture changes, smaller objects, and larger objects. Across all variations, TDP shows a smaller performance drop than Diffusion Policy. The paper further notes that increasing object size can help because the required execution precision is lower and the trajectory cue becomes more effective. This is presented as evidence that trajectory conditioning reduces the degree to which the policy must recover the entire plan from raw RGB alone.

The paper’s explanation for these gains has four components. First, sub-task decomposition reduces temporal complexity. Second, trajectory conditioning reduces visual dependence by supplying an explicit spatial prior. Third, voxel trajectories provide planar position and height information, rather than only a 2D sketch. Fourth, the trajectory offers local guidance under perturbation: even if the global image is noisy, the path can still guide the end-effector toward the correct region. A plausible implication is that the principal benefit of VLM-TDP lies not only in semantic decomposition but also in the introduction of a structured spatial latent that regularizes policy behavior under distribution shift.

Several limitations are stated or implied. Trajectory ambiguity increases when multiple objects are close together. Grid resolution introduces a trade-off: higher resolution makes trajectory generation harder for the VLM, while lower resolution blurs important distinctions. The method depends on the VLM’s ability to produce reasonably accurate spatial plans. Long-horizon tasks remain challenging, and performance still drops as the number of sequential choices increases. These constraints indicate that the voxel representation and the quality of VLM prompting are central bottlenecks rather than incidental implementation details.

## 6. Relation to Adjacent VLM-Guided Diffusion Frameworks

VLM-TDP belongs to a broader 2025 pattern in which VLMs provide high-level semantic or strategic structure while diffusion models handle generative control. In autonomous driving, "DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving" combines a **VLM command-guidance module**, a **sparse–dense perception stack**, and a **truncated diffusion planner** to address computationally heavy BEV construction, multi-modal driving behavior, and sub-optimal decision-making in reactive closed-loop traffic. There, the VLM produces high-level driving commands that condition trajectory generation rather than directly generating steering or throttle [2505.19381].

A second adjacent example appears in "VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion," where the VLM operates at a strategic layer, infers a safety-critical scenario generation objective, selects an adversarial background vehicle, and configures differentiable guidance templates for adaptive guided diffusion in closed-loop simulation. That framework is explicitly organized into strategic, tactical, and operational layers, with the VLM acting as strategist rather than a direct low-level controller [2512.02844].

Taken together, these works suggest a shared architectural motif: VLMs are increasingly used to supply semantically rich structure—commands, sub-task decompositions, risk hypotheses, or guidance functions—while diffusion models remain responsible for generating temporally extended control outputs. VLM-TDP is distinctive within this pattern because its conditioning object is a **voxel-based end-effector trajectory** for each manipulation sub-task. In that sense, it instantiates a specific form of VLM-guided trajectory-conditioned diffusion policy in which the interface between semantic reasoning and action generation is a discrete 3D spatial plan rather than a textual command or a scalar objective.

Within robotic manipulation, the paper’s broader claim is therefore narrower than a general theory of VLM-robot control and more precise than a generic multimodal policy. It proposes that long-horizon manipulation becomes more tractable when the VLM supplies sub-task structure and a coarse voxel trajectory, and when the diffusion policy is trained to denoise actions conditioned on that trajectory. The reported results support that formulation, while the stated limitations show that the effectiveness of the approach depends materially on the fidelity of sub-task decomposition and voxel-level trajectory generation [2507.04524].

Source: https://www.emergentmind.com/topics/vlm-guided-trajectory-conditioned-diffusion-policy-vlm-tdp