Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)
Papers
Topics
Authors
Recent
Search
2000 character limit reached

VLM-guided Trajectory-conditioned Diffusion Policy

Updated 6 July 2026
  • The paper demonstrates a novel approach where a VLM decomposes high-level instructions into voxel-based sub-task trajectories for guiding diffusion policies.
  • The diffusion policy conditions on the generated voxel trajectory to improve action generation robustness and long-horizon performance under visual noise.
  • Experiments show up to a 44% success rate increase and significant robustness gains compared to standard diffusion policies in both simulation and real-world settings.

to=functions.arxiv_search ฝ่ายขายออนไลน์ 无码不卡高清免费 天天送钱json {"query":"(Huang et al., 6 Jul 2025) OR \"VLM-TDP\" OR \"VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation\"","max_results":5} to=functions.arxiv_search 娱乐总代 彩票天天乐json {"query":"\"VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation\"","max_results":10} to=functions.arxiv_search 彩神争霸邀请码 早点加盟json {"query":"trajectory-conditioned diffusion policy manipulation diffusion policy long-horizon VLM 2025","max_results":10} to=functions.arxiv_search 夜夜啪 ฝ่ายขายข่าวেলjson {"query":"Diffusion Policy robotic manipulation arXiv long-horizon robustness vision-LLM trajectory-conditioned","max_results":10} VLM-guided Trajectory-conditioned Diffusion Policy (VLM-TDP) is a robotic manipulation framework in which a vision-LLM (VLM) decomposes a long-horizon instruction into concise sub-tasks and generates a voxel-based end-effector trajectory for each sub-task, while a trajectory-conditioned diffusion policy (TDP) uses that trajectory as an additional conditioning signal for action generation. The method is designed to address two limitations identified for standard diffusion policies in manipulation: degradation on long-horizon tasks and fragility under visual perturbations. In simulation, the reported gains include an average 44% increase in success rate, over 100% improvement in long-horizon tasks, and a 20% reduction in performance degradation in challenging conditions such as noisy images or altered environments; the paper further reports that the performance gap becomes more pronounced in real-world long-horizon tasks (Huang et al., 6 Jul 2025).

1. Problem Setting and Core Idea

The framework is motivated by two failure modes of standard diffusion policies for manipulation. First, long-horizon execution is difficult because multi-stage tasks require the policy to decide which object to interact with next, where to move the gripper, and how to transition between stages. The paper illustrates this with tasks such as grasping a handle, pulling a drawer, grasping an item, and placing the item into the drawer. Second, RGB-conditioned policies are sensitive to image noise, altered backgrounds, object texture changes, object size changes, clutter, and occlusion (Huang et al., 6 Jul 2025).

VLM-TDP addresses these issues by separating high-level task structuring from low-level visuomotor control. The VLM decomposes a high-level instruction into manageable sub-tasks and generates a voxel-based trajectory for each sub-task. The diffusion policy then predicts actions conditioned not only on the observation but also on that trajectory prior. The paper’s central claim is that a policy guided by a VLM-supplied trajectory prior can follow a more structured plan and become more robust.

A common misunderstanding is to treat the VLM as the direct action generator. In VLM-TDP, the VLM does not replace the control policy. Instead, it supplies sub-task descriptions and spatial trajectory guidance, while the diffusion model remains responsible for action generation. The method therefore implements a division between high-level decomposition and low-level execution rather than a direct language-to-action pipeline.

2. Task Decomposition and Voxel Trajectory Generation

The pipeline begins with a high-level instruction I\mathcal{I} and an initial observation O0O_0. The VLM M\mathcal{M} decomposes the task into a sequence of sub-tasks,

M(O0,I)={Si}i=1n.\mathcal{M}(O_0, \mathcal{I}) = \{\mathcal{S}_i\}_{i=1}^n .

Each sub-task Si\mathcal{S}_i is defined as a manipulation phase that begins and ends around a gripper open/close event. This definition is operational: it partitions long-horizon behavior into discrete interaction phases. For the task Put Item in Drawer, the paper gives the sequence “grasp the bottom drawer handle,” “pull the drawer out,” “grasp the item,” and “put the item into the bottom drawer” (Huang et al., 6 Jul 2025).

For each sub-task, the system asks the VLM to generate a trajectory,

M(Oi,0,Si)=Ti.\mathcal{M}(O_{i,0}, \mathcal{S}_i) = \mathcal{T}_i .

The visual prompting process uses the front RGB image, projects it into a top-down view, fills shadow regions with table texture for consistency, and creates a height map to encode elevation. The image is divided into an M×NM \times N grid, vertical space is divided into KK levels, and the paper sets

M=N=K=6.M = N = K = 6 .

The VLM outputs a sequence of region indices or waypoints in this M×N×KM \times N \times K voxel grid.

The resulting trajectory is encoded as a sparse voxel tensor of size M×N×KM \times N \times K. Voxels on the trajectory are labeled according to their order in the path, while non-trajectory voxels are set to $0$. This representation preserves planar position, height information, and ordering along the path. The paper emphasizes that it is more compact and easier to represent than continuous 2D or 2.5D sketch trajectories. Features are extracted from this trajectory representation using three layers of 3D CNNs, and those features are passed to the policy as an additional conditioning modality (Huang et al., 6 Jul 2025).

3. Trajectory-Conditioned Diffusion Policy

The policy backbone is the Trajectory-conditioned Diffusion Policy (TDP), which models

p(At∣Ot,T),p(A_t \mid O_t, \mathcal{T}),

where AtA_t is the action at time tt, OtO_t is the observation, and T\mathcal{T} is the trajectory condition. The implementation uses a denoising diffusion probabilistic model (DDPM). At inference time, the model starts from Gaussian noise,

AtK∼N(0,I),A_t^K \sim \mathcal{N}(0, I),

and iteratively denoises it until it obtains the final action At0A_t^0. The denoising update is written as

Atk−1=α(Atk−γϵθ(Ot,T,Atk,k)+N(0,σ2I)).A^{k-1}_{t} = \alpha \left(A^k_{t} - \gamma \epsilon_\theta(O_{t}, \mathcal{T}, A^k_t, k) + \mathcal{N}(0, \sigma ^ 2 I)\right).

Here, ϵθ\epsilon_\theta conditions jointly on the observation OtO_t, the trajectory T\mathcal{T}, the noisy action AtkA_t^k, and the diffusion step kk (Huang et al., 6 Jul 2025).

Training uses demonstration data. For each demonstration, the method takes the clean action At0A_t^0, samples a random diffusion step kk, adds random noise ϵk\epsilon^k, and trains the network to predict that noise through the loss

L=MSE(ϵk, ϵθ(Ot,T,At0+ϵk,k)).\mathcal{L} = MSE\big(\epsilon^k,\ \epsilon_\theta(O_t, \mathcal{T}, A^0_t + \epsilon^k, k)\big).

The encoded trajectory is concatenated with the encoded image-observation history and the flattened robot state. The paper specifies that trajectory information is included only once and is assumed constant within a sub-task.

The distinction between TDP and VLM-TDP is methodological. In TDP, the trajectory condition comes from ground-truth demonstrations. In VLM-TDP, the trajectory is generated by the VLM from the current observation and the task or sub-task description. The training workflow therefore uses demonstration trajectories to teach the policy how to follow trajectories, whereas deployment uses VLM-generated trajectories to guide behavior. This suggests that VLM-TDP is not merely a new diffusion objective; it is a deployment architecture in which a policy trained with trajectory supervision is driven at test time by language-conditioned trajectory synthesis.

4. Evaluation Protocol and Reported Performance

The reported experiments cover simulation on RLBench, generalization and variation tests on Colosseum, and real-world trials on a Franka Emika Panda robot with a parallel gripper and an Intel RealSense D435i camera. The RLBench benchmark uses 7 tasks, 40 demonstrations per task, 20 test episodes per task, a single-view setup for the main benchmark, a multi-view setup for long-horizon tasks, training for 500 epochs, evaluation every 50 epochs, and averaging over the five highest success-rate checkpoints. The baselines are Diffusion Policy (DP), 3D Diffusion Policy (DP3), and Language-guided object-centric Diffusion Policy (Lang-o3dp) (Huang et al., 6 Jul 2025).

The main simulation result on the 7-task RLBench benchmark reports average success rates of 0.49 for Diffusion Policy, 0.71 for TDP, 0.69 for VLM-TDP, 0.50 for 3D Diffusion Policy, and 0.69 for Lang-o3dp. The long-horizon stacking study evaluates 1-block, 2-block, and 4-block settings, with repeated Pick and Place sub-tasks. For 1 block, the combined success rates are 0.26 for DP, 0.48 for TDP, and 0.52 for VLM-TDP. For 2 blocks, the combined rates are 0.02, 0.17, and 0.15, respectively. For 4 blocks, they are 0.00, 0.05, and 0.04. The paper interprets this as evidence that trajectory conditioning reduces the burden of the difficult Pick phase and that sub-task decomposition makes each stage easier to solve, even though performance still declines as task complexity increases.

In the real-world study, the tasks are Pick One Orange, Pick Two Bananas, and Pick Three Bananas. Object positions are randomized, banana orientations are also randomized, and the dataset contains 40 successful teleoperated demonstrations per task collected with a 6-DOF SpaceMouse. Evaluation uses 5 randomized setups, 4 trials per setup, and 20 evaluations per task. The reported success rates are 0.70 versus 0.85 for DP and TDP on One Orange, 0.70 versus 0.95 on Two Bananas, and 0.20 versus 0.70 on Three Bananas (Huang et al., 6 Jul 2025).

Setting Systems Reported result
RLBench, 7-task average DP / TDP / VLM-TDP / DP3 / Lang-o3dp 0.49 / 0.71 / 0.69 / 0.50 / 0.69
Stack 1 Block, combined DP / TDP / VLM-TDP 0.26 / 0.48 / 0.52
Stack 2 Blocks, combined DP / TDP / VLM-TDP 0.02 / 0.17 / 0.15
Stack 4 Blocks, combined DP / TDP / VLM-TDP 0.00 / 0.05 / 0.04
Real world, One Orange DP / TDP 0.70 / 0.85
Real world, Two Bananas DP / TDP 0.70 / 0.95
Real world, Three Bananas DP / TDP 0.20 / 0.70

These results show an asymmetry between TDP and VLM-TDP. On the main 7-task RLBench average, TDP slightly exceeds VLM-TDP, whereas on Stack 1 Block the VLM-guided system is higher. This pattern is consistent with the paper’s distinction between ground-truth trajectory conditioning during TDP analysis and VLM-generated trajectory conditioning during deployment: VLM guidance is effective, but its quality is bounded by the quality of the generated spatial plan.

5. Robustness, Mechanism, and Limitations

The robustness analysis addresses both noisy images and altered environments. In the noisy-image test, Gaussian noise is added to image inputs without additional training, using Stack 1 Block as the evaluation task. The paper reports that standard Diffusion Policy degrades sharply under noise, especially in Pick sub-tasks where object localization is crucial, whereas TDP degrades more slowly because the trajectory provides a strong geometric cue. At sufficiently high noise, the baseline can fall to 0% success in Pick, while TDP retains meaningful performance. Specifically, TDP keeps about 17.0% of its original performance in Pick at σ=0.64\sigma=0.64, and about 10.4% of its original complete-task performance (Huang et al., 6 Jul 2025).

The Colosseum experiments test background texture changes, object texture changes, smaller objects, and larger objects. Across all variations, TDP shows a smaller performance drop than Diffusion Policy. The paper further notes that increasing object size can help because the required execution precision is lower and the trajectory cue becomes more effective. This is presented as evidence that trajectory conditioning reduces the degree to which the policy must recover the entire plan from raw RGB alone.

The paper’s explanation for these gains has four components. First, sub-task decomposition reduces temporal complexity. Second, trajectory conditioning reduces visual dependence by supplying an explicit spatial prior. Third, voxel trajectories provide planar position and height information, rather than only a 2D sketch. Fourth, the trajectory offers local guidance under perturbation: even if the global image is noisy, the path can still guide the end-effector toward the correct region. A plausible implication is that the principal benefit of VLM-TDP lies not only in semantic decomposition but also in the introduction of a structured spatial latent that regularizes policy behavior under distribution shift.

Several limitations are stated or implied. Trajectory ambiguity increases when multiple objects are close together. Grid resolution introduces a trade-off: higher resolution makes trajectory generation harder for the VLM, while lower resolution blurs important distinctions. The method depends on the VLM’s ability to produce reasonably accurate spatial plans. Long-horizon tasks remain challenging, and performance still drops as the number of sequential choices increases. These constraints indicate that the voxel representation and the quality of VLM prompting are central bottlenecks rather than incidental implementation details.

6. Relation to Adjacent VLM-Guided Diffusion Frameworks

VLM-TDP belongs to a broader 2025 pattern in which VLMs provide high-level semantic or strategic structure while diffusion models handle generative control. In autonomous driving, "DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving" combines a VLM command-guidance module, a sparse–dense perception stack, and a truncated diffusion planner to address computationally heavy BEV construction, multi-modal driving behavior, and sub-optimal decision-making in reactive closed-loop traffic. There, the VLM produces high-level driving commands that condition trajectory generation rather than directly generating steering or throttle (Jiang et al., 26 May 2025).

A second adjacent example appears in "VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion," where the VLM operates at a strategic layer, infers a safety-critical scenario generation objective, selects an adversarial background vehicle, and configures differentiable guidance templates for adaptive guided diffusion in closed-loop simulation. That framework is explicitly organized into strategic, tactical, and operational layers, with the VLM acting as strategist rather than a direct low-level controller (Wu et al., 2 Dec 2025).

Taken together, these works suggest a shared architectural motif: VLMs are increasingly used to supply semantically rich structure—commands, sub-task decompositions, risk hypotheses, or guidance functions—while diffusion models remain responsible for generating temporally extended control outputs. VLM-TDP is distinctive within this pattern because its conditioning object is a voxel-based end-effector trajectory for each manipulation sub-task. In that sense, it instantiates a specific form of VLM-guided trajectory-conditioned diffusion policy in which the interface between semantic reasoning and action generation is a discrete 3D spatial plan rather than a textual command or a scalar objective.

Within robotic manipulation, the paper’s broader claim is therefore narrower than a general theory of VLM-robot control and more precise than a generic multimodal policy. It proposes that long-horizon manipulation becomes more tractable when the VLM supplies sub-task structure and a coarse voxel trajectory, and when the diffusion policy is trained to denoise actions conditioned on that trajectory. The reported results support that formulation, while the stated limitations show that the effectiveness of the approach depends materially on the fidelity of sub-task decomposition and voxel-level trajectory generation (Huang et al., 6 Jul 2025).

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to VLM-guided Trajectory-conditioned Diffusion Policy (VLM-TDP).