WorldDP: Hierarchical Robotic Manipulation

• WorldDP is a hierarchical world model that leverages object-centric representations to decouple planning and execution in multi-stage robotic tasks.
• It utilizes a high-level world model for subgoal optimization and a low-level diffusion policy for robust, short-horizon action planning.
• The framework demonstrates superior performance through ablation studies and benchmark comparisons on complex, sequential multi-object manipulation tasks.

WorldDP is a hierarchical world model framework specifically designed for sequential, multi-stage robotic manipulation. It integrates object-centric representations, a high-level world model for subgoal optimization, and a low-level Diffusion Policy for robust action execution. WorldDP addresses the limitations of prior visual world models—most notably their inability to reliably handle complex, multi-stage planning—by decoupling environment entities and orchestrating planning and control across both high- and low-level hierarchies to sequentially solve manipulation benchmarks (Goswami et al., 7 Jun 2026).

1. Hierarchical Model Structure

The WorldDP framework is explicitly hierarchical, consisting of two principal components: a high-level world model that proposes subgoals in an object-centric latent space, and a low-level generative policy that efficiently executes action sequences to reach those subgoals.

1.1 High-Level World Model as Transition Function

• State Representation: The system encodes observations as an object-centric latent vector $s_t \in \mathbb{R}^{N \times d}$, where $N$ is the slot count (robot, background, each object) and $d$ is slot dimensionality.
• Action Representation: At time $t$, the robot action is $a_t \in \mathbb{R}^5$ (Δx, Δy, Δz, Δyaw, Δgripper). Actions are batched in chunks of length $K$ and encoded as a latent $\mathcal{a}_t \in \mathbb{R}^{32}$ via a small Transformer.
• Learned Transition: A Conditional Diffusion Transformer $f_\theta$ parameterizes the world model,

$\hat{s}_{t+1} = f_\theta(s_t, \mathcal{a}_t)$

The model is trained via an MSE loss,

$$\mathcal{L}_{mse} = \| s_{t+1} - \hat{s}_{t+1} \|_2^2$$

For planning, $N$0 is unrolled autoregressively for $N$1 steps.

1.2 Low-Level Diffusion Policy

Adapted from Diffusion Policy (Chi et al. '25), a Transformer-based conditional denoising diffusion model generates short-horizon trajectories.

• The model conditions on the current embedding $N$2 and a subgoal $N$3; it generates actions $N$4 to move toward $N$5.
• Denoising loss:

$N$6

where the context includes $N$7, $N$8, and the end-effector pose/velocity.

• Horizon $N$9 steps is set as a trade-off between reliable execution and subgoal distance.

The hierarchical coupling enables long-horizon, physically grounded planning at the high level and efficient, robust execution at the low level.

2. Object-Centric Latent Representation

Central to WorldDP is its object-centric encoding, which decouples entities for targeted sequential planning.

2.1 Slot-Attention Encoder

• Input: Each RGB observation $d$0 passes through a frozen DINOv2 backbone, yielding patch features $d$1.
• Slot Extraction: Learnable slots $d$2 are iteratively refined via Cross-Attention plus GRU for $d$3 steps to yield final slots $d$4.

2.2 Reconstruction and Segmentation Supervision

• Slot Decoder: Each slot predicts its reconstruction $d$5 and mask logits $d$6, softmaxed over slots. Full patch features are aggregated:

$d$7

• Losses:
  • Reconstruction: $d$8
  • Segmentation: Given privileged masks $d$9 (from SAM2), Tversky loss [Salehi et al. ‘17] is applied,

  $t$0

  with $t$1, $t$2, $t$3. - Combined: $t$4, where $t$5.

This object-slot representation is foundational, enabling fine-grained, entity-specific planning.

3. Subgoal Optimization and Execution at Runtime

3.1 Particle-Filter-Based Planning

For each control iteration:

• Initialization: From the current robotic state, $t$6 action sequence means $t$7 are sampled.
• Particle Generation: Each mean is perturbed with Gaussian noise, producing $t$8 particles.
• Trajectory Rollout: Each candidate is rolled out via $t$9 for $a_t \in \mathbb{R}^5$0 steps, yielding predicted slot sequences $a_t \in \mathbb{R}^5$1.
• Cost Evaluation: The planning cost combines object-slot distance and contact probability,

$a_t \in \mathbb{R}^5$2

where $a_t \in \mathbb{R}^5$3 and $a_t \in \mathbb{R}^5$4 are goal and predicted object slots, $a_t \in \mathbb{R}^5$5 is predicted contact trajectory, $a_t \in \mathbb{R}^5$6 is the one-hot object contact target, and $a_t \in \mathbb{R}^5$7 balances terms.

• Means Update: After each of $a_t \in \mathbb{R}^5$8 planning iterations, the top $a_t \in \mathbb{R}^5$9 particles become new means. The best sequence is retained.

3.2 Subgoal Extraction and Handoff

• The highest-scoring particle’s sequence $K$0 defines subgoals $K$1.
• For each subgoal $K$2, the low-level Diffusion Policy is invoked to generate an action plan toward $K$3 (horizon $K$4), executed until completion or timeout.
• For multi-object tasks, this planning-execution pipeline is sequentially repeated per object.

4. Training Regimen and Supervision

WorldDP’s components are trained in distinct phases to ensure robust and disentangled representations and dynamics modeling.

4.1 Training Workflow

• Object-Centric Encoder: Trained solely on “play” data via $K$5 for 10 epochs, supporting both reconstruction and segmentation.
• World Model (CDiT): With frozen encoder, $K$6 is trained for 10 epochs using MSE $K$7 on variable skipframes.
• Contact Predictor: A 2-layer MLP is trained (10 epochs) on the same data using cross-entropy loss for contact classification.
• Diffusion Policy: Trained for 3 epochs with $K$8, matching the denoising diffusion objective.

4.2 Loss Function Summary

Component	Loss Function
Object-Centric Encoder	$K$9
World Model	$\mathcal{a}_t \in \mathbb{R}^{32}$0
Contact Predictor	$\mathcal{a}_t \in \mathbb{R}^{32}$1
Diffusion Policy	$\mathcal{a}_t \in \mathbb{R}^{32}$2 as above

This staged, modular training is critical for stability and performance in sequential manipulation.

5. Empirical Results and Ablation Analysis

5.1 Benchmarks

WorldDP is evaluated across several standard robotic manipulation tasks:

• Cube-Single: 1-cube rearrangement (success if ≤12 cm error).
• Cube-Triple: 3-cube sequential arrangement, success at $\mathcal{a}_t \in \mathbb{R}^{32}$31, $\mathcal{a}_t \in \mathbb{R}^{32}$42, and all 3 cubes.
• Scene-Single-Direct: Pressing buttons, opening drawers/windows (≤4/3 cm tolerances).
• Scene-Single-Composite: Unlocking then manipulating via multi-stage tasks.

5.2 Quantitative Results

Task	DinoWM	LeWM	HECRL*	DP40	DP100	WorldDP
Cube-Single	0	0	98	0	98	72
Scene-Single-Direct (avg)	18	16	30	44	26	74
Cube-Triple (3 cubes)	0	0	12	0	4	30
Scene-Comp. (full task)	0	0	18	0	14	20

WorldDP surpasses all baselines on multi-stage and compositional tasks, demonstrating superior sequential planning and execution.

5.3 Ablation Studies

Key findings include:

• Without Diffusion Policy (DP): Direct action optimization produces large drops on multi-stage tasks.
• Without Object-Centric Encoder (OCE): Simple patch features reduce Scene-Single-Direct performance by $\mathcal{a}_t \in \mathbb{R}^{32}$530 points on average.
• Slot-Cost vs. Image-Embeddings: Cube-Triple success for all cubes increases from 6% to 30% with slot-based cost.
• Particle Filter (PF) vs. CEM: PF increases 3-cube success from 6% (CEM) to 30%.
• Contact Predictor Ablation: Omitting contact supervision halves success rates for grasp-requiring tasks.

This suggests that the integration of object-decoupled representations, physically grounded long-horizon planning, and robust execution policies is necessary to achieve multi-stage robotic manipulation reliability (Goswami et al., 7 Jun 2026).

6. Context, Significance, and Applicability

WorldDP represents a significant step in enabling model-based agents to autonomously decompose and solve sequential object manipulation tasks of substantial complexity. By leveraging object-centric latent states and hierarchical goal-conditioned planning, it overcomes the limitations of prior approaches, which typically fail at long-horizon, multi-object benchmarks or in tasks demanding intermediate goal sequencing. The architectural decoupling of planning and execution—exploiting a world model for generative subgoal design and a diffusion policy for robust short-horizon control—demonstrates a new compositional paradigm in robotics, validated by ablations and benchmark comparisons.

The framework’s modular design and loss structure are broadly extensible to similar domains where compositional manipulation, entity reasoning, and robust sequential planning are critical.