4D Diffusion Policy (DP4)

• 4D Diffusion Policy (DP4) is a visual imitation learning method that enhances robot visuomotor control by integrating explicit 3D spatial and 4D spatiotemporal awareness.
• It employs a dynamic Gaussian world model and deformable MLP to reconstruct current scenes and predict future 3D states from single-view RGB-D inputs.
• Empirical results show DP4 significantly improves success rates in both simulation and real-robot tasks compared to prior 2D and 3D imitation learning baselines.

Searching arXiv for the DP4 paper and closely related baseline papers so the article can include proper arXiv citations. 4D Diffusion Policy (DP4) is a visual imitation learning method that augments diffusion-based visuomotor control with explicit 3D spatial awareness and 4D spatiotemporal awareness, where “4D” denotes 3D space plus time. It is designed for settings in which a robot observes a scene through single-view RGB-D input and must generate current and short-horizon actions that depend on geometry, contact, alignment, and predicted scene evolution. Rather than relying only on behavior cloning of historical trajectories, DP4 uses a dynamic Gaussian world model to supervise current-scene reconstruction and future-scene prediction, and conditions a diffusion policy on the resulting multi-level 3D representations (Liu et al., 9 Jul 2025).

1. Concept and problem formulation

DP4 is motivated by a limitation attributed to standard behavior cloning for visuomotor policy learning. In the reported formulation, behavior cloning trains policies from supervised observation–action pairs and can fit historical trajectories with strong 2D-image or 3D-point-cloud encoders, but it lacks explicit 3D spatial awareness and 4D spatiotemporal awareness. The former concerns the geometry needed for accurate contact, alignment, and object-centric reasoning; the latter concerns predicting how the 3D scene evolves under actions. DP4 addresses this by learning the current 3D scene structure and predicting future 3D scene states conditioned on actions, with the policy then guided by these structured representations rather than purely by cloned trajectories (Liu et al., 9 Jul 2025).

The method is defined for a time step $t$ with observation $O^{(t)} = (C^{(t)}, D^{(t)})$, where $C^{(t)}$ and $D^{(t)}$ are single-view RGB and depth. The policy predicts the current action $a^{(t)}$ and a short-horizon action sequence $a^{(t+1:\ldots)}$. The single-view constraint is central: the setting is described as realistic for eye-to-hand cameras, but 3D reconstruction is partial and noisy, and temporal prediction is difficult because of occlusions and ambiguous depth or texture.

The 3D construction step begins from standard back-projection. From a pixel $(u,v)$, depth $D(u,v)$, camera intrinsics $K$, and extrinsics $T_{wc}$, the reported back-projection is

$O^{(t)} = (C^{(t)}, D^{(t)})$0

These points are converted into local crops for a 3D Local Encoder and a global voxel for a 3D Global Encoder.

2. Dynamic Gaussian world model and 4D scene representation

The central structured component in DP4 is the Dynamic Gaussian World Model (DGWM). From a single RGB-D frame, the method builds a 3D point cloud and extracts two forms of representation: local 3D features from cropped point clouds and global 3D features from a $O^{(t)} = (C^{(t)}, D^{(t)})$1 voxel grid. A generalizable Gaussian regressor then maps these 3D features into a 3D Gaussian Splatting representation $O^{(t)} = (C^{(t)}, D^{(t)})$2 of the current scene, while rendering consistency on RGB and depth provides explicit spatial supervision (Liu et al., 9 Jul 2025).

The Gaussian scene representation is parameterized as

$O^{(t)} = (C^{(t)}, D^{(t)})$3

where $O^{(t)} = (C^{(t)}, D^{(t)})$4 is 3D position, $O^{(t)} = (C^{(t)}, D^{(t)})$5 color, $O^{(t)} = (C^{(t)}, D^{(t)})$6 rotation, $O^{(t)} = (C^{(t)}, D^{(t)})$7 scale, and $O^{(t)} = (C^{(t)}, D^{(t)})$8 opacity. The covariance $O^{(t)} = (C^{(t)}, D^{(t)})$9 is derived from rotation and scale. Rendering uses tile-based rasterization and alpha-blending:

$C^{(t)}$0

$C^{(t)}$1

with

$C^{(t)}$2

Here $C^{(t)}$3 is the $C^{(t)}$4-depth after world-to-camera projection, and $C^{(t)}$5 is the number of Gaussians projected into the pixel’s tile.

The 4D component is implemented through parameter propagation. A deformable MLP $C^{(t)}$6 predicts Gaussian parameter updates from the current scene representation and action:

$C^{(t)}$7

A differentiable renderer then projects $C^{(t)}$8 into the next camera view. In the reported interpretation, temporal data association is implicit: $C^{(t)}$9 learns to move, rotate, scale, and re-opacity the Gaussians so that the predicted future RGB-D is consistent with how actions change the scene.

3. Diffusion policy architecture and optimization

DP4 couples the DGWM to a conditional diffusion policy. The 3D Local Encoder is a three-layer MLP with max-pooling over a cropped point cloud that is FPS downsampled to 512 points, producing order-equivariant local features. The 3D Global Encoder is a 3D CNN operating on the $D^{(t)}$0 voxel grid from the full point cloud and outputs $D^{(t)}$1; trilinear sampling provides $D^{(t)}$2 at arbitrary 3D coordinates $D^{(t)}$3. The generalizable Gaussian regressor consumes 3D coordinates and sampled features to regress $D^{(t)}$4, while the deformable MLP supplies the dynamics head for $D^{(t)}$5 (Liu et al., 9 Jul 2025).

The diffusion policy is conditioned on the multi-level 3D representation $D^{(t)}$6 and robot state $D^{(t)}$7. At inference time, the denoising process uses a DDIM-style scheduler with $D^{(t)}$8 steps:

$D^{(t)}$9

The action denoising objective is

$a^{(t)}$0

The overall training objective combines action denoising with current-frame 3D supervision and future-frame 4D supervision. In reported usage, the diffusion head predicts a short horizon of $a^{(t)}$1 actions conditioned on $a^{(t)}$2 observed steps, and during inference only the last $a^{(t)}$3 are executed. The default configuration follows the adopted DP3 setting with $a^{(t)}$4, $a^{(t)}$5, and $a^{(t)}$6. Actions are continuous vectors; the method conditions on robot state $a^{(t)}$7 but does not fix a single action space across all benchmarks.

A recurring clarification in the paper is that the policy is not merely conditioned on point clouds. It is conditioned on a representation trained jointly with $a^{(t)}$8 and $a^{(t)}$9, so the reported intention is that $a^{(t+1:\ldots)}$0 explicitly encodes both spatial and temporal dependencies useful for action generation.

4. Training protocol, data, and implementation

The reported training pipeline is joint. For each training iteration, the method constructs a point cloud from RGB-D, crops a task-relevant region, voxelizes the full point cloud, encodes local and global 3D features, regresses the Gaussian scene $a^{(t+1:\ldots)}$1, renders the current view for $a^{(t+1:\ldots)}$2, predicts $a^{(t+1:\ldots)}$3 through the deformable MLP for $a^{(t+1:\ldots)}$4, and trains the diffusion model on noisy actions conditioned on $a^{(t+1:\ldots)}$5 and $a^{(t+1:\ldots)}$6. The optimization uses a warm-up in which $a^{(t+1:\ldots)}$7 is frozen for the first 500 iterations to stabilize the spatial representation and Gaussian regressor, after which all modules are trained end-to-end (Liu et al., 9 Jul 2025).

The data regime spans both simulation and real robotics. The simulation suites are Adroit with Door, Pen, and Hammer; DexArt with Laptop, Faucet, Bucket, and Toilet; and RLBench with 10 tasks and 166 variants. Adroit and DexArt demonstrations come from RL agents, specifically VRL3 for Adroit and PPO otherwise. RLBench uses 20 planner demonstrations per task. The real-robot platform is a KINOVA GEN2 arm with a RealSense D455 in an eye-to-hand configuration and single-view RGB-D sensing. The three reported tasks are Grasping Bottles, Pouring Water, and Stacking Cups.

The principal hyperparameters are also explicit. Diffusion uses a DDIM scheduler with $a^{(t+1:\ldots)}$8 training steps and 10 inference steps. Training runs for 3,000 epochs with batch size 32. The reported best loss weights are $a^{(t+1:\ldots)}$9 and $(u,v)$0, selected by hyperparameter sweep and identified as best on Adroit Pen. LayerNorm is used in the local encoder, standard MSE is used for diffusion, and no additional occupancy or Chamfer losses are reported. The experiments were run on an NVIDIA H100 80GB GPU and a 192-vCPU Intel Xeon Platinum 8468. Training is described as feasible on a single H100 GPU.

5. Empirical performance and ablation findings

Across the reported experiments, DP4 is evaluated on 17 simulation tasks with 173 variants and on 3 real-world robotic tasks. The paper states that DP4 improves the average simulation task success rate by 16.4% on Adroit, 14.0% on DexArt, and 6.45% on RLBench, and improves the average real-world robotic task success rate by 8.6% (Liu et al., 9 Jul 2025).

Benchmark	Scope	Reported DP4 outcome
Adroit	Door, Pen, Hammer	84.7% overall success; +16.4% over prior SOTA
DexArt	Laptop, Faucet, Bucket, Toilet	82.5% overall; +14.0% over DP3
RLBench	10 tasks, 166 variants	63.3% and 39.9% on two 5-task subsets
Real robot	3 tasks	+8.6% average success over baselines

For Adroit, the reported comparison is DP at 41.0%, DP3 at 68.3%, and DP4 at 84.7%, with task-level values of Hammer 100.0, Door 80.0, and Pen 75.0. For DexArt, DP4 reaches 82.5% overall and is described as much stronger than 2D baselines such as DP at 49.0%. On RLBench, DP4 is reported as state-of-the-art with averages of 63.3% and 39.9% on two five-task subsets, outperforming PerAct, GNFactor, and ManiGaussian. Task-level means and standard deviations are provided, including close jar $(u,v)$1, open drawer $(u,v)$2, sweep to dustpan $(u,v)$3, meat off grill $(u,v)$4, turn tap $(u,v)$5, slide block $(u,v)$6, put in drawer $(u,v)$7, drag stick $(u,v)$8, push buttons $(u,v)$9, and stack blocks $D(u,v)$0.

The real-robot results report DP, DP3, and DP4 respectively as 36.0, 42.0, and 48.0 on Grasping Bottles; 44.0, 62.0, and 72.0 on Stacking Cups; and 28.0, 34.0, and 44.0 on Pouring Water.

The Adroit ablation studies isolate the contributions of Gaussian supervision and dynamics. The base model without 3D or 4D supervision achieves Hammer 94.0/6.40, Door 64.0/6.44, and Pen 45.0/6.46, where each pair denotes success and inference time in seconds. Adding RGB-only Gaussian world supervision yields 96.0/6.52, 68.0/6.45, and 48.0/6.48. Adding the 3D spatial loss with RGB and depth yields 95.0/6.53, 68.0/6.46, and 47.0/6.48. Full DP4 with RGB, depth, and dynamics achieves 100.0/6.57, 80.0/6.45, and 75.0/6.50. The reported finding is that 4D dynamics supervision drives the largest improvement, notably on Pen, while inference time remains nearly unchanged because DGWM is used only for training-time supervision and is not constructed at test time.

6. Relation to prior methods, limitations, and prospective directions

DP4 is situated against both 2D and 3D imitation-learning baselines. The paper states that 2D policies such as IBC, BCRNN, and Diffusion Policy underperform because they lack explicit 3D and 4D structure. It further states that DP3 improves spatial reasoning by conditioning on point clouds but still lacks explicit 4D scene evolution. Relative to other 3D-aware methods such as PerAct, GNFactor, and ManiGaussian, DP4 is distinguished by using a dynamic 3D Gaussian Splatting world model to create training-time spatial and temporal supervision that directly informs the policy (Liu et al., 9 Jul 2025).

Several limitations and assumptions are explicit. Under the single-view constraint, reconstructions can miss fine details and occluded geometry, and the renderings are described as “coarse but sufficient.” Heavy occlusions and dynamic clutter can still challenge prediction. The method requires known camera intrinsics and extrinsics, so calibration error degrades the quality of 3D and 4D supervision. The deformable MLP assumes that future scenes are well approximated by smooth deformations of current Gaussians; nonrigid, fluid, or highly nonlocal changes may therefore be difficult. The policy is model-free and guided by learned supervision, and no special safety mechanism is introduced beyond the need for standard robot safety checks.

The computational profile also shapes interpretation. Inference latency is reported as roughly 6.4–6.6 seconds per episode on Adroit ablations, with runtime dominated by diffusion sampling and policy forward passes rather than online construction of the DGWM. Memory use is supported by the reported 80 GB GPU configuration, and Gaussian splatting is described as memory-friendly compared to volumetric grids. This suggests that the method’s principal cost lies in training the joint representation rather than in deploying a world model online.

The future directions identified in the paper are multi-view sensing to reduce ambiguity and occlusion sensitivity, uncertainty-aware planning using DGWM uncertainties such as opacity or learned variance, integration with model-based control or MPC over predicted $D(u,v)$1, and richer dynamics for nonrigid or fluid phenomena via extended Gaussian fields or hybrid representations. A plausible implication is that DP4 is best understood not as a replacement for diffusion-policy control, but as a training-time mechanism for imposing scene-structured spatial and temporal supervision on that control stack.