Structured 4D Latent Predictive Model for Robot Planning

Abstract: Video predictive models are emerging as a powerful paradigm in robotics, offering a promising path toward task generalization, long-horizon planning, and flexible decision-making. However, prevailing approaches often operate on 2D video sequences, inherently lacking the 3D geometric understanding necessary for precise spatial reasoning and physical consistency. We introduce a Structured 4D Latent Predictive Model, which predicts the evolution of a scene's 3D structure in a structured latent space conditioned on observations and textual instructions. Our representation encodes the scene holistically and can be decoded into diverse 3D formats, enabling a more complete and 3D consistent scene understanding. This structured 4D latent predictive model serves as a planner, generating future scenes that are translated into executable actions by a goal-conditioned inverse dynamics module. Experiments demonstrate that our model generates futures with strong visual quality, substantially better 3D consistency and multi-view coherence compared to state-of-the-art video-based planners. Consequently, our full planning pipeline achieves superior performance on complex manipulation tasks, exhibits robust generalization to novel visual conditions, and proves effective on real-world robotic platforms. Our website is available at https://structured-4d-model.github.io/.

• The paper introduces a structured 4D latent predictive model that leverages sparse voxel grids and multi-view fusion to forecast future 3D scene dynamics with language-conditioned planning.
• The model employs a two-stage pipeline—Single Dynamics and Latent Generator—with flow-matching objectives, ensuring robust semantic grounding and detailed geometric refinement.
• The framework demonstrates superior multi-view consistency, robustness to visual and viewpoint shifts, and effective real-world robot manipulation, surpassing state-of-the-art methods.

Structured 4D Latent Predictive Model for Robot Planning

Introduction and Motivation

Contemporary robotic planning systems leveraging video prediction and generative policies predominantly operate in 2D pixel space, inherently lacking consistent, explicit 3D geometric reasoning. This shortcoming creates significant challenges for generalization, spatial reasoning, and the execution of physically plausible manipulation—especially under distribution shifts in environmental appearance or sensor viewpoint. The paper "Structured 4D Latent Predictive Model for Robot Planning" (2607.01166) introduces a predictive framework that learns scene dynamics in a structured 3D latent space, integrating multi-view perception and language-conditioned task planning, thus addressing core limitations of prior approaches.

Figure 1: Our structured 4D latent predictive model integrates multi-view images and text instructions to forecast future 3D dynamics for robot planning and execution, demonstrated in simulation (top) and on a real robot (bottom).

Model Architecture and Structured Latent Representation

The proposed framework departs from traditional video-centric approaches by building a latent space based on sparse, structured voxel grids. Each active voxel stores a feature vector encoding local geometry and appearance, resulting in efficient latent compression while maintaining explicit 3D spatial semantic priors. Multi-view RGB-D inputs are fused using a calibrated projection procedure followed by a DINOv2-based patch embedding and a specialized encoder, producing the structured latent $z_t$. This representation is sufficiently expressive for detailed prediction yet compact for scalable dynamics modeling, setting a foundation for multiview and temporally coherent 3D generation.

Figure 2: The structured 4D latent predictive model reconstructs a 3D latent from multi-view images, predicts future latents conditioned on current state and text instructions, decodes these into explicit 3D formats, and translates them into actions with a goal-conditioned inverse dynamics model.

The decoding process recovers explicit 3D scene formats (e.g., point clouds, 3D Gaussian splats), supporting rendering from arbitrary viewpoints and enabling downstream geometric reasoning.

Latent Predictive Modeling Dynamics

Temporal evolution of the 3D scene is modeled by autoregressively predicting future structured latents given the current state and a language instruction. To address the compositional complexity of 3D prediction, a two-stage pipeline is deployed:

• Single Dynamics Model (SD): Predicts coarse structural scene transformations using a conditional flow-matching generative objective implemented via a transformer backbone. Text and structured latent conditioning are combined with cross-attention to enhance robustness and semantic grounding.
• Latent Generator (LG): Fills in voxel-level features conditioned on the structure proposed by SD, refining the geometric and visual detail necessary for semantically rich planning and rendering.

Both modules are trained with independent flow-matching objectives and support classifier-free guidance and conditional augmentation, improving performance with partial or noisy observations.

Figure 3: 4D generation visualizations show the unrolled future scene dynamics via the model’s latent evolution—both in multi-view renderings and point cloud space.

Planning and Policy Integration

Predicted latent rollouts serve as subgoal specifications for a goal-conditioned inverse dynamics policy—either differentiable (learned on point cloud encodings of latent subgoals) or learning-free (via geometric registration and classical motion planning). This modular design supports hierarchical planning: the structured predictive model produces semantically and physically plausible futures, while the downstream controller infers robot actions to accomplish the predicted geometric transitions.

Quantitative evaluation demonstrates that decoded point cloud representations present the optimal tradeoff for the inverse dynamics module, providing high planning accuracy while remaining computationally lightweight relative to both dense voxel and raw latent approaches.

Empirical Evaluation

Multi-view Consistency and Visual Fidelity

Experiments across the ManiSkill3 and RLBench benchmarks validate strong 3D consistency, visual fidelity, and temporal coherence in the model's generations:

• The model yields superior multi-view consistency statistics (Chamfer Distance, depth error, cPSNR/cSSIM/cLPIPS) when compared to leading video and hybrid 2.5D methods.
• The mask IoU between the generated and ground-truth robot masks is substantially higher, providing a robust link between perceptual consistency and effective control execution.

Robustness to Visual and Viewpoint Shifts

A critical advantage of this structured 3D approach is robust generalization to changes in lighting, sensor noise, background, and camera viewpoint—outperforming video-based and direct policy learning baselines by large margins.

Figure 4: Generalization to novel viewpoints: the model generates geometrically plausible rollouts from previously unseen camera poses, unlike 2D video-based baselines which fail to maintain object consistency and physicality.

Downstream Manipulation and Real-world Transfer

On multiple manipulation tasks, the proposed method closes or surpasses the performance gap with specialized 3D policy and diffusion-based planners, establishing new state-of-the-art for both open-loop and closed-loop robot execution across synthetic and real-world environments.

Figure 5: Real-world deployment pipeline: from multi-view RGB-D observations, the framework reconstructs the input scene, predicts the sequence of subgoal latents, registers the gripper motion, and executes via either learned or learning-free inverse dynamics—achieving higher real-world manipulation success rates.

Additional figures, including (Figure 6), reinforce the model’s performance on viewpoint generalization, highlighting consistent 3D reconstructions and behavior despite training only on global viewpoints.

Limitations and Future Directions

The framework’s reliance on calibrated multi-view perception to form initial scene latents may restrict applicability under monocular or weakly calibrated sensor regimes. Furthermore, fine-grained contact dynamics remain a challenge; errors in prediction and control accumulate markedly in high-precision or contact-rich manipulation. Extending the architecture to support self-supervised latent construction from sparse or uncalibrated sensors and integrating differentiable physics or higher-frequency dynamic priors represent natural future work.

Conclusion

This work introduces a structured 4D latent predictive modeling framework for robot planning that operates explicitly in a 3D spatial latent space. It achieves robust, generalizable, and high-fidelity long-horizon planning, yielding strong quantitative and qualitative improvements over 2D video and hybrid baselines across simulated and real-world settings. Its modular prediction-control structure, geometric grounding, and language-conditioned flexibility mark significant advancement in general-purpose robot planning architectures, with implications for robust manipulation, safety-critical deployment, and future generalist embodied agents.

See (2607.01166) for a comprehensive bibliography.