Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation

Abstract: Simulating robot-world interactions is a cornerstone of Embodied AI. Recently, a few works have shown promise in leveraging video generations to transcend the rigid visual/physical constraints of traditional simulators. However, they primarily operate in 2D space or are guided by static environmental cues, ignoring the fundamental reality that robot-world interactions are inherently 4D spatiotemporal events that require precise interactive modeling. To restore this 4D essence while ensuring the precise robot control, we introduce Kinema4D, a new action-conditioned 4D generative robotic simulator that disentangles the robot-world interaction into: i) Precise 4D representation of robot controls: we drive a URDF-based 3D robot via kinematics, producing a precise 4D robot control trajectory. ii) Generative 4D modeling of environmental reactions: we project the 4D robot trajectory into a pointmap as a spatiotemporal visual signal, controlling the generative model to synthesize complex environments' reactive dynamics into synchronized RGB/pointmap sequences. To facilitate training, we curated a large-scale dataset called Robo4D-200k, comprising 201,426 robot interaction episodes with high-quality 4D annotations. Extensive experiments demonstrate that our method effectively simulates physically-plausible, geometry-consistent, and embodiment-agnostic interactions that faithfully mirror diverse real-world dynamics. For the first time, it shows potential zero-shot transfer capability, providing a high-fidelity foundation for advancing next-generation embodied simulation.

• The paper introduces Kinema4D, combining deterministic 4D kinematic control with generative diffusion modeling for precise and scalable embodied simulation.
• It utilizes URDF-driven kinematics and DiT-based diffusion transformers to project continuous robot trajectories and generate synchronized RGB and pointmap sequences.
• Empirical results demonstrate superior geometric fidelity and visual realism, enabling accurate policy evaluation and robust generalization across diverse domains.

Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation

Motivation and Contributions

Kinema4D addresses the fundamental limitations of contemporary embodied simulators, which historically operate within either constrained physical solvers or 2D-generative paradigms. The principal innovation is the disentanglement of robot-world interaction into (i) deterministic, kinematics-driven action representation in 4D, and (ii) generative spatiotemporal modeling of environmental reactions. This enables both precise policy evaluation and scalable visual realism. The framework leverages a novel large-scale dataset, Robo4D-200k, encompassing over 201k robot interaction episodes with high-quality 4D annotations, facilitating the emergence of robust foundation models for embodied AI.

Figure 1: Kinema4D simulator generates future robot-world interactions in a unified 4D space conditioned on initial world state and action sequences.

Methodology

Precise 4D Kinematic Control

Kinema4D grounds abstract robot actions (joint angles, end-effector poses) via URDF-driven kinematics, projecting dynamic trajectories into 4D space. For non-canonical robots, high-fidelity 3D meshes are reconstructed using Grounded-SAM2 and ReconViaGen pipelines, ensuring articulation compatibility by aligning physical joints and reconstruction space. Action sequences are converted either through IK (for Cartesian control) or direct mapping (for joint control), producing temporally continuous link poses. Primary viewpoint projection utilizes calibrated camera transforms, generating pixel-aligned pointmap sequences storing $(x, y, z)$ camera-space coordinates.

Figure 2: Kinema4D architecture: kinematics-controlled trajectory projection followed by generative diffusion-based modeling.

Generative 4D Modeling

The framework employs a DiT-based diffusion transformer operating over the joint latent space of RGB images and robot pointmaps, guided by occupancy masks. This multi-modal fused input enforces pixel-level control and cross-modal geometric coherence. The generative pipeline predicts synchronized RGB and pointmap sequences, transforming the simulation task into spatiotemporal reasoning, where all environmental responses are strictly conditioned on physically grounded robot trajectories.

Robo4D-200k Dataset

Kinema4D’s training leverages Robo4D-200k which aggregates diverse real-world and synthetic demonstrations sourced from DROID, Bridge, RT-1, and LIBERO. Real-world RGB sequences are lifted into 4D space via ST-V2 reconstruction, ensuring pixel-level geometric alignment. Synthetic episodes are rendered with native depth maps for absolute precision, and failure modes are synthesized to diversify environmental responses.

Figure 3: Robo4D-200k provides comprehensive spatial coverage of robot-world interactions across multiple domains.

Experimental Results

Quantitative and Qualitative Evaluation

Kinema4D outperforms state-of-the-art generative simulators (e.g., Ctrl-World [2D], TesserAct [4D]) in both visual and geometric metrics. PSNR, SSIM, FID, FVD, and LPIPS are superior or second-best across all modalities, with significant improvements in geometric fidelity: lower Chamfer Distance and higher [email protected]. Crucially, Kinema4D generalizes to out-of-distribution domains with zero-shot transfer performance.

Figure 4: Qualitative comparison: Kinema4D exhibits higher action fidelity and environmental realism in 2D compared to Ctrl-World.

Figure 5: Kinema4D achieves rigorous 4D spatiotemporal consistency, accurately reflecting ground-truth outcomes and subtle failures versus TesserAct.

Ablation studies demonstrate robustness to noisy pointmap conditions, minimal overfitting from multi-domain training, and severely degraded accuracy when robot control is replaced by weaker representations (e.g., text or embeddings).

Policy Evaluation

Kinema4D’s 4D-aware simulation platform enables accurate policy benchmarking both in simulation and real-world OOD environments. Policy success rates are closely aligned with actual executions under standardized benchmarks. Notably, the method reliably synthesizes "near-miss" failures by correctly interpreting spatial gaps, even with ambiguous 2D textures.

Figure 6: Real-world zero-shot evaluation: Kinema4D closely mirrors GT outcomes, handling OOD environments and noisy sensor inputs robustly.

Figure 7: Simulation platform quantitative results: physically plausible rollouts and nuanced failure modes are synthesized with geometric rigor.

Discussion and Implications

The integration of kinematics-driven action representation with generative 4D modeling resolves the trilemma of dynamics, precision, and spatiotemporal awareness, previously unaddressed in earlier embodied simulators. Kinema4D’s methodology facilitates scalable, high-fidelity simulation suitable for policy evaluation, visual planning, and downstream RL applications. Practically, the decoupling of robot intent and environmental reaction enhances cross-domain transfer and robustness to embodiment variation. Theoretically, it opens the path for foundational research in generative world modeling, where learned dynamics are constrained by precise geometric control.

Figure 8: Extensive 4D qualitative results showcase generalization over diverse domains, robot actions, and manipulated objects.

Limitations and Future Directions

While Kinema4D achieves strong visual and geometric fidelity, its reliance on statistical synthesis rather than explicit physical constraints occasionally produces violations of physical laws (e.g., penetration, non-conservation of mass). Future directions include integrating differentiable physics solvers and enforcing conservation laws within the generative process, leveraging hybrid models that combine learned world dynamics with analytical constraints.

Conclusion

Kinema4D establishes a novel paradigm in embodied simulation, combining kinematic precision with generative modeling in unified 4D space. Empirical results highlight substantial gains in fidelity, generalization, and robustness, providing a strong foundation for scalable embodied AI research and practical deployment. Integrating explicit physical constraints remains an open avenue for further improving the reliability and applicability of generative world models in robotics.

(2603.16669)