NeuROK: Generative 4D Neural Object Kinematics

Abstract: Data-driven approaches have revolutionized 3D vision, enabling transformers to effectively reconstruct and generate static 3D objects. However, generating simulative 4D dynamics -- realistic temporal deformations of static objects under various physical conditions -- remains challenging and often ad hoc, despite its importance in building comprehensive 3D world models. Most existing methods assume a predefined physical model and use system identification to estimate parameters, restricting these methods to specific categories and small-scale datasets. We propose that these restrictions can be overcome by learning a data-driven kinematic state parameterization for object-centric physical systems. Specifically, we learn both a latent space representing all possible states of the object and a decoder that maps any sampled latent to a plausibly deformed shape of the object. We refer to this parameterization as Neural Object Kinematics (NeuROK), and learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset. This formulation and the learned model significantly simplify the generation of simulative dynamics since we only need to consider the dynamics within a low-dimensional latent space from the Lagrangian mechanics' perspective in classical physics. We demonstrate the effectiveness and generality of this neural simulation framework across diverse dynamic object types, showing clear advantages over prior works. Project page: https://chen-geng.com/neurok

• The paper introduces a universal framework by learning a compact latent space for synthesizing 4D deformations without explicit physical labels.
• It employs a transformer-based conditional VAE integrated with Lagrangian mechanics to ensure physically plausible and temporally coherent motion.
• Experimental evaluations demonstrate improved Chamfer distances, IoU scores, and robust generalization across diverse and unseen object categories.

NeuROK: A Universal Data-Driven Framework for Generative 4D Neural Object Kinematics

Motivation and Problem Formulation

Precise generative modeling of temporally consistent 4D dynamics (sequences of 3D object deformations over time) is essential for many applications in vision, graphics, and robotics, yet existing approaches are typically limited by category-specific models, strong inductive biases, or the need for explicit physical annotations. Conventional paradigms in physics-based animation and simulation demand hand-crafted structural priors and system identification of parameters tailored to particular materials or articulated types, severely restricting scalability and out-of-distribution generalization.

In contrast, NeuROK ("Generative 4D Neural Object Kinematics") (2605.30347) proposes a framework in which simulative 4D dynamics are generated for arbitrary static 3D objects under diverse physical conditions, entirely without physical labels or category-specific constraints. The central theoretical innovation is the automatic discovery of a low-dimensional, learned kinematic state parameterization: the NeuROK manifold—tightly linking Lagrangian mechanics with classical generative modeling to construct a universal latent coordinate system for object deformations.

Learned Kinematic State Parameterization

The core insight formalized in NeuROK is that, for practical object-centric physical systems, only a low-intrinsic-dimensional subset of $R^{3n}$ corresponds to physically plausible object configurations, with $k_\text{int} \ll 3n$. Existing works often adopt geometry-derived parameterizations (e.g., dense point/vertex sets or mesh displacements), which are highly over-parameterized, yielding under-constrained and category-specific models. NeuROK instead introduces a data-driven parameterization by learning a latent manifold $\mathcal{Z}$ coupled with a neural decoder $\mathcal{F}$, such that any latent vector $z \in \mathcal{Z}$ deterministically generates a plausible object configuration.

Figure 1: Schematic comparison of symbolic, geometry-derived, and data-driven parameterizations, with NeuROK identifying a compact, learnable latent space for general dynamic object modeling.

This representation enables the elimination of explicit inter-particle constraints by ensuring that any traversed trajectory in latent space decodes to physically plausible object shapes. Thus, the entire dynamics generation task reduces to modeling trajectories in this compact latent state space, which is then interpreted through the lens of Lagrangian mechanics.

Model Architecture and Generative Training

NeuROK employs a modular transformer-based conditional variational autoencoder to realize this parameterization:

• The prior encoder maps a static 3D mesh to parameters of the kinematic latent prior.
• The variational encoder takes pairs of mesh and deformation field, estimating the posterior over latents.
• The decoder reconstructs deformations from the latent, enabling direct synthesis of plausible deformed geometries.

  Figure 2: Pipeline overview—NeuROK encodes a 3D mesh into a latent manifold and generates deformations by simulating trajectories in latent space conditioned on physical actions.

Generative supervision is performed by minimizing a combination of reconstruction error and Kullback–Leibler divergence, with an additional dimension reduction step (the Active Subspace Method) to distill the high-density regression manifold down to intrinsic degrees of freedom.

Figure 3: Training diagram illustrating random sampling of meshes and their associated deformations, with joint optimization of prior, encoder, and decoder module targets in the conditional VAE.

Latent Dynamics via Lagrangian Mechanics

Leveraging the learned NeuROK manifold, object dynamics are generated by simulating time-indexed latent vectors $\{z_t\}$ under the Euler–Lagrange equations:

$$\frac{\partial L}{\partial \dot{z}} - \frac{\partial L}{\partial z} = 0$$

with the Lagrangian $L(z, \dot z) = T(z, \dot z) - V(z)$, using neural estimates of kinetic and potential energy, and the decoder Jacobian $J_z$ for mapping to observable coordinates. Initial conditions and external actions are incorporated through constrained optimization over the starting latent and its time derivative. This approach synthesizes temporally coherent 4D mesh sequences purely from an initial mesh and user-specified actions or physical quantities.

Experimental Evaluation

The NeuROK framework is extensively benchmarked on curated and simulated large-scale 4D mesh datasets, compared against state-of-the-art systems for object kinematics and physically-inspired 4D generation across diverse object types.

Learning Compact Object Kinematics

NeuROK demonstrates consistent improvements in matching ground-truth deformations with state reconstructions, as measured by Chamfer distances and IoU metrics, across both articulated and deformable object categories. The method outperforms canonical neural and analytic kinematic representations due to its capacity for generalizable, data-driven structure discovery.

Figure 4: Qualitative comparison of kinematic space learning between NeuROK and major baselines, showing enhanced shape reconstruction fidelity and latent smoothness.

Physically Consistent and Generalizable 4D Dynamics

NeuROK-generated sequences display high physical plausibility, action alignment, and visual realism—validated by both quantitative metrics (VBench/WorldScore) and large-scale human user studies. Unlike baselines, which frequently collapse or overfit to specific categories (e.g., MPM for elastics, articulation graphs for robots), the NeuROK model generalizes to both simulated and real-captured objects—showing successful dynamics transfer even to categories unseen during training.

Figure 5: Examples of physically plausible 4D motion generation conditioned on actions, evaluated against multiple baselines.

Figure 6: Application to simulating dynamic behavior of real-world captured object geometries.

Energy conservation analysis confirms that NeuROK's latent simulation framework maintains physically consistent trajectories, with near-constant total mechanical energy over simulated motion.

Figure 7: Empirical demonstration of approximate energy conservation in generated motion, evidencing the effectiveness of the Lagrangian-based latent simulation.

Notably, NeuROK retains robust generalization: it learns reusable kinematic subspaces and control policies that immediately extrapolate to entirely new object types.

Figure 8: Results illustrating successful simulation and plausible 4D dynamics on object classes absent from training data.

Design Analysis and Ablation

Ablation studies confirm the substantial contributions of model reduction, data augmentation, and advanced deformation representations (e.g., dual quaternions) in achieving competitive reconstruction and simulation performance. Removing model reduction, for instance, degrades both reconstruction metrics and latent compactness, underscoring the importance of an efficiently parameterized kinematic state space.

Theoretical and Practical Implications

NeuROK achieves a class of object-agnostic 4D generative modeling that obviates the need for hand-designed physical models or dense physical annotation. The abstraction of physically plausible motions into a learned latent mechanism enables scalability across unstructured shape collections, rapid generalization, and direct integration into upstream applications in vision (e.g., 4D scene understanding, motion-based object reasoning), robotics (e.g., manipulation, simulation-based planning), and graphics (e.g., high-diversity animation synthesis).

Crucially, the merger of generative latent models with a Lagrangian formalism highlights the broader potential for synthesizing physically consistent, low-entropy samplings for data-driven simulation, potentially extending to inverse control, differentiable planning, or AI agents in 3D interactive worlds. Challenges remain in increasing trajectory diversity, integrating contact-rich phenomena, and scaling to non-manifold or topologically changing geometries.

Conclusion

NeuROK introduces a universal, data-driven approach to generative 4D neural object kinematics—systematically bridging classical mechanics and modern deep generative models in a latent-permissive and physically interpretable framework. The method consistently delivers superior numerical results over both analytic and learning-based baselines, provides strong generalization to real and unseen domains, and establishes a foundational paradigm for future research in scalable 4D world simulation and AI-based reasoning with complex physical systems.