3D Dynamics Predictor

• 3D dynamics predictor is a forecasting model that estimates future three-dimensional states using observations, geometrical data, and control inputs.
• It employs diverse representations such as SDFs, point clouds, and mesh models and integrates physics-based and deep learning methods.
• It supports robotics and simulation by providing rich spatial context beyond simple pose predictions for improved control and planning.

A 3D dynamics predictor is a model or planning component that forecasts the future evolution of a three-dimensional state from observations, geometry, and often actions, forces, or control inputs. In recent work, the predicted state may be a future workspace signed-distance field for obstacle avoidance, the rest state or full trajectory of a rigid body, a 3D human mesh sequence, a deformable object’s control-particle motion, a future radiance field or triplane for a dynamic scene, or a mesh-wide deformation field for vehicle collision analysis (Finean et al., 2020, Rempe et al., 2019, Zhang et al., 2024, Kim et al., 25 Mar 2025, Qi et al., 28 Jan 2025). The term is therefore broader than pose forecasting alone: several systems are explicitly motivated by the inadequacy of image-space prediction or pose-only forecasting for planning, control, simulation, and uncertainty-aware scene reasoning (Paparusso et al., 2021, Lee et al., 25 Jun 2026).

1. Scope of the concept

The literature uses the concept across several distinct problem formulations. In robotics and motion planning, the predictor may forecast how dynamic obstacles will occupy a 3D workspace and then convert those predictions into future signed-distance fields that a planner can query online. In rigid-body mechanics, it may map object geometry together with an impulse, or geometry together with initial velocities, to either a final rest state or an entire trajectory. In human motion, it may infer current, past, and future 3D mesh motion from video, or reconstruct physically plausible motion from a moving camera by explicitly modeling forces, contacts, and control. In deformable-object manipulation, it may predict sparse particle motion conditioned on robot end-effector trajectories and then interpolate that motion back to a dense renderable representation. At the scene level, it may predict an explicit future 3D radiance field, a viewpoint-invariant object-factorized latent scene, or a probabilistic distribution over multimodal scene variables under partial observations and incomplete 3D transforms (Finean et al., 2020, Rempe et al., 2020, Kanazawa et al., 2018, Zhang et al., 2024, Tung et al., 2020, Lee et al., 25 Jun 2026).

Two recurrent distinctions organize these formulations. One is final-state prediction versus rollout prediction. “Learning Generalizable Physical Dynamics of 3D Rigid Objects” predicts final planar position and total rotation after an impulse, whereas “Predicting the Physical Dynamics of Unseen 3D Objects” predicts per-step changes in position, rotation, velocities, and stability over time (Rempe et al., 2019, Rempe et al., 2020). The other is object-centric versus scene-centric prediction. Object-centric models often infer latent physical quantities such as mass, moment of inertia, or contact geometry from a point cloud, while scene-centric models represent the evolving world through SDFs, voxel feature maps, Gaussians, triplanes, or probabilistic memory structures (Rempe et al., 2019, Finean et al., 2020, Zhang et al., 2024, Qi et al., 28 Jan 2025).

A further extension appears in work that treats “3D” as a spatiotemporal tensor rather than a physical Euclidean scene. DeepSPred models a history of spectrograms as a 3D spatiotemporal object and predicts either future spectrograms or spectrum occupancy rate, using 3D patching, 3D self-attention windows, 3D patch merging and expanding, and 3D convolutions (Pan et al., 2024). This suggests that the term is used both for physically grounded 3D dynamics and for 3D sequence modeling where time-frequency evolution is the target structure.

2. Representations of 3D state

The representation of state is the defining design choice in this literature, because the predictor’s inductive bias is usually encoded directly in the state space rather than added only through the loss.

Representation	Predicted quantity	Example papers
Composite signed-distance field / ESDF	Future obstacle field in the workspace	(Finean et al., 2020)
3D point cloud + PointNet features	Rigid-body rest state or trajectory	(Rempe et al., 2019, Rempe et al., 2020)
SMPL mesh and pose parameters	3D human mesh motion	(Kanazawa et al., 2018, Zhang et al., 2019, Li et al., 2022)
Tracked 3D Gaussians and sparse control particles	Action-conditioned deformable-object motion and rendered future video	(Zhang et al., 2024)
Object-factorized 3D scene map / ego-centric unbounded triplane	Future scene state and novel views	(Tung et al., 2020, Qi et al., 28 Jan 2025)
Probabilistic scene variables with persistent scene memory	Multimodal future scene states under partial transforms	(Lee et al., 25 Jun 2026)
Mesh nodes with rigid-motion and deformation decomposition	3D spatio-temporal vehicle collision response	(Kim et al., 25 Mar 2025)

In motion planning, “Predicted Composite Signed-Distance Fields for Real-Time Motion Planning in Dynamic Environments” defines an ESDF as “a 3D voxelgrid in which each cell contains the signed-distance to the closest obstacle surface,” and models the future workspace SDF as the superposition of a static SDF and moving-object SDFs combined by a voxelwise minimum (Finean et al., 2020). This representation is tailored to collision checking and to GPMP2-style obstacle factors rather than to appearance or reconstruction.

Rigid-body predictors instead use geometry-centric representations. Both rigid-object papers represent object shape as a 3D point cloud with 1024 surface points and use PointNet to extract global shape features tied to volume, mass, inertia, and contact geometry (Rempe et al., 2019, Rempe et al., 2020). Human dynamics papers use the SMPL parameterization, where the 3D state is expressed through shape, pose, and camera variables and the mesh is generated as $\mathcal{M}(\beta,\theta)\in\mathbb{R}^{N\times 3}$ (Kanazawa et al., 2018, Zhang et al., 2019).

More recent scene models make the representation itself dynamically renderable. Dynamic 3D Gaussian tracking represents the scene as a set of Gaussians with center $\mu$, covariance $\Sigma$, color, opacity, and orientation/scale parameters, then down-samples the dense tracked Gaussians to sparse control particles that serve as graph vertices for a GNN dynamics model (Zhang et al., 2024). 3D-OES uses a viewpoint-invariant 3D scene feature map $M\in\mathbb{R}^{w\times h\times d\times c}$ and object-factorizes it into object-centric crops, allowing the simulator to “move” object features through latent 3D space rather than re-encode every predicted frame (Tung et al., 2020). The monocular dynamic-radiance-field predictor adopts an ego-centric unbounded triplane $$\mathbf{T}=\{\mathbf{T}_{xy},\mathbf{T}_{yz},\mathbf{T}_{xz}\}$$ and uses mip-NeRF 360 contraction to represent unbounded scenes in a fixed-size structure (Qi et al., 28 Jan 2025).

Some representations are explicitly constrained by mechanics. The collision predictor composes node positions from global translation, global rotation in $SO(3)$, and node-wise deformation residuals, while the freely rotating rigid-body papers map images to a latent representation homeomorphic to $\mathbf{SO}(3)$ and use learned inertia or Hamiltonian structure to propagate orientation (Kim et al., 25 Mar 2025, Mason et al., 2023, Mason et al., 2022). P3Sim pushes this further by treating a perceptual scene as a probabilistic graphical model over local multimodal variables indexed by pointers, with persistent memory integrating geometry, motion fields, observed surface, and unobserved volume over time (Lee et al., 25 Jun 2026).

3. Predictive formulations and architectures

The predictive mechanisms range from explicit geometric updates to graph neural networks, recurrent models, temporal convolutions, transformers, and physics-informed integrators.

A non-neural but practical formulation appears in composite SDF prediction. The moving workspace is decomposed into stationary objects $\mathcal{O}_i^s$ and moving objects $\mathcal{O}_j^d$; each moving object is tracked through centroid position $\mathbf{x}_j(t)$ and velocity $\mu$0; a constant-velocity model propagates the centroid forward; and the predicted SDF is obtained by placing precomputed object-centric SDFs at future poses and taking a pointwise minimum with the static SDF. The paper states that the composite SDF is “guaranteed to be accurate up to the distance $\mu$1,” where $\mu$2 is the safety margin used in collision avoidance (Finean et al., 2020). This is a 3D dynamics predictor in the sense that it forecasts an evolving 3D environment representation rather than only obstacle positions.

Rigid-body predictors commonly split geometry processing from dynamics processing. “Learning Generalizable Physical Dynamics of 3D Rigid Objects” uses an impulse branch, a PointNet shape branch, and a 6-layer MLP that jointly predicts final position and total rotation; the architecture is motivated by the relations $\mu$3 and $\mu$4 and includes pairwise products of impulse and position components to help represent the angular impulse term (Rempe et al., 2019). “Predicting the Physical Dynamics of Unseen 3D Objects” instead uses PointNet once per object and then reuses the resulting feature across an LSTM rollout with hidden state size 1024, predicting changes in 13 state parameters at each time step (Rempe et al., 2020). The deep residual rigid-body model uses a ResNet-style network with residual blocks of two fully connected layers with 256 neurons and predicts final position, quaternion orientation, linear velocity, and angular velocity from the initial state, forces, and torques (Oketunji, 2024).

Graph structure dominates multi-body and deformable-object settings. Dynamic 3D Gaussian tracking learns a graph neural dynamics model over sparse control particles and a robot end-effector node, with node features formed from recent displacement vectors, a node-type indicator, and distance to the table, and with edge features given by relative 3D position and relation type (Zhang et al., 2024). 3D-IntPhys builds a graph over points extracted from a conditional NeRF-style visual frontend and applies edge encoders, node encoders, and multi-step propagation to predict the next point states, training without dense point correspondences (Xue et al., 2023). 3D-OES similarly applies a graph interaction network over object-centric 3D feature maps and predicts per-object translation and rotation increments, then performs cumulative motion updates and warps the original object features through 3D spatial transformers (Tung et al., 2020).

Human dynamics papers favor temporal feature encoding. HMMR uses a 1D fully convolutional temporal encoder with 3 residual blocks and receptive field 13 frames to produce a “movie strip” representation, from which it regresses current SMPL parameters and predicts pose changes at $\mu$5 (Kanazawa et al., 2018). Predicting Human Dynamics makes the temporal encoder causal and adds an autoregressive latent predictor

$\mu$6

so that the model can forecast future 3D meshes from past video input alone (Zhang et al., 2019). D&D keeps an initial kinematics backbone but refines the motion through inertial force control, probabilistic contact torque, and an attentive PD controller in the non-inertial local camera frame, with forward dynamics providing pose acceleration (Li et al., 2022).

Physics-informed dynamics predictors make the equations themselves part of the model class. The freely rotating rigid-body papers encode images into $\mu$7 latents, estimate angular velocity from latent pairs, compute angular momentum $\mu$8, and integrate rigid-body Hamiltonian dynamics using Euler’s equation

$\mu$9

They then decode the predicted latent rotations back into images (Mason et al., 2023, Mason et al., 2022). The vehicle-collision model explicitly decouples rigid-body motion and deformation:

$\Sigma$0

using a quaternion-based Rigid Net for global rotation and translation and a coordinate-based Deformation Net for node-wise residual motion (Kim et al., 25 Mar 2025).

Transformer-based world models appear in scene prediction. The monocular future-radiance-field predictor uses a 4D-aware transformer with temporal-aware view-attention, axis-attention, and plane-attention to update a future triplane from past monocular video (Qi et al., 28 Jan 2025). P3Sim converts multimodal scene inference into random-access sequence modeling with a causal transformer trained to predict values at queried pointers conditioned on arbitrary subsets of observed pointers and values, while a geometrizer provides sparse target depth and flow and persistent scene memory maintains global consistency (Lee et al., 25 Jun 2026).

4. Supervision, objectives, and empirical behavior

The supervision regimes span synthetic physics engines, multi-view robot interaction videos, RGB-D observations, motion-capture trajectories, and self-supervised rendering. Bullet integrated with Unity is used to generate large rigid-body datasets in both rigid-object papers, with 793 distinct shapes and either 98,826 simulations for final-state prediction or 65,715 simulations for rollout prediction (Rempe et al., 2019, Rempe et al., 2020). Dynamic 3D Gaussian tracking is trained from real multi-view RGB-D videos with 4 cameras at 15 FPS over 8 deformable object instances, while 3D-OES is trained in Bullet on object pushing and object falling and then transferred to real robotic pushing with an RGB-D camera (Zhang et al., 2024, Tung et al., 2020). The dynamic radiance-field predictor is trained self-supervised on EPIC Fields, Plenoptic Video, and the nuScenes train split, and evaluated by rendering predicted future 3D fields into novel views (Qi et al., 28 Jan 2025).

Loss design usually reflects the target representation. Relative-error losses are preferred when raw $\Sigma$1 would be dominated by large motions: the rigid-object final-state predictor uses $\Sigma$2, and the rollout rigid-body model combines relative errors for position and rotation with $\Sigma$3 losses for velocities and binary cross-entropy for stability (Rempe et al., 2019, Rempe et al., 2020). Human dynamics papers combine 2D reprojection, optional 3D supervision, adversarial pose priors, and shape priors, and enforce constant shape over time (Kanazawa et al., 2018, Zhang et al., 2019). 3D-IntPhys uses Chamfer distance plus a spacing loss because Chamfer loss alone can collapse points in dense granular scenes (Xue et al., 2023). Dynamic 3D Gaussian tracking uses multi-step prediction MSE together with optional edge-length and rigidity regularization (Zhang et al., 2024). P3Sim uses next-token cross-entropy for the world model and rendering losses inherited from 3DGS/Dyn3DGS with $\Sigma$4, $\Sigma$5 (Lee et al., 25 Jun 2026).

Reported empirical behavior varies with domain but is often organized around real-time feasibility, generalization, or physical plausibility. In motion planning, composite SDF prediction produces “an 81--97% reduction in time for subsequent predictions,” reaches 540 Hz versus 36 Hz for full benchmark SDF generation in a $\Sigma$6 workspace, and generates predicted SDFs at about 400 Hz in the Panda-arm demonstration (Finean et al., 2020). In rigid-object final-state prediction, combined-category training yields about 6.2% position error and about 11.8% rotation error, while auxiliary probes show 3.5% relative error for mass prediction and 8.7% for moment of inertia (Rempe et al., 2019). In rollout rigid-body prediction, mean errors remain under about 1.2 cm and 4 degrees in the hardest leave-one-category-out setting for non-toppling sequences, and the toppling classifier achieves an average F-score of 0.64 on Boxes, Cylinders, and Bottles (Rempe et al., 2020).

Human-motion predictors report improvements in smoothness and in-the-wild generalization. HMMR reduces acceleration error by 56% relative to a comparable single-view approach and reaches MPJPE 116.5 and PA-MPJPE 72.6 with InstaVariety on 3DPW (Kanazawa et al., 2018). D&D reports, on 3DPW, MPJPE 73.7, PA-MPJPE 42.7, PVE 88.6, and ACCEL 7.0, and on Human3.6M, MPJPE 52.5, PA-MPJPE 35.5, PVE 72.9, ACCEL 6.1, FS 5.8, and GP 1.5 (Li et al., 2022).

Deformable and scene-level models are typically evaluated through geometry, rendering, or planning metrics. Dynamic 3D Gaussian tracking reports 3D MTE values of 6.90 mm for Rope, 13.14 mm for Cloth, and 12.83 mm for Toy animals, and achieves lower error and higher success rate than MPM and FleX in planning tasks (Zhang et al., 2024). The collision predictor reports 40.93% lower test loss than DeepONet and 18.60% lower test loss than a Single MLP, while training on only 10% of samples reduces training time from 208.32 minutes to 34.71 minutes (Kim et al., 25 Mar 2025). The monocular future-radiance-field predictor reaches PSNR 22.43, SSIM 0.706, and LPIPS 16.29 on NVIDIA Dynamic Scenes, and PSNR 21.05, SSIM 0.535, and LPIPS 32.42 on DAVIS (Qi et al., 28 Jan 2025).

5. Planning, control, and simulation use

A central reason to build 3D dynamics predictors is that the predicted state can be inserted into a planner or controller without collapsing back to pixels. In the composite-SDF system, predicted future SDFs are fed to GPMP2 obstacle factors, so that each time-indexed factor can receive its own predicted SDF in a time-configuration-space formulation. The framework is demonstrated on a 7-DoF WAM arm in simulation and on a Franka Emika Panda 7-DoF arm in simulation and hardware, where the Panda reaches across a gap between tables while a Toyota HSR mobile robot moves through the gap as a dynamic obstacle (Finean et al., 2020).

In deformable-object manipulation, the learned dynamics model can function as a world model for model-based planning. Dynamic 3D Gaussian tracking rolls out candidate robot actions with a GNN dynamics predictor and optimizes an MPC objective with MPPI to minimize the distance between predicted and target states; the reported planning tasks include rope straightening and toy animal relocation (Zhang et al., 2024). 3D-OES uses a viewpoint-invariant object-factorized simulator inside model predictive control for cluttered pushing, with success rates 0.86 in simulation and 0.78 in the real setup after training solely in simulation (Tung et al., 2020).

The same pattern appears outside tabletop manipulation. DriVe-forecast is explicitly motivated by control and simulation applications that need future dynamics rather than only future pose. Its encoder–decoder predictor fuses past driver–vehicle dynamics with future road geometry and is evaluated in a motion cueing case study, where the average prediction-window time is about 32 ms on a CPU compared with about 500 ms for a comparable multibody simulation (Paparusso et al., 2021). P3Sim is organized around novel view synthesis, object manipulation, rigid edits, deformable manipulation, collision reasoning, and multi-agent motion, using a learned world model, geometric conditioning, and persistent memory rather than a conventional renderer (Lee et al., 25 Jun 2026).

This planning role explains why many papers insist on explicit 3D representations. Several of them argue that image-space forecasting does not provide the object state, action trajectory, geometry, or uncertainty structure required for closed-loop manipulation, robot safety, scene editing, or control (Zhang et al., 2024, Qi et al., 28 Jan 2025, Lee et al., 25 Jun 2026).

6. Assumptions, limitations, and recurring debates

A common misconception is that a 3D dynamics predictor must be a learned neural network. The composite-SDF planner is a counterexample: it uses a constant-velocity tracker and min-composition over object-centric SDFs rather than a learned predictor, yet it is explicitly presented as a practical 3D dynamics predictor for motion planning (Finean et al., 2020). Another misconception is that “3D” implies prediction of object pose alone. The literature includes predictors of contact-aware human motion, future SDFs, point sets, radiance fields, multimodal scene variables, and mesh-level deformation fields, all of which are 3D but not reducible to pose forecasting (Li et al., 2022, Qi et al., 28 Jan 2025, Kim et al., 25 Mar 2025).

The assumptions are often strong and domain-specific. Composite SDF prediction assumes segmented moving objects with known occupied voxels, reliable centroid and velocity tracking, and in simulation uses noiseless sensor data and accurate occupancy grids; its constant-velocity model fails when objects suddenly change direction, stop unexpectedly, or follow curved trajectories, and storing many time-indexed SDFs can be memory-intensive, with a $\Sigma$7 workspace over 31 time steps requiring about 6.24 GB RAM (Finean et al., 2020). The rigid-object rollout model is restricted to objects sliding on a plane, does not model collisions, treats toppling mostly as a classification signal, and only lightly explores rich physical parameter variation (Rempe et al., 2020). The residual rigid-body model reports degradation with many interacting bodies, limited generalization to unseen geometries and materials, error growth beyond about 10 seconds, and imperfect energy conservation relative to RK4 (Oketunji, 2024).

Scene and perception-based predictors confront partial observability rather than only parametric uncertainty. P3Sim is explicitly framed around incomplete 3D transformation signals, occlusion, disocclusion, and hidden induced motion, arguing that real-world 3D simulation should be treated as probabilistic inference over multimodal scene variables rather than deterministic rendering (Lee et al., 25 Jun 2026). The monocular future-radiance-field predictor notes that temporal view-attention alone is noisy in dynamic regions because dynamic scenes break strict epipolar consistency, which motivates its axis-attention and plane-attention refinements (Qi et al., 28 Jan 2025). 3D-OES argues that 2D projective representations are brittle because object appearance changes with viewpoint, object identity becomes unstable under occlusion, and long-horizon rollout in 2D accumulates severe error (Tung et al., 2020).

The literature’s stated future directions are correspondingly pragmatic. For composite SDFs, the constant-velocity tracker is described as replaceable by more advanced trackers or predictors such as a KLT tracker or Unscented Kalman Filter, and the authors suggest parallelization and live ESDF updates from point clouds (Finean et al., 2020). The residual rigid-body predictor proposes graph neural networks, data augmentation, meta-learning, physics-informed loss terms, and hybrid deep-learning-plus-physics methods for better generalization (Oketunji, 2024). The collision predictor suggests that decoupling fundamentally distinct physical phenomena is an effective route to accuracy and sampling efficiency, while P3Sim suggests a broader synthesis in which learned inference, explicit geometry, and persistent memory are all necessary for physically grounded prediction under partial observation (Kim et al., 25 Mar 2025, Lee et al., 25 Jun 2026).