MotionPhysics: Text-Guided Simulation

• MotionPhysics is a text-guided simulation framework that infers physically plausible material parameters for 3D scenes by translating natural language prompts.
• It integrates a pretrained GPT-4 initializer, differentiable MLS-MPM simulation, and a learnable motion extractor distilled from video diffusion models to refine parameters.
• Evaluations across diverse scenarios demonstrate high realism and prompt adherence, significantly reducing the need for manual tuning in physics-based graphics.

Searching arXiv for the named paper and closely related motion-physics works to ground the article in current literature. arXiv search: "MotionPhysics Learnable Motion Distillation for Text-Guided Simulation" MotionPhysics denotes an end-to-end differentiable framework for text-guided physical simulation in which plausible material parameters are inferred from a natural-language prompt for a chosen 3D scene, without guidance from ground-truth trajectories or annotated videos. The framework combines multimodal LLM initialization, differentiable MLS-MPM simulation over Gaussian scene representations, and a learnable motion distillation objective that extracts motion priors from pretrained video diffusion models while minimizing appearance and geometry inductive biases. In the broader motion-physics literature, it belongs to a family of methods that couple explicit physical structure with learned visual priors for dynamic scene understanding, generation, and control (Wang et al., 1 Jan 2026).

1. Problem setting and scope

MotionPhysics is motivated by a specific bottleneck in physics-based graphics and simulation: accurately simulating existing 3D objects and a wide variety of materials often demands expert knowledge and time-consuming physical parameter tuning to achieve the desired dynamic behavior. Its stated objective is to remove that manual tuning burden by inferring physically plausible parameters directly from user language for a selected scene, then optimizing those parameters inside a differentiable simulation loop (Wang et al., 1 Jan 2026).

The framework is evaluated across more than thirty scenarios, including real-world, human-designed, and AI-generated 3D objects, and spans a wide range of materials such as elastic solids, metals, foams, sand, and both Newtonian and non-Newtonian fluids. The reported scope therefore covers heterogeneous constitutive behavior rather than a single elasticity regime or a single object category (Wang et al., 1 Jan 2026).

This places MotionPhysics in a research area that has expanded from classical motion models toward learned physics-aware systems. At one end of that spectrum, educational and analytical work on projectile motion emphasizes closed-form kinematics and quantitative agreement between analytic formulas and simulated trajectories (Silva et al., 2014), while quadratic-drag projectile analysis introduces nonlinear motion equations under the assumption $Re\gg 1$ for real projectiles (Hernández et al., 2020). At the other end, contemporary methods embed simulators, diffusion models, and differentiable renderers into end-to-end pipelines for human motion capture, multi-person interaction, and scene dynamics (Ju et al., 2023, Ugrinovic et al., 2024, Li et al., 9 Jun 2025, Lu et al., 11 May 2026). MotionPhysics occupies the text-guided simulation branch of that continuum.

2. End-to-end architecture

The MotionPhysics pipeline proceeds in five stages. It first converts a static scene, represented as a mesh, images, or 3DGS, into an initial collection of Gaussians $\mathcal{G}^0$. It then queries a pretrained multimodal LLM, specifically GPT-4, using a text prompt and an optional reference image to obtain an initial estimate of material parameters $\theta^{\rm ini}$. A differentiable MLS-MPM simulator runs forward under external forces $\mathbf{f}^{\rm ext}$, producing time-varying Gaussians $\{\mathcal{G}^t\}$ and rendered frames $\{\mathcal{I}_l\}$. Motion priors are extracted from a frozen video-diffusion model $\phi$ through a learnable motion extractor $M$, yielding a motion-distillation loss $L_{\rm LMD}$. Finally, the framework optionally combines the LLM prior with score distillation or optical-flow supervision and backpropagates through rendering and simulation to refine $\theta$ (Wang et al., 1 Jan 2026).

The system is described as a single autodiff graph connecting text, GPT-4 initialization, Gaussian scene state, MLS-MPM simulation, rendering, diffusion-based motion supervision, and gradient flow back to the physical parameters. All components except the LLM initialization are therefore part of a unified differentiable optimization procedure (Wang et al., 1 Jan 2026).

Within adjacent literature, this architecture is closely related to methods that also couple Gaussian scene representations with continuum simulation. PhysMotion, for example, reconstructs a feed-forward 3D Gaussian from a single image, time-steps that representation using a differentiable Material Point Method with continuum mechanics-based elastoplasticity models, and then refines the result using a text-to-image diffusion model with cross-frame attention (Tan et al., 2024). This suggests that MotionPhysics is distinguished less by the use of MPM itself than by its text-conditioned parameter inference and its motion-distillation objective.

3. Multimodal material parameter estimation

The multimodal parameter-estimation stage maps prompt semantics into a structured material vector

$\mathcal{G}^0$0

where $\mathcal{G}^0$1 is density, $\mathcal{G}^0$2 is the material class, and $\mathcal{G}^0$3 contains class-specific coefficients such as Young’s modulus and yield stress. GPT-4 receives the text prompt $\mathcal{G}^0$4 and an optional reference image, and returns an initial material guess in that parameter space (Wang et al., 1 Jan 2026).

To prevent LLM hallucinations, MotionPhysics constrains each scalar parameter to a plausible interval,

$\mathcal{G}^0$5

and then clamps the raw estimate $\mathcal{G}^0$6 through

$\mathcal{G}^0$7

When a parameter spans many orders of magnitude, such as Young’s modulus, the query and clamp are performed in log-space for numerical stability (Wang et al., 1 Jan 2026).

A common misconception is that MotionPhysics performs direct laboratory-grade material identification. The reported limitations state the opposite: the estimated $\mathcal{G}^0$8 are visually plausible but are not guaranteed to match real-world material tests, so the method is explicitly “not a substitute for laboratory parameter identification” (Wang et al., 1 Jan 2026).

This parameter-prior mechanism differs from video-only scene-dynamics approaches that encode physics in latent per-particle descriptors. FreeGave augments each 3D Gaussian with a low-dimensional physics code $\mathcal{G}^0$9 intended to capture latent physical attributes such as mass, force direction, and motion class, while VeloGauss assigns each Gaussian a latent Physics Code $\theta^{\rm ini}$0 and combines it with explicit affine-basis velocity modeling and global physical constraints (Li et al., 9 Jun 2025, Lu et al., 11 May 2026). MotionPhysics instead begins from language-conditioned material initialization and then refines those parameters by differentiable simulation.

4. Learnable motion distillation and differentiable simulation

The central methodological claim of MotionPhysics is that standard score-distillation or framewise optical-flow losses tend to conflate appearance and geometry with motion. To address that, it introduces a small learnable motion extractor $\theta^{\rm ini}$1, described as a two-layer convolutional module initialized to identity, whose function is to isolate motion cues in the diffusion-model latent space (Wang et al., 1 Jan 2026).

The framework defines a clean latent code $\theta^{\rm ini}$2 from a mildly noised scene and a noisy code $\theta^{\rm ini}$3. Using the motion extractor, the target and prediction become

$\theta^{\rm ini}$4

The learnable motion-distillation objective is a Charbonnier loss variant,

$\theta^{\rm ini}$5

Gradients propagate through rendering and simulation according to

$\theta^{\rm ini}$6

This is the core mechanism by which motion priors from a frozen video-diffusion model are converted into parameter gradients for a physics simulator (Wang et al., 1 Jan 2026).

The simulation backend is a differentiable MLS-MPM adapted to 3D Gaussian Splatting. Each splat $\theta^{\rm ini}$7 has center $\theta^{\rm ini}$8, covariance $\theta^{\rm ini}$9, and color $\mathbf{f}^{\rm ext}$0, and the one-step update is written as

$\mathbf{f}^{\rm ext}$1

The operator $\mathbf{f}^{\rm ext}$2 is implemented in NVIDIA Warp so that it is fully differentiable with respect to $\mathbf{f}^{\rm ext}$3 (Wang et al., 1 Jan 2026).

Joint optimization uses

$\mathbf{f}^{\rm ext}$4

with the reported practical setting $\mathbf{f}^{\rm ext}$5 and $\mathbf{f}^{\rm ext}$6 (Wang et al., 1 Jan 2026).

In broader context, this design echoes other systems that use diffusion models as physics guidance rather than as purely appearance generators. In physics-guided human motion capture, a reverse diffusion process is explicitly guided by gradients derived from a physics-tracking module, and several iterations cause physics-based tracking and kinematic denoising to promote each other (Ju et al., 2023). MotionPhysics applies a comparable division of labor—learned priors for motion regularization, simulator for physical plausibility—but uses that coupling to optimize material parameters for text-guided dynamics.

5. Evaluation and reported behavior

The reported experiments span human-designed 3DGS reconstructions such as Torus, Bird, Playdoh, and Toothpaste; real-world captures such as Alocasia, Carnation, Hat, and Telephone; and AI-generated meshes such as Urchin, Alien, Axe, and Gentleman. Baselines are PhysDreamer, DreamPhysics, OmniPhysGS, and PhysFlow. Evaluation uses Overall Consistency (OC) $\mathbf{f}^{\rm ext}$7, defined through ViCLIP video-text cosine; CLIPSIM $\mathbf{f}^{\rm ext}$8, defined as per-frame CLIP cosine average; ECMS, the Energy-Constrained Motion Score; and a 2AFC user study on physical realism and prompt adherence (Wang et al., 1 Jan 2026).

The quantitative comparison reports the following averaged values: PhysDreamer achieves $\mathbf{f}^{\rm ext}$9 OC, $\{\mathcal{G}^t\}$0 CLIPSIM, $\{\mathcal{G}^t\}$1 ECMS, and $\{\mathcal{G}^t\}$2 min; DreamPhysics achieves $\{\mathcal{G}^t\}$3, $\{\mathcal{G}^t\}$4, $\{\mathcal{G}^t\}$5, and $\{\mathcal{G}^t\}$6 min; OmniPhysGS achieves $\{\mathcal{G}^t\}$7, $\{\mathcal{G}^t\}$8, $\{\mathcal{G}^t\}$9, and approximately $\{\mathcal{I}_l\}$0 h; PhysFlow achieves $\{\mathcal{I}_l\}$1, $\{\mathcal{I}_l\}$2, $\{\mathcal{I}_l\}$3, and $\{\mathcal{I}_l\}$4 min; MotionPhysics reports $\{\mathcal{I}_l\}$5, $\{\mathcal{I}_l\}$6, $\{\mathcal{I}_l\}$7, and $\{\mathcal{I}_l\}$8 min (Wang et al., 1 Jan 2026).

The user study includes $\{\mathcal{I}_l\}$9 participants and $\phi$0 video-pairs each. The reported preference rate is greater than $\phi$1 of the time on human-designed and AI-generated scenes for both realism and prompt-adherence, and greater than $\phi$2 on real-world scenes (Wang et al., 1 Jan 2026).

Qualitatively, the paper highlights three behaviors. An elastic AI mesh labeled “Alien bird” bounces correctly, unlike the baselines. Water-like jam spreads smoothly in the Newtonian case rather than oscillating. Non-Newtonian toothpaste flows and then arrests, matching the prompt (Wang et al., 1 Jan 2026). A plausible implication is that the method’s principal contribution lies in preserving prompt-conditioned motion semantics across materially different regimes, rather than only improving photometric fidelity.

6. Position within the broader motion-physics literature

The broader contemporary literature on motion physics comprises several neighboring paradigms. One group learns physical structure directly from dynamic videos. FreeGave models 3D scene geometry, appearance, and underlying physics purely from multi-view videos, introduces a physics code and a divergence-free module for estimating a per-Gaussian velocity field, and avoids PINN losses; VeloGauss extends this direction by learning a physically grounded velocity field over Gaussian particles with a Physics Code, a Particle Dynamics System, and Global Physical Constraints enforced via a PINN-style loss (Li et al., 9 Jun 2025, Lu et al., 11 May 2026).

A second group focuses on physics-aware correction of motion estimates. MultiPhys feeds monocular multi-person motion estimated by a kinematic-based method into MuJoCo in an autoregressive manner and reports large reductions in inter-person penetration, ground penetration, and foot-skating while maintaining competitive motion accuracy (Ugrinovic et al., 2024). Physics-informed Ground Reaction Dynamics from Human Motion Capture estimates ground reaction forces from motion-capture data using Euler’s integration scheme and a PD algorithm, and uses those physics-based reactive forces as supervision to improve dynamics estimation from MoCap alone (Le et al., 2 Jul 2025).

A third group integrates physics modules with generative priors. Physics-Guided Human Motion Capture with Pose Probability Modeling uses a latent gaussian model for 2D-to-3D uncertainty and a physics-guided reverse diffusion process to reconstruct physically plausible human motion (Ju et al., 2023). PhysMotion combines single-image 3D Gaussian reconstruction, differentiable MPM with elastoplastic constitutive models, and text-to-image diffusion with cross-frame attention to generate physically plausible video from one image and input conditions such as applied force and torque (Tan et al., 2024).

MotionPhysics shares elements with all three groups but is not identical to any of them. It uses differentiable simulation as in PhysMotion, learned motion priors as in physics-guided diffusion approaches, and Gaussian scene representations as in FreeGave and VeloGauss, yet its specific task is text-guided inference of material parameters for a chosen 3D scene without ground-truth trajectories or annotated videos (Wang et al., 1 Jan 2026). This suggests that the distinctive research question is not merely reconstructing motion or enforcing compliance, but translating language into a physically plausible parameterization that remains optimizable through simulation.

7. Limitations, misconceptions, and open directions

MotionPhysics reports four principal strengths: zero-shot text-guided estimation of a wide variety of materials, including non-Newtonian fluids and plastics; robustness to out-of-distribution geometry and texture via motion distillation; a fully differentiable end-to-end path from LLM prior through simulation to gradient-based refinement; and competitive optimization times of approximately $\phi$3 min on A100 without per-scene engineering (Wang et al., 1 Jan 2026).

The limitations are equally explicit. Shadows and lighting changes are not modelled inside the MPM loop, so visual realism could improve through coupling to a differentiable shading model. The estimated $\phi$4 are visually plausible but not guaranteed to match real-world material tests. Current metrics—OC, CLIPSIM, and ECMS—are described as imperfectly correlated with human perception, motivating new benchmarks that jointly evaluate appearance, dynamics, and prompt adherence. Extension to multi-object scenes and real-time interactive control is identified as a promising next step (Wang et al., 1 Jan 2026).

Two misconceptions are clarified by these limitations. First, MotionPhysics is not a purely first-principles simulator; it depends on GPT-4 initialization and motion priors distilled from a frozen video-diffusion model. Second, it is not a direct metrology tool for constitutive parameter recovery. Its target is visually realistic dynamic simulation guided by natural language, with automatically determined physically plausible parameters, rather than certified physical identification (Wang et al., 1 Jan 2026).

In a wider sense, these limitations align with open problems across the motion-physics field. VLM-based physics reasoning still struggles with subtle motion dynamics and spatial interactions, which has motivated explicit spatial-temporal grounding and motion tracking in models such as MASS (Wu et al., 23 Nov 2025). A plausible implication is that future MotionPhysics-style systems will increasingly be evaluated not only by rendered trajectories, but also by their compatibility with grounded physical reasoning, long-range temporal consistency, and language-conditioned interpretability.