PhysTalk: Language-Driven Visual Simulation

• PhysTalk is a train-free, language-driven visual simulation system that integrates LLM-based text-to-physics translation with GPU-accelerated multi-material simulation.
• It employs a modular architecture combining convex-hull proxy extraction, K-nearest neighbor skinning, and real-time Gaussian rendering to achieve interactive 4D animation.
• The system delivers rapid simulation performance by bypassing offline mesh extraction, enabling seamless, physics-based transformations in diverse 3DGS scenarios.

PhysTalk is a train-free, language-driven visual simulation system enabling real-time, open-vocabulary, physically realistic 4D animation within 3D Gaussian Splatting (3DGS) scenes. By coupling a LLM with a GPU-accelerated physics engine, PhysTalk interprets arbitrary user text prompts and interacts with 3DGS representations through executable physics-based code, generating live visual effects without offline mesh extraction or traditional render-wait pipelines (Collorone et al., 31 Dec 2025).

1. System Architecture and Workflow

The PhysTalk system integrates three core modules: a Text-to-Physics Translator (LLM in in-context learning mode), a GPU-based multi-material Physics Simulator (Genesis engine), and a Gaussian Skinning plus real-time Renderer. The workflow is initiated by user inputs comprising a static 3DGS scene, a free-form text prompt (e.g., “make the vase jump up and fall back”), and grounding documents such as an API spec, few-shot exemplars, and a simulation function template.

The pipeline unfolds as follows:

• The LLM fills in a Python template (with build_scene, step, and query stubs) based on the prompt and context documents.
• The build_scene function extracts a convex-hull proxy over Gaussian centers, discretizes this hull into particles, and assigns material models (rigid, elastic, or fluid).
• The step function advances the Genesis simulator for fixed time steps, recording per-particle positions and deformation gradients.
• The query function maps simulation-particle deformation to Gaussian parameters (center, covariance) via nearest neighbors and blending, enabling dynamic skinning.
• The renderer projects and composites the updated Gaussians in image space via front-to-back blending.

All operations are GPU-parallelized, enabling interactive 4D animation and editing at live frame rates.

2. 3D Gaussian Splatting (3DGS) Representation

3DGS represents volumetric scenes as collections of anisotropic Gaussians, each parameterized by a center $c_i \in \mathbb{R}^3$, covariance $\Sigma_i \in \mathbb{R}^{3 \times 3}$, RGB color $C_i$, and opacity $\alpha_i \in [0, 1]$. Covariance matrices are constructed as $\Sigma_i = R_i S_i S_i^T R_i^T$, where $R_i \in SO(3)$ (from a quaternion) and $S_i = \text{diag}(s_{i,1}, s_{i,2}, s_{i,3})$ provides scale.

A Gaussian’s contribution at $x \in \mathbb{R}^3$ is

$g_i(x) = \exp[-(x-c_i)^T \Sigma_i^{-1} (x-c_i)]$

For rendering, each $\Sigma_i$ is projected with a view transform and Jacobian, and overlapping Gaussians along a viewing ray are composited front-to-back:

$\Sigma_i \in \mathbb{R}^{3 \times 3}$0

GPU buffers store the centers and transformations; skinning overwrites positional and covariance parameters based on physics simulation. Colors and opacities can be per-Gaussian modulated for dynamic effects (e.g., fading, variable brightness).

3. Language-to-Physics Interface and Code Generation

PhysTalk leverages an LLM (GPT-5 in the described experiments) via in-context learning, conditioning on API specifications and example scripts. The prompt is reduced to a completion task where the model fills three template functions:

• build_scene(): sets up convex hull, assigns materials (MPM-elastic, rigid, SPH-fluid), configures gravity, objects, and initializes the Genesis scene.
• step(): steps the Genesis simulator by $\Sigma_i \in \mathbb{R}^{3 \times 3}$1, records per-particle positions and deformations.
• query(): for each Gaussian, gathers $\Sigma_i \in \mathbb{R}^{3 \times 3}$2 nearest simulation particles (via a k-d tree), computes blended displacements and deformations, and updates centers and covariances.

This approach circumvents model hallucinations and concentrates LLM output on parameterization rather than boilerplate code. Errors are correctable via single regeneration cycles.

Proxy objects are constructed by convex-hull extraction over Gaussians, enabling orders-of-magnitude acceleration over mesh extraction and dynamic remeshing. The convex hull is voxelized or tetrahedralized into simulation particles, each with tags and parameters reflecting the intended physical behavior.

4. Physics Engine Coupling and Simulation Dynamics

Genesis, the underlying simulation engine, accommodates rigid body dynamics, MPM-based continuum elasticity, and SPH-based particle fluids—all within a single GPU framework accessible via Python. The primary simulation entities are particles $\Sigma_i \in \mathbb{R}^{3 \times 3}$3 with mass $\Sigma_i \in \mathbb{R}^{3 \times 3}$4, position $\Sigma_i \in \mathbb{R}^{3 \times 3}$5, velocity $\Sigma_i \in \mathbb{R}^{3 \times 3}$6.

Dynamic evolution follows

$\Sigma_i \in \mathbb{R}^{3 \times 3}$7

Stepping utilizes discrete Euler integration:

$\Sigma_i \in \mathbb{R}^{3 \times 3}$8

Within elastic regions, the internal force arises from stress divergence

$\Sigma_i \in \mathbb{R}^{3 \times 3}$9

where $C_i$0 and $C_i$1. In fluid regions, interactions are governed by SPH methods incorporating pressure, density, and viscosity.

Genesis resolves inter-material and contact phenomena via unified constraint solvers, supporting real-time multi-material interaction and collision. Material assignments can vary spatially within the proxy hulls.

5. Implementation Characteristics and Performance

PhysTalk is computationally lightweight and training-free, as the LLM performs only code generation without optimizing model weights. Proxy hull extraction is $C_i$2 for up to $C_i$3. Skinning employs a $C_i$4-nearest neighbor search (e.g., $C_i$5 via k-d tree), scaling as $C_i$6. All simulation, blending, and rasterization loops are GPU-parallelized.

Typical benchmarks on an NVIDIA RTX 4090 report:

• Code generation and build_scene: $C_i$7 ms
• Full physics simulation (120 frames at $C_i$8 ms/frame): $C_i$9 ms
• Skinning and rendering: $\alpha_i \in [0, 1]$0 ms total ($\alpha_i \in [0, 1]$1 ms/frame) End-to-end, 120 frames (2 s at 60 fps) are generated under 100 ms, allowing interactive editing rates of approximately 8–10 fps.

6. Exemplary Use Cases

PhysTalk demonstrates versatility across diverse prompts:

• Elastic vase jump: A prompt such as “Make the vase jump up, fall back, and have its flowers wobble elastically” causes the LLM to assign MPM-elastic properties to flower subregions, rigid materials to the vase body, and introduces realistic gravity and initial velocities. The rendered animation exhibits realistic post-impact elastic recovery.
• Lava flow transformation: For “Turn the horse into molten lava pouring off its back,” the proxy hull is tagged as SPH fluid, parameterized by fluid variables (density, surface tension), and simulated as a cohesive flow. A hole-filling procedure spawns new Gaussians to fill visual discontinuities, producing seamless animations of material transitions.
• Interactive manipulation: Real-time, browser-based manipulation is supported via user drag/click, injecting external forces at the proxy level and updating the skinning pipeline instantly, thus enabling interactive, physically plausible responses.

7. Significance and Implications

PhysTalk advances the field of visual simulation by integrating language guidance, physics-based realism, and high-performance volumetric graphics without mesh extraction or offline optimization. The workflow enables direct, editable, and efficient transformation of user intent into 4D physically plausible animations for arbitrary multi-material 3DGS scenes. This design paradigm shifts workflows from traditional slow, batch-based paradigms to interactive, dialogue-driven creative processes, broadening accessibility and control for simulation-based animation (Collorone et al., 31 Dec 2025). A plausible implication is the facilitation of rapid prototyping and experimentation in both scientific visualization and creative content generation contexts.