ContactGaussian-WM: Differentiable World Model

• ContactGaussian-WM is a differentiable, physics-grounded rigid-body world model that employs a unified Gaussian representation for both visual rendering and collision detection.
• It integrates SG-GS scene initialization and phys-geo refinement to jointly refine geometry, appearance, and dynamics from sparse, contact-rich video sequences.
• Experimental evaluations highlight its robustness in data-scarce and discontinuous-contact scenarios, outperforming baselines in simulation and real-world tests.

Searching arXiv for ContactGaussian-WM and closely related world-model / differentiable-contact papers. ContactGaussian-WM is a differentiable, physics-grounded rigid-body world model for learning complex physical interactions directly from sparse and contact-rich video sequences. Its defining premise is a unified Gaussian representation that serves simultaneously as a visual primitive for 3D Gaussian splatting and as an explicit collision proxy for contact dynamics, coupled to an end-to-end differentiable learning framework that back-propagates image-level supervision through collision detection, a closed-form contact dynamics engine, and a renderer. The model is designed for regimes in which existing methods struggle with data scarcity and discontinuous contact phenomena, and it is evaluated in both simulation and real-world settings, including downstream use in data synthesis and real-time model predictive control (Wang et al., 11 Feb 2026).

1. Problem setting and architectural decomposition

ContactGaussian-WM addresses world-model learning under sparse visual supervision and contact-rich dynamics by organizing the pipeline into two stages: SG-GS scene initialization and phys-geo refinement. In Stage I, calibrated multi-view images or sparse video frames are converted into a spherical Gaussian splat representation. In Stage II, geometry and dynamics are jointly refined through a differentiable simulator plus renderer. The architecture is explicitly modular: a collision detector computes smooth distances to a union of spherical primitives, a complementarity-free contact dynamics engine advances the rigid-body state in closed form, and a 3DGS renderer projects the transformed Gaussians back to images (Wang et al., 11 Feb 2026).

Component	Representation or state	Role
SG-GS scene initialization	$\{\mathbf c_i,s_i\}$, $\{\alpha_i,f_i\}$	Initializes geometry and appearance
Collision detector	Smooth distances to spherical primitives	Supplies contact points, normals, Jacobians
Contact dynamics engine	$$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$	Closed-form rigid-body integration
3DGS renderer	Transformed Gaussians	Produces predicted images

The paper characterizes this coupling as a “what-you-see-is-what-you-collide” loop: the same Gaussian scene description participates in rendering and collision reasoning. This is operationally significant because it eliminates the separation between a high-fidelity visual model and an unrelated collision mesh, allowing pixel-level losses to update both physical parameters and geometric proxies.

2. Unified Gaussian representation for appearance and collision

The scene is represented as

$$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$

where $\mathbf c_i\in\mathbb R^3$ is a 3D center, $s_i>0$ is an isotropic scale, $\alpha_i$ is opacity, and $f_i$ is a learned color feature. The geometric and visual descriptions are therefore aligned at the primitive level. By enforcing isotropic scales, each Gaussian becomes a sphere with radius

$r_i=2s_i,$

which serves as an explicit collision proxy (Wang et al., 11 Feb 2026).

For rendering, the framework relies on the standard analytic overlap of two 3D Gaussians,

$$\int_{\mathbb R^3}\mathcal N(x\mid \mu_i,\Sigma_i)\mathcal N(x\mid \mu_j,\Sigma_j)\,dx = \frac{(2\pi)^{3/2}}{|\Sigma_i+\Sigma_j|^{1/2}} \exp\!\Bigl(-\tfrac12(\mu_i-\mu_j)^\top(\Sigma_i+\Sigma_j)^{-1}(\mu_i-\mu_j)\Bigr),$$

with spherical covariance $\{\alpha_i,f_i\}$0. In the model description, this analytic overlap drives silhouette and radiance rendering. The important methodological consequence is that collision geometry is not derived from an auxiliary mesh or occupancy field; it is built directly from the same Gaussian primitives that support view synthesis.

A common misunderstanding is to treat the Gaussian representation as merely a visual parameterization. In ContactGaussian-WM, the isotropic constraint makes the geometric Gaussians physically consequential, because their centers and scales define a union of spheres that participates directly in contact computation. This suggests that the representation is not only differentiable and renderable, but also intentionally chosen to make contact geometry tractable.

3. Differentiable contact geometry and closed-form dynamics

Given a query point $\{\alpha_i,f_i\}$1, the exact signed distance to the union of $\{\alpha_i,f_i\}$2 spheres is

$\{\alpha_i,f_i\}$3

which is non-differentiable when the active sphere changes. ContactGaussian-WM replaces this with a smoothed minimum,

$\{\alpha_i,f_i\}$4

followed by a sigmoid-clamped penalization,

$\{\alpha_i,f_i\}$5

where $\{\alpha_i,f_i\}$6 controls sharpness, $\{\alpha_i,f_i\}$7 the sigmoid steepness, and $\{\alpha_i,f_i\}$8 a fixed penalty depth. From this differentiable distance, the closest surface point and normal are

$\{\alpha_i,f_i\}$9

These quantities provide contact Jacobians and signed distances to every sphere (Wang et al., 11 Feb 2026).

The rigid-body state is

$$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$0

and the learnable physical parameters are

$$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$1

with $$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$2 the mass-inertia matrix, $$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$3 the Coulomb friction coefficient, $$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$4 a stiffness gain, and $$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$5 a damping gain. Free motion is first computed as

$$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$6

where $$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$7 includes gravity, actuation, and Coriolis terms. After contact detection, Jacobians and distances are stacked into $$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$8 and $$(\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})$$9, and the contact impulse is obtained by the complementarity-free rule

$$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$0

The state update is then

$$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$1

Because $$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$2 is smooth, the integrator is closed-form and fully differentiable, while still handling normal and friction forces, including stick, slide, and separation modes, without solving an LCP or NCP. This places the model in a specific niche within differentiable physics: it is neither a purely latent dynamics model nor a traditional complementarity solver wrapped in finite-difference gradients.

4. End-to-end learning objective and two-stage optimization

The model predicts images through a composition of dynamics and rendering,

$$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$3

and is trained with the image-level objective

$$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$4

Because the collision detector, dynamics engine, and renderer are all differentiable, gradients propagate from $$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$5 back to both $$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$6 and $$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$7 through the chain rule (Wang et al., 11 Feb 2026).

Training proceeds in two stages. Stage I (SG-GS) first fixes appearance and learns $$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$8 to match rendered geometry maps $$\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},$$9 by minimizing

$\mathbf c_i\in\mathbb R^3$0

It then fixes geometry and refines appearance through

$\mathbf c_i\in\mathbb R^3$1

Stage II (phys-geo refinement) uses the learned SG-GS as initialization and jointly optimizes $\mathbf c_i\in\mathbb R^3$2 while fine-tuning $\mathbf c_i\in\mathbb R^3$3 through the full image loss and gradient flow across the differentiable world model.

This decomposition separates scene initialization from physical identification. A plausible implication is that the first stage supplies a geometry and appearance prior strong enough to let the second stage allocate supervision to contact and inertial parameters rather than rediscovering basic visual structure.

5. Experimental protocol and reported performance

The reported simulation experiments use two modalities in MuJoCo: Fall, consisting of free-fall and bounce of a single object, and Push, consisting of quasi-static pushing by a virtual hand. Training uses a single short trajectory of 15 frames, while evaluation uses much longer unseen trajectories with different initial poses or external velocities. The baselines are DreamerV3, CEM + MuJoCo + R, and PIN-WM. The evaluation metrics are translation error,

$\mathbf c_i\in\mathbb R^3$4

orientation error,

$\mathbf c_i\in\mathbb R^3$5

and PSNR over pixels (Wang et al., 11 Feb 2026).

The paper reports that, in the Fall scenario, ContactGaussian-WM dramatically outperforms all baselines in the highly discontinuous, sparse-contact case, while in the Push scenario it matches or slightly exceeds the alternatives. Real-world tests include free fall and LEAP Hand manipulations over varied ground materials, evaluated with long-horizon open-loop PSNR. Jointly learning $\mathbf c_i\in\mathbb R^3$6 is reported to shrink the sim-to-real gap by $\mathbf c_i\in\mathbb R^3$7–$\mathbf c_i\in\mathbb R^3$8 dB, which the paper interprets as evidence of robust generalization and accurate long-horizon prediction from pure RGB videos.

These results are methodologically notable because the training regime is deliberately data-limited. The evaluation design therefore tests not only visual reconstruction quality but also whether physical parameters inferred from sparse observations remain predictive outside the training rollout.

6. Downstream use, scope, and terminological disambiguation

Two downstream applications are emphasized. First, ContactGaussian-WM is used for simulation data synthesis: because it is explicit, fast at approximately $\mathbf c_i\in\mathbb R^3$9 Hz on a single RTX 4090, and physically faithful, it can generate labeled, contact-rich 4D data that is both visually photorealistic through 3DGS and physically consistent. Second, it is used for real-time MPC, with control sequences obtained from

$s_i>0$0

The closed-form differentiable dynamics provide analytic gradients, and the reported MuJoCo in-hand redirection experiments show the LEAP Hand reorienting objects in real time at tens of Hertz (Wang et al., 11 Feb 2026).

The model’s scope is correspondingly specific. It is a rigid-body world model with explicit spherical collision proxies, not a general-purpose latent simulator for arbitrary deformable media. Likewise, its “Gaussian” designation refers to 3D Gaussian splats and spherical Gaussian collision primitives, not to the Gaussian Wigner-function formalism used in statistical optics. The latter appears in a different line of work, where “Gaussian-WM” denotes a Gaussian Wigner-function model propagated through matrix-aperture beamlines (Pogorelov et al., 2023). Disambiguating these uses is useful because the shared acronym “WM” refers to distinct objects: a world model in ContactGaussian-WM, and a Wigner model in the optics context.

Within few-body physics and quantum chemistry, “contact Gaussian” language also appears in unrelated Gaussian regularization and Gaussian-expansion schemes (Kievsky et al., 2020, Kristyan, 2019). Those usages are conceptually separate from ContactGaussian-WM. In the present context, the term denotes a unified Gaussian scene representation embedded in differentiable contact dynamics and visual prediction, with the central claim that geometry, appearance, and physical parameters can be learned jointly from sparse videos in contact-rich settings.