---
title: 'ContactGaussian-WM: Differentiable World Model'
url: https://www.emergentmind.com/topics/contactgaussian-wm
type: topic
---

# ContactGaussian-WM: Differentiable World Model

Searching arXiv for ContactGaussian-WM and closely related world-model / differentiable-contact papers.
ContactGaussian-WM is a differentiable, physics-grounded rigid-body world model for learning complex physical interactions directly from sparse and contact-rich video sequences. Its defining premise is a unified Gaussian representation that serves simultaneously as a visual primitive for 3D Gaussian splatting and as an explicit collision proxy for contact dynamics, coupled to an end-to-end differentiable learning framework that back-propagates image-level supervision through collision detection, a closed-form contact dynamics engine, and a renderer. The model is designed for regimes in which existing methods struggle with data scarcity and discontinuous contact phenomena, and it is evaluated in both simulation and real-world settings, including downstream use in data synthesis and real-time model predictive control [2602.11021].

## 1. Problem setting and architectural decomposition

ContactGaussian-WM addresses world-model learning under sparse visual supervision and contact-rich dynamics by organizing the pipeline into two stages: **SG-GS scene initialization** and **phys-geo refinement**. In Stage I, calibrated multi-view images or sparse video frames are converted into a spherical Gaussian splat representation. In Stage II, geometry and dynamics are jointly refined through a differentiable simulator plus renderer. The architecture is explicitly modular: a collision detector computes smooth distances to a union of spherical primitives, a complementarity-free contact dynamics engine advances the rigid-body state in closed form, and a 3DGS renderer projects the transformed Gaussians back to images [2602.11021].

| Component | Representation or state | Role |
|---|---|---|
| SG-GS scene initialization | \(\{\mathbf c_i,s_i\}\), \(\{\alpha_i,f_i\}\) | Initializes geometry and appearance |
| Collision detector | Smooth distances to spherical primitives | Supplies contact points, normals, Jacobians |
| Contact dynamics engine | \((\mathbf q_t,\mathbf v_t)\to(\mathbf q_{t+1},\mathbf v_{t+1})\) | Closed-form rigid-body integration |
| 3DGS renderer | Transformed Gaussians | Produces predicted images |

The paper characterizes this coupling as a “what-you-see-is-what-you-collide” loop: the same Gaussian scene description participates in rendering and collision reasoning. This is operationally significant because it eliminates the separation between a high-fidelity visual model and an unrelated collision mesh, allowing pixel-level losses to update both physical parameters and geometric proxies.

## 2. Unified Gaussian representation for appearance and collision

The scene is represented as
$$
\mathcal G=\bigl\{(\mathbf c_i,s_i)_{i=1}^n,\;(\alpha_i,f_i)_{i=1}^n\bigr\},
$$
where \(\mathbf c_i\in\mathbb R^3\) is a 3D center, \(s_i>0\) is an isotropic scale, \(\alpha_i\) is opacity, and \(f_i\) is a learned color feature. The geometric and visual descriptions are therefore aligned at the primitive level. By enforcing isotropic scales, each Gaussian becomes a sphere with radius
$$
r_i=2s_i,
$$
which serves as an explicit collision proxy [2602.11021].

For rendering, the framework relies on the standard analytic overlap of two 3D Gaussians,
$$
\int_{\mathbb R^3}\mathcal N(x\mid \mu_i,\Sigma_i)\mathcal N(x\mid \mu_j,\Sigma_j)\,dx
=
\frac{(2\pi)^{3/2}}{|\Sigma_i+\Sigma_j|^{1/2}}
\exp\!\Bigl(-\tfrac12(\mu_i-\mu_j)^\top(\Sigma_i+\Sigma_j)^{-1}(\mu_i-\mu_j)\Bigr),
$$
with spherical covariance \(\Sigma=\mathrm{diag}(s^2,s^2,s^2)\). In the model description, this analytic overlap drives silhouette and radiance rendering. The important methodological consequence is that collision geometry is not derived from an auxiliary mesh or occupancy field; it is built directly from the same Gaussian primitives that support view synthesis.

A common misunderstanding is to treat the Gaussian representation as merely a visual parameterization. In ContactGaussian-WM, the isotropic constraint makes the geometric Gaussians physically consequential, because their centers and scales define a union of spheres that participates directly in contact computation. This suggests that the representation is not only differentiable and renderable, but also intentionally chosen to make contact geometry tractable.

## 3. Differentiable contact geometry and closed-form dynamics

Given a query point \(q\in\mathbb R^3\), the exact signed distance to the union of \(n\) spheres is
$$
d(q)=\min_{i=1,\dots,n}\bigl(\|q-\mathbf c_i\|_2-r_i\bigr),
$$
which is non-differentiable when the active sphere changes. ContactGaussian-WM replaces this with a smoothed minimum,
$$
d_{\mathrm{soft}}(q)
=
-\frac1\beta
\log\sum_{i=1}^n
\exp\!\bigl(-\beta(\|q-\mathbf c_i\|_2-r_i)\bigr),
$$
followed by a sigmoid-clamped penalization,
$$
d(q)\approx
\sigma\!\bigl(\gamma d_{\mathrm{soft}}(q)\bigr)\,d_{\mathrm{soft}}(q)
+
\bigl(1-\sigma\!\bigl(\gamma d_{\mathrm{soft}}(q)\bigr)\bigr)(-\delta),
$$
where \(\beta\) controls sharpness, \(\gamma\) the sigmoid steepness, and \(\delta>0\) a fixed penalty depth. From this differentiable distance, the closest surface point and normal are
$$
q_c=q-d(q)\frac{\nabla_q d(q)}{\|\nabla_q d(q)\|},
\qquad
n=\frac{\nabla_q d(q)}{\|\nabla_q d(q)\|}.
$$
These quantities provide contact Jacobians and signed distances to every sphere [2602.11021].

The rigid-body state is
$$
x_t=\{q_t,v_t\},
$$
and the learnable physical parameters are
$$
\Theta=(M,\mu,k,c),
$$
with \(M\) the mass-inertia matrix, \(\mu\) the Coulomb friction coefficient, \(k\) a stiffness gain, and \(c\) a damping gain. Free motion is first computed as
$$
v^{(0)}_{t+1}=v_t+h\,M^{-1}F(q_t,v_t,u_t),
$$
where \(F(\cdot)\) includes gravity, actuation, and Coriolis terms. After contact detection, Jacobians and distances are stacked into \(\tilde J_t\) and \(\tilde D_t\), and the contact impulse is obtained by the complementarity-free rule
$$
\lambda_t=
\mathrm{softplus}\Bigl(
-k\bigl(h\,\tilde J_t\,v^{(0)}_{t+1}+\tilde D_t\bigr)
-c\,(\tilde J_t\,v^{(0)}_{t+1})
\Bigr).
$$
The state update is then
$$
v_{t+1}=v^{(0)}_{t+1}+h\,M^{-1}\tilde J_t^\top\lambda_t,
\qquad
q_{t+1}=q_t\oplus\bigl(h\,v_{t+1}\bigr).
$$

Because \(\mathrm{softplus}(x)=\log(1+e^x)\) is smooth, the integrator is closed-form and fully differentiable, while still handling normal and friction forces, including stick, slide, and separation modes, without solving an LCP or NCP. This places the model in a specific niche within differentiable physics: it is neither a purely latent dynamics model nor a traditional complementarity solver wrapped in finite-difference gradients.

## 4. End-to-end learning objective and two-stage optimization

The model predicts images through a composition of dynamics and rendering,
$$
\hat x_{t+1}=\mathrm{Dyn}\bigl(\hat x_t,u_t;\Theta,\mathcal G_{\mathrm{geo}}\bigr),
\qquad
\hat I_t=R(\hat x_t,\mathcal G),
$$
and is trained with the image-level objective
$$
\mathcal L(\Theta,\mathcal G)
=
\sum_{t=1}^T
\Bigl[
L_{\mathrm{LoFT}}(\hat I_t,I_t)
+
L_1(\hat I_t,I_t)
\Bigr].
$$
Because the collision detector, dynamics engine, and renderer are all differentiable, gradients propagate from \(\partial\mathcal L/\partial \hat I_t\) back to both \(\Theta\) and \(\mathcal G_{\mathrm{geo}}\) through the chain rule [2602.11021].

Training proceeds in two stages. **Stage I (SG-GS)** first fixes appearance and learns \(\mathcal G_{\mathrm{geo}}\) to match rendered geometry maps \(g\) by minimizing
$$
\min\sum_i\|\widehat g_i-g_i\|_1.
$$
It then fixes geometry and refines appearance through
$$
\min\sum_i\Bigl[(1-\lambda)\|I_i-\hat I_i\|_1+\lambda\,L_{\mathrm{D\mkern1mu SSIM}}(\hat I_i,I_i)\Bigr].
$$
**Stage II (phys-geo refinement)** uses the learned SG-GS as initialization and jointly optimizes \(\Theta\) while fine-tuning \(\mathcal G_{\mathrm{geo}}\) through the full image loss and gradient flow across the differentiable world model.

This decomposition separates scene initialization from physical identification. A plausible implication is that the first stage supplies a geometry and appearance prior strong enough to let the second stage allocate supervision to contact and inertial parameters rather than rediscovering basic visual structure.

## 5. Experimental protocol and reported performance

The reported simulation experiments use two modalities in MuJoCo: **Fall**, consisting of free-fall and bounce of a single object, and **Push**, consisting of quasi-static pushing by a virtual hand. Training uses a single short trajectory of 15 frames, while evaluation uses much longer unseen trajectories with different initial poses or external velocities. The baselines are **DreamerV3**, **CEM + MuJoCo + R**, and **PIN-WM**. The evaluation metrics are translation error,
$$
E_{\mathrm{trans}}=\frac1{NT}\sum\|\mathbf p_t-\hat{\mathbf p}_t\|,
$$
orientation error,
$$
E_{\mathrm{rot}}=\frac1{NT}\sum 2\arccos\bigl|\langle q_t,\hat q_t\rangle\bigr|,
$$
and PSNR over pixels [2602.11021].

The paper reports that, in the Fall scenario, ContactGaussian-WM dramatically outperforms all baselines in the highly discontinuous, sparse-contact case, while in the Push scenario it matches or slightly exceeds the alternatives. Real-world tests include free fall and LEAP Hand manipulations over varied ground materials, evaluated with long-horizon open-loop PSNR. Jointly learning \(\Theta\) is reported to shrink the sim-to-real gap by \(2\)–\(3\) dB, which the paper interprets as evidence of robust generalization and accurate long-horizon prediction from pure RGB videos.

These results are methodologically notable because the training regime is deliberately data-limited. The evaluation design therefore tests not only visual reconstruction quality but also whether physical parameters inferred from sparse observations remain predictive outside the training rollout.

## 6. Downstream use, scope, and terminological disambiguation

Two downstream applications are emphasized. First, ContactGaussian-WM is used for **simulation data synthesis**: because it is explicit, fast at approximately \(40\) Hz on a single RTX 4090, and physically faithful, it can generate labeled, contact-rich 4D data that is both visually photorealistic through 3DGS and physically consistent. Second, it is used for **real-time MPC**, with control sequences obtained from
$$
\min_{u_{0:H-1}}
\sum_{t=0}^{H-1}\|x_t-x_{\mathrm{goal}}\|_Q^2+\|u_t\|_R^2
\quad
\text{s.t. }
x_{t+1}=\mathrm{Dyn}(x_t,u_t;\Theta,\mathcal G_{\mathrm{geo}}).
$$
The closed-form differentiable dynamics provide analytic gradients, and the reported MuJoCo in-hand redirection experiments show the LEAP Hand reorienting objects in real time at tens of Hertz [2602.11021].

The model’s scope is correspondingly specific. It is a rigid-body world model with explicit spherical collision proxies, not a general-purpose latent simulator for arbitrary deformable media. Likewise, its “Gaussian” designation refers to 3D Gaussian splats and spherical Gaussian collision primitives, not to the Gaussian Wigner-function formalism used in statistical optics. The latter appears in a different line of work, where “Gaussian-WM” denotes a Gaussian Wigner-function model propagated through matrix-aperture beamlines [2309.11008]. Disambiguating these uses is useful because the shared acronym “WM” refers to distinct objects: a **world model** in ContactGaussian-WM, and a **Wigner model** in the optics context.

Within few-body physics and quantum chemistry, “contact Gaussian” language also appears in unrelated Gaussian regularization and Gaussian-expansion schemes [2011.06828] [1910.02987]. Those usages are conceptually separate from ContactGaussian-WM. In the present context, the term denotes a unified Gaussian scene representation embedded in differentiable contact dynamics and visual prediction, with the central claim that geometry, appearance, and physical parameters can be learned jointly from sparse videos in contact-rich settings.

Source: https://www.emergentmind.com/topics/contactgaussian-wm