PG-3DGS: Optimizing 3D Gaussian Splatting to Satisfy Physics Objectives

Abstract: Recent advances in Gaussian Splatting have enabled fast, high-fidelity 3D scene generation, yet these methods remain purely visual and lack an understanding of how shapes behave in the physical world. We introduce Physics-Guided 3D Gaussian Splatting (PG-3DGS), a framework that couples differentiable physics simulation with 3D Gaussian representations to generate 3D structures satisfying physics functionalities. By allowing physical objectives to guide the shape optimization process alongside visual losses, our approach produces geometries that are not only photometrically accurate but also physically functional. The model learns to adjust shapes so that the generated objects exhibit physically meaningful behaviors, for example, teapots that can pour and airplanes that can generate lift, without sacrificing visual quality. Experiments on pouring and aerodynamic lift tasks show that PG-3DGS improves physical functionality while preserving visual quality. In addition to simulation gains, bench-top physical lift tests with 3D-printed aircraft (Cessna, B-2 Spirit, and paper plane) under identical airflow conditions show higher scale-measured lift for PG-3DGS, generated structures than an appearance-matching baseline in all three cases. Our unified framework connects appearance-based reconstruction with physics-based reasoning, enabling end-to-end generation of 3D structures that both look realistic and function correctly.

• The paper introduces PG-3DGS, a joint optimization method that combines multi-view photometric losses with differentiable fluid and aerodynamic simulation to resolve physically important ambiguities in reconstructed geometry.
• The method uses a differentiable soft Gaussian-to-grid occupancy mask and Brinkman penalization to model fluid–solid interaction, enabling gradients from objectives such as liquid pouring and aerodynamic lift to guide 3D reconstruction.
• Physical tests on 3D-printed aircraft showed higher lift for PG-3DGS across all three shapes, increasing mean lift from 3.27 g for appearance-only 3DGS to 11.53 g while largely preserving rendering quality.

Motivation and problem statement

Appearance-based 3D reconstruction is an incomplete supervision signal when reconstructed assets are meant to be used rather than merely rendered. The image formation process underlying multi-view reconstruction is largely insensitive to internal structure, watertightness, and connectivity, so inverse-graphics objectives admit many visually consistent geometries that are physically nonfunctional. The paper's motivating examples are concrete: a teapot whose interior is sealed from the spout despite photometric agreement across views, and an airplane that matches silhouette and texture yet produces insufficient lift. PG-3DGS (Physics-Guided 3D Gaussian Splatting) addresses this by casting functional 3D structure generation as a joint optimization over a 3D Gaussian representation, combining visual reconstruction losses with differentiable physics objectives (2605.11266).

Formulation as joint MAP inference

The method formulates the task probabilistically via a Gibbs posterior over geometry parameters $\theta$, with a joint objective

$$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$

where $\mathcal L_{\mathrm{vis}}$ measures multi-view photometric consistency (an $\ell_1$ + SSIM combination through the standard 3DGS splatting renderer) and $\mathcal L_{\mathrm{phys}} = \Phi(\tau(\theta))$ evaluates a task-specific functional over the trajectory $\tau(\theta)$ produced by unrolling a differentiable simulator. MAP inference reduces to minimizing $\mathcal J$. This framing connects to guidance mechanisms in diffusion models: the visual loss acts as the base gradient while the physics term acts as a guidance gradient steering geometry toward task-consistent configurations.

Technical core: Gaussian-to-grid coupling

The central technical challenge is bridging continuous anisotropic Gaussians and grid-aligned Eulerian fluid solvers. The authors construct a soft solid occupancy mask

$$\chi(\mathbf x;\theta) = 1 - \prod_i \big(1 - \hat\alpha_i(\mathbf x)\big),$$

a permutation-invariant, fully differentiable probabilistic union of per-Gaussian occupancy contributions evaluated at grid cell centers. A tiled rasterization-style evaluation with conservative AABB culling reduces the naive $O(N|\Omega_h|)$ cost to $O(|\Omega_h|\bar P)$ without changing the definition or introducing discontinuities.

This soft mask replaces binary voxelization, which would yield vanishing or undefined gradients almost everywhere as geometry crosses voxel boundaries. Fluid–solid interaction is enforced entirely through Brinkman penalization in the momentum equation, using a semi-implicit update stable for large penalization strength $$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$0; for intermediate $$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$1 values the update acts as fractional drag interpolating between free fluid and no-slip solid motion. Hydrodynamic forces and torques on the body are recovered from the discrete momentum exchange across the Brinkman step, making them differentiable with respect to $$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$2. For moving solids, rigid-body state (position, quaternion orientation, angular momentum) is integrated with symplectic Euler translationally and an implicit Euler angular update—the latter chosen because explicit rotation was numerically unstable under strong coupling—reducing each step to a $$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$3 linear solve.

Optimization proceeds in two stages: a visual-only initialization phase followed by a joint phase in which physics gradients are enabled alongside the visual term, avoiding clashes between physics gradients and immature visual representations.

Experimental results

Two task families are evaluated: liquid pouring from teapots (fixed solids, objective based on time-averaged mass-weighted negative $$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$4-position of tracked fluid) and aerodynamic flight from airplanes (objective defined as negative average $$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$5-position under gravity and constant thrust, with fixed orientation to isolate shape-dependent lift). Baselines include standard 3DGS, VolSDF, and NeRF; the cleanest controlled comparison is 3DGS versus PG-3DGS, which share depth-map initialization, since cross-family comparisons are not initialization-matched—a caveat the authors state explicitly.

Qualitatively, only PG-3DGS consistently recovers connected cavities and spouts enabling pouring; baselines seal interiors or introduce disconnected voids, and the NeRF baseline places spurious mass away from the object that distorts simulated dynamics. On flight tasks, baseline geometries lose altitude while PG-3DGS produces sustained lift, with training trajectories showing steady transitions from descending to ascending paths.

The strongest quantitative evidence comes from benchtop physical validation: 3D-printed aircraft tested under identical blower airflow showed higher scale-measured lift for PG-3DGS than the appearance-only baseline on all three shapes:

Shape	Baseline lift (g)	PG-3DGS lift (g)	Δ (g)
Cessna	13.7	20.5	6.8
Paper plane	−13.4	0.3	13.7
B-2 Spirit	9.5	13.8	4.3
Mean	3.27	11.53	8.27

The paper-plane case is notable: the baseline produced net downward force (−13.4 g) while PG-3DGS reached slightly positive lift. This result matters because it suggests the simulation-driven geometric changes capture real aerodynamic effects rather than simulator artifacts, though the authors correctly characterize the setup as proxy validation rather than a controlled wind-tunnel study.

An ablation sweeping the physics weight across eight values (40 runs total on a B-2 Spirit configuration) shows that physics satisfaction improves substantially with increasing weight while SSIM remains nearly constant up to roughly $$\mathcal J(\theta) = \mathcal L_{\mathrm{vis}}(\theta) + \mathcal L_{\mathrm{phys}}(\theta) + \mathcal L_{\mathrm{reg}}(\theta),$$6, with mild SSIM degradation and diminishing physics returns only at the largest weights. This supports the interpretation that physics guidance selects among visually plausible geometries rather than trading off appearance against function.

Limitations and open questions

The paper concedes several limitations at points where they bear directly on results. The soft-mask Brinkman coupling trades boundary sharpness for differentiability: overly smooth masks blur fine geometric features relevant to certain flow regimes, while overly sharp masks yield poorly conditioned gradients. Grid resolution and timestep constrain fidelity for thin structures, narrow channels, and boundary-layer effects important to aerodynamics. Optimization is task-conditional—recovered geometry improves the specified objective and may not match the true physical shape if the objective is misspecified. Unrolled differentiable simulation is computationally expensive and introduces hyperparameters (physics weight schedules, horizon, loss frequency) affecting convergence. Open questions left by the work include whether adaptive gradient normalization can replace manual physics-weight tuning, how the Gaussian-to-physics interface extends to spatially varying material properties such as compliance or permeability, and whether closed-loop calibration against small numbers of physical measurements can correct simulation bias.

Conclusion

PG-3DGS demonstrates that differentiable physics can serve as a practical second supervision signal for 3D Gaussian reconstruction, resolving geometric ambiguities invisible to image formation—interior connectivity, watertightness, aerodynamic effectiveness—without meaningful degradation of rendering quality over a broad range of physics-loss weights. Its reusable ingredients are the differentiable soft occupancy mask, Brinkman-based fluid–solid coupling, and momentum-exchange force estimation, all of which admit substitution of new simulators and objectives. The consistent benchtop lift improvements across three aircraft shapes provide evidence that these gains transfer beyond the training simulator, while the acknowledged constraints on solver fidelity and objective specification delineate where the approach currently applies.