Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats

Abstract: 3D Gaussian Splatting (3DGS) has achieved state-of-the-art photorealistic rendering, but the representation gap prevents these assets from being physically interactive. Production-grade physics engines do not understand the 3DGS representation, while prior physics-for-3DGS methods are monolithic silos. These prior works are fundamentally limited, demonstrating only object-centric physics in isolated environments, such as on an ideal plane. They are incapable of interacting with complex static collision geometry or heterogeneous assets. We propose a novel framework that, for the first time, bridges this gap by enabling 3DGS assets to participate in scene-level, heterogeneous, multi-solver physical simulations. Our core contribution is a Representation Abstraction Framework that translates all diverse assets, including 3DGS, virtual meshes, and fluids, into a unified physical particle set. This abstraction is key to enabling complex behaviors, such as the non-rigid deformation of 3DGS assets, within a unified physics pipeline. This particle set, along with the static scene collision boundaries derived from scene capture, is processed within a solver-agnostic physics kernel. The physical results are then mapped back to drive each asset's specific visual reconstruction. This architecture unlocks capabilities impossible with prior art. We demonstrate complex, two-way interactions between deformable 3DGS assets, standard CG assets such as fluids and meshes, and large-scale captured static environments, showcasing realistic coupled phenomena that were previously unattainable.

• The paper introduces a Representation Abstraction Framework (RAF) that unifies 3D Gaussian splats with traditional assets for heterogeneous physics simulation.
• It demonstrates dynamic, two-way interactions across multiple solver systems (MPM, SPH, PBD) using a unified particle-based simulation kernel.
• The framework achieves photorealistic rendering and accurate scene-level collisions, validated through extensive ablation studies and complex experimental scenarios.

Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats: An Expert Analysis

Introduction

"Scene-Level Heterogeneous Physics Simulation with 3D Gaussian Splats" (2606.21753) presents a comprehensive bridging of the gap between photorealistic 3D Gaussian Splat (3DGS) representations and industrial-grade, heterogeneous physics engines. The work addresses two major shortcomings that have historically limited the use of 3DGS assets in interactive, physically-plausible environments: the non-physicality of native 3DGS assets and the isolation and non-extensibility of prior physics-for-3DGS efforts. By introducing a Representation Abstraction Framework (RAF), the authors enable dynamic, two-way physical interactions between diverse asset types (captured 3DGS, rigid bodies, deformables, fluids) in statically and physically complex scenes, all rendered with photorealistic fidelity in Unreal Engine 5.

Prior Work and Limitations

All prior "physics-for-3DGS" methods, including PhysGaussian, GausSim, DecoupledGaussian, and others, are fundamentally constrained by their homogeneous, "siloed" architectures. They treat 3DGS kernels as special material points but only within closed, single-solver systems (typically MPM or neural simulators), unable to handle multi-material settings, standard CG assets, or complex, real captures as boundary geometry. Critically, these approaches are not scene-level and degenerate to simple setups—voids or planes—without genuine scene context. Consequently, state coupling between mesh-based solvers, particles, and 3DGS assets is unachievable with these schemes.

The Representation Abstraction Framework (RAF)

RAF's architecture abstracts heterogeneous assets—including captured 3DGS, virtual meshes, and dynamical particle-based systems—into a unified particle-based physical context, supported by a modular, solver-agnostic physics kernel. The framework comprises three stages:

1. Asset Preparation and Abstraction: All static geometry (captured or imported) is merged into a single static collision mesh, while dynamics (GS, mesh, fluid, rigid bodies, articulated assets) are ingested and consistently "particleized." For meshes, Poisson disk sampling is employed, while 3DGS assets use an opacity field-derived internal filling. All per-entity constitutive parameters, as well as visual state and barycentric bindings, are cached for later recoupling.
2. Unified Simulation Kernel: The scene’s unified particle list and collision mesh are processed by a multi-solver backend (MPM, SPH, PBD, rigid, articulated). Inter-solver coupling, continuous force exchanges, and friction/contact handling are managed through shared physics primitives (impulse-based and grid-mediated responses). The kernel iteratively resolves solver order and propagates dynamical state across assets and solvers within each simulation step.
3. Heterogeneous Recoupling and Rendering: After simulation, each asset’s visual state is reconstructed from updated physical state: 3DGS covariance is updated by deformation gradients, mesh vertex positions by barycentric weights, and particle systems are rendered directly. Final output is serialized into Unreal Engine 5, supporting advanced illumination and cinematics.

Figure 1: Architecture of the Representation Abstraction Framework, highlighting asset ingestion, unified simulation, and asset-specific recoupling for rendering.

Experimental Scenarios and New Capabilities

The authors thoroughly validate their claims with five heterogeneous, scene-level scenarios that are as follows:

1. Fluid (SPH) on virtual mesh: Demonstrates fluid–rigid body interaction in a captured GS setting.
2. Fluid (SPH) on GS soft body (MPM): Exhibits two-way, multi-solver coupling between a captured GS-based soft body and an external fluid.
3. Articulated rigid manipulation: Validates support for kinematically driven assets and their interaction with fully dynamic bodies.
4. Cloth (PBD) on captured geometry: Showcases high-fidelity cloth–complex geometry coupling, leveraging non-planar, captured GS collision meshes.
5. Rigid bodies in 3DGS container: Demonstrates non-trivial rigid body collisions in intricate GS-recovered statics, previously impossible with mesh-only physics or GS-only approaches.

Each scenario distinctly exemplifies operation beyond prior art and the extensibility and compositionality of the RAF.

Figure 2: Qualitative showcase of heterogeneous, physics-based interactions within highly realistic 3DGS environments, spanning fluids, deformables, articulated bodies, and cloth.

Ablation Studies

To rigorously establish the necessity of RAF's major components, the authors perform three ablations:

• Unified Kernel Removal: Disabling heterogeneous coupling (operating solvers in isolation) causes physically implausible results—for SPH and MPM, entities pass through each other with no interaction.
• Scene Geometry Ablation: Without static mesh construction, simulation defaults to trivial planes, eliminating interaction with complex, detailed geometry and resulting in nonphysical artifacting (e.g., cloth passing through real statues).
• Representation Abstraction Ablation: Omitting visual recoupling (using only simulation particles) destroys photorealistic fidelity; only sparse, non-continuous states are rendered.

Figure 3: Comparison of unified kernel ablation—without full coupling, complex multi-material interactions are fundamentally broken.

Figure 4: Ablation of static mesh—complex geometry interaction is only possible with proper scene-level collision mesh integration.

Figure 5: Ablation of representation abstraction reveals the imperative for photorealistic recoupling to reconstruct high-fidelity asset appearance.

Comparison with Photogrammetry

A further visual comparison with a standard multi-view photogrammetry workflow (e.g., KiriEngine meshes) reveals that using classic mesh representations yields considerable loss of geometric detail and completeness, especially on high-frequency structures (e.g., mechanical parts, non-watertight areas). In contrast, 3DGS-based pipeline preserves sharp, intricate details and provides a continuous, artifact-free geometric basis for further abstraction and simulation.

Figure 6: Photogrammetry baseline (left) fails to reconstruct sharp, detailed geometry; 3DGS pipeline (right) enables full-fidelity asset simulation and rendering.

Practical and Theoretical Implications

This framework establishes a paradigm for integrating captured reality—through high-fidelity photorealistic methods like 3DGS—directly into advanced, multi-material physics simulation environments. Theoretically, it demonstrates that explicit, volumetric abstraction, combined with modular solver interfaces, is an effective approach for bridging disparate visual and physical representations. Practically, it unlocks a new regime for VFX, simulation-driven content generation, robotics simulation, and digital twin construction, where complicated, interactable assets—both captured and synthetic—are composed naturally.

Current runtime is optimized for offline, cinematic workflows rather than real-time, interactive performance. The pipeline's generalization potential to VR/AR applications hinges on further acceleration of both the abstraction and simulation stages and on tighter integration with real-time rendering pipelines.

Conclusion

The Representation Abstraction Framework convincingly closes the loop between photorealistic asset capture and complex, production-grade, multi-solver physical simulation, a barrier unbreached by previous siloed approaches. The authors establish the necessity of unified coupling, scene-level geometry handling, and visual recoupling, validated through diverse, nontrivial experimental demonstrations and ablation studies. This architecture sets a robust baseline for future research in interactive 3D scene simulation, heterogeneous asset integration, and neural scene representation interoperability with symbolic physical engines.

Further research directions include real-time kernel acceleration, enhanced instance decoupling for more complex scene compositions, and the extension of RAF toward generalizable, online learning-based solver synthesis and inverse design pipelines.