PhysGaia: A Physics-Aware Dataset of Multi-Body Interactions for Dynamic Novel View Synthesis (2506.02794v1)

Abstract: We introduce PhysGaia, a novel physics-aware dataset specifically designed for Dynamic Novel View Synthesis (DyNVS), encompassing both structured objects and unstructured physical phenomena. Unlike existing datasets that primarily focus on photorealistic reconstruction, PhysGaia is created to actively support physics-aware dynamic scene modeling. Our dataset provides complex dynamic scenarios with rich interactions among multiple objects, where they realistically collide with each other and exchange forces. Furthermore, it contains a diverse range of physical materials, such as liquid, gas, viscoelastic substance, and textile, which moves beyond the rigid bodies prevalent in existing datasets. All scenes in PhysGaia are faithfully generated to strictly adhere to physical laws, leveraging carefully selected material-specific physics solvers. To enable quantitative evaluation of physical modeling, our dataset provides essential ground-truth information, including 3D particle trajectories and physics parameters, e.g., viscosity. To facilitate research adoption, we also provide essential integration pipelines for using state-of-the-art DyNVS models with our dataset and report their results. By addressing the critical lack of datasets for physics-aware modeling, PhysGaia will significantly advance research in dynamic view synthesis, physics-based scene understanding, and deep learning models integrated with physical simulation -- ultimately enabling more faithful reconstruction and interpretation of complex dynamic scenes. Our datasets and codes are available in the project website, http://cvlab.snu.ac.kr/research/PhysGaia.

• The paper introduces PhysGaia, a novel dataset providing ground-truth physics data and complex multi-body interactions for dynamic novel view synthesis.
• It leverages specialized physics solvers (FLIP, Pyro, MPM, Vellum) to accurately model liquids, gases, viscoelastic materials, and textiles.
• Experimental results reveal that current DyNVS methods struggle with physical realism, highlighting the need for improved dynamic scene modeling.

PhysGaia: A Physics-Aware Dataset for Dynamic Novel View Synthesis

The paper introduces PhysGaia, a novel dataset designed to advance physics-aware Dynamic Novel View Synthesis (DyNVS). Addressing the limitations of existing datasets that primarily focus on photorealistic reconstruction, PhysGaia provides complex dynamic scenarios with rich interactions among multiple objects and diverse physical materials. By offering ground-truth information such as 3D particle trajectories and physics parameters, this dataset enables quantitative evaluation of physical modeling and facilitates research in dynamic view synthesis, physics-based scene understanding, and deep learning models integrated with physical simulation.

Dataset Properties and Contributions

PhysGaia distinguishes itself through several key properties: complex physics-aware dynamics, ground-truth physics information, support for diverse DyNVS tasks, customizability, and accessibility. Unlike existing datasets that often lack multi-body interactions and comprehensive ground-truth data, PhysGaia includes 17 scenes featuring interactions among multiple objects composed of liquids, gases, viscoelastic substances, and textiles (Figure 1). The dataset provides ground-truth 3D particle trajectories and accurate physics parameters, such as viscosity and Young's modulus, enabling precise evaluation of physical reasoning in dynamic scenes.

Figure 1: Examples from the dataset showcasing complex physical interactions between diverse materials.

Moreover, PhysGaia supports both multiview and monocular DyNVS tasks, offering flexibility in training configurations. The provision of simulation node graphs and parameter settings allows users to generate customized scenes or additional modalities like depth maps and surface normals. The dataset includes integration pipelines for state-of-the-art DyNVS models and COLMAP-reconstructed point clouds, fostering its adoption in the research community.

Related Work and Uniqueness

The paper thoroughly reviews related work in DyNVS and 4D datasets, highlighting PhysGaia's uniqueness. Existing datasets often focus on photorealistic reconstruction with limited object motion or lack accurate physics solvers and multi-object interactions. While some datasets include scenes with physical phenomena, they typically lack ground-truth physical information and complex interactions. In contrast, PhysGaia offers scenes with multi-body interactions involving diverse physical materials, simulation parameters, and ground-truth trajectories, positioning it as a valuable resource for advancing physics-aware dynamic scene modeling.

Dataset Construction Methodology

The dataset construction leverages SideFX Houdini 20.5 to integrate multiple physics solvers within a unified procedural environment. The dataset includes four material types: liquids (using the FLIP solver), gases (using the Pyro solver), viscoelastic materials (using the MPM), and textiles (using the Vellum solver). The authors carefully selected these material-specific solvers for accurately calculating force exchange among multiple bodies.

For liquid scenes, the FLIP solver is preferred for its ability to maintain particle velocities and model realistic liquid behavior. In gas simulations, the Pyro solver accurately models the temperature field, crucial for capturing buoyancy effects. The MPM is used for viscoelastic substances, effectively capturing deformation and internal force propagation, while the Vellum solver, based on the XPBD framework, simulates textile materials.

Experimental Analysis and Results

The paper evaluates recent DyNVS methods, including D-3DGS, 4DGS, STG, and SOM, on the PhysGaia dataset. Experiments conducted on an NVIDIA A6000 GPU reveal the limitations of existing methods in capturing complex multi-body dynamics and achieving physical realism. Quantitative results, measured using PSNR, SSIM, and LPIPS, indicate that multiview setups generally outperform monocular ones, yet even they often yield PSNR scores below 30, underscoring the difficulty of capturing complex dynamics. Textile materials are generally reconstructed more accurately, while viscoelastic materials pose greater challenges due to their complex motion. Qualitative results demonstrate that all methods struggle to accurately capture multi-body interactions, leading to artifacts and under-reconstruction of dynamic regions.

Conclusion and Future Directions

PhysGaia offers a novel physics-aware dataset for understanding physics in dynamic scenes and for advancing DyNVS. The dataset's complex multi-body interactions, diverse materials, and ground-truth physics data will enable the evaluation of physics reasoning. The paper reveals the limitations of current DyNVS methods in achieving physical realism, highlighting the potential for improvement. The authors propose future research directions, including advancing physical reasoning in dynamic scenes, exploring multi-body interactions, and integrating material-specific physics solvers.