Multi-material Physical Gaussians (M-PhyGs)

• M-PhyGs are an explicit, continuous framework that augments 3D Gaussian Splatting by associating each splat with detailed photometric and physical descriptors.
• They leverage Bayesian and neural inference to update material classifications and estimate continuous physical properties with uncertainty modeling.
• The framework enables physically accurate simulation, relightable rendering, and robust multi-material interaction for applications in vision, graphics, and robotics.

Multi-material Physical Gaussians (M-PhyGs) are an explicit, spatially continuous framework that extends 3D Gaussian Splatting to jointly represent geometry, photometric properties, and dense spatially varying physical parameters—including multi-material composition—at per-Gaussian resolution. M-PhyGs fuse visual cues from multi-view imagery with physical reasoning and, depending on the approach, leverage Bayesian, deep, or LMM-based methods to enable uncertainty-aware estimation, heterogeneous mechanical simulation, and high-fidelity physically-based rendering. The methodology unifies recent developments across graphics, vision, and computational mechanics, with applications ranging from dynamics simulation to physically grounded material estimation and relightable scene synthesis.

1. Gaussian Splatting Framework: Geometric and Material Parameterization

M-PhyGs extend 3D Gaussian Splatting by associating, with each oriented 3D Gaussian atom or “splat” ($\mu_i,\Sigma_i$), a set of physical and material descriptors in addition to standard photometric attributes. In PhysGS (Chopra et al., 23 Nov 2025), each Gaussian is parameterized by

• Mean position $\mu_i\in\mathbb{R}^3$, anisotropic covariance $\Sigma_i\in\mathbb{R}^{3\times3}$,
• RGB color $c_i\in[0,1]^3$, opacity $\alpha_i\in[0,1]$,
• Discrete material belief $\theta_i=(\theta_{i1},…,\theta_{iK})$ (Dirichlet over $K$ material classes),
• Continuous beliefs over each material’s physical properties ($\mu_f$, $k$, $H$, $\rho$).

In physically integrated pipelines (PIG (Xiao et al., 9 Jun 2025), PhysGaussian (Xie et al., 2023)), each Gaussian or derived MPM particle is further annotated with mechanical parameters such as Young’s modulus $E$, Poisson’s ratio $\nu$, density $\rho$, friction coefficient $\mu$, and more general constitutive descriptors as physical material fields.

PBR-centric pipelines (TexGaussian (Xiong et al., 29 Nov 2024), MGM (Ye et al., 26 Sep 2025), GS-2M (Nguyen et al., 26 Sep 2025)) inject roughness $r_i$, metallic $m_i$, and physical albedo $a_i$ at each primitive, enabling material-aware rendering under arbitrary environments.

Table: Core Attributes of a Multi-material Physical Gaussian

Attribute	Description	Typical Source
$\mu_i$	3D position	Splat/particle
$\Sigma_i$	3D anisotropic covariance (orientation + scale)	Splat/particle
$c_i$ or $a_i$	RGB color or physical albedo	Photometric fitting
$\alpha_i$	Opacity/density parameter	Photometric fitting
$r_i, m_i$	Roughness, metallic channels (PBR)	Photometric fitting
$\theta_i$	Material class distribution (Dirichlet/Categorical)	Bayesian inference
$\rho_i, E_i, \nu_i, \mu_i$	Physical/mechanical parameters	Direct estimation

The explicit, per-Gaussian attribution enables localized, uncertainty-aware multi-property inference and physical simulation across heterogeneous materials.

2. Bayesian and Neural Inference of Material Properties

PhysGS (Chopra et al., 23 Nov 2025) formalizes the estimation of both material class and continuous physical properties as recursive Bayesian inference. At each splat $i$, given sequential observations $o_t$ from vision-LLMs (VLMs), the posterior $p(z_i, \psi_i | o_{1:t})$ over material label $z_i$ (discrete) and property $\psi_i$ (continuous) is maintained and iteratively updated:

• Material class: Dirichlet-Categorical, $\theta_i \sim \mathrm{Dir}(\alpha_i^{(0)})$, updated via

$$\tilde{\alpha}_{ik} \leftarrow \alpha_{ik}^{(0)} + \sum_{m:c_m=k} \lambda p_m$$

$$p(z_i = k| o_{1:t}) = \frac{\tilde{\alpha}_{ik}}{\sum_{j=1}^K \tilde{\alpha}_{ij}}$$

• Physical property: class-conditional Gaussian (or NIG for uncertainty), moments updated via confidence-weighted sample sums, predictive mixture

$$p(\psi_i | o_{1:t}) = \sum_{k=1}^K p(z_i = k | o_{1:t}) \mathcal{N}(\psi_i; \mu_{ik}, \sigma^2_{ik})$$

Here, aleatoric and epistemic uncertainties are disentangled using a Normal–Inverse–Gamma hierarchy, enabling uncertainty-aware selection and automatic disambiguation as more evidence arrives.

Neural approaches (OmniPhysGS (Lin et al., 31 Jan 2025), Physics3D (Liu et al., 6 Jun 2024), PIDG (Hong et al., 9 Nov 2025)) instead parameterize per-Gaussian physical behavior via deep feature-encoded mixtures of constitutive submodels or time-evolving material fields, whose parameters are estimated by backpropagating gradients from visual or physics-based loss functions—often using score distillation sampling (SDS) from video diffusion models or matching Lagrangian flows to tracked optical flow.

3. Multi-material Simulation and Constitutive Laws

M-PhyGs realize physically accurate, multi-material simulation by endowing each Gaussian (or derived MPM particle) with locally assigned constitutive parameters and performing continuum-mechanics-based time integration.

• PhysGaussian (Xie et al., 2023), PIG (Xiao et al., 9 Jun 2025), Physics3D (Liu et al., 6 Jun 2024), and OmniPhysGS (Lin et al., 31 Jan 2025) use the Material Point Method (MPM), where each particle or group of Gaussians carries $E$, $\nu$, $\rho$, friction, and, in the general case, a mixture weight over a finite set of expert constitutive laws (hyperelastic, plastic, viscoelastic, fluid).
• The stress update at each time step is

$$\mathbf{P}_j = \sum_k w_{j,k} \mathbf{P}_k(\mathbf{F})$$

where $w_{j,k}$ are mixture weights (OmniPhysGS), and $\mathbf{P}_k$ is the Piola–Kirchhoff stress for submodel $k$.

• Deformation gradients are clamped or regularized (PIG) to prevent geometric instabilities during large-strain deformations.

For heterogeneous scenes, each semantic region or material segment (defined via over-segmentation, feature clustering, or functional mapping) is assigned a distinct physical parameter vector, but simulation is performed on the unified grid so that different material domains interact through shared forces and momenta.

4. Material Segmentation and Physical Property Annotation

Material segmentation in M-PhyGs proceeds via a combination of explicit geometric mapping, vision-language understanding, and multi-view aggregation:

• GaussianProperty (Xu et al., 15 Dec 2024), PhysGS (Chopra et al., 23 Nov 2025), and M-PhyGs (Wada et al., 18 Dec 2025) segment input RGB images with SAM; each segment is assigned a material class and properties using VLMs or LMMs, whose outputs are projected onto the 3D Gaussians through geometric visibility checks and aggregated by frequency voting.
• Physics3D (Liu et al., 6 Jun 2024) and M-PhyGs (Wada et al., 18 Dec 2025) cluster features (e.g., DINO, affinity) to define over-segmented groups, assigning coherent local bonds for jointly optimizing mechanical parameters.
• PIG (Xiao et al., 9 Jun 2025) maintains topology-preserving correspondences between 2D masks and 3D Gaussians for accurate object-level splitting.

The annotation pipeline delivers per-Gaussian or per-segment sets of physically meaningful parameters, ready for downstream simulation or grasp synthesis.

5. Physically-Based Rendering and Material Appearance Estimation

PBR pipelines (TexGaussian (Xiong et al., 29 Nov 2024), MGM (Ye et al., 26 Sep 2025), GS-2M (Nguyen et al., 26 Sep 2025)) incorporate physically meaningful albedo, roughness, and metallic channels into the parameter set of each Gaussian, enabling relightable synthesis and robust material decomposition:

• Rendering combines volumetric/explicit splatting with Cook–Torrance microfacet BRDF evaluation:

$$L_o(\mathbf{x},\mathbf{v}) = \int L_i(\mathbf{x}, \ell)\; f_r(\ell,\mathbf{v}; a_i, r_i, m_i)\; (\mathbf{n}\cdot \ell) d\ell$$

• Joint regression pipelines (TexGaussian, MGM) train 3D U-Nets or volume transformers to directly output all geometric and PBR channels, with loss terms mixing RGB, LPIPS, and parameter-space MSE for globally consistent appearance and material estimates.

GS-2M employs multi-view photometric-variation supervision, correlating cross-view reflectance consistency with roughness to enforce plausible material separation in challenging, specular surfaces.

6. Uncertainty Modeling, Supervision, and Evaluation

M-PhyGs frameworks leverage rigorous uncertainty modeling at both the class (material) and scalar property levels:

• PhysGS (Chopra et al., 23 Nov 2025) models per-splat material ambiguity as posterior entropy of the Dirichlet weights and quantifies continuous property uncertainty with mixture variances.
• Normal–Inverse–Gamma parametrizations decompose epistemic and aleatoric uncertainties, supporting robust physical reasoning under partial observability.
• Evaluation metrics include ADE, ALDE, APE, MnRE for physical property fidelity; segmentation mean IoU; PSNR, SSIM for photo-geometry alignment; and task-centric metrics for downstream simulation (e.g., success rate in robotic grasping (Xu et al., 15 Dec 2024)).

Supervision pipelines—ranging from LMM-powered human feedback to deep model-based SDS—enable optimization even from limited or ambiguous visual signals, efficiently resolving multi-material composition and physical constants from short video clips or sparse imagery.

7. Applications: Physical Simulation, Grasp Planning, and Relightable Generation

M-PhyGs methods have demonstrated significant impact across several domains:

• Dense physical property estimation and uncertainty-aware scene understanding for robotic vision (Chopra et al., 23 Nov 2025).
• Accurate multi-material interaction simulation (elastic, plastic, fluid, viscoelastic) under real-world forces and constraints (Xie et al., 2023, Liu et al., 6 Jun 2024, Lin et al., 31 Jan 2025, Wada et al., 18 Dec 2025).
• Physically realistic grasp-force prediction and affordance-aware manipulation in robotic systems, achieving state-of-the-art task and damage rates over prior approaches (Xu et al., 15 Dec 2024).
• Fully relightable 3D asset generation with feed-forward text conditioning, enabling downstream editing and visualization under arbitrary illumination (Ye et al., 26 Sep 2025, Xiong et al., 29 Nov 2024).
• Mesh/material decomposition and surface reconstructions robust to complex reflectances and geometry (Nguyen et al., 26 Sep 2025).
• New datasets and benchmarks, e.g., Phlowers (Wada et al., 18 Dec 2025), for evaluating multi-material parameter inference on compound, real-world objects.

Collectively, M-PhyGs frameworks establish a unified paradigm for scene representation that bridges geometry, material appearance, and the full spectrum of mechanical behaviors—opening new research directions at the confluence of graphics, vision, robotics, and computational mechanics.