MaterialRefGS: Physically-Based Gaussian Splatting

• MaterialRefGS is a method that couples physically-based material parameterization with deferred PBR shading to enable precise material reflectance inference using 2D Gaussian Splatting.
• It enforces multi-view consistency and employs photometric reflection priors to reduce ambiguities in specular reconstruction across varying viewpoints.
• Experimental results demonstrate state-of-the-art performance on reflective scenes, outperforming benchmarks in PSNR, SSIM, LPIPS, and normal accuracy.

MaterialRefGS is a class of methods and a reference pipeline for physically-based, multi-view-consistent material reflectance inference and rendering in the context of 2D Gaussian Splatting. Developed to address the limitations of traditional Gaussian Splatting approaches on reflective and highly specular surfaces, MaterialRefGS couples explicit physically-based spatial material parameterization with environment- and occlusion-aware illumination models, delivering faithful reflective appearance reconstruction and novel-view synthesis, especially for scenes exhibiting strong specular reflection and complex lighting (Zhang et al., 13 Oct 2025).

1. Physically-Based Material Representation and Deferred Shading

MaterialRefGS augments standard screen-space Gaussian primitives with physically-based material parameters (albedo, metallicity, roughness), enabling deferred PBR shading on top of the splatted geometry. For each image or rendered view, the pipeline computes screen-space G-buffers of geometry and material attributes by front-to-back alpha-compositing of attributes from the Gaussians:

$$\hat \Psi(\mathbf u) = \sum_{i=1}^K \psi_i(\mathbf u)\;o_i(\mathbf u)\;p_i(\mathbf u)\;\prod_{j<i}(1 - o_j(\mathbf u))$$

where $\psi \in \{c_d, a, m, r\}$ denotes per-Gaussian color, albedo, metallic, and roughness, and $o_i, p_i$ are per-splat opacity and probability. The resulting spatially-varying screen-space maps are then input to a simplified Disney BSDF comprising diffuse and microfacet-specular lobes:

$$L_o(\omega_o) = (1 - m)\,c_d + L_s(\omega_o; a, m, r, n)$$

$$L_s(\omega_o) = \int_{\Omega^+} L_i(\omega_i)\, f_s(\omega_i, \omega_o)\, (\omega_i\!\cdot\!n)\;d\omega_i$$

where $L_i$ encodes incident radiance (including direct and indirect components) and $f_s$ is a microfacet BRDF with normal distribution, Fresnel, and shadow-masking terms.

This deferred-shading approach ensures spatial and angular decoupling, critical for reliable estimation of reflectance properties in overlapping splats and preventing the "baking" of view-dependent color into material maps (Zhang et al., 2024).

2. Multi-view Consistency and Ill-posedness Mitigation

A central innovation of MaterialRefGS is the enforcement of multi-view material consistency. Without such constraints, ambiguity in BRDF fitting (due to entanglement with lighting and environment) leads to per-view aliasing and inconsistent specular inference. MaterialRefGS enforces that rasterized material maps are stable under cross-view warps:

$$\mathcal L_{mv} = \left\|\,\Psi_i[P] \;-\;\Psi_j[P'] \right\|_2^2$$

where a 7×7 patch $P$ around pixel $u$ (view $\psi \in \{c_d, a, m, r\}$0) is back-projected and warped to $\psi \in \{c_d, a, m, r\}$1 in view $\psi \in \{c_d, a, m, r\}$2 using local depth and normals. This consistency is imposed for each of the BRDF channels (diffuse, roughness, metallic). Multi-view integration drastically reduces the solution space by requiring that per-pixel attributes not change with viewpoint except due to visibility, enforcing robust global material inference in the presence of both direct and secondary light transport (Zhang et al., 13 Oct 2025).

3. Photometric Reflection Priors and Highly Reflective Region Detection

To further constrain metallic and specular inference, MaterialRefGS integrates photometric variability tracking across views. For each region, the variation in appearance as a function of camera pose provides a direct prior on reflection strength, decoupled from geometric texture. The standard deviation of a given patch's appearance across aligned views is computed in luminance (intensity) space and projected into 3D. These per-point reflection scores are used as soft supervision weights in the loss:

$\psi \in \{c_d, a, m, r\}$3

where $\psi \in \{c_d, a, m, r\}$4 is a binary mask of likely-reflective regions, $\psi \in \{c_d, a, m, r\}$5 the desired metallic value for the material, and $\psi \in \{c_d, a, m, r\}$6 the photometric prior. This datadriven constraint enables the system to distinguish truly reflective (specular) areas from regions with merely bright highlights or textured variation (Zhang et al., 13 Oct 2025).

4. Physically-Based Environment Illumination and Occlusion Modeling

MaterialRefGS introduces occlusion- and environment-aware illumination modeling. Each pixel's outgoing radiance is a combination of (a) direct radiance from the environment map gated by visibility, and (b) indirect radiance captured via differentiable Gaussian ray tracing along the reflection direction:

$\psi \in \{c_d, a, m, r\}$7

$\psi \in \{c_d, a, m, r\}$8

$\psi \in \{c_d, a, m, r\}$9

where $o_i, p_i$0 are the opacity, probability, and diffuse color of Gaussians hit along the reflected ray, and $o_i, p_i$1 is a small residual for higher-order effects. This physically-based compositing enables the renderer to capture secondary interreflections and occlusion-driven radiance, mitigating the limitations of naïve cubemap lookup and strengthening material estimation fidelity (Zhang et al., 13 Oct 2025).

5. Joint Optimization and Implementation Details

The full MaterialRefGS objective combines photometric loss, geometric (normal/depth) priors, multi-view consistency, and reflection supervision:

$o_i, p_i$2

$o_i, p_i$3

where $o_i, p_i$4 enforces depth-normal consistency, $o_i, p_i$5 is a monocular normal prior (early-stage only), and $o_i, p_i$6, $o_i, p_i$7 are as above. Optimization proceeds in phases: geometry and depth/normal priors are trained first, then PBR parameters, and finally multi-view consistency and reflection supervisions are added for full joint inference of attributes, environment, and geometry (Zhang et al., 13 Oct 2025).

6. Experimental Results and Benchmarking

MaterialRefGS achieves state-of-the-art performance across synthetic and real benchmarks for reflective and specular scenes, exceeding prior methods (Ref-NeRF, ENVIDR, Ref-Gaussian, GaussianShader, and others) in PSNR, SSIM, LPIPS, and normal-MAE. Representative results:

Dataset	PSNR (dB)	SSIM	LPIPS	Normal MAE
ShinyBlender	35.57	0.976	0.049	2.04°
GlossySynthetic	30.83	0.962	0.046	–
Ref-Real	25.04	0.703	0.185	–
Mip-NeRF360	27.06	0.809	0.181	–

MaterialRefGS delivers sharper, more accurate specular highlights, improved secondary interreflection, and more coherent, artifact-free material maps relative to all baselines. Notably, it produces uniform per-pixel metallic/roughness for analytic BRDF evaluation and editing (Zhang et al., 13 Oct 2025).

7. Limitations and Future Directions

Limitations of MaterialRefGS include increased computational overhead due to multi-view warping and differentiable ray tracing, and dependence on accurate initial geometry and environment modeling. Potential avenues for further research include more efficient warping, learned or adaptive lighting priors, applicability to dynamic and nonrigid scenes, and integration with learned priors for sparsely labeled or real-world environments (Zhang et al., 13 Oct 2025).