---
title: 'GS-2M: Joint Mesh & Material Decomposition'
url: https://www.emergentmind.com/topics/gs-2m
type: topic
---

# GS-2M: Joint Mesh & Material Decomposition

GS-2M, short for **Gaussian Splatting for Joint Mesh Reconstruction and Material Decomposition**, is a method for recovering from calibrated multi-view images both accurate 3D geometry and material properties, notably albedo and roughness, within a single optimization framework built on 3D Gaussian Splatting [2509.22276]. Its central premise is that mesh reconstruction and material decomposition should be solved jointly rather than as separate pipelines, because reflective and specular appearance can otherwise be misinterpreted as geometry, leading to distorted surfaces, holes, non-watertight meshes, or oversmoothed reconstructions [2509.22276]. The method extends an explicit Gaussian scene representation with per-Gaussian material parameters and couples geometry supervision, physically based rendering, and a roughness supervision signal derived from multi-view photometric variation. This design is intended to retain the speed and explicitness of Gaussian splatting-based reconstruction while improving robustness to reflective surfaces without introducing large neural components such as MLPs, encoder-decoder networks, pretrained geometric priors, SDF backbones, or tensor factorization modules [2509.22276].

## 1. Problem formulation and conceptual basis

GS-2M is defined around two coupled tasks. The first is **mesh reconstruction**: given multiple images and camera parameters, the goal is to estimate a faithful surface representation that preserves fine geometric detail, smooth regions, and watertightness where appropriate, with a triangle mesh as the final output [2509.22276]. The second is **material decomposition**: the method seeks to decompose appearance into intrinsic material properties, with explicit per-Gaussian albedo $\mathbf{a}_i \in \mathbb{R}^3$ and roughness $\rho_i \in [0,1]$, which are then used in a physically based rendering model [2509.22276].

The rationale for joint optimization follows directly from the ambiguities of the two tasks. Geometry reconstruction methods typically emphasize multi-view consistency and depth or surface regularity, whereas material decomposition methods focus on separating illumination, reflectance, and geometry under a rendering model [2509.22276]. This division becomes problematic on reflective or specular surfaces. If view-dependent highlights are not explicitly explained by reflectance, a reconstruction system may attribute them to surface variation, producing geometric artifacts. Conversely, methods that model material decomposition more thoroughly often rely on sophisticated neural machinery and may not preserve sharp or detailed geometry as effectively [2509.22276].

The method is positioned against two categories of prior work. Relative to 3DGS-based reconstruction methods such as 2DGS, GOF, PGSR, and GausSurf, GS-2M adds physically motivated material parameters and deferred rendering in order to disentangle reflectance from geometry and improve robustness to specular appearance [2509.22276]. Relative to neural inverse rendering pipelines, it avoids SDF backbones, large MLPs, and pretrained priors, trading some decomposition sophistication for scalability and speed while remaining an explicit splatting-based method [2509.22276].

## 2. Representation and rendering architecture

GS-2M represents a scene as a set of anisotropic Gaussians
\[
\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},
\]
where each Gaussian stores a center $\mu_i \in \mathbb{R}^3$, scale $\mathbf{s}_i \in \mathbb{R}^3$, rotation quaternion $\mathbf{q}_i \in \mathbb{R}^4$, opacity $\phi_i$, spherical harmonics appearance coefficients, albedo $\mathbf{a}_i$, and roughness $\rho_i$ [2509.22276]. Geometry is encoded through Gaussian positions, anisotropic covariances, opacities, and supervision on rendered depth and normals, while material decomposition is encoded through albedo and roughness rendered into G-buffers and consumed by a deferred physically based rendering pipeline [2509.22276].

The geometric parameterization follows standard Gaussian splatting practice. If $S_i$ is the diagonal scaling matrix induced by $\mathbf{s}_i$ and $R_i \in \mathrm{SO}(3)$ is the rotation matrix induced by $\mathbf{q}_i$, then the covariance is
\[
\Sigma_i = R_i S_i S_i^\top R_i^\top,
\]
and the Gaussian evaluated at a 3D point $\mathbf{x}\in\mathbb{R}^3$ is
\[
\mathcal{G}_i(\mathbf{x}) = \exp\left(-\frac{1}{2}(\mathbf{x}-\mu_i)^\top \Sigma_i^{-1}(\mathbf{x}-\mu_i)\right).
\]
After projection to image space, alpha compositing is given by
\[
\hat{\mathcal{I}}(\mathbf{p}) = \sum_i T_i \alpha_i \mathbf{c}_i, \qquad T_i = \prod_{j=0}^{i-1}(1-\alpha_j). \tag{1}
\]
Here $\mathbf{p}$ denotes a pixel, $\alpha_i$ the projected opacity contribution of Gaussian $i$, $\mathbf{c}_i$ its view-dependent color, and $T_i$ the transmittance up to Gaussian $i$ [2509.22276].

This representation simultaneously supports rendering of RGB appearance, depth, normals, albedo, and roughness [2509.22276]. The optimized attributes affect the outputs in distinct ways: $\mu_i, \mathbf{s}_i, \mathbf{q}_i, \phi_i$ affect projected splat extent, geometry, visibility, alpha compositing, depth, and normals; spherical harmonics coefficients affect RGB appearance during the bootstrap stage; $\mathbf{a}_i$ affects the rendered albedo $\mathcal{A}$; and $\rho_i$ affects the rendered roughness $\mathcal{R}$, BRDF sharpness, and the weighting of smoothness supervision on normals [2509.22276]. A plausible implication is that the Gaussians serve as a common latent substrate from which both surface and reflectance estimates are produced.

## 3. Geometry reconstruction mechanism

A defining geometric component of GS-2M is its **surface-oriented Gaussian representation**. The method encourages Gaussians to become locally planar by penalizing one scaling component through a plane loss $\mathcal{L}_{\mathrm{plane}}$, so that the shortest scale axis can be interpreted as the local surface normal [2509.22276]. This orientation is crucial because it turns Gaussians into surface-aligned primitives, supports unbiased plane-depth rendering, and provides a normal estimate directly from Gaussian parameters [2509.22276].

Instead of blending camera-space $z$-depth, GS-2M uses **unbiased plane depth rendering**, inherited from PGSR [2509.22276]. For each Gaussian, its normal $\mathbf{n}_i$ is defined as the orientation axis corresponding to the shortest scaling direction, normalized and flipped toward the camera. Let $\mu'_i$ be the Gaussian center transformed into camera space. The distance from the camera to the hypothetical local tangent plane is
\[
\bar{d}_i = \mu'_i \cdot (R_i \mathbf{n}_i).
\]
By alpha-blending these values one obtains a distance map $\bar{\mathcal{D}}$ and, similarly, a normal map $\mathcal{N}$. The unbiased depth is then
\[
\mathcal{D}(\mathbf{p}) = \frac{\bar{\mathcal{D}}(\mathbf{p})}{\mathcal{N}(\mathbf{p}) \cdot \left(K^{-1}\tilde{\mathbf{p}}\right)}. \tag{2}
\]
Here $K\in\mathbb{R}^{3\times 3}$ is the camera intrinsic matrix and $\tilde{\mathbf{p}}=[\mathbf{p},1]^\top$ is the homogeneous pixel coordinate [2509.22276]. The method explicitly states that this yields cleaner depth and normals than naive $z$-depth blending, especially for slanted surfaces [2509.22276].

Geometry optimization combines several constraints: unbiased plane-based depth rendering, depth-normal consistency, plane regularization, multi-view geometric and photometric constraints, multi-view normal consistency, and occlusion-aware correspondence filtering [2509.22276]. The multi-view loss $\mathcal{L}_{\mathrm{mv}}$ is a weighted sum of a multi-view geometric term $L_g$ and a multi-view photometric term $L_p$,
\[
\mathcal{L}_{\mathrm{mv}} = \lambda_g L_g + \lambda_p L_p,
\]
with the geometric term enhanced by adding multi-view normal consistency across corresponding world-space points [2509.22276]. Occlusion-aware filtering discards correspondences if a back-projected 3D point is occluded in the neighboring view, determined by comparing rendered neighbor-view depth with the point’s camera-space $z$-depth [2509.22276]. The paper contrasts this with empirical reprojection-threshold rejection and reports that explicit visibility checking avoids invalid supervision [2509.22276].

After optimization, GS-2M extracts a triangle mesh by rendering color and depth from all training viewpoints, optionally filtering depth by masks, fusing all RGB-D frames into a TSDF volume using Open3D, extracting a mesh with marching cubes, and post-processing by keeping only one connected cluster [2509.22276]. The resulting polygon mesh has RGB color baked to vertices. Mesh extraction is strictly post-training; the paper explicitly states that it does not use a differentiable mesh extraction module during training [2509.22276].

## 4. Material decomposition and physically based rendering

GS-2M explicitly estimates albedo $\mathbf{a}_i\in\mathbb{R}^3$ and roughness $\rho_i\in[0,1]$, and approximates metallic through
\[
\mathcal{M}=1-\mathcal{R},
\]
where $\mathcal{R}$ is the rendered roughness map [2509.22276]. The rendered decomposition outputs include at least an albedo map, roughness map, specular appearance, and environment lighting visualization [2509.22276].

The rendering model is a deferred PBR pipeline based on a **Cook–Torrance microfacet BRDF** with a differential environment cubemap [2509.22276]. Albedo and roughness are alpha-blended from per-Gaussian values into G-buffers for $\mathcal{A}$, $\mathcal{R}$, $\mathcal{N}$, and $\mathcal{D}$, then composited using environment lighting to produce a rendered image $\bar{\mathcal{I}}$ [2509.22276]. The PBR photometric loss is described analogously to the RGB photometric loss as a combination of pixel-wise $L_1$ and SSIM between $\bar{\mathcal{I}}$ and the ground-truth image $\mathcal{I}$:
\[
\mathcal{L}_{\mathrm{pbr}} = \lambda_{L_1}\|\bar{\mathcal{I}}-\mathcal{I}\|_1 + \lambda_{\mathrm{ssim}}\mathcal{L}_{\mathrm{SSIM}}(\bar{\mathcal{I}}, \mathcal{I}).
\]
The exact weights are not given in the main text [2509.22276].

Lighting is represented with a differential environment cubemap and mipmapped prefiltering: diffuse lighting is sampled from the base cubemap level, specular lighting uses roughness-dependent mip levels, and a 2D LUT stores BRDF response terms via the split-sum approximation [2509.22276]. The paper notes that this follows common real-time PBR practice and is best suited to object-centric scenes; it is less favorable for unbounded scenes and cannot handle self-reflection or indirect effects faithfully [2509.22276].

The intended disentanglement between geometry and materials is grounded in multi-view behavior. Geometry should remain view consistent, whereas material-induced specular effects vary with viewpoint [2509.22276]. GS-2M leverages this distinction by combining multi-view geometric loss, NCC-based roughness supervision, and PBR rendering so that viewpoint-sensitive appearance can be explained by low-roughness reflectance rather than geometric inconsistency [2509.22276]. This suggests that the BRDF model is not merely a rendering add-on but a structural mechanism for preventing geometry from absorbing specular artifacts.

## 5. NCC-based roughness supervision

The most distinctive component of GS-2M is its **roughness supervision strategy based on multi-view photometric variation** [2509.22276]. The underlying heuristic is that diffuse surfaces remain relatively stable across nearby views after geometric warping, whereas specular or reflective surfaces exhibit strong view-dependent changes because reflected environment content shifts with viewpoint [2509.22276]. High photometric variation across warped multi-view patches is therefore treated as evidence of low roughness, while low variation is treated as evidence of high roughness [2509.22276].

For a sampled reference pixel $\mathbf{p}$, the method takes a $3\times 3$ grayscale patch $\mathcal{P}_{\mathrm r}$ in the reference image, warps it to a neighboring view to obtain $\hat{\mathcal{P}}_{\mathrm n}$, and computes an NCC error
\[
L_{\mathrm{NCC}}(\mathbf{p}) = 1 - \frac{ \sum_{p} \left(\mathcal{P}_{\mathrm r}(p)-\mu_{\mathcal{P}_{\mathrm r}}\right) \left(\hat{\mathcal{P}}_{\mathrm n}(p)-\mu_{\hat{\mathcal{P}}_{\mathrm n}}\right) }{ \sqrt{\sum_p \left(\mathcal{P}_{\mathrm r}(p)-\mu_{\mathcal{P}_{\mathrm r}}\right)^2} \sqrt{\sum_p \left(\hat{\mathcal{P}}_{\mathrm n}(p)-\mu_{\hat{\mathcal{P}}_{\mathrm n}}\right)^2} }. \tag{3}
\]
Here $\mu_{\mathcal{P}_{\mathrm r}}$ and $\mu_{\hat{\mathcal{P}}_{\mathrm n}}$ are patch means [2509.22276]. Small $L_{\mathrm{NCC}}$ indicates cross-view consistency suggestive of diffuse behavior, whereas large $L_{\mathrm{NCC}}$ indicates viewpoint-sensitive behavior suggestive of specularity or reflectivity [2509.22276].

Given a rendered roughness map $\mathcal{R}$, the roughness supervision loss is
\[
\mathcal{L}_{\mathrm{ro}} = \frac{1}{|\mathcal{R}|} \sum_{\mathbf{p}} \tanh\Big(k_{\mathrm{ro}}\big(L_{\mathrm{NCC}}(\mathbf{p}) - \lambda_{\mathrm{ref}}\big)\Big) \mathcal{R}(\mathbf{p}). \tag{4}
\]
The parameter $k_{\mathrm{ro}}$ controls transition sharpness, and the paper uses $k_{\mathrm{ro}}=8.0$ [2509.22276]. The threshold $\lambda_{\mathrm{ref}}$ is treated as an optimization parameter or manually tuned hyperparameter depending on scene type; for reflective objects the paper uses $\lambda_{\mathrm{ref}}<1.0$, otherwise $1.0$ [2509.22276]. The intended effect is explicitly described as follows: high photometric variation implies smooth or specular surface behavior and thus lower roughness, while low photometric variation implies diffuse behavior and thus higher roughness [2509.22276].

To address instability in textureless regions, GS-2M computes a patch standard deviation
\[
\sigma_r(\mathbf{p}) = \sqrt{ \sum_p \left(\mathcal{P}_{\mathrm r}(p)-\mu_{\mathcal{P}_{\mathrm r}}\right)^2 },
\]
and if $\sigma_r(\mathbf{p}) < 0.01$, replaces raw grayscale NCC with NCC computed on gradient-based patches [2509.22276]. This is intended to avoid incorrectly labeling flat, textureless diffuse regions as reflective [2509.22276].

The roughness signal also influences geometry. A total variation loss $\mathcal{L}_{\mathrm{tv}}$ is applied to the rendered normal map $\mathcal{N}$, with weights derived from roughness $\mathcal{R}$ and detached from the graph; smooth regions predicted as specular or smooth receive stronger normal smoothing, which helps suppress geometry artifacts arising from reflections [2509.22276]. Together with depth-normal consistency $\mathcal{L}_{\mathrm{dn}}$, this couples reflectance estimation back into shape recovery [2509.22276]. The paper explicitly frames this as a mechanism by which roughness supervision improves mesh reconstruction, not only material maps [2509.22276].

## 6. Optimization schedule, objective, and implementation

GS-2M uses a **two-stage optimization schedule** [2509.22276]. In the **bootstrap stage**, the first **5,000 iterations** activate only $\mathcal{L}_{\mathrm{rgb}}$, $\mathcal{L}_{\mathrm{plane}}$, and $\mathcal{L}_{\mathrm{alpha}}$ if masks exist [2509.22276]. This stage is intended to stabilize the initial Gaussian representation before introducing PBR and BRDF variables [2509.22276]. The bootstrap objective is summarized as
\[
\mathcal{L}_{\mathrm{boot}} = \mathcal{L}_{\mathrm{rgb}} + \lambda_{\mathrm{plane}}\mathcal{L}_{\mathrm{plane}} + \lambda_{\mathrm{alpha}}\mathcal{L}_{\mathrm{alpha}},
\]
with $\mathcal{L}_{\mathrm{alpha}}$ present only when masks are available [2509.22276].

In the **joint stage**, $\mathcal{L}_{\mathrm{rgb}}$ is replaced by $\mathcal{L}_{\mathrm{pbr}}$, all geometry and material losses are activated, and roughness learning proceeds jointly with geometry [2509.22276]. Training runs up to **30,000 iterations** total [2509.22276]. A faithful summary of the full objective is
\[
\mathcal{L}_{\mathrm{total}} = \lambda_{\mathrm{pbr}}\mathcal{L}_{\mathrm{pbr}} + \lambda_{\mathrm{plane}}\mathcal{L}_{\mathrm{plane}} + \lambda_{\mathrm{dn}}\mathcal{L}_{\mathrm{dn}} + \lambda_{\mathrm{mv}}\mathcal{L}_{\mathrm{mv}} + \lambda_{\mathrm{ro}}\mathcal{L}_{\mathrm{ro}} + \lambda_{\mathrm{tv}}\mathcal{L}_{\mathrm{tv}} + \lambda_{\mathrm{sm}}\mathcal{L}_{\mathrm{sm}} + \lambda_{\mathrm{alpha}}\mathcal{L}_{\mathrm{alpha}},
\]
though exact coefficients are not specified in the main text [2509.22276]. The paper describes $\mathcal{L}_{\mathrm{sm}}$ as a BRDF smoothness regularizer and $\mathcal{L}_{\mathrm{alpha}}$ as a binary cross-entropy loss between rendered alpha and the ground-truth foreground mask when masks are available [2509.22276].

The implementation inherits standard 3DGS-style learnable Gaussian parameters and the unmodified adaptive density control (ADC) of 3DGS [2509.22276]. A limitation explicitly noted is that in large unbounded scenes this can generate too many Gaussians and cause out-of-memory issues [2509.22276].

The reported implementation details are summarized below.

| Item | Reported value or choice |
|---|---|
| Hardware | single RTX 4090 (24GB) |
| Bootstrap length | 5,000 iterations |
| Maximum training length | 30,000 iterations |
| Roughness patch size | $3\times 3$ |
| Roughness loss slope | $k_{\mathrm{ro}}=8.0$ |
| Textureless-patch threshold | $0.01$ standard deviation |
| Lighting | differential environment cubemap lighting with prefiltered mip levels and LUT-based split-sum approximation |
| Mesh extraction | TSDF fusion and marching cubes in Open3D |

Runtime on DTU is reported as about **22.4 min** for **Ours w/o BRDF** and about **51.0 min** for **Ours** [2509.22276]. The paper attributes the near doubling of runtime mainly to deferred PBR rendering and overhead from $\mathcal{L}_{\mathrm{ro}}$, which the authors describe as under-optimized [2509.22276].

## 7. Empirical evaluation, contributions, and limitations

The evaluation uses **DTU**, **Shiny Blender Synthetic**, and **Tanks and Temples (TnT)** [2509.22276]. DTU is used for geometry evaluation by Chamfer Distance and for novel-view synthesis by PSNR; Shiny Blender Synthetic is used mainly for qualitative evaluation on reflective surface reconstruction; and TnT is used to test scalability in larger unbounded settings through F1 mesh metrics on Barn and Truck [2509.22276]. On TnT, only the **w/o BRDF** variant is evaluated because the PBR setup is object-centric [2509.22276].

On DTU geometry, Table 1 reports mean Chamfer Distance values of **0.51** for **Ours w/o BRDF** and **0.53** for **Ours**, compared with **0.52** for **PGSR** and **0.52** for **GausSurf** [2509.22276]. The interpretation given in the paper is that joint BRDF optimization does not significantly degrade reconstruction quality, and that the reconstruction-only variant slightly outperforms prior state-of-the-art explicit methods on average [2509.22276]. On DTU novel-view synthesis, Table 2 reports mean PSNR values of **34.22** for **Ours w/o BRDF**, **33.86** for **Ours**, **33.33** for **PGSR**, and **33.84** for **Neuralangelo** [2509.22276]. The paper attributes the slight PSNR drop in the full model to noisy optimization of albedo and environment lighting, which remain under-constrained [2509.22276].

On Tanks and Temples, Table 3 reports F1-scores for the **w/o BRDF** variant: Barn **0.57**, Truck **0.67**, and mean **0.62**, compared with PGSR at **0.66** mean and Neuralangelo at **0.59** mean [2509.22276]. The method is only tested on two TnT scenes due to memory limits [2509.22276]. This suggests that the modified multi-view constraints remain competitive in larger settings, though the paper does not extend the full PBR configuration to those scenes.

Qualitative results on Shiny Blender Synthetic are presented as a major practical finding. Compared with 2DGS, GOF, and PGSR, prior methods often produce distorted or non-watertight meshes on shiny objects and sacrifice geometry to explain specular highlights, whereas GS-2M is reported to yield smoother, more uniform, more plausible surfaces [2509.22276]. The paper specifically attributes this behavior to the combination of $\mathcal{L}_{\mathrm{ro}}$, roughness-weighted $\mathcal{L}_{\mathrm{tv}}$, and $\mathcal{L}_{\mathrm{dn}}$ [2509.22276].

Ablation results further isolate the method’s components. In Table 4, **Ours w/o mv normal** gives CD **0.58** and PSNR **26.73**, **Ours w/o filtering** gives CD **0.53** and PSNR **33.76**, and **Ours** gives CD **0.53** and PSNR **33.86** [2509.22276]. The paper interprets these results as showing that multi-view normal consistency gives a major improvement and occlusion-aware filtering a smaller but positive novel-view synthesis gain [2509.22276]. A qualitative ablation shows that removing $\mathcal{L}_{\mathrm{ro}}$ causes noisy environment lighting, albedo contaminated by highlights, poorer roughness maps, and specular bleeding into intrinsic color, indicating that roughness supervision is essential for decomposition quality [2509.22276].

The paper explicitly claims three principal contributions: a unified 3DGS framework for joint mesh reconstruction and material decomposition from multi-view images; a roughness supervision strategy based on multi-view photometric variation that avoids neural priors or components; and improved multi-view geometry constraints through occlusion-aware filtering and multi-view normal consistency, which also improve novel-view synthesis quality [2509.22276]. It also identifies several limitations: under-constrained albedo and lighting, inability to faithfully model self-reflection and self-shadowing, a crude metallic approximation via $m = 1-\rho$, scalability issues from unchanged 3DGS densification in unbounded scenes, and object-centric assumptions in the PBR and environment-lighting setup [2509.22276].

Taken together, GS-2M can be understood as a compromise between the efficiency and explicitness of 3D Gaussian Splatting and the physical plausibility of inverse rendering [2509.22276]. Its central technical insight is that reflective appearance should be absorbed by material and lighting variables rather than geometry, and that a lightweight multi-view photometric heuristic can provide enough supervision on roughness to make this feasible without heavy neural backbones [2509.22276].

Source: https://www.emergentmind.com/topics/gs-2m