GS-2M: Joint Mesh & Material Decomposition

• GS-2M jointly optimizes mesh reconstruction and material decomposition to accurately recover geometry and material properties from calibrated multi-view images.
• The method extends 3D Gaussian splatting with per-Gaussian material parameters, enabling physically based rendering without heavy neural components.
• A novel roughness supervision strategy based on multi-view photometric variation decouples specular highlights from geometric artifacts for improved reconstruction.

GS-2M, short for Gaussian Splatting for Joint Mesh Reconstruction and Material Decomposition, is a method for recovering from calibrated multi-view images both accurate 3D geometry and material properties, notably albedo and roughness, within a single optimization framework built on 3D Gaussian Splatting (Nguyen et al., 26 Sep 2025). Its central premise is that mesh reconstruction and material decomposition should be solved jointly rather than as separate pipelines, because reflective and specular appearance can otherwise be misinterpreted as geometry, leading to distorted surfaces, holes, non-watertight meshes, or oversmoothed reconstructions (Nguyen et al., 26 Sep 2025). The method extends an explicit Gaussian scene representation with per-Gaussian material parameters and couples geometry supervision, physically based rendering, and a roughness supervision signal derived from multi-view photometric variation. This design is intended to retain the speed and explicitness of Gaussian splatting-based reconstruction while improving robustness to reflective surfaces without introducing large neural components such as MLPs, encoder-decoder networks, pretrained geometric priors, SDF backbones, or tensor factorization modules (Nguyen et al., 26 Sep 2025).

1. Problem formulation and conceptual basis

GS-2M is defined around two coupled tasks. The first is mesh reconstruction: given multiple images and camera parameters, the goal is to estimate a faithful surface representation that preserves fine geometric detail, smooth regions, and watertightness where appropriate, with a triangle mesh as the final output (Nguyen et al., 26 Sep 2025). The second is material decomposition: the method seeks to decompose appearance into intrinsic material properties, with explicit per-Gaussian albedo $\mathbf{a}_i \in \mathbb{R}^3$ and roughness $\rho_i \in [0,1]$, which are then used in a physically based rendering model (Nguyen et al., 26 Sep 2025).

The rationale for joint optimization follows directly from the ambiguities of the two tasks. Geometry reconstruction methods typically emphasize multi-view consistency and depth or surface regularity, whereas material decomposition methods focus on separating illumination, reflectance, and geometry under a rendering model (Nguyen et al., 26 Sep 2025). This division becomes problematic on reflective or specular surfaces. If view-dependent highlights are not explicitly explained by reflectance, a reconstruction system may attribute them to surface variation, producing geometric artifacts. Conversely, methods that model material decomposition more thoroughly often rely on sophisticated neural machinery and may not preserve sharp or detailed geometry as effectively (Nguyen et al., 26 Sep 2025).

The method is positioned against two categories of prior work. Relative to 3DGS-based reconstruction methods such as 2DGS, GOF, PGSR, and GausSurf, GS-2M adds physically motivated material parameters and deferred rendering in order to disentangle reflectance from geometry and improve robustness to specular appearance (Nguyen et al., 26 Sep 2025). Relative to neural inverse rendering pipelines, it avoids SDF backbones, large MLPs, and pretrained priors, trading some decomposition sophistication for scalability and speed while remaining an explicit splatting-based method (Nguyen et al., 26 Sep 2025).

2. Representation and rendering architecture

GS-2M represents a scene as a set of anisotropic Gaussians

$$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$

where each Gaussian stores a center $\mu_i \in \mathbb{R}^3$, scale $\mathbf{s}_i \in \mathbb{R}^3$, rotation quaternion $\mathbf{q}_i \in \mathbb{R}^4$, opacity $\phi_i$, spherical harmonics appearance coefficients, albedo $\mathbf{a}_i$, and roughness $\rho_i$ (Nguyen et al., 26 Sep 2025). Geometry is encoded through Gaussian positions, anisotropic covariances, opacities, and supervision on rendered depth and normals, while material decomposition is encoded through albedo and roughness rendered into G-buffers and consumed by a deferred physically based rendering pipeline (Nguyen et al., 26 Sep 2025).

The geometric parameterization follows standard Gaussian splatting practice. If $S_i$ is the diagonal scaling matrix induced by $\rho_i \in [0,1]$0 and $\rho_i \in [0,1]$1 is the rotation matrix induced by $\rho_i \in [0,1]$2, then the covariance is

$\rho_i \in [0,1]$3

and the Gaussian evaluated at a 3D point $\rho_i \in [0,1]$4 is

$\rho_i \in [0,1]$5

After projection to image space, alpha compositing is given by

$\rho_i \in [0,1]$6

Here $\rho_i \in [0,1]$7 denotes a pixel, $\rho_i \in [0,1]$8 the projected opacity contribution of Gaussian $\rho_i \in [0,1]$9, $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$0 its view-dependent color, and $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$1 the transmittance up to Gaussian $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$2 (Nguyen et al., 26 Sep 2025).

This representation simultaneously supports rendering of RGB appearance, depth, normals, albedo, and roughness (Nguyen et al., 26 Sep 2025). The optimized attributes affect the outputs in distinct ways: $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$3 affect projected splat extent, geometry, visibility, alpha compositing, depth, and normals; spherical harmonics coefficients affect RGB appearance during the bootstrap stage; $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$4 affects the rendered albedo $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$5; and $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$6 affects the rendered roughness $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$7, BRDF sharpness, and the weighting of smoothness supervision on normals (Nguyen et al., 26 Sep 2025). A plausible implication is that the Gaussians serve as a common latent substrate from which both surface and reflectance estimates are produced.

3. Geometry reconstruction mechanism

A defining geometric component of GS-2M is its surface-oriented Gaussian representation. The method encourages Gaussians to become locally planar by penalizing one scaling component through a plane loss $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$8, so that the shortest scale axis can be interpreted as the local surface normal (Nguyen et al., 26 Sep 2025). This orientation is crucial because it turns Gaussians into surface-aligned primitives, supports unbiased plane-depth rendering, and provides a normal estimate directly from Gaussian parameters (Nguyen et al., 26 Sep 2025).

Instead of blending camera-space $$\{\mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_{n-1}\},$$9-depth, GS-2M uses unbiased plane depth rendering, inherited from PGSR (Nguyen et al., 26 Sep 2025). For each Gaussian, its normal $\mu_i \in \mathbb{R}^3$0 is defined as the orientation axis corresponding to the shortest scaling direction, normalized and flipped toward the camera. Let $\mu_i \in \mathbb{R}^3$1 be the Gaussian center transformed into camera space. The distance from the camera to the hypothetical local tangent plane is

$\mu_i \in \mathbb{R}^3$2

By alpha-blending these values one obtains a distance map $\mu_i \in \mathbb{R}^3$3 and, similarly, a normal map $\mu_i \in \mathbb{R}^3$4. The unbiased depth is then

$\mu_i \in \mathbb{R}^3$5

Here $\mu_i \in \mathbb{R}^3$6 is the camera intrinsic matrix and $\mu_i \in \mathbb{R}^3$7 is the homogeneous pixel coordinate (Nguyen et al., 26 Sep 2025). The method explicitly states that this yields cleaner depth and normals than naive $\mu_i \in \mathbb{R}^3$8-depth blending, especially for slanted surfaces (Nguyen et al., 26 Sep 2025).

Geometry optimization combines several constraints: unbiased plane-based depth rendering, depth-normal consistency, plane regularization, multi-view geometric and photometric constraints, multi-view normal consistency, and occlusion-aware correspondence filtering (Nguyen et al., 26 Sep 2025). The multi-view loss $\mu_i \in \mathbb{R}^3$9 is a weighted sum of a multi-view geometric term $\mathbf{s}_i \in \mathbb{R}^3$0 and a multi-view photometric term $\mathbf{s}_i \in \mathbb{R}^3$1,

$\mathbf{s}_i \in \mathbb{R}^3$2

with the geometric term enhanced by adding multi-view normal consistency across corresponding world-space points (Nguyen et al., 26 Sep 2025). Occlusion-aware filtering discards correspondences if a back-projected 3D point is occluded in the neighboring view, determined by comparing rendered neighbor-view depth with the point’s camera-space $\mathbf{s}_i \in \mathbb{R}^3$3-depth (Nguyen et al., 26 Sep 2025). The paper contrasts this with empirical reprojection-threshold rejection and reports that explicit visibility checking avoids invalid supervision (Nguyen et al., 26 Sep 2025).

After optimization, GS-2M extracts a triangle mesh by rendering color and depth from all training viewpoints, optionally filtering depth by masks, fusing all RGB-D frames into a TSDF volume using Open3D, extracting a mesh with marching cubes, and post-processing by keeping only one connected cluster (Nguyen et al., 26 Sep 2025). The resulting polygon mesh has RGB color baked to vertices. Mesh extraction is strictly post-training; the paper explicitly states that it does not use a differentiable mesh extraction module during training (Nguyen et al., 26 Sep 2025).

4. Material decomposition and physically based rendering

GS-2M explicitly estimates albedo $\mathbf{s}_i \in \mathbb{R}^3$4 and roughness $\mathbf{s}_i \in \mathbb{R}^3$5, and approximates metallic through

$\mathbf{s}_i \in \mathbb{R}^3$6

where $\mathbf{s}_i \in \mathbb{R}^3$7 is the rendered roughness map (Nguyen et al., 26 Sep 2025). The rendered decomposition outputs include at least an albedo map, roughness map, specular appearance, and environment lighting visualization (Nguyen et al., 26 Sep 2025).

The rendering model is a deferred PBR pipeline based on a Cook–Torrance microfacet BRDF with a differential environment cubemap (Nguyen et al., 26 Sep 2025). Albedo and roughness are alpha-blended from per-Gaussian values into G-buffers for $\mathbf{s}_i \in \mathbb{R}^3$8, $\mathbf{s}_i \in \mathbb{R}^3$9, $\mathbf{q}_i \in \mathbb{R}^4$0, and $\mathbf{q}_i \in \mathbb{R}^4$1, then composited using environment lighting to produce a rendered image $\mathbf{q}_i \in \mathbb{R}^4$2 (Nguyen et al., 26 Sep 2025). The PBR photometric loss is described analogously to the RGB photometric loss as a combination of pixel-wise $\mathbf{q}_i \in \mathbb{R}^4$3 and SSIM between $\mathbf{q}_i \in \mathbb{R}^4$4 and the ground-truth image $\mathbf{q}_i \in \mathbb{R}^4$5: $\mathbf{q}_i \in \mathbb{R}^4$6 The exact weights are not given in the main text (Nguyen et al., 26 Sep 2025).

Lighting is represented with a differential environment cubemap and mipmapped prefiltering: diffuse lighting is sampled from the base cubemap level, specular lighting uses roughness-dependent mip levels, and a 2D LUT stores BRDF response terms via the split-sum approximation (Nguyen et al., 26 Sep 2025). The paper notes that this follows common real-time PBR practice and is best suited to object-centric scenes; it is less favorable for unbounded scenes and cannot handle self-reflection or indirect effects faithfully (Nguyen et al., 26 Sep 2025).

The intended disentanglement between geometry and materials is grounded in multi-view behavior. Geometry should remain view consistent, whereas material-induced specular effects vary with viewpoint (Nguyen et al., 26 Sep 2025). GS-2M leverages this distinction by combining multi-view geometric loss, NCC-based roughness supervision, and PBR rendering so that viewpoint-sensitive appearance can be explained by low-roughness reflectance rather than geometric inconsistency (Nguyen et al., 26 Sep 2025). This suggests that the BRDF model is not merely a rendering add-on but a structural mechanism for preventing geometry from absorbing specular artifacts.

5. NCC-based roughness supervision

The most distinctive component of GS-2M is its roughness supervision strategy based on multi-view photometric variation (Nguyen et al., 26 Sep 2025). The underlying heuristic is that diffuse surfaces remain relatively stable across nearby views after geometric warping, whereas specular or reflective surfaces exhibit strong view-dependent changes because reflected environment content shifts with viewpoint (Nguyen et al., 26 Sep 2025). High photometric variation across warped multi-view patches is therefore treated as evidence of low roughness, while low variation is treated as evidence of high roughness (Nguyen et al., 26 Sep 2025).

For a sampled reference pixel $\mathbf{q}_i \in \mathbb{R}^4$7, the method takes a $\mathbf{q}_i \in \mathbb{R}^4$8 grayscale patch $\mathbf{q}_i \in \mathbb{R}^4$9 in the reference image, warps it to a neighboring view to obtain $\phi_i$0, and computes an NCC error

$\phi_i$1

Here $\phi_i$2 and $\phi_i$3 are patch means (Nguyen et al., 26 Sep 2025). Small $\phi_i$4 indicates cross-view consistency suggestive of diffuse behavior, whereas large $\phi_i$5 indicates viewpoint-sensitive behavior suggestive of specularity or reflectivity (Nguyen et al., 26 Sep 2025).

Given a rendered roughness map $\phi_i$6, the roughness supervision loss is

$\phi_i$7

The parameter $\phi_i$8 controls transition sharpness, and the paper uses $\phi_i$9 (Nguyen et al., 26 Sep 2025). The threshold $\mathbf{a}_i$0 is treated as an optimization parameter or manually tuned hyperparameter depending on scene type; for reflective objects the paper uses $\mathbf{a}_i$1, otherwise $\mathbf{a}_i$2 (Nguyen et al., 26 Sep 2025). The intended effect is explicitly described as follows: high photometric variation implies smooth or specular surface behavior and thus lower roughness, while low photometric variation implies diffuse behavior and thus higher roughness (Nguyen et al., 26 Sep 2025).

To address instability in textureless regions, GS-2M computes a patch standard deviation

$\mathbf{a}_i$3

and if $\mathbf{a}_i$4, replaces raw grayscale NCC with NCC computed on gradient-based patches (Nguyen et al., 26 Sep 2025). This is intended to avoid incorrectly labeling flat, textureless diffuse regions as reflective (Nguyen et al., 26 Sep 2025).

The roughness signal also influences geometry. A total variation loss $\mathbf{a}_i$5 is applied to the rendered normal map $\mathbf{a}_i$6, with weights derived from roughness $\mathbf{a}_i$7 and detached from the graph; smooth regions predicted as specular or smooth receive stronger normal smoothing, which helps suppress geometry artifacts arising from reflections (Nguyen et al., 26 Sep 2025). Together with depth-normal consistency $\mathbf{a}_i$8, this couples reflectance estimation back into shape recovery (Nguyen et al., 26 Sep 2025). The paper explicitly frames this as a mechanism by which roughness supervision improves mesh reconstruction, not only material maps (Nguyen et al., 26 Sep 2025).

6. Optimization schedule, objective, and implementation

GS-2M uses a two-stage optimization schedule (Nguyen et al., 26 Sep 2025). In the bootstrap stage, the first 5,000 iterations activate only $\mathbf{a}_i$9, $\rho_i$0, and $\rho_i$1 if masks exist (Nguyen et al., 26 Sep 2025). This stage is intended to stabilize the initial Gaussian representation before introducing PBR and BRDF variables (Nguyen et al., 26 Sep 2025). The bootstrap objective is summarized as

$\rho_i$2

with $\rho_i$3 present only when masks are available (Nguyen et al., 26 Sep 2025).

In the joint stage, $\rho_i$4 is replaced by $\rho_i$5, all geometry and material losses are activated, and roughness learning proceeds jointly with geometry (Nguyen et al., 26 Sep 2025). Training runs up to 30,000 iterations total (Nguyen et al., 26 Sep 2025). A faithful summary of the full objective is

$\rho_i$6

though exact coefficients are not specified in the main text (Nguyen et al., 26 Sep 2025). The paper describes $\rho_i$7 as a BRDF smoothness regularizer and $\rho_i$8 as a binary cross-entropy loss between rendered alpha and the ground-truth foreground mask when masks are available (Nguyen et al., 26 Sep 2025).

The implementation inherits standard 3DGS-style learnable Gaussian parameters and the unmodified adaptive density control (ADC) of 3DGS (Nguyen et al., 26 Sep 2025). A limitation explicitly noted is that in large unbounded scenes this can generate too many Gaussians and cause out-of-memory issues (Nguyen et al., 26 Sep 2025).

The reported implementation details are summarized below.

Item	Reported value or choice
Hardware	single RTX 4090 (24GB)
Bootstrap length	5,000 iterations
Maximum training length	30,000 iterations
Roughness patch size	$\rho_i$9
Roughness loss slope	$S_i$0
Textureless-patch threshold	$S_i$1 standard deviation
Lighting	differential environment cubemap lighting with prefiltered mip levels and LUT-based split-sum approximation
Mesh extraction	TSDF fusion and marching cubes in Open3D

Runtime on DTU is reported as about 22.4 min for Ours w/o BRDF and about 51.0 min for Ours (Nguyen et al., 26 Sep 2025). The paper attributes the near doubling of runtime mainly to deferred PBR rendering and overhead from $S_i$2, which the authors describe as under-optimized (Nguyen et al., 26 Sep 2025).

7. Empirical evaluation, contributions, and limitations

The evaluation uses DTU, Shiny Blender Synthetic, and Tanks and Temples (TnT) (Nguyen et al., 26 Sep 2025). DTU is used for geometry evaluation by Chamfer Distance and for novel-view synthesis by PSNR; Shiny Blender Synthetic is used mainly for qualitative evaluation on reflective surface reconstruction; and TnT is used to test scalability in larger unbounded settings through F1 mesh metrics on Barn and Truck (Nguyen et al., 26 Sep 2025). On TnT, only the w/o BRDF variant is evaluated because the PBR setup is object-centric (Nguyen et al., 26 Sep 2025).

On DTU geometry, Table 1 reports mean Chamfer Distance values of 0.51 for Ours w/o BRDF and 0.53 for Ours, compared with 0.52 for PGSR and 0.52 for GausSurf (Nguyen et al., 26 Sep 2025). The interpretation given in the paper is that joint BRDF optimization does not significantly degrade reconstruction quality, and that the reconstruction-only variant slightly outperforms prior state-of-the-art explicit methods on average (Nguyen et al., 26 Sep 2025). On DTU novel-view synthesis, Table 2 reports mean PSNR values of 34.22 for Ours w/o BRDF, 33.86 for Ours, 33.33 for PGSR, and 33.84 for Neuralangelo (Nguyen et al., 26 Sep 2025). The paper attributes the slight PSNR drop in the full model to noisy optimization of albedo and environment lighting, which remain under-constrained (Nguyen et al., 26 Sep 2025).

On Tanks and Temples, Table 3 reports F1-scores for the w/o BRDF variant: Barn 0.57, Truck 0.67, and mean 0.62, compared with PGSR at 0.66 mean and Neuralangelo at 0.59 mean (Nguyen et al., 26 Sep 2025). The method is only tested on two TnT scenes due to memory limits (Nguyen et al., 26 Sep 2025). This suggests that the modified multi-view constraints remain competitive in larger settings, though the paper does not extend the full PBR configuration to those scenes.

Qualitative results on Shiny Blender Synthetic are presented as a major practical finding. Compared with 2DGS, GOF, and PGSR, prior methods often produce distorted or non-watertight meshes on shiny objects and sacrifice geometry to explain specular highlights, whereas GS-2M is reported to yield smoother, more uniform, more plausible surfaces (Nguyen et al., 26 Sep 2025). The paper specifically attributes this behavior to the combination of $S_i$3, roughness-weighted $S_i$4, and $S_i$5 (Nguyen et al., 26 Sep 2025).

Ablation results further isolate the method’s components. In Table 4, Ours w/o mv normal gives CD 0.58 and PSNR 26.73, Ours w/o filtering gives CD 0.53 and PSNR 33.76, and Ours gives CD 0.53 and PSNR 33.86 (Nguyen et al., 26 Sep 2025). The paper interprets these results as showing that multi-view normal consistency gives a major improvement and occlusion-aware filtering a smaller but positive novel-view synthesis gain (Nguyen et al., 26 Sep 2025). A qualitative ablation shows that removing $S_i$6 causes noisy environment lighting, albedo contaminated by highlights, poorer roughness maps, and specular bleeding into intrinsic color, indicating that roughness supervision is essential for decomposition quality (Nguyen et al., 26 Sep 2025).

The paper explicitly claims three principal contributions: a unified 3DGS framework for joint mesh reconstruction and material decomposition from multi-view images; a roughness supervision strategy based on multi-view photometric variation that avoids neural priors or components; and improved multi-view geometry constraints through occlusion-aware filtering and multi-view normal consistency, which also improve novel-view synthesis quality (Nguyen et al., 26 Sep 2025). It also identifies several limitations: under-constrained albedo and lighting, inability to faithfully model self-reflection and self-shadowing, a crude metallic approximation via $S_i$7, scalability issues from unchanged 3DGS densification in unbounded scenes, and object-centric assumptions in the PBR and environment-lighting setup (Nguyen et al., 26 Sep 2025).

Taken together, GS-2M can be understood as a compromise between the efficiency and explicitness of 3D Gaussian Splatting and the physical plausibility of inverse rendering (Nguyen et al., 26 Sep 2025). Its central technical insight is that reflective appearance should be absorbed by material and lighting variables rather than geometry, and that a lightweight multi-view photometric heuristic can provide enough supervision on roughness to make this feasible without heavy neural backbones (Nguyen et al., 26 Sep 2025).