DN-Splatter: Depth & Normal Priors for 3D Reconstruction

• DN-Splatter is a 3D scene reconstruction framework that integrates depth and normal priors to enhance Gaussian splatting for realistic indoor scenes.
• It employs photometric, depth, and normal supervision with a multi-stage optimization pipeline to achieve state-of-the-art geometric accuracy and real-time rendering.
• The method converts rendered depth and normal cues into watertight triangle meshes via Poisson reconstruction, outperforming previous methods on metrics like Chamfer-L₁ and F-score.

DN-Splatter (Depth and Normal Priors for Gaussian Splatting and Meshing) is a 3D scene reconstruction framework designed to address the limitations of 3D Gaussian splatting for novel view synthesis and mesh extraction, particularly in the context of challenging indoor environments. By incorporating geometric priors—specifically, depth and surface normal cues—DN-Splatter enhances the physical plausibility and fidelity of Gaussian-based reconstructions and enables efficient conversion to watertight triangle meshes. The method leverages photometric, depth, and normal supervision, utilizing sensor data or monocular predictions, and achieves state-of-the-art geometric accuracy while maintaining the real-time efficiency of 3D Gaussian splatting (Turkulainen et al., 2024).

1. Scene Representation and Rendering Pipeline

DN-Splatter models a scene via a set of $N$ 3D Gaussian primitives, each parameterized by:

• $\mu_i \in \mathbb{R}^3$: mean (geometry center)
• $\Sigma_i \in \mathbb{R}^{3 \times 3}$: covariance, decomposed as $s_i \in \mathbb{R}^3$ (axis lengths) and $q_i \in \mathbb{R}^4$ (orientation quaternion)
• $o_i \in \mathbb{R}$: opacity (density scaling)
• $c_i \in \mathbb{R}^3$: color via spherical harmonics

Rendering proceeds by projecting each 3D Gaussian onto the image plane, yielding a 2D Gaussian. For a pixel $p$ along a camera ray, the alpha contribution is computed as:

$$\alpha_i(p) = o_i \exp\left(\frac{1}{2}(p - \mu_i)^\top \Sigma_i^{-1}(p - \mu_i)\right)$$

Gaussians are sorted along the viewing direction by mean $z$-depth and composited in a front-to-back order using discrete volume rendering:

$\mu_i \in \mathbb{R}^3$0

2. Loss Functions and Geometric Regularization

The DN-Splatter loss function is a weighted sum of photometric, depth, and normal terms, with additional smoothness regularization:

$\mu_i \in \mathbb{R}^3$1

• $\mu_i \in \mathbb{R}^3$2: per-pixel photometric loss between rendered and observed color
• Depth rendering (approximate):

$\mu_i \in \mathbb{R}^3$3

with $\mu_i \in \mathbb{R}^3$4

• Gradient-aware, logarithmic depth loss:

$\mu_i \in \mathbb{R}^3$5

• Depth smoothness (total variation):

$\mu_i \in \mathbb{R}^3$6

• Gaussian-based surface normals:

$\mu_i \in \mathbb{R}^3$7

Rendered per-pixel normal:

$\mu_i \in \mathbb{R}^3$8

• Normal supervision (L1 loss):

$\mu_i \in \mathbb{R}^3$9

• Normal smoothness (total variation):

$\Sigma_i \in \mathbb{R}^{3 \times 3}$0

Typical weights: $\Sigma_i \in \mathbb{R}^{3 \times 3}$1, $\Sigma_i \in \mathbb{R}^{3 \times 3}$2, $\Sigma_i \in \mathbb{R}^{3 \times 3}$3.

3. Integration of Monocular Depth and Surface Normal Priors

DN-Splatter enhances geometric supervision by incorporating monocular depth and normal prediction when sensor data is absent.

• Monocular depth (e.g., ZoeDepth, DepthAnything) outputs are aligned to sparse SfM depths via affine scale and shift:

$\Sigma_i \in \mathbb{R}^{3 \times 3}$4

• The aligned monocular depth $\Sigma_i \in \mathbb{R}^{3 \times 3}$5 serves as the ground truth for $\Sigma_i \in \mathbb{R}^{3 \times 3}$6.
• Surface normal priors are produced with a pretrained Omnidata network, providing pseudo-ground truth $\Sigma_i \in \mathbb{R}^{3 \times 3}$7 for normal supervision.

This suggests DN-Splatter is robust to the lack of dense sensor depth and can utilize learned scene priors from monocular predictors without additional hardware.

4. Training Pipeline and Optimization

The DN-Splatter optimization follows a multi-stage iterative protocol:

1. Input: calibrated RGB images, poses, optionally sensor depths.
2. Initialization via COLMAP SfM:
   • $\Sigma_i \in \mathbb{R}^{3 \times 3}$8: 3D points, $\Sigma_i \in \mathbb{R}^{3 \times 3}$9: nearest pixel colors, $s_i \in \mathbb{R}^3$0: average neighbor distance, $s_i \in \mathbb{R}^3$1: local normals.
3. Precompute monocular depths and normals; align as required.
4. Iterative optimization loop:
   • Randomly sample a training view.
   • Render $s_i \in \mathbb{R}^3$2, $s_i \in \mathbb{R}^3$3, $s_i \in \mathbb{R}^3$4.
   • Compute all loss terms.
   • Backpropagate and update all Gaussian parameters ($s_i \in \mathbb{R}^3$5) using Adam.
   • Maintenance: periodically cull low-opacity Gaussians, split high-gradient Gaussians, and optionally adapt learning rate.

5. Mesh Extraction from Gaussian Representations

DN-Splatter bypasses the need for explicit signed distance fields or occupancy grids for mesh extraction. Instead, the system back-projects rendered depth and normal maps from all training views into world coordinates, generating a set of oriented 3D points $s_i \in \mathbb{R}^3$6. These points are then fused into a watertight triangle mesh via a single Poisson surface reconstruction:

$s_i \in \mathbb{R}^3$7

A plausible implication is that this approach retains physical correctness while enabling direct meshable representations, even in textureless scenes.

6. Experimental Results and Comparative Performance

Evaluation on multiple indoor datasets (MuSHRoom, ScanNet++, Replica, Tanks & Temples) demonstrates notable improvements in both view synthesis and geometric accuracy.

Method	Chamfer-L₁ ↓	F-score ↑
Splatfacto	4.72	71.84
SuGaR-coarse	3.90	80.18
SuGaR-refined	3.94	79.39
DN-Splatter (ours)	1.94	93.10

Additional key results:

• Novel view synthesis on MuSHRoom: PSNR=24.40, SSIM=0.8424, LPIPS=0.1338 for DN-Splatter (vs. Splatfacto: 24.21/0.8375/0.1421).
• Depth metrics (MuSHRoom): Abs Rel reduced from 0.0826 (Splatfacto) to 0.0294, RMSE log from 0.0956 to 0.0300.
• ScanNet++ (mesh metric): Chamfer-L₁=6.80 (vs. 17.19 Splatfacto), F-score=69.41 (vs. 17.90).

DN-Splatter achieves competitive novel-view synthesis performance while producing meshes with substantially lower Chamfer-L₁ distance and higher F-score compared to Splatfacto and SuGaR variants. When compared to SDF-based (MonoSDF) and NeRF-based (Depth-Nerfacto) techniques, DN-Splatter is competitive on photometric metrics, significantly superior on geometric metrics, and retains the real-time rendering and fast training (∼30 k iterations) characteristic of Gaussian-based methods.

7. Advantages, Limitations, and Future Directions

Key advantages include:

• Enforcing real-world depth and normal priors during Gaussian splatting
• Avoiding complex SDF or occupancy grids while supporting direct mesh extraction
• Fast training and real-time inference

Limitations:

• Depth rendering is approximate due to global Gaussian sorting
• Performance relies on moderately dense camera trajectories; highly dynamic or sparse captures are more difficult
• Mesh quality remains sensitive to depth noise; the joint optimization of mesh and Gaussians is an open question

This suggests further progress may require integrated mesh-Gaussian optimization and advancements in depth denoising for even higher fidelity reconstructions (Turkulainen et al., 2024).