Normal-GS: Normals in 3D Gaussian Splatting

• Normal-GS is a novel method for 3D Gaussian Splatting that natively incorporates surface normals to enhance both image fidelity and geometric accuracy.
• It employs anchor-based IDIV encoding and IDE for specular effects, enabling efficient lighting simulation and physically-based rendering.
• The approach leverages specialized loss functions and gradient propagation to optimize surface normals, improving photometric metrics and reconstruction precision.

Normal-GS refers to algorithms and representations in 3D Gaussian Splatting (3DGS) that explicitly integrate surface normal vectors into the rendering, reconstruction, and optimization process to achieve both high-fidelity image synthesis and physically accurate surface geometry. While classic 3DGS employs anisotropic Gaussian primitives for real-time view synthesis, the addition of normal-guidance mechanisms fundamentally changes both representation and optimization pipelines, offering improved geometric precision and material-aware rendering. The recent Normal-GS method (Wei et al., 2024) is the first to make normal integration intrinsic in 3DGS rendering, enabling direct connection between surface orientation, appearance, and lighting. This paradigm now anchors many state-of-the-art approaches for differentiable rendering, scene reconstruction, and physically-based editing.

1. Standard 3D Gaussian Splatting and Limitations

3D Gaussian Splatting encodes a scene as a collection of anisotropic Gaussians $\mathcal G(p;\mu,\Sigma)$, where each Gaussian has center $\mu \in \mathbb{R}^3$, covariance $\Sigma$, per-Gaussian opacity $\alpha$, and color parameterized by Spherical Harmonics (SH). Rendering proceeds via projecting each $\mathcal G$ to the image plane, yielding $G'(x)$, followed by depth-sorted, front-to-back $\alpha$-blending: $$\sigma_i(x) = \alpha_i G'_i(x), \qquad C(x) = \sum_{i\in N(x)} c_i\,\sigma_i(x) \prod_{j<i}[1-\sigma_j(x)]$$ In this standard formulation, colors are treated as independent, intrinsic attributes. Surface normals do not appear in the forward rendering pass; any “normal” is either heuristically read off by selecting the shortest ellipsoid axis (i.e., minimal eigenvector of $\Sigma$) or reconstructed post hoc. This disconnection leads to a fundamental tension in prior work: enforcing geometric accuracy (e.g., normal/depth losses) undermines photometric fidelity, while richer appearance models (e.g., improved BRDF) do not feed back to surface estimation (Wei et al., 2024).

2. Physically-Based Rendering in Normal-GS

Normal-GS integrates surface normals $\mathbf{n}$ into the rendering equation by explicitly parameterizing color as a function of both normal and incoming lighting. The rendering equation (Kajiya 1986) describes the outgoing radiance at point $x$ with normal $\mathbf{n}$ under view direction $\omega_o$ as: $$L_{\mathrm{out}}(x, \omega_o) = L_E(x, \omega_o) + \int_{\Omega^+} L_i(x, \omega_i) f_r(\mathbf{n}, \omega_i, \omega_o) \max(0, \mathbf{n} \cdot \omega_i) d\omega_i$$ For Lambertian surfaces, this reduces to: $$L_D(x) = k_D(x) \int_{\Omega^+} L_i(x, \omega_i) (\mathbf{n} \cdot \omega_i) d\omega_i$$ Normal-GS re-parameterizes this as: $L_D(x) = k_D(x) [\mathbf{n}(x) \cdot \ell(x)]$ where the Integrated Directional Illumination Vector (IDIV) $$\ell(x) \equiv \int_{\Omega^+} L_i(x,\omega_i)\,\omega_i\,d\omega_i$$. This allows the forward and backward passes to directly propagate gradients from photometric loss into $\mathbf{n}$—making normals first-class, optimizable entities (Wei et al., 2024).

3. Efficient Encoding: Anchor-Based IDIV and IDE for Specular Effects

Storing a per-Gaussian $\ell$ is memory-prohibitive. Normal-GS adopts a spatially anchored encoding: a sparse set of anchor Gaussians $v$ each possess a low-dimensional feature $f_v \in \mathbb{R}^D$, decoded by an MLP into local IDIVs for affiliated Gaussians. This architecture regularizes illumination, ensuring spatial smoothness and memory efficiency. For view-dependent specular effects, Normal-GS employs Integrated Directional Encoding (IDE), which encodes the specular reflection direction $$\omega_r = 2(\mathbf{n}\cdot\omega_o)\mathbf{n} - \omega_o$$ via either Fourier or spherical harmonic bases, with the result fed into a neural specular predictor (Wei et al., 2024).

The per-Gaussian color thus becomes: $c_i(x) = L_{D,i}(x) + L_{S,i}(x)$ where $L_{D,i}$ is the anchor-IDIV-driven diffuse term and $L_{S,i}$ is the IDE-MLP-driven specular component. This design enables scene-level consistency of lighting and material effects while enabling per-splat normal optimization.

4. Loss Functions and Normal Self-Regularization

Normal-GS employs a composite loss composed of photometric reconstruction, volume regularization, and a specifically designed normal self-regularization term: $$\mathcal{L}_N = \mathbb{E}_x\left[1 - N(x) \cdot N_D(x)\right]$$ where $N(x)$ is the rendered normal-map and $N_D(x)$ is the normal derived from finite-differences of the depth map. This gently constrains the learned surface normals of the ellipsoid axes to align with the observed geometric normal without excessively penalizing rendering quality. The overall objective is: $$\mathcal{L} = \mathcal{L}_P + \lambda_{\text{vol}}\mathcal{L}_{\text{vol}} + \lambda_N\mathcal{L}_N$$ with $\lambda_N \approx 10^{-2}$ in practice and photometric loss warm-started to allow normals and depths to stabilize first.

5. Pipeline and Implementation Details

The end-to-end pipeline proceeds as follows:

• Scene structure-from-motion provides initial points and camera poses.
• Gaussians are initialized for position, scale (covariance), opacity, color, and anchors determined.
• Training iteratively updates all Gaussian and illumination parameters by differentiable rasterization, with normal-involved color and normal, depth (and optionally specular) buffers splatted per-tile in CUDA.
• During each batch, overlapping Gaussians for every pixel are identified, their normals and IDIV/specular components decoded, and the forward rendering pass completed.
• Losses are computed, and gradients are backpropagated to all parameters.

Rendering speed remains real-time ( >30 Hz), with practical memory usage compatible with scenes containing up to millions of Gaussians (Wei et al., 2024).

6. Empirical Performance and Comparative Analysis

Normal-GS achieves strong performance on both geometric and photometric metrics. On the NeRF Synthetic dataset, it attains a normal mean angular error of ∼20.7°, outperforming 3DGS, Scaffold-GS, GaussianShader, and SpecGaussian (which range from ∼23° to 46°), while matching or exceeding state-of-the-art in PSNR, SSIM, and LPIPS for novel-view synthesis. The integrated normals capture local lighting, complex specularities, and do not suffer from the observed trade-off in previous works between rendering and geometry. Training typically converges in ∼30k iterations, and rendering remains real time on commodity GPUs (Wei et al., 2024).

7. Extensions, Variants, and the Broader Normal-GS Paradigm

The Normal-GS paradigm generalizes to a family of approaches in 3DGS that leverage normals for improved geometry and appearance:

• GSSR employs per-Gaussian normal alignment, pixel-level and Gaussian-level normal/photometric supervision, opacity pruning, and periodic resampling to enforce uniform spatial coverage and tight normal-to-surface correspondence for accurate reconstruction and scene editing. Its losses ensure that each Gaussian’s normal aligns with the underlying surface rather than only the principal axis of the ellipsoid (Huang et al., 25 Jul 2025).
• GS-IR introduces a depth-derivation-based regularization where normals are obtained from local depth gradients, and penalizes the deviation of per-Gaussian normal predictions from these estimates in the context of inverse rendering and physically-based relighting (Liang et al., 2023).
• RaDe-GS derives closed-form depth and normals per Gaussian and writes them to buffers during rasterization, enabling direct geometric normal computation at minimal cost and improving fidelity over post-hoc normal estimation (Zhang et al., 2024).

Collectively, these methods reinforce the principle that endowing 3D Gaussian Splatting representations with intrinsic, optimized normal fields—directly coupled to lighting and appearance—enables highly accurate, editable, and physically-grounded 3D reconstruction and synthesis.