Normal-Guided Sparse Depth Sampling

• Normal-guided sparse depth sampling is a geometry-aware technique that leverages estimated surface normals to identify and sample reliable depth measurements.
• It employs normal-aligned, reliability-weighted, and combined depth/normal priors to optimize depth completion, 3D mapping, and neural rendering performance.
• Empirical results show reduced errors and improved sensor fidelity, validating its application in RGB-D depth completion, implicit SDF mapping, and NeRF-based scene modeling.

Normal-guided sparse depth sampling encompasses a family of geometry-aware strategies for selecting sparse depth measurements in computer vision and robotics, leveraging estimated surface normals to bias sample selection toward regions of high reliability or informativeness. Unlike uniform or projective sampling protocols, normal-guided approaches focus on the underlying geometry, yielding improved supervisory signals for depth completion, large-scale mapping, and neural rendering frameworks. These techniques have recently driven advances in high-fidelity RGB-D depth completion, implicit neural SDF mapping, and NeRF-based scene modeling (Song et al., 7 Jan 2024, Salloom et al., 9 Dec 2025, Guo et al., 8 Jul 2024).

1. Principles of Normal-Guided Sparse Depth Sampling

Normal-guided sparse depth sampling relies on local surface orientation—typically computed via principal component analysis (PCA) of the local point cloud—to estimate surface normals at each sampled or candidate location. The core motivation is twofold: (1) physical depth sensors exhibit viewpoint- and geometry-dependent reliability, especially degrading at large incidence angles or on highly curved surfaces, and (2) accurate geometric supervision is crucial for learning-based depth completion and mapping models.

The essential mechanisms in normal-guided sampling can be grouped into:

• Normal-aligned sampling: Query points are generated along the estimated normal direction, typically forming a narrow band around the observed surface for more accurate signed distance supervision or geometry priors (Song et al., 7 Jan 2024).
• Normal-weighted reliability: Each surface point is assigned a reliability score based on its normal’s alignment with the sensor’s viewing direction, guiding the distribution from which sparse measurements are drawn; this approach emphasizes planar, front-facing regions (Salloom et al., 9 Dec 2025).
• Combined depth/normal priors: For models such as NeRF, sparse normals are used together with sparse depths to inform completion priors, optimizing both sampling density and loss weighting in geometry-aware ways (Guo et al., 8 Jul 2024).

2. Methodologies for Normal and Reliability Estimation

Surface normals are typically estimated using PCA on local neighborhoods of the depth or point cloud data:

$$\mathbf{p}_i = D(u_i,v_i)\,K^{-1}\,\mathbf{x}_i \in \mathbb{R}^3,\quad \mathcal{N}(i)=\text{neighbors}$$

$$\boldsymbol{\mu}_i = \frac{1}{|\mathcal{N}(i)|} \sum_{j\in\mathcal{N}(i)} \mathbf{p}_j$$

$$\mathbf{C}_i = \frac{1}{|\mathcal{N}(i)|} \sum_{j\in\mathcal{N}(i)} (\mathbf{p}_j - \boldsymbol{\mu}_i) (\mathbf{p}_j - \boldsymbol{\mu}_i)^\top$$

The eigenvector corresponding to the smallest eigenvalue defines the normal $\mathbf{n}_i$ at point $i$.

Reliability for sparse sampling is derived from the normalized dot product between the estimated normal $\mathbf{n}_i$ and the normalized viewing (sensor-to-point) direction $\mathbf{v}_i$, raised to a power parameter $\beta$: $r_i = |\mathbf{n}_i^\top \mathbf{v}_i|^{\beta}$ This term $r_i \in [0,1]$ reflects that points with normals facing the sensor are more likely to yield accurate depth readings.

3. Sampling Strategies and Distributions

3.1 Depth Completion: Geometry-aware Subsampling

The categorical sampling distribution over $N$ candidate points is given by: $P(i) = \frac{r_i}{\sum_{j=1}^{N} r_j}$ $K$ sample indices are drawn without replacement according to $P(i)$ to form the sparse measurement map $D_{\rm sp}(u,v)$. This selects a spatially non-uniform, geometry-aware set of sparse points that concentrates samples in reliable regions, particularly avoiding noisy edges or grazing angles (Salloom et al., 9 Dec 2025).

3.2 Implicit Neural Mapping: Surface-normal Poisson Band

In mapping frameworks such as N$^{3}$-Mapping, normal-guided sampling is performed by generating query points at $x_i(t) = x_i + t n_i$, where $t \sim \mathcal{N}(0, \sigma)\cap[-tr, tr]$. Each sample is labeled with a signed distance target $d_i(t) = -t$, directly encoding the Euclidean offset along the normal direction. Sparse samples in free space (along the sensor ray, outside the truncation band) are also included to enforce SDF boundary conditions (Song et al., 7 Jan 2024).

4. Supervisory Losses and Training Dynamics

Normal-guided sparse depth sampling is tightly coupled to the model losses and training schedules:

• Occupancy-BCE/SDF Loss: For normal-aligned SDF supervision, the Binary Cross-Entropy (BCE) loss between predicted occupancy (from SDF) and ground-truth occupancy is used, together with an Eikonal regularization term enforcing $||\nabla_x f_\theta(x)|| \approx 1$ within the narrow band (Song et al., 7 Jan 2024).
• Diffusion-based Completion Loss: In geometry-aware depth completion, the denoising loss trains a diffusion model conditioned on the RGB image and the sparse depth input. Only the sampling of sparse points changes; loss and architecture remain unchanged (Salloom et al., 9 Dec 2025).
• Hierarchical/Spatially-balanced Batching: In large-scale mapping, hierarchical per-voxel sampling schemes are employed to ensure even spatial coverage and efficient use of computational resources. The hierarchical sampler adaptively downsamples in under-populated voxels, supporting real-time bounded-memory operation (Song et al., 7 Jan 2024).

5. Applications in Mapping, Depth Completion, and Neural Rendering

Table: Diverse Use-Cases of Normal-Guided Sparse Depth Sampling

Application Area	Sampling Scheme	Model Types
3D Mapping (N$^{3}$-Mapping)	Gaussian band along normals	Implicit SDF (MLP + grid)
Depth Completion (Marigold-DC)	PCA-normal reliability-weighted pixels	Diffusion, Denoising UNet
Neural Rendering (CP NeRF)	Sparse depth/normal from SfM, completion	NeRF, completion decoders

In mapping scenarios, normal-guided sampling enables high-fidelity SDF construction, reducing approximation errors compared to projective-distance-based supervision. In diffusion-based depth completion, it creates more realistic, sensor-faithful sparse patterns for training and evaluation, directly improving RMSE and MAE metrics, especially at moderate sparsity (e.g., RMSE decreases by 7.5% at 300 sparse points, Table 1 in (Salloom et al., 9 Dec 2025)). For scene representation learning (NeRF), normal-guided sparse priors are densified and used for guided ray sampling and uncertainty-weighted supervision, resulting in improved rendering from limited viewpoints (Guo et al., 8 Jul 2024).

6. Realism, Benefits, and Empirical Outcomes

Normal-guided selection more closely reflects the spatially heterogeneous reliability of physical RGB-D sensors, striking a balance between informativeness and robustness:

• Realism: By concentrating samples in regions with high normal–viewing alignment, these methods better match true sensor behavior, which degrades at large incidence angles or high-curvature areas. This leads to improved fidelity in downstream tasks and reduces artifacts at object boundaries (Salloom et al., 9 Dec 2025).
• Supervisory Efficiency: Normal-guided patterns reduce the burden on learning-based models to "correct" unreliable or noisy measurements, allowing the models to propagate accurate geometric cues and focus supervision where it is most actionable.
• Empirical Metrics: Geometry-aware sampling yields substantially lower error compared to uniform random sampling, both for RMSE/MAE (as demonstrated on NYU-Depth-v2) and for large-scale mapping accuracy (see N$^{3}$-Mapping's state-of-the-art results).

7. Connections to Broader Geometry-Aware Perception

Normal-guided sparse depth sampling reflects a broader trend in robotic perception and computational imaging toward geometry-aware data selection and supervision. Similar PCA-based normal estimation underpins robust surface modeling, while reliability-driven subsampling is critical for simulation-to-real transfer and realistic benchmarking. In neural rendering, the fusion of normal and depth priors, completion decoders, and guided ray sampling exemplifies the utility of normal-guided signals in complex, view-dependent learning problems (Guo et al., 8 Jul 2024).

A plausible implication is that future depth sampling and completion benchmarks are likely to transition further toward realism by adopting geometry-aware, normal-aligned protocols as standard in both evaluation and training. This shift aligns with a growing recognition that sparse input patterns must reflect true sensor behavior to maximize real-world transferability and learning efficiency.