Sparse-View Depth Estimation

Updated 22 February 2026

Sparse-view depth estimation is the process of reconstructing dense depth maps from minimal, sparse sensor measurements and multi-view data.
It leverages deep fusion models, planar priors, and multi-branch architectures to optimize geometric fidelity under severe data constraints.
The techniques enable robust 3D reconstruction in robotics, autonomous driving, and SLAM despite challenges like occlusion and calibration errors.

Sparse-view depth estimation refers to the inference of dense depth maps or 3D structure from highly limited, spatially sparse input measurements—either few calibrated images, a small subset of depth samples (pixels or points), or hybrid multi-modal inputs (e.g., camera + radar/LiDAR). The field addresses fundamental challenges in perception and reconstruction, with major applications in robotics, autonomous driving, SLAM, view synthesis, and depth completion from cost-constrained or low-power sensors. Research on this topic combines algorithmic, architectural, and mathematical innovations to maximize geometric fidelity and efficiency under strong information constraints.

1. Core Concepts and Problem Setting

Sparse-view depth estimation is characterized by a severe restriction on input coverage.

Sparse-image baselines: Multi-view stereo (MVS) or novel view synthesis from as few as 2–5 camera views, typically with large angular or spatial baselines, resulting in incomplete or ambiguous scene geometry (Lu et al., 28 May 2025, Ma et al., 29 Sep 2025, Sinha et al., 2020, Chuchvara et al., 2018).
Sparse point measurements: Depth sensors (e.g., LiDAR, radar, stereo) often provide depth for <1% of pixels—either due to sensing limitations or to minimize energy/cost. These measurements are irregular, non-uniform, and noisy (Lo et al., 2021, Kumar et al., 2018, Zhang et al., 2019, Chen et al., 2018, Sartipi et al., 2020, Liu et al., 2014).
Hybrid/deep priors: Fusion with monocular images, surface normals, or semantic features is common to address missing or unreliable regions (Lo et al., 2021, Sartipi et al., 2020, Chen et al., 2018).

The goal is to reconstruct a dense, metrically accurate depth map or 3D scene representation. Typical metrics include RMSE, AbsRel, accuracy at various thresholds (δ₁, δ₂, δ₃ < 1.25ᶦ), PSNR, SSIM, and perceptual measures such as LPIPS depending on the application domain.

2. Algorithmic Frameworks and Methodologies

A wide array of algorithmic strategies have emerged, tailored to the unique sparsity domain and available side information.

Learned Sparse-to-Dense Inference

Deep fusion models: These take as input an RGB image and a highly sparse depth (or binary mask + values). Effective fusion architectures exploit:
- Nearest-neighbor fill and distance-transform input channels (Chen et al., 2018)
- Encoder–decoder networks with skip-style injection of sparse features (Chen et al., 2018).
- Late/early fusion with learned residuals for detail recovery (Lo et al., 2021).
Surface normal and planar priors: Scene planar structure is used to enrich sparse depth maps by fitting planes over detected planar regions, guided by estimated normals and Mask-RCNN segmentation. Gravity-aligned warping corrects for pose discrepancies in normal estimation from RGB (Sartipi et al., 2020).
Multi-branch CNNs: Panoptic/FPN-style backbones for joint encoding of RGB, normals, and enriched sparse depth (Sartipi et al., 2020).

Sparse Multi-view Stereo (MVS) / View Synthesis

Interest-point triangulation and densification: Learning to detect, match, and triangulate sparse keypoints (via epipolar-constrained descriptors), followed by CNN-based densification (Sinha et al., 2020).
Patch/superpixel-based plane refinement: Images are over-segmented and each superpixel is initialized and refined as a depth plane. Plane-sweeping, propagation, and slant refinement are combined with photo-consistency and geometric terms for global optimization (Chuchvara et al., 2018).
Self-supervised and hierarchical Gaussian Splatting (3DGS): 3DGS models scene structure as anisotropic Gaussian “splats,” optimized to match photometric and depth constraints across real and synthesized (virtual) views (Lu et al., 28 May 2025, Ma et al., 29 Sep 2025, Xiao et al., 4 Jun 2025).

Physics- and Representation-driven Approaches

Diffusion-based depth imputation: Sparse 3D points are rendered as image-space Gaussians (“seeds”), and a differentiable diffusion PDE fills in the depth domain while respecting seeds, smoothness, and visibility via analytic radiative transfer (Khan et al., 2021).
Compressive sensing (CS) and analysis priors: Depth maps allow sparse representations in wavelet/contourlet dictionaries and TV regularization. Combinatorial sampling and ADMM-based reconstruction (with multi-scale warm start) are applied to maximize quality for a given measurement budget (Liu et al., 2014).

Radar and LiDAR as depth anchors: Sparse, noisy radar returns are height-extended and temporally accumulated, then fused with camera inputs via dedicated network branches and late feature fusion, with ordinal regression for ordering constraints (Lo et al., 2021).

3. Loss Functions and Optimization

Loss function design in sparse-view depth estimation must address the ambiguity, scale, and noise inherent in the limited data regime.

Ordinal regression: Discretizing depth and using cumulative link loss enforces correct ordering, robust to outliers in sparse radar (Lo et al., 2021).
Cascade/multi-scale correlation loss: Pearson correlation loss is computed between rendered and monocular depths at coarse-to-fine scales. This enforces structure while tolerating scale misalignment from monocular priors (Lu et al., 28 May 2025).
Hybrid-likelihood and geometric constraints: HLDE combines reprojection loss, point propagation loss (matching chained correspondences), and total variation smoothness to regularize sparse-MVS-based splatting (Ma et al., 29 Sep 2025).
Self-supervised multi-view consistency: Training losses include photometric reprojection, SSIM, edge-aware smoothness, and confidence-weighted consistency between predicted and warped depths (Khan et al., 2021, Xiao et al., 4 Jun 2025).
L₁/L₂ regression and mask-based losses: For supervised cases, L₂ or L₁ error over valid (nonzero) predictions is the default (Chen et al., 2018, Sartipi et al., 2020).

4. Architectures, Representations, and Fusion Schemes

Architectural choices follow the problem structure and desired output.

Multi-branch and encoder–decoder CNNs: Essential for integrating multimodal features or spatially sparse cues, sometimes employing separate encoders for RGB, depth, and normals, followed by shared decoding (Sartipi et al., 2020, Lo et al., 2021).
Gaussian Splatting and flow-depth fusion: 3DGS representations handle view synthesis and 3D scene rendering from limited views. JointSplat introduces probabilistic fusion of flow-based and hybrid depth, modulated per-pixel by matching confidence, and a confidence-weighted depth-consistency loss (Xiao et al., 4 Jun 2025).
Plane, superpixel, and point cloud templates: Superpixel-based planes (for light field/multi-view) and diffusion from sparse points exploit scene geometry and spatial regularity (Chuchvara et al., 2018, Khan et al., 2021).
Sparse transform dictionaries: CS-inspired $\ell_1$ -TV models in wavelet/contourlet space with multiscale warm-start ADMM provide strong “classical” baselines (Liu et al., 2014).

5. Benchmarks, Quantitative Results, and Limitations

Sparse-view depth estimation methods are evaluated on challenging benchmarks, with the following results (as reported):

Method	Input Structure	Domain / Benchmark	Representative Results
DELTAS (Sinha et al., 2020)	3 views, sparse points	ScanNet/Sun3D (AbsRel)	0.093 (3 views); generalizes to unseen, low-cost
HDGS (Lu et al., 28 May 2025)	3 views, RGB	LLFF, DTU (PSNR/SSIM)	20.9dB/0.735 (LLFF); 21.45/0.87 (DTU); finest with multi-scale loss
DWGS (Ma et al., 29 Sep 2025)	3–5 views, 3DGS-based	LLFF (PSNR/LPIPS)	21.13dB/0.189 (LLFF/3-view full); HLDE provides >0.18dB over baseline
JointSplat (Xiao et al., 4 Jun 2025)	3 views, RGB	RealEstate10K (PSNR/LPIPS)	27.53dB/0.113; improves over all previous feed-forward Splat methods
Radar/DORN (Lo et al., 2021)	RGB + sparsified radar	nuScenes (AbsRel/RMSE)	0.107/5.082m; especially strong at night, over pure monocular
Fisheye + LiDAR (Kumar et al., 2018)	Sparse LiDAR, RGB	Custom/test (RMSE/δ₁)	1.72m / 0.816 (to 50m); better than top KITTI monocular
Sparse2Dense (Chen et al., 2018)	<0.2% sampled pixels	NYU, KITTI (RMSE/δ₁)	0.118m/99.5% (NYU, 0.17% sparsity); robust to extremely sparse input
FastSuperpixel (Chuchvara et al., 2018)	3--5 lightfield images	Middlebury, Unicorn (err%)	~2.5% error @ 3x3 views; ~1s per full-HD view
Diff. Diffusion (Khan et al., 2021)	Sparse points, 2–5 views	HCI/real (MSE)	0.56 (MVS, ~700 points), ~0.2 MSE for dense seed sets
VI-SLAM (Sartipi et al., 2020)	RGB + 0.5% VI-SLAM pts	ScanNet/NYU/Azure Kinect	0.2m RMSE, 97.5% δ₁ (NYU, 200 pts); superior to CSPN on SLAM features

Limitations:

Strong dependence on input pose accuracy; errors in egomotion/SLAM degrade depth.
Generalization to dynamic, uncalibrated, or severely textureless scenes remains an unsolved challenge.
Some methods require monocular depths as priors—errors can propagate.
Occlusion remains a significant challenge, especially with extreme sparsity or modality mismatches.
Large patch/region strategies may smear high-frequency details.

6. Applications and Future Directions

Sparse-view depth estimation underpins tasks such as 3D reconstruction for view synthesis (cinematic video, mixed reality), autonomous driving (radar + camera, occupancy mapping), robotics (low-overhead mapping), and low-power/low-cost depth sensing. The field’s methodological innovations—probabilistic flow-depth fusion (Xiao et al., 4 Jun 2025), cascade scale-invariant losses (Lu et al., 28 May 2025), hybrid geometric learning (Ma et al., 29 Sep 2025), compressive measurement strategies (Liu et al., 2014)—are being rapidly translated into pipelines for real-time and memory-efficient 3D understanding.

Promising future research includes:

Fusion of sparse depth, pose, and multimodal cues for dynamic, unstructured environments.
Robust handling of occlusions and reflective/sparse texture zones.
Plug-and-play depth completion modules for SLAM, AR, and robotic perception.
Adaptation to novel sensors and sampling patterns, including learned optimally informative measurement strategies.

Sparse-view depth estimation remains one of the most demanding and practically important problems in 3D vision, characterized by interdisciplinary advances at the intersection of geometry, deep learning, and computational optimization.