SparseSat-NeRF: Dense Depth Supervised Neural Radiance Fields for Sparse Satellite Images

Abstract: Digital surface model generation using traditional multi-view stereo matching (MVS) performs poorly over non-Lambertian surfaces, with asynchronous acquisitions, or at discontinuities. Neural radiance fields (NeRF) offer a new paradigm for reconstructing surface geometries using continuous volumetric representation. NeRF is self-supervised, does not require ground truth geometry for training, and provides an elegant way to include in its representation physical parameters about the scene, thus potentially remedying the challenging scenarios where MVS fails. However, NeRF and its variants require many views to produce convincing scene's geometries which in earth observation satellite imaging is rare. In this paper we present SparseSat-NeRF (SpS-NeRF) - an extension of Sat-NeRF adapted to sparse satellite views. SpS-NeRF employs dense depth supervision guided by crosscorrelation similarity metric provided by traditional semi-global MVS matching. We demonstrate the effectiveness of our approach on stereo and tri-stereo Pleiades 1B/WorldView-3 images, and compare against NeRF and Sat-NeRF. The code is available at https://github.com/LulinZhang/SpS-NeRF

• The paper introduces dense depth supervision via SGM to guide NeRF for precise digital surface model reconstruction from sparse satellite views.
• It employs correlation-based uncertainty and a depth-informed ray sampling strategy to overcome limitations of traditional multi-date image methods.
• Experimental results on urban and rural datasets demonstrate superior view synthesis and altitude extraction compared to existing NeRF variants.

SparseSat-NeRF: Dense Depth Supervised Neural Radiance Fields for Sparse Satellite Images

The paper "SparseSat-NeRF: Dense Depth Supervised Neural Radiance Fields for Sparse Satellite Images" introduces an innovative extension to Neural Radiance Fields (NeRF), termed SparseSat-NeRF (SpS-NeRF), which is particularly tailored for handling sparse satellite images. SpS-NeRF provides an expanded capability in producing robust digital surface models (DSMs) and novel view synthesis from satellite imagery, even when only few views are available—a frequent limitation in remote sensing applications.

Background and Motivation

Traditional methods for generating DSMs from satellite imagery, such as Multi-View Stereo (MVS) matching, often face significant challenges on non-Lambertian surfaces, asynchronous image captures, and surface discontinuities. NeRF, which represents scenes as continuous volumetric radiance fields, offers an alternative by leveraging many visual inputs to model these fields. However, NeRF's performance declines sharply when typical satellite data constraints—like limited angular views—are present. Previous adaptations, like Sat-NeRF, attempt to mitigate these issues but still rely extensively on multi-date images, relying on temporal data beyond spatial context.

Methodological Enhancements in SpS-NeRF

SparseSat-NeRF innovates by incorporating dense depth supervision via Stereo Global Matching (SGM) algorithms to provide reliable depth priors. This integration serves as a corrective mechanism to NeRF's tendencies to misfit geometries under sparse view conditions. The methodology includes:

• Dense Depth Supervision: Uses low-resolution depth maps derived from SGM, which guide SpS-NeRF without oversized bias towards SGM's outcomes.
• Correlation-Based Uncertainty: Implements cross-correlation from SGM as a metric for depth prediction reliability, enhancing robustness.
• Ray Sampling Strategy: Implements a depth-informed guided sampling technique, balancing NeRF’s computations between RGB and depth loss to prioritize feature sharpness.

Experimental Evaluation

The research evaluates SpS-NeRF using two datasets: the urban landscape of the DFC2019 (Jacksonville, USA) and the rural terrain of Djibouti, using 2 and 3 view configurations. The experiments reveal:

• In novel view synthesis, SpS-NeRF outperforms both NeRF and Sat-NeRF, particularly overcoming their limitations in generating visually convincing resolutions from sparse, high-angle vantage points.
• For altitude extraction, SpS-NeRF competently reconstructs DSMs, showing both qualitative and quantitative improvements. Notably, it handles vegetation and occlusions more effectively than traditional SGM and NeRF variants.

Implications and Future Directions

By successfully integrating dense depth supervision into the NeRF framework, SpS-NeRF encapsulates a significant step towards more reliable photogrammetric applications using scarce satellite views. From a practical perspective, it suggests new potential for satellite image applications across urban planning, environmental monitoring, and disaster mapping, where DSM accuracy is critical.

The study opens several future avenues:

1. Exploring semantic data integrations to address flat surface irregularities.
2. Extending SpS-NeRF applicability to diverse satellite platforms with further constraint handling.
3. Continuous optimization enhancements to adjust for variable satellite image qualities and scales.

In conclusion, SpS-NeRF stands out as a promising advancement in satellite-based 3D reconstruction, ensuring superior outcome quality even from limited input data—a characteristic inherent to remote sensing scenarios. This methodology adds a layer of feasibility to satellite image automations, pushing towards unified and dependable photogrammetric outcomes in real-world applications.