AG-NeRF: Attention-guided Neural Radiance Fields for Multi-height Large-scale Outdoor Scene Rendering (2404.11897v1)

Abstract: Existing neural radiance fields (NeRF)-based novel view synthesis methods for large-scale outdoor scenes are mainly built on a single altitude. Moreover, they often require a priori camera shooting height and scene scope, leading to inefficient and impractical applications when camera altitude changes. In this work, we propose an end-to-end framework, termed AG-NeRF, and seek to reduce the training cost of building good reconstructions by synthesizing free-viewpoint images based on varying altitudes of scenes. Specifically, to tackle the detail variation problem from low altitude (drone-level) to high altitude (satellite-level), a source image selection method and an attention-based feature fusion approach are developed to extract and fuse the most relevant features of target view from multi-height images for high-fidelity rendering. Extensive experiments demonstrate that AG-NeRF achieves SOTA performance on 56 Leonard and Transamerica benchmarks and only requires a half hour of training time to reach the competitive PSNR as compared to the latest BungeeNeRF.

• The paper introduces AG-NeRF, an attention-guided framework that selects optimal images and fuses features to improve multi-height outdoor scene rendering.
• It achieves a 6 dB PSNR improvement while reducing training time from days to just 30 minutes on a single RTX 4090 GPU.
• This efficient approach broadens NeRF applicability by enabling high-quality rendering across varying altitudes without extensive preprocessing.

Attention-guided Neural Radiance Fields for Efficient Multi-height Large-scale Outdoor Scene Rendering

Introduction

In novel view synthesis for large-scale outdoor scenes, existing Neural Radiance Fields (NeRF) approaches are primarily constrained to single-altitude data, requiring prior knowledge of camera height and scene extent. This limitation hampers their practical application when there are variations in camera altitude. The paper presents AG-NeRF, an end-to-end framework that significantly decreases the training overhead required for high-quality reconstructions over varying altitudes, from low (drone-level) to high (satellite-level). Utilizing a novel source image selection mechanism and an attention-based feature fusion method, AG-NeRF delivers state-of-the-art performance efficiently.

Problem Statement

Traditional NeRF implementations face challenges in capturing the high variability in detail when images are captured from different altitudes, often resulting in feature loss or excessive blurriness in synthesized views. This issue is compounded by the extensive computational resources and time required for training such models, making them less feasible for practical applications.

AG-NeRF Approach

AG-NeRF addresses these limitations by:

• Source Image Selection: It selects a set of images from different altitudes that most closely approximate the target view. This selection enriches the input feature space, enabling the model to adaptively activate high- or low-frequency channels in NeRF’s positional encoding based on detail requirements dictated by varying altitudes.
• Attention-based Feature Fusion: This technique allows for the integration of features extracted from the selected images. By maximizing the relevance of these features to the target view, the approach ensures high-quality feature representation across the entire image synthesis process.

Key Results

AG-NeRF was evaluated against benchmarks like the 56 Leonard and Transamerica datasets, where it:

• Achieved a 6 dB improvement in Peak Signal-to-Noise Ratio (PSNR) over the state-of-the-art BungeeNeRF.
• Required only half an hour of training time on a single RTX 4090 GPU for performance comparable to that of BungeeNeRF, which otherwise necessitates up to five days of training.

Implications

The rapid training capability and reduced resource demand of AG-NeRF not only facilitate quicker deployments but also lower the entry barrier for utilizing advanced view synthesis techniques in practical scenarios. Furthermore, the ability to handle image data from varying altitudes without extensive preprocessing or manual intervention potentially expands the applicability of NeRF techniques to a wider range of real-world environments and applications.

Future Directions

The advancement shown by AG-NeRF opens several avenues for future research, particularly in exploring how these techniques can be adapted for dynamic environments where both camera position and scene content may vary over time. Additionally, further reducing the computational requirements and enhancing the model's ability to generalize across more diverse datasets are areas ripe for exploration.

The introduction of AG-NeRF not only sets a new standard in the field of neural radiance fields but also significantly pushes the boundaries of what can be achieved in the domain of large-scale outdoor scene rendering through deep learning techniques.