AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis (2504.13157v1)

Abstract: We explore the task of geometric reconstruction of images captured from a mixture of ground and aerial views. Current state-of-the-art learning-based approaches fail to handle the extreme viewpoint variation between aerial-ground image pairs. Our hypothesis is that the lack of high-quality, co-registered aerial-ground datasets for training is a key reason for this failure. Such data is difficult to assemble precisely because it is difficult to reconstruct in a scalable way. To overcome this challenge, we propose a scalable framework combining pseudo-synthetic renderings from 3D city-wide meshes (e.g., Google Earth) with real, ground-level crowd-sourced images (e.g., MegaDepth). The pseudo-synthetic data simulates a wide range of aerial viewpoints, while the real, crowd-sourced images help improve visual fidelity for ground-level images where mesh-based renderings lack sufficient detail, effectively bridging the domain gap between real images and pseudo-synthetic renderings. Using this hybrid dataset, we fine-tune several state-of-the-art algorithms and achieve significant improvements on real-world, zero-shot aerial-ground tasks. For example, we observe that baseline DUSt3R localizes fewer than 5% of aerial-ground pairs within 5 degrees of camera rotation error, while fine-tuning with our data raises accuracy to nearly 56%, addressing a major failure point in handling large viewpoint changes. Beyond camera estimation and scene reconstruction, our dataset also improves performance on downstream tasks like novel-view synthesis in challenging aerial-ground scenarios, demonstrating the practical value of our approach in real-world applications.

AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis

In the field of computer vision, the task of geometric reconstruction from multi-view images constitutes a fundamental challenge. The paper "AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis" by Vuong et al. addresses a significant impediment in this domain: the difficulty in handling extreme viewpoint variations inherent in aerial-ground image pairs. Contemporary learning-based methodologies underperform in such circumstances primarily due to the absence of high-quality, co-registered aerial-ground datasets for training. The authors propose a novel, scalable framework designed to overcome these limitations by integrating pseudo-synthetic renderings from 3D city meshes with real, ground-level crowd-sourced images to formulate the AerialMegaDepth dataset.

Methodology and Dataset Construction

The paper introduces an innovative data generation strategy that leverages geospatial platforms like Google Earth. The approach involves rendering pseudo-synthetic images from 3D city-wide meshes to simulate a comprehensive range of aerial viewpoints. These images are subsequently paired with real images sourced from datasets like MegaDepth, which provide accurate ground-level captures. This pseudo-synthetic data aids in maintaining visual fidelity across varying viewpoints, particularly in scenarios where mesh renderings fall short on detail.

A central feature of AerialMegaDepth is its scalability, as it provides over 132,000 geo-registered images across 137 diverse landmarks. This vast dataset allows for the fine-tuning of state-of-the-art algorithms, which significantly enhances performance in real-world applications.

Empirical Evaluation and Results

The authors subjected their methodology to rigorous empirical validation across two key computer vision tasks: multi-view geometry prediction and novel view synthesis. Fine-tuning of algorithms such as DUSt3R and MASt3R on the AerialMegaDepth dataset resulted in substantial improvements. For DUSt3R, the accuracy in localizing aerial-ground image pairs within 5 degrees of camera rotation error improved from less than 5% to nearly 56%. This highlights a remarkable advancement in addressing the challenges posed by large viewpoint variations.

Furthermore, in novel view synthesis tasks, using a single input image coupled with geographical context, the fine-tuned models demonstrated better visual and geometric coherence, validating the practical utility and effectiveness of the hybrid dataset.

Implications and Future Directions

The work of Vuong et al. extends far-reaching implications for both theoretical and practical applications in AI and computer vision. By bridging the gap between aerial and ground-level images, AerialMegaDepth enriches the training landscape, enabling models to generalize to real-world scenarios more effectively. This achievement is pivotal for applications like autonomous navigation, remote sensing, and urban planning, where the need for accurate 3D reconstructions is paramount.

Looking forward, this research opens avenues for incorporating drone and satellite imagery to create even more comprehensive datasets. Achieving seamless integration across varying altitudes and resolutions could potentially propel advancements towards the ambitious goal of planet-scale 3D reconstruction. Additionally, exploring self-supervised strategies on such datasets could further reduce dependency on labeled data, thus broadening the scope and accessibility of advanced computer vision techniques.

In summary, "AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis" represents a significant contribution to multi-view image reconstruction. By addressing the data scarcity challenge with a novel hybrid dataset, the authors lay down a robust foundation for future research in enhancing AI's capability to understand and reconstruct complex 3D environments across disparate viewpoints.