Improved Scene Landmark Detection for Camera Localization (2401.18083v1)

Abstract: Camera localization methods based on retrieval, local feature matching, and 3D structure-based pose estimation are accurate but require high storage, are slow, and are not privacy-preserving. A method based on scene landmark detection (SLD) was recently proposed to address these limitations. It involves training a convolutional neural network (CNN) to detect a few predetermined, salient, scene-specific 3D points or landmarks and computing camera pose from the associated 2D-3D correspondences. Although SLD outperformed existing learning-based approaches, it was notably less accurate than 3D structure-based methods. In this paper, we show that the accuracy gap was due to insufficient model capacity and noisy labels during training. To mitigate the capacity issue, we propose to split the landmarks into subgroups and train a separate network for each subgroup. To generate better training labels, we propose using dense reconstructions to estimate visibility of scene landmarks. Finally, we present a compact architecture to improve memory efficiency. Accuracy wise, our approach is on par with state of the art structure based methods on the INDOOR-6 dataset but runs significantly faster and uses less storage. Code and models can be found at https://github.com/microsoft/SceneLandmarkLocalization.

• The paper presents an ensemble approach that partitions landmarks into subgroups to address model capacity issues.
• It refines training labels using dense scene reconstructions to enhance the accuracy of landmark detection.
• The novel architecture achieves over 40 times faster localization and 20 times improved storage efficiency compared to standard methods.

Improved Scene Landmark Detection for Camera Localization

Overview of SLD Enhancements

This paper presents enhancements to the Scene Landmark Detection (SLD) framework, which plays a critical role in camera pose estimation tasks essential for applications such as robotics and augmented reality. The prior state of SLD demonstrated promising results by utilizing a Convolutional Neural Network (CNN) trained to detect specific 3D points within a scene. Despite outperforming other learning-based methods, SLD lagged behind structure-based methods, attributed primarily to insufficient model capacity and the presence of noisy training labels.

To overcome these limitations, this research introduces a novel approach that involves partitioning landmarks into subgroups and training individual networks on each subgroup. Concurrently, the generation of training labels sees a significant improvement through dense reconstructions, which aid in the accurate estimation of landmark visibility. The fusion of these strategies culminates in a new compact network architecture, proving to be both memory efficient and remarkably more accurate.

Model Capacity and Training Labels

Investigations into the factors that hindered SLD's accuracy revealed that the models were inadequate when handling a large count of landmarks. The capacity issues were evident from the increased angular errors as the number of landmarks grew for training. The solution proposed here is the use of ensemble networks where each is dedicated to a batch of landmarks, allowing the overall framework to scale and accommodate more landmarks without a decline in performance.

Addressing the quality of training labels, the authors refine the traditional use of structure from motion (SfM) derived labels. By incorporating dense scene reconstructions, the model gains a more robust representation of scene visibility, leading to significantly reduced erroneous labels and, ultimately, more precise landmark detections.

Architecture and Efficiency

The novel network architecture introduced, SLD, is a less memory-intensive variant of its predecessor and achieves higher performance benchmarks. This stripped-down architecture eliminates an upsampling layer without compromising the accuracy of landmark prediction. In practice, this directly translates into a reduction in parameters and a minimized memory footprint.

In addition to technical improvements, practical application benefits such as speed and storage efficiency make SLD highly conducive to diverse deployment scenarios. It remains more than 40 times faster during localization and 20 times more storage efficient than its structure-based counterparts such as hloc, while matching their accuracy.

Results and Conclusion

Benchmark tests on the challenging INDOOR-6 dataset validate that SLD approximates the accuracy of leading structure-based methods yet delivers a dramatic increase in computational speed. Further, an ablation paper solidifies the observed advantages of ensemble size and weighted pose estimation in the model's success. The amalgamation of improved label generation methods, efficient network architecture, and the use of ensembles for scalability renders SLD a significant advancement in the domain of camera localization. Future work could explore avenues to expedite the C training process, pushing the boundaries of rapid, scalable, and precise localization.