Yes, we CANN: Constrained Approximate Nearest Neighbors for local feature-based visual localization (2306.09012v3)

Abstract: Large-scale visual localization systems continue to rely on 3D point clouds built from image collections using structure-from-motion. While the 3D points in these models are represented using local image features, directly matching a query image's local features against the point cloud is challenging due to the scale of the nearest-neighbor search problem. Many recent approaches to visual localization have thus proposed a hybrid method, where first a global (per image) embedding is used to retrieve a small subset of database images, and local features of the query are matched only against those. It seems to have become common belief that global embeddings are critical for said image-retrieval in visual localization, despite the significant downside of having to compute two feature types for each query image. In this paper, we take a step back from this assumption and propose Constrained Approximate Nearest Neighbors (CANN), a joint solution of k-nearest-neighbors across both the geometry and appearance space using only local features. We first derive the theoretical foundation for k-nearest-neighbor retrieval across multiple metrics and then showcase how CANN improves visual localization. Our experiments on public localization benchmarks demonstrate that our method significantly outperforms both state-of-the-art global feature-based retrieval and approaches using local feature aggregation schemes. Moreover, it is an order of magnitude faster in both index and query time than feature aggregation schemes for these datasets. Code: \url{https://github.com/google-research/google-research/tree/master/cann}

• The paper introduces CANN, a novel technique that uses only local features to significantly reduce indexing and query time while maintaining high accuracy.
• It integrates both geometric and appearance constraints into a unified framework, enabling order-of-magnitude speedups compared to global feature methods.
• Empirical evaluations on four large-scale datasets demonstrate that CANN outperforms state-of-the-art approaches in complex and resource-constrained scenarios.

Overview of "Yes, we CANN: Constrained Approximate Nearest Neighbors for Local Feature-Based Visual Localization"

The paper "Yes, we CANN: Constrained Approximate Nearest Neighbors for Local Feature-Based Visual Localization" presents a novel approach to visual localization, addressing the challenges inherent in traditional methods that rely on both global and local features for image retrieval and matching. The authors, Aiger, Araujo, and Lynen from Google Research, propose an innovative solution termed Constrained Approximate Nearest Neighbors (CANN) that leverages only local features, aiming to enhance efficiency and performance in visual localization tasks.

Key Contributions and Methodology

1. Constrained Approximate Nearest Neighbors (CANN): The primary contribution is the introduction of CANN, a technique that performs nearest neighbor searches across both geometric and appearance spaces using solely local features. This approach challenges the conventional reliance on global embeddings for initial image retrieval, which often involves computational redundancy and complexity.
2. Theoretical Framework: The paper lays down a theoretical foundation for k-nearest-neighbor retrieval across multiple metrics, integrating both geometric constraints and descriptor similarity.
3. Performance and Efficiency: Empirical results show that CANN significantly outperforms state-of-the-art global feature-based retrieval methods in terms of accuracy. The method also boasts an order of magnitude reduction in both indexing and query time compared to existing local feature aggregation schemes.
4. Innovative Algorithms: The authors present two solutions for the colored nearest neighbor search problem—CANN-RS and CANN-RG. The CANN-RG, leveraging Random Grids, is particularly efficient, achieving rapid query times while maintaining high-quality matches.

Experimental Validation

CANN's effectiveness is demonstrated across four large-scale datasets: "Baidu-Mall", "Gangnam Station", "RobotCar Seasons", and "Aachen Day-Night v1.1". The results indicate that local feature-based methods not only rival but often surpass the performance of global feature-based approaches, especially in complex or partial view scenarios. The paper also details the computational benefits, with CANN achieving considerable speedups and efficient runtime performance even without leveraging GPU resources.

Implications and Future Directions

The implications of this work are significant for the field of visual localization and related applications. By demonstrating that effective localization can be achieved using only local features, CANN opens up pathways for more efficient and compact localization systems. This could lead to broader applications in robotics, autonomous navigation, and augmented reality where computational resources and speed are critical.

In envisioning future developments, this work could prompt further exploration into hybrid approaches that appropriately balance global and local features. Additionally, the potential adaptation of CANN to accommodate evolving descriptor types and more complex environments could extend its applicability and robustness.

Conclusion

The CANN methodology presented in this paper represents an important advancement in solving the classic problem of visual localization. By introducing a practical solution to the joint optimization of appearance and geometry in local feature spaces, the authors set a new benchmark in terms of both efficiency and accuracy. This work not only challenges existing paradigms but also invites further innovation and research into more refined and adaptable localization systems.