MESA: Matching Everything by Segmenting Anything (2401.16741v2)

Abstract: Feature matching is a crucial task in the field of computer vision, which involves finding correspondences between images. Previous studies achieve remarkable performance using learning-based feature comparison. However, the pervasive presence of matching redundancy between images gives rise to unnecessary and error-prone computations in these methods, imposing limitations on their accuracy. To address this issue, we propose MESA, a novel approach to establish precise area (or region) matches for efficient matching redundancy reduction. MESA first leverages the advanced image understanding capability of SAM, a state-of-the-art foundation model for image segmentation, to obtain image areas with implicit semantic. Then, a multi-relational graph is proposed to model the spatial structure of these areas and construct their scale hierarchy. Based on graphical models derived from the graph, the area matching is reformulated as an energy minimization task and effectively resolved. Extensive experiments demonstrate that MESA yields substantial precision improvement for multiple point matchers in indoor and outdoor downstream tasks, e.g. +13.61% for DKM in indoor pose estimation.

• The paper introduces MESA, which uses advanced SAM segmentation to reduce matching redundancy and enhance feature matching accuracy by over 13%.
• It builds a multi-relational graph and employs a Graph Cut algorithm to solve area matching as an energy minimization problem.
• Experimental results demonstrate significant gains in indoor pose estimation, visual odometry, and outdoor localization, setting new state-of-the-art benchmarks.

Introduction

Feature matching is a critical component in computer vision, essential for tasks like Simultaneous Localization and Mapping (SLAM), Structure from Motion (SfM), and visual localization. Existing methods for feature matching, including sparse, semi-dense, and dense approaches, have their respective challenges, often resulting from the need to address matching redundancy. To mitigate these challenges, a novel approach called Matching Everything by Segmenting Anything (MESA) has been introduced, incorporating a foundation model known as the Segment Anything Model (SAM) for image segmentation. The paper extensively tests MESA's performance and quantitatively demonstrates its superiority over existing methods in various scenarios.

Problem Addressed

MESA addresses the issue of reducing matching redundancy to enhance the accuracy of point matching between images. Current methods for matching feature points across images suffer from several issues that affect precision, such as scale variations, viewpoint changes, illumination differences, and the presence of repetitive patterns. Classical approaches either focus on those keypoints that have detectable features or involve dense methods that are computation-intensive and error-prone. MESA proposes a smart reduction in matching redundancy by leveraging area-level correspondences.

Methodology

The methodology of MESA revolves around the advanced SAM, which provides high-quality segmentation inputs. These segments are then used to build a multi-relational graph that defines the spatial structure and scale hierarchy of areas. Area matching is then solved as an energy minimization problem through a Graph Cut algorithm, enhanced by the precise calculation of area similarities using a learning-based model. This method effectively leverages SAM's image understanding capabilities to reduce the redundancy in feature matching.

Experimental Results

MESA's experimental results are striking. For instance, in indoor pose estimation, there is a significant precision increase across multiple point matchers. Notably, there's a +13.61% enhancement for Dense Knowledge Mining (DKM) in accuracy. MESA also demonstrates substantial improvements in visual odometry benchmarks and outdoor pose estimation, setting new state-of-the-art records. These strong numerical results evidence the potential of MESA to serve as a robust and accurate solution for matching area correspondences, substantially advancing the field of feature matching in computer vision.

Conclusion

The MESA approach marks a significant stride forward in the domain of feature matching by effectively reducing matching redundancy. Its clever use of advanced image segmentation to inform area matching challenges the prevailing norms in feature comparison computations and opens avenues for more efficient and accurate correspondences in computer vision tasks. The robust experimental validations across various benchmarks attest to the method's profound impact on the accuracy and reliability of feature matching processes.