Semantic Correspondence: Unified Benchmarking and a Strong Baseline (2505.18060v3)

Abstract: Establishing semantic correspondence is a challenging task in computer vision, aiming to match keypoints with the same semantic information across different images. Benefiting from the rapid development of deep learning, remarkable progress has been made over the past decade. However, a comprehensive review and analysis of this task remains absent. In this paper, we present the first extensive survey of semantic correspondence methods. We first propose a taxonomy to classify existing methods based on the type of their method designs. These methods are then categorized accordingly, and we provide a detailed analysis of each approach. Furthermore, we aggregate and summarize the results of methods in literature across various benchmarks into a unified comparative table, with detailed configurations to highlight performance variations. Additionally, to provide a detailed understanding on existing methods for semantic matching, we thoroughly conduct controlled experiments to analyse the effectiveness of the components of different methods. Finally, we propose a simple yet effective baseline that achieves state-of-the-art performance on multiple benchmarks, providing a solid foundation for future research in this field. We hope this survey serves as a comprehensive reference and consolidated baseline for future development. Code is publicly available at: https://github.com/Visual-AI/Semantic-Correspondence.

Overview of Semantic Correspondence Methods

The paper "Semantic Correspondence: Unified Benchmarking and a Strong Baseline" provides a comprehensive survey of semantic correspondence methods, a key task in computer vision aimed at matching keypoints with semantic equivalence across diverse images. Over recent years, substantial progress has been made, primarily driven by advances in deep learning. Nevertheless, the paper points out a lack of comprehensive analysis and review of semantic correspondence methodologies. It aims to bridge this gap by proposing a taxonomy for classification, presenting a detailed analysis, and highlighting a strong baseline strategy.

Taxonomy and Methodological Analysis

The authors categorize existing semantic correspondence methodologies into three primary classes based on their designs: handcrafted methods, architectural improvements, and training strategies.

1. Handcrafted Methods: Initially, techniques such as SIFT and HOG were employed to establish correspondences based on predefined features. Despite their utility, these approaches typically falter under substantial appearance variations.
2. Architectural Improvements: This category emphasizes enhancements in feature extraction and matching quality. Key strategies include feature assembly and feature adaptation. The assembly methods leverage different levels of network features to construct hyper-features, increasing the robustness of semantic correspondence. Adaptation, conversely, involves transferring pre-trained features to new tasks via adaptation modules.
3. Matching Refinement: Techniques here refine the established matches through cost volumes, flow fields, or parameterized transformations. Convolutional methods like Neighbourhood Consensus and transformer approaches like Match-to-Match attention represent key advancements, offering improved match filtering.
4. Training Strategies: These strategies aim to reduce reliance on manual annotations. Non-keypoint-based methods use categories or binary masks, while consistency-based methods employ cycle consistency principles. Pseudo-label generation also offers a mechanism to create extensive training data efficiently.

Experimental Evaluation and Benchmarking

The paper provides detailed experiments evaluating the performance of various innovations across different stages of the semantic matching pipeline. DINOv2 emerges as a particularly potent feature backbone, demonstrating significant gains when fine-tuning the feature parameters. The paper further validates that feature enhancement modules considerably bolster performance in frozen backbone settings, though deliver limited gains during joint training due to saturation in backbone design.

Matching refinement approaches, particularly CNN-based cost aggregators, excel in optimizing semantic correspondence results, outperforming transformer counterparts that suffer computational inefficiencies at higher resolutions.

Strong Baseline Proposal and Implications

Based on extensive analysis, the authors propose a strong baseline framework leveraging DINOv2 for both zero-shot and supervised settings. Fine-tuning the last two layers of DINOv2 substantively improves matching performance across benchmark datasets such as SPair-71k and PF-PASCAL, attaining state-of-the-art results.

The proposed baseline underscores the importance of robust feature backbones and effective feature adaptation techniques, offering critical insights into the architectural optimization of semantic correspondence methodologies.

Future Directions and Conclusions

The findings presented highlight the untapped potential of foundational models like DINOv2 in semantic correspondence. Future research may explore advanced adaptation techniques and pseudo-label generation strategies, aiming to unleash models' full capabilities while minimizing annotation dependencies.

This paper effectively serves as both a consolidated reference and a foundation for upcoming advancements in semantic correspondence, navigating both theoretical and practical terrains with clarity.