Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence

Abstract: This paper introduces a Transformer-based integrative feature and cost aggregation network designed for dense matching tasks. In the context of dense matching, many works benefit from one of two forms of aggregation: feature aggregation, which pertains to the alignment of similar features, or cost aggregation, a procedure aimed at instilling coherence in the flow estimates across neighboring pixels. In this work, we first show that feature aggregation and cost aggregation exhibit distinct characteristics and reveal the potential for substantial benefits stemming from the judicious use of both aggregation processes. We then introduce a simple yet effective architecture that harnesses self- and cross-attention mechanisms to show that our approach unifies feature aggregation and cost aggregation and effectively harnesses the strengths of both techniques. Within the proposed attention layers, the features and cost volume both complement each other, and the attention layers are interleaved through a coarse-to-fine design to further promote accurate correspondence estimation. Finally at inference, our network produces multi-scale predictions, computes their confidence scores, and selects the most confident flow for final prediction. Our framework is evaluated on standard benchmarks for semantic matching, and also applied to geometric matching, where we show that our approach achieves significant improvements compared to existing methods.

• The paper introduces a unified Transformer architecture that integrates feature and cost aggregation for enhanced correspondence estimation.
• It employs a coarse-to-fine strategy with integrative self- and cross-attention mechanisms to jointly refine feature descriptors and matching costs.
• Evaluation on benchmarks such as SPair-71k and ETH3D confirms significant improvements in both semantic and geometric matching accuracy.

Unifying Feature and Cost Aggregation for Dense Matching Tasks Using Transformers

Introduction

The task of finding visual correspondences between images is vital for numerous applications in computer vision, ranging from augmented reality (AR) to simultaneous localization and mapping (SLAM). Traditional approaches to this problem have evolved from sparse correspondence methods, where only a limited number of key points are matched between images, to dense correspondence techniques that aim to match all pixel points across images. Recent studies in dense matching have highlighted two prominent techniques – feature aggregation and cost aggregation. Feature aggregation primarily focuses on aligning similar features between images, while cost aggregation attempts to enhance the flow estimates' coherence by leveraging matching similarities.

This paper presents a novel architecture that effectively combines the strengths of feature and cost aggregation using Transformers, demonstrating substantial improvements over existing methods in both semantic and geometric matching tasks. The method capitalizes on self- and cross-attention mechanisms to offer a unified approach to feature and cost aggregation, thereby enabling more accurate correspondence estimation. The proposed method is thoroughly evaluated across standard benchmarks, showing significant performance enhancements.

Feature and Cost Aggregation: Distinct Characteristics

Both feature and cost aggregation serve different purposes and possess unique characteristics. Feature aggregation is aimed at integrating similar features within and across images, enhancing the matching accuracy, particularly for images with semantic similarities. On the other hand, cost aggregation focuses on enforcing smoothness and coherence in flow estimates, proving robust against repetitive patterns and clutter by leveraging the similarities encoded in cost volumes.

Despite their individual benefits, integrating both approaches can unlock further potential. This paper demonstrates that a careful design incorporating both feature and cost aggregation can lead to enriched feature representations and more disciplined flow predictions.

Proposed Method: Unified Feature and Cost Aggregation with Transformers (UFC)

At the heart of the UFC architecture is the integrative self-attention mechanism that jointly processes feature descriptors and cost volumes, thereby capitalizing on the strengths of both feature and cost aggregation. The network leverages a coarse-to-fine strategy that progressively refines the correspondence estimates across multiple scales.

Key components of the UFC architecture include:

• Integrative Self-Attention: A mechanism that jointly aggregates feature descriptors and cost volumes, allowing for mutual enhancement.
• Cross-Attention with Matching Distribution: A novel approach that uses aggregated cost volumes to further refine feature representations via cross-attention.
• Hierarchical Processing: A coarse-to-fine method that iteratively refines the correspondences, improving the accuracy of fine-scale estimates.

The UFC framework is extensively evaluated on standard benchmarks for both semantic and geometric matching, demonstrating notable improvements in accuracy and robustness against variations and complexities in the images.

Evaluation and Results

The UFC framework achieves state-of-the-art performance across several semantic matching benchmarks, including SPair-71k, PF-PASCAL, and PF-WILLOW. It showcases significant improvements over existing methods, particularly in challenging conditions involving extreme viewpoints and scale variations. Moreover, when applied to geometric matching on HPatches and ETH3D benchmarks, UFC demonstrates its versatility by outperforming prior works by a considerable margin, proving its efficacy in accurately estimating dense correspondences under various transformations.

Conclusion and Future Directions

This study introduces a powerful architecture that unifies the strengths of feature and cost aggregation through Transformers for dense matching tasks. It demonstrates the potential of combining these two aggregation techniques, leading to significant performance gains across different matching tasks. Future work could explore extending this framework to include other forms of attention mechanisms or integrating additional cues such as texture or edge information to further enhance the matching accuracy.