Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters

Abstract: We introduce a novel approach for keypoint detection task that combines handcrafted and learned CNN filters within a shallow multi-scale architecture. Handcrafted filters provide anchor structures for learned filters, which localize, score and rank repeatable features. Scale-space representation is used within the network to extract keypoints at different levels. We design a loss function to detect robust features that exist across a range of scales and to maximize the repeatability score. Our Key.Net model is trained on data synthetically created from ImageNet and evaluated on HPatches benchmark. Results show that our approach outperforms state-of-the-art detectors in terms of repeatability, matching performance and complexity.

• The paper introduces a novel hybrid approach that combines handcrafted filters with CNN-based learning to enhance keypoint detection.
• It leverages a multi-scale pyramid architecture and a custom loss function to boost feature repeatability while reducing model complexity.
• Extensive evaluation on synthetic data and the HPatches benchmark demonstrates significant improvements in matching performance and computational efficiency.

Insightful Overview of Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters

This paper introduces Key.Net, an innovative approach to keypoint detection that synergizes handcrafted and learned CNN filters within a shallow multi-scale architecture. The amalgamation of these diverse filters facilitates the detection, localization, and scoring of repeatable image features. The proposed model utilizes handcrafted filters as anchor structures for the learned filters, leveraging scale-space representation to secure multi-level keypoint extraction. A specially designed loss function aims to enhance feature robustness across a range of scales, optimizing the repeatability score.

Key.Net is trained on synthetic data derived from the ImageNet dataset and evaluated using the HPatches benchmark, exhibiting superior performance over state-of-the-art detectors concerning repeatability, matching, and computational complexity.

Methodological Framework

1. Hybrid Filter Design: Key.Net integrates handcrafted and CNN-based features to extract salient image features efficiently. Handcrafted filters, inspired by traditional detectors like Harris and Hessian, incorporate lower-order image derivatives to propose potential corners and blobs, thereby reducing learnable parameters and enhancing convergence stability.
2. Multi-Scale Pyramid Architecture: By processing images at multiple scales, this network enhances robustness to scale transformations, a common challenge in keypoint detection tasks. Feature maps from varied scale streams are upsampled and combined, leading to the final response map.
3. Loss Function with Multi-Scale Index Proposal: The novel Multi-Scale Index Proposal (M-SIP) operator proposes keypoint locations across multiple scales, addressing both local and contextual information. The loss function, based on covariant constraint principles, promotes feature stability and consistency across geometric and photometric transformations.

Numerical Results and Implications

Key.Net demonstrates enhanced repeatability and matching performance. Notably, the approach balances model complexity and performance, evidenced by the high scores obtained with a significantly reduced number of learnable parameters compared to prior techniques. For example, the Tiny-Key.Net version, with minimal learnable layers, maintains competitive performance.

The implications of this research are manifold:

• Theoretical Advancements: By highlighting the efficacy of blending deterministic and stochastic methods in deep learning frameworks, this work bolsters understanding of filter designs' interplay for computer vision tasks.
• Practical Applications: Key.Net's efficiency and accuracy make it highly suitable for real-time applications like augmented reality and mobile robotics that demand fast and reliable local feature detection.

Key.Net's success suggests further explorations into hybrid models combining traditional computer vision insights with advanced learning techniques may unlock new frontiers in feature detection and related tasks. Future developments might explore integrating these paradigms within a unified learning pipeline optimizing descriptor performance alongside detection, thereby fostering more robust and reliable computer vision systems.