HPatches: A benchmark and evaluation of handcrafted and learned local descriptors (1704.05939v1)

Abstract: In this paper, we propose a novel benchmark for evaluating local image descriptors. We demonstrate that the existing datasets and evaluation protocols do not specify unambiguously all aspects of evaluation, leading to ambiguities and inconsistencies in results reported in the literature. Furthermore, these datasets are nearly saturated due to the recent improvements in local descriptors obtained by learning them from large annotated datasets. Therefore, we introduce a new large dataset suitable for training and testing modern descriptors, together with strictly defined evaluation protocols in several tasks such as matching, retrieval and classification. This allows for more realistic, and thus more reliable comparisons in different application scenarios. We evaluate the performance of several state-of-the-art descriptors and analyse their properties. We show that a simple normalisation of traditional hand-crafted descriptors can boost their performance to the level of deep learning based descriptors within a realistic benchmarks evaluation.

• The paper introduces a benchmark that standardizes descriptor evaluation by using patch-based tests for matching, retrieval, and verification.
• The benchmark combines diverse real-world scenes and controlled transformations to mimic challenging imaging conditions.
• The results show that simple normalized handcrafted descriptors can perform competitively with deep learning-based methods across tasks.

Evaluation of Local Descriptors through HPatches Benchmark

The paper "HPatches: A benchmark and evaluation of handcrafted and learned local descriptors" presents a detailed study of local image descriptors, introducing a new benchmark designed to enhance the evaluation coherence and reliability in this domain. The authors address ambiguities and inconsistencies identified in current datasets and evaluation protocols, proposing a comprehensive dataset that enables a more dependable comparison of descriptors across different application scenarios.

Abstract Overview

The paper critiques existing datasets, pointing out their limitations in size and diversity which hinder effective evaluations of modern descriptors. The authors introduce a new, large dataset specifically tailored for training and testing descriptors, with explicit evaluation protocols across tasks like matching, retrieval, and classification. The paper underscores that a simple normalization of traditional handcrafted descriptors can elevate their performance to the level of deep learning-based descriptors when evaluated against realistic benchmark conditions.

Introduction

Local feature descriptors are foundational for image matching and retrieval systems, playing a central role in computer vision research. The paper identifies that recent advancements in learning-based descriptors have not been matched by equivalent improvements in benchmarking datasets and evaluation protocols, causing discrepancies in reported performance. The proposed HPatches benchmark addresses these discrepancies, providing a robust platform for more general descriptor evaluation.

Benchmark Design and Data Collection

HPatches is constructed to surpass existing datasets by incorporating the following characteristics:

• Reproducibility and Patch-based Evaluation: Evaluations are performed on patches, removing detector-related variables and enabling standardized comparisons.
• Diversity and Real-World Relevance: Includes a broad range of scenes and real-world variabilities, including geometric and photometric transformations.
• Scale and Multitask Evaluation: Larger scale than traditional datasets, allowing assessments across multiple tasks (patch verification, image matching, and patch retrieval).

The data is sourced from a combination of newly captured sequences and existing datasets, with patches extracted using scale-invariant interest points and various noise conditions (Easy, Hard, Tough) to simulate detection inaccuracies.

Evaluation Metrics and Protocols

Three distinct evaluation tasks are outlined:

1. Patch Verification tasks involve the binary classification of patch pairs to assess descriptor matching capabilities.
2. Image Matching evaluates the ability to correctly identify corresponding patches between images, a scenario akin to real-world applications.
3. Patch Retrieval mimics image retrieval systems by assessing how well a descriptor can identify a query patch from a large, distractor-filled dataset.

The metrics used include AP and PR curves, which are integral for accounting for the unbalanced nature of patch data.

Experimental Results

The paper evaluates a spectrum of descriptors, including traditional handcrafted ones like SIFT and binary descriptors like BRIEF, alongside state-of-the-art deep learning-based ones such as TFeat and DeepDesc. A key observation is the substantial role that data normalization and transformation techniques (e.g., ZCA whitening) play in enhancing descriptor performance.

The findings reveal inconsistencies in descriptor performance across different tasks, underscoring the necessity of a multitask evaluation framework. Particularly intriguing is the ability of simple normalized descriptors to rival deep learning-based ones, highlighting the potential for optimization through effective pre-processing.

Conclusion

The authors posit that HPatches effectively addresses the current inadequacies in descriptor evaluation benchmarks by providing a larger, diverse, and rigorously defined testing environment. This benchmark is poised to set a new standard, enabling more generalizable and conclusive evaluations of descriptors. Moving forward, HPatches can guide future research, potentially influencing developments in areas such as feature detectors and performance-driven descriptor enhancements.

The HPatches benchmark and its accompanying evaluation protocols have been made publicly accessible, promoting transparency and widespread adoption within the research community. This benchmark represents a significant stride towards establishing more consistent and reliable evaluations of local descriptors, a crucial step for advancing the field of computer vision.