Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning (1904.06627v3)

Abstract: A family of loss functions built on pair-based computation have been proposed in the literature which provide a myriad of solutions for deep metric learning. In this paper, we provide a general weighting framework for understanding recent pair-based loss functions. Our contributions are three-fold: (1) we establish a General Pair Weighting (GPW) framework, which casts the sampling problem of deep metric learning into a unified view of pair weighting through gradient analysis, providing a powerful tool for understanding recent pair-based loss functions; (2) we show that with GPW, various existing pair-based methods can be compared and discussed comprehensively, with clear differences and key limitations identified; (3) we propose a new loss called multi-similarity loss (MS loss) under the GPW, which is implemented in two iterative steps (i.e., mining and weighting). This allows it to fully consider three similarities for pair weighting, providing a more principled approach for collecting and weighting informative pairs. Finally, the proposed MS loss obtains new state-of-the-art performance on four image retrieval benchmarks, where it outperforms the most recent approaches, such as ABE\cite{Kim_2018_ECCV} and HTL by a large margin: 60.6% to 65.7% on CUB200, and 80.9% to 88.0% on In-Shop Clothes Retrieval dataset at Recall@1. Code is available at https://github.com/MalongTech/research-ms-loss.

Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning

This paper introduces an innovative framework for deep metric learning known as the General Pair Weighting (GPW), which unifies the approach to pair-based loss functions. The researchers focus on developing a comprehensive understanding and enhancement of pair-based metric learning by addressing the intricacies of pair sampling and weighting.

GPW Framework Overview

The GPW framework addresses the challenge of pair-based deep metric learning by formulating it as a holistic pair weighting problem. This brings about a novel perspective on how pairs are sampled and weighted in loss functions. The significance of GPW lies in its ability to explain existing methods through gradient analysis and offer a robust foundation for better pair-based techniques.

Analysis of Pair-Based Loss Functions

The research revisits several classic pair-based loss functions including contrastive loss, triplet loss, lifted structure loss, and binomial deviance loss. It reveals that while these methods provide varying approaches to sampling and weighting pairs, they often focus on singular aspects of pair similarity. The GPW framework allows for a nuanced comparison and highlights the limitations inherent in focusing solely on individual similarities.

Multi-Similarity Loss (MS Loss)

Based on the insights gleaned from GPW, the paper presents the Multi-Similarity Loss (MS loss), which promises a more sophisticated approach by integrating three types of similarities:

1. Self-Similarity: This measures the cosine similarity between a negative sample and an anchor.
2. Positive Relative Similarity: This contrasts a negative pair's similarity to that of positive pairs.
3. Negative Relative Similarity: This evaluates differences between the similarity of a pair and other negative pairs.

The MS loss framework operates in two iterative steps: pair mining and pair weighting. Pair mining involves selecting informative samples by considering relative similarities, while pair weighting assigns importance using both self-similarity and relative similarities. This dual-step process aims to better exploit the information inherent in data pairs, minimizing redundancy and maximizing learning efficiency.

Implications and Results

Empirical evaluations demonstrate that the proposed MS loss method achieves competitive or superior performance across several image retrieval benchmarks, including CUB-200-2011, Cars-196, Stanford Online Products, and In-Shop Clothes Retrieval datasets. The improvements are noteworthy over state-of-the-art methods, including methods that use ensemble techniques.

Theoretical and Practical Implications

The GPW framework and MS loss have significant implications for both the theory and application of deep metric learning. By unifying various sampling and weighting methods, they offer a new lens through which to understand pair-based learning methodologies. Practically, these approaches enhance model performance in tasks requiring nuanced understanding of image similarities, such as image retrieval, face recognition, and person re-identification.

Speculation on Future Developments

The research opens avenues for further exploration into more advanced weighting schemes that incorporate additional aspects of pair dependencies. Future work could focus on refining the iterative process of mining and weighting to further enhance model robustness and performance in real-time applications. Additionally, adaptations of this framework could be explored in non-visual domains, potentially expanding its utility across various data modalities.

In conclusion, this paper's approach to deep metric learning through GPW and multi-similarity loss significantly impacts the landscape of pair-based learning by addressing the complexities of pair sampling and weighting in a unified manner. This advancement paves the way for more efficient and understood implementations of deep metric learning strategies.