No Fuss Distance Metric Learning using Proxies (1703.07464v3)

Abstract: We address the problem of distance metric learning (DML), defined as learning a distance consistent with a notion of semantic similarity. Traditionally, for this problem supervision is expressed in the form of sets of points that follow an ordinal relationship -- an anchor point $x$ is similar to a set of positive points $Y$, and dissimilar to a set of negative points $Z$, and a loss defined over these distances is minimized. While the specifics of the optimization differ, in this work we collectively call this type of supervision Triplets and all methods that follow this pattern Triplet-Based methods. These methods are challenging to optimize. A main issue is the need for finding informative triplets, which is usually achieved by a variety of tricks such as increasing the batch size, hard or semi-hard triplet mining, etc. Even with these tricks, the convergence rate of such methods is slow. In this paper we propose to optimize the triplet loss on a different space of triplets, consisting of an anchor data point and similar and dissimilar proxy points which are learned as well. These proxies approximate the original data points, so that a triplet loss over the proxies is a tight upper bound of the original loss. This proxy-based loss is empirically better behaved. As a result, the proxy-loss improves on state-of-art results for three standard zero-shot learning datasets, by up to 15% points, while converging three times as fast as other triplet-based losses.

• The paper’s key contribution is redefining triplet loss with proxies to tighten upper bounds and accelerate convergence.
• It employs proxies to simplify training by approximating data points and reducing the computational complexity of triplet mining.
• Experiments on zero-shot datasets reveal up to 15% improvement in Recall@1 and convergence speeds three times faster than traditional methods.

Overview of "No Fuss Distance Metric Learning using Proxies"

The paper "No Fuss Distance Metric Learning using Proxies" by Movshovitz-Attias et al. presents an innovative approach to Distance Metric Learning (DML) that focuses on optimizing the traditional triplet-based methodology by introducing proxies. These proxies simplify the optimization space and alleviate the challenges associated with the conventional triplet mining strategies. The authors propose a proxy-based loss that not only facilitates faster convergence but also achieves superior performance on benchmark zero-shot learning datasets.

Key Contributions

The core contribution lies in redefining triplet-based losses to a proxy-based formulation. Instead of directly optimizing the triplets of data points, the method employs proxies that approximate these data points. This reformulation ensures that the proxy-based triplet loss is a tighter upper bound of the original triplet loss, thus enhancing the stability and convergence speed of the learning process. Notably, the experimental results exhibit improvements in state-of-the-art performance on standard datasets by up to 15 percentage points and achieve convergence three times faster than traditional methods.

Methodology

The approach addresses DML by learning a distance metric through proxies, enabling the following advancements:

• Proxy-Based Loss: Proxies act as compact representatives of data points, significantly reducing the complexity associated with sampling effective triplets. This is achieved by defining a triplet over an anchor point and its respective proxy positives and negatives.
• Upper Bound Tightness: The proxy-based loss serves as an empirical upper bound to the original triplet loss, allowing the model to more effectively approximate the desired distance relationships within the data.
• Reduction in Complexity: By condensing the training set into a smaller set of proxies, the number of triplets to be considered reduces from $O(n^3)$ to $O(n|P|^2)$, where $|P|$ is the number of proxies, greatly simplifying the computational overhead.

Experimental Results

The proposed method was evaluated on three widely-used zero-shot learning datasets: CUB200, Cars196, and Stanford Products. The results demonstrated notable achievements:

• Cars196 Dataset: Achieves a 15% improvement in Recall@1 measure, coupled with faster convergence, demonstrating the method's efficiency and effectiveness.
• Stanford Online Products: Proxy-NCA results in a significant gain of over 6% in the Recall@1 metric over previous methods, affirming the utility of proxies for capturing nuanced semantic distances.
• CUB200 Dataset: Marked improvements were seen in clustering quality, indicated by an NMI of 59.53, showcasing enhanced retrieval and classification effectiveness.

Implications and Future Directions

The introduction of proxies provides substantial practical benefits by reducing training times and improving metric learning performance. The use of proxies in DML opens avenues for further exploration into scalable distance learning methods that can generalize effectively across varied domains with large class labels. Future research could extend this proxy paradigm to additional applications and evaluate the adaptability of proxies under varying data distributions and complex settings.

In conclusion, leveraging proxies for optimizing distance metric learning offers a promising direction for efficiently tackling the inherent complexity in traditional methods. This work underscores the utility of relaxing direct triplet dependencies and introduces a methodological advance that could influence future research in DML and related fields.