Learning Metrics from Teachers: Compact Networks for Image Embedding (1904.03624v1)

Abstract: Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)

• The paper introduces novel network distillation methods for metric learning by leveraging absolute and relative teacher loss functions.
• The relative teacher loss significantly boosts performance, increasing Recall@1 from 27.5% to 44.6% in compact networks.
• The approach enables efficient image embedding on mobile devices, achieving teacher-level results with substantially smaller models.

Learning Metrics from Teachers: Compact Networks for Image Embedding

The paper "Learning Metrics from Teachers: Compact Networks for Image Embedding" presents a novel approach to construct image embedding networks efficiently by employing network distillation techniques. The research primarily focuses on metric learning, which is integral to tasks such as image retrieval and face recognition, by preserving semantic similarity in embedding spaces. The authors propose innovative modifications to the distillation process, contributing to metric learning applications.

Overview

The paper illustrates the limitations of conventional deep neural networks due to computational complexity, particularly in scenarios involving mobile or resource-constrained environments. To tackle this, the authors leverage the concept of network distillation, traditionally utilized for image classification, to metric learning networks. They introduce two novel loss functions aimed at optimizing the transfer of knowledge from large teacher networks to smaller student networks.

Novel Contributions

1. Distillation Loss Functions: The paper introduces two distinctive loss functions—the absolute teacher and the relative teacher. The absolute teacher minimizes the distance between student and teacher embeddings. In contrast, the relative teacher optimizes the preservation of pairwise distances within embedding spaces. The latter approach, as demonstrated, consistently surpasses the former, indicating its suitability for embedding applications.
2. Empirical Evaluations: The efficiency of the proposed methods was evaluated through rigorous experiments conducted on datasets like CUB-200-2011, Cars-196, and Stanford Online Products. Remarkably, the distillation methods yield substantial improvements, such as a significant increase from 27.5\% to 44.6\% in Recall@1 for compact networks on mobile devices.

Key Results

The results showcase robust performance improvements across various configurations. For instance, student networks trained with the relative distillation method not only matched but in some cases exceeded teacher performance, despite having substantially fewer parameters. The method effectively transcends traditional constraints posed by network size and labeled dataset availability, facilitating broader applicability.

Practical and Theoretical Implications

From a practical viewpoint, this research opens pathways for deploying sophisticated image embedding networks efficiently in environments with limited computational capacity, such as mobile devices. Theoretically, the paper enriches the field of metric learning by extending network distillation's applicability beyond classification to embedding tasks, thereby adding depth to the existing body of knowledge.

Speculations on Future Developments

This research sets the foundation for further exploration into embedding distillation using diverse types of networks and data domains. The concept of incorporating unlabeled data via distillation holds promise for semi-supervised learning methods, potentially advancing further developments in AI, particularly in the field of unsupervised and transfer learning.

Overall, the paper delivers a compelling perspective on optimizing image embedding networks, enhancing efficacy without compromising performance through innovative distillation techniques.