Embedding Propagation: Smoother Manifold for Few-Shot Classification (2003.04151v2)

Abstract: Few-shot classification is challenging because the data distribution of the training set can be widely different to the test set as their classes are disjoint. This distribution shift often results in poor generalization. Manifold smoothing has been shown to address the distribution shift problem by extending the decision boundaries and reducing the noise of the class representations. Moreover, manifold smoothness is a key factor for semi-supervised learning and transductive learning algorithms. In this work, we propose to use embedding propagation as an unsupervised non-parametric regularizer for manifold smoothing in few-shot classification. Embedding propagation leverages interpolations between the extracted features of a neural network based on a similarity graph. We empirically show that embedding propagation yields a smoother embedding manifold. We also show that applying embedding propagation to a transductive classifier achieves new state-of-the-art results in mini-Imagenet, tiered-Imagenet, Imagenet-FS, and CUB. Furthermore, we show that embedding propagation consistently improves the accuracy of the models in multiple semi-supervised learning scenarios by up to 16\% points. The proposed embedding propagation operation can be easily integrated as a non-parametric layer into a neural network. We provide the training code and usage examples at https://github.com/ElementAI/embedding-propagation.

• The paper presents Embedding Propagation as a novel non-parametric regularizer that smooths the embedding manifold for improved few-shot classification.
• It leverages a propagator matrix via label propagation on feature similarity graphs to enhance transductive and semi-supervised learning.
• Empirical results on miniImagenet and CUB demonstrate significant accuracy gains, including 59.32% on 1-shot miniImagenet and 87.75% on 1-shot CUB.

Embedding Propagation: Smoother Manifold for Few-Shot Classification

In the domain of few-shot classification (FSC), the challenge lies in learning to recognize new classes from a limited number of labeled examples. The paper "Embedding Propagation: Smoother Manifold for Few-Shot Classification" proposes an innovative technique called Embedding Propagation (EP) that serves as a non-parametric regularizer to improve generalization in FSC by smoothing the manifold. The authors demonstrate the effectiveness of EP in enhancing the performance of both transductive and semi-supervised learning scenarios and achieve state-of-the-art results on benchmark datasets such as miniImagenet, tieredImagenet, Imagenet-FS, and CUB.

Theoretical Underpinnings and Methodology

The proposed EP method builds upon the concept of smoothing decision boundaries within the embedding space, a notion previously associated with methodologies like manifold mixup. By interpolating between network-extracted features based on a similarity graph, EP provides an embedding manifold that is inherently smoother. This approach is distinctively different from other regularization techniques as it operates in an unsupervised manner and is applicable as a non-parametric layer. The propagator matrix is derived through label propagation using the Laplacian of the adjacency matrix formed by feature similarities. The adaptability of EP allows it to be seamlessly integrated into existing neural networks as demonstrated with the introduction of EPNet.

Empirical Results

The experimental results underscore the significance of EP in FSC by reporting substantial performance gains. For instance, EPNet surpasses previous methods by notable margins in various scenarios — achieving 59.32% accuracy on 1-shot miniImagenet and a remarkable 87.75% on 1-shot CUB with WRN-28-10 as the backbone. Even more telling is the consistent improvement observed in semi-supervised learning setups, where EPNet exhibits accuracy improvements of up to 16 percentage points in the 1-shot SSL settings with additional unlabeled samples.

Algorithmic Insights and Ablations

The authors provide comprehensive ablation studies to dissect the contributions of the EP component relative to other model components such as label propagation and rotation losses. The evidence suggests that EP, when combined with label propagation, significantly smooths the decision boundaries and leverages neighborhood information for improved classification performance. This is further supported by their ablation of the propagator matrix, illustrating the necessity of leveraging off-diagonal neighbor information for maximum benefit.

Implications and Future Directions

The introduction of EP not only advances the performance boundaries in FSC but also opens pathways for further exploration in manifold regularization strategies across varied machine learning paradigms. The methodology's scalability and its demonstrated efficacy on large-scale datasets such as Imagenet-FS point towards potential applications in real-world settings where data scarcity is a constraint. Future research could explore broader applications of EP beyond few-shot scenarios, such as adversarial robustness or unsupervised domain adaptation.

In summary, the embedding propagation method represents an effective advancement in smoothing the learning manifold for few-shot classification, demonstrating superior generalization capabilities and offering a valuable technique applicable across diverse machine learning models.