Data-Efficient Image Recognition with Contrastive Predictive Coding (1905.09272v3)

Abstract: Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-the-art linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5x less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained ImageNet classifiers.

• The paper demonstrates that enhanced CPC v2 improves linear classification accuracy on ImageNet from 48.7% to 71.5% through revised architecture and training protocols.
• The method achieves robust data efficiency, reaching a 78% Top-5 accuracy with just 1% of ImageNet labels, significantly outperforming supervised baselines.
• The learned representations transfer effectively, yielding 76.6% mAP on PASCAL VOC 2007 for object detection compared to 74.7% from fully supervised pre-training.

Data-Efficient Image Recognition with Contrastive Predictive Coding

This paper addresses the challenge of data-efficient image recognition, proposing an enhanced version of the Contrastive Predictive Coding (CPC) framework. CPC, originally formulated by \citep{oord2018representation}, is an unsupervised method designed to learn representations by predicting future observations from past ones. The improved implementation, referred to as CPC v2, demonstrates significant advances in image recognition tasks with limited labeled data, achieving superior performance compared to conventional supervised learning approaches.

Key Contributions

1. Enhanced Architecture and Training Protocol: The authors present a revisited architecture and training methodology, which improve the linear classification accuracy on ImageNet from 48.7% to 71.5% in Top-1 accuracy, a remarkable enhancement of 23% absolute improvement.
2. Data-Efficient Supervised Learning: Leveraging CPC v2 representations, the paper demonstrates significant improvements in image classification accuracy using vastly fewer labeled examples. For instance, when trained with 1% of ImageNet labels, CPC-based models achieve 78% Top-5 accuracy, a 34% absolute improvement over baseline supervised models.
3. Superiority in Full Dataset Scenarios: CPC v2 representations not only excel in low-data regimes but also outperform fully supervised classifiers even when trained with the entire ImageNet dataset, achieving a Top-1 accuracy of 83.4%, compared to 80.2% for the best supervised baseline.
4. Effective Transfer Learning: The paper validates the generality of CPC v2 representations through transfer learning to object detection tasks on the PASCAL VOC 2007 dataset, reporting a performance (76.6% mAP) that surpasses fully supervised pre-training (74.7% mAP).

Experimental Setup

Contrastive Predictive Coding: The CPC model is framed to predict future feature vectors in spatial contexts within images. The approach involves dividing images into patches, encoding each patch independently, and using a masked convolutional network to predict future patches based on past context. Predictions are evaluated using a contrastive loss, ensuring that the model distinguishes between correct and incorrect future patches.

Representation Utilization: After training the CPC model, the authors employ the learned encoder representations for image recognition tasks. The representations are either used directly for linear classification or fine-tuned with a deeper network stack based on the labeled data available.

Improvements in CPC v2

The revision of CPC involved several key enhancements:

• Model Capacity: Increasing the depth and width of the ResNet encoder, transitioning from ResNet-101 to ResNet-161.
• Training Efficiency: Introducing layer normalization to replace batch normalization, avoiding dependencies that could lead to trivial solutions.
• Prediction Complexity: Extending spatial predictions in multiple directions (not only bottom-up but also from left, right, and top).
• Patch-Based Augmentation: Extensive augmentations were applied to individual patches, including random color-dropping and spatial transformations.

By combining these improvements, the CPC v2 significantly outperforms previous self-supervised methods, setting a new benchmark in linear classification tasks.

Implications and Future Directions

The findings of this paper underscore the potential of contrastive prediction methods in achieving data-efficient learning, presenting a viable alternative to fully supervised approaches. The demonstrated improvements suggest several implications for the field:

• Enhanced Models for Low-Data Domains: The techniques described could be especially beneficial in domains where labeled data is scarce, such as medical imaging or specialized scientific data.
• Broader Applications: Given that CPC is modality-agnostic, its application could extend to other domains, including audio, video, and multimodal environments.
• Increasing Representation Power: Future work may focus on integrating additional self-supervised tasks and modalities, further enhancing the richness of learned representations.

Conclusion

Overall, this paper provides a comprehensive overview of an improved self-supervised learning framework that significantly enhances data-efficient image recognition. The strong numerical results achieved with CPC v2 highlight its efficacy in both low-data and fully-supervised environments, paving the way for broader applications and further research in self-supervised learning and representation learning methodologies.