SuperMix: Supervising the Mixing Data Augmentation (2003.05034v2)

Abstract: This paper presents a supervised mixing augmentation method termed SuperMix, which exploits the salient regions within input images to construct mixed training samples. SuperMix is designed to obtain mixed images rich in visual features and complying with realistic image priors. To enhance the efficiency of the algorithm, we develop a variant of the Newton iterative method, $65\times$ faster than gradient descent on this problem. We validate the effectiveness of SuperMix through extensive evaluations and ablation studies on two tasks of object classification and knowledge distillation. On the classification task, SuperMix provides comparable performance to the advanced augmentation methods, such as AutoAugment and RandAugment. In particular, combining SuperMix with RandAugment achieves 78.2\% top-1 accuracy on ImageNet with ResNet50. On the distillation task, solely classifying images mixed using the teacher's knowledge achieves comparable performance to the state-of-the-art distillation methods. Furthermore, on average, incorporating mixed images into the distillation objective improves the performance by 3.4\% and 3.1\% on CIFAR-100 and ImageNet, respectively. {\it The code is available at https://github.com/alldbi/SuperMix}.

• The paper introduces a supervised mixing augmentation technique that leverages salient image regions to enrich training data.
• It formulates an optimization problem with mixing masks, achieving a 65x speed-up over SGD on large datasets like ImageNet.
• Empirical results on CIFAR-100 and ImageNet demonstrate enhanced accuracy and robust performance in classification and knowledge distillation tasks.

Overview of SuperMix: Supervising the Mixing Data Augmentation

The paper "SuperMix: Supervising the Mixing Data Augmentation" presents a method named SuperMix, designed to enhance the quality and performance of training datasets by employing a supervised approach to mixing data augmentation. The core idea of SuperMix is to generate mixed training samples that are enriched with salient features by leveraging the input images' most informative regions. This is particularly significant in the context of deep neural networks (DNNs), which are frequently susceptible to overfitting when faced with limited or qualitatively deficient datasets.

Methodology

SuperMix operates by identifying and combining salient regions across multiple images to construct new, informative images used in training. The method is distinct from previous unsupervised mixing augmentation techniques like MixUp and CutMix, which often blend images without regard to spatially salient areas, leading to diluted visual patterns and ineffective pseudo labels.

The methodology includes:

• A formalized problem of supervised mixing augmentation, utilizing mixing masks that associate pixel importance across multiple images.
• An optimization problem ensuring generated images are rich in features while adhering to realistic image priors, balancing local smoothness and sparsity constraints.
• A modified Newton iterative approach that achieves significant computational efficiency, demonstrating a 65x speed-up compared to stochastic gradient descent (SGD) on large datasets like ImageNet.

SuperMix's design allows for the extraction and combination of salient features across several images, supervised by either the target model itself or a more advanced teacher model. This is achieved through knowledge distillation, where the combination learns from the superior model’s output.

Results

The empirical evaluation covers two tasks: object classification and knowledge distillation, with extensive assessments on CIFAR-100 and ImageNet benchmarks. In the object classification task, SuperMix performs on par with advanced augmentation methods like AutoAugment and RandAugment. Specifically, SuperMix coupled with RandAugment achieves remarkable results, such as a 78.2% top-1 accuracy on ImageNet with a ResNet50 architecture.

In knowledge distillation tasks, SuperMix mixed images that were classified using a teacher's knowledge, yielding results comparable to state-of-the-art methods. Notably, the integration of SuperMix resulted in average performance improvements of 3.4% on CIFAR-100 and 3.1% on ImageNet.

Implications and Future Directions

The implications of this research are substantial for both practical applications and theoretical advancements in machine learning and computer vision. SuperMix showcases an approach to data augmentation that not only increases dataset size but also enhances the dataset quality by ensuring that the augmented data aligns more closely with realistic image priors and salient features. The promising results in both classification and distillation tasks indicate that supervised mixing could be a powerful tool in training more robust and generalized models.

As artificial intelligence continues to evolve, future work might explore the integration of SuperMix into more complex learning paradigms, potentially extending beyond image data to include other domains like text and signal processing. Additionally, further exploration into the scalability of supervised mixing augmentations for larger datasets or higher-dimensional data could offer significant advancements in areas such as autonomous driving, drug discovery, and beyond. The intersection of supervised data augmentation and advanced model architectures holds potential for creating more versatile AI systems.