Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity (2102.03065v1)

Abstract: While deep neural networks show great performance on fitting to the training distribution, improving the networks' generalization performance to the test distribution and robustness to the sensitivity to input perturbations still remain as a challenge. Although a number of mixup based augmentation strategies have been proposed to partially address them, it remains unclear as to how to best utilize the supervisory signal within each input data for mixup from the optimization perspective. We propose a new perspective on batch mixup and formulate the optimal construction of a batch of mixup data maximizing the data saliency measure of each individual mixup data and encouraging the supermodular diversity among the constructed mixup data. This leads to a novel discrete optimization problem minimizing the difference between submodular functions. We also propose an efficient modular approximation based iterative submodular minimization algorithm for efficient mixup computation per each minibatch suitable for minibatch based neural network training. Our experiments show the proposed method achieves the state of the art generalization, calibration, and weakly supervised localization results compared to other mixup methods. The source code is available at https://github.com/snu-mllab/Co-Mixup.

• The paper presents a saliency-based mixup method that optimizes data mixing for improved neural network performance.
• The method leverages supermodular functions to ensure diverse mixup samples, enhancing calibration and robustness.
• Experiments on benchmarks like CIFAR-100 and ImageNet demonstrate tangible gains in accuracy and object localization.

Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity

The paper introduces Co-Mixup, a novel data augmentation technique designed to improve the generalization performance and robustness of deep neural networks. The core idea of Co-Mixup lies in its advanced data mixing strategy, which intelligently maximizes saliency measures and diversity when constructing mixup examples. This approach challenges existing mixup methods by addressing the limitations of using only random pairs of input data for interpolation, thereby proposing a more structured and optimization-driven solution to batch mixup.

Methodology

The authors present a novel optimization perspective on batch mixup, which is framed as a discrete optimization problem. This problem aims to maximize the saliency of each mixup example while encouraging supermodular diversity within the batch. The approach considers mix-matching multiple salient regions across multiple inputs, which aids in accumulating rich supervisory signals that are often lost in classical mixup methods.

Key Contributions:

• Saliency Optimization: The Co-Mixup method leverages gradient-based saliency maps to inform the mixup process, ensuring that the generated data is both salient and relevant to the task.
• Diversity through Supermodular Functions: Using a supermodular function to ensure diversity among mixup examples helps prevent the common pitfall of generating overly similar data.
• Efficient Algorithm: The paper proposes a modular approximation method for iterative submodular minimization, facilitating practical computation for neural network training.

Experimental Results

The experimental results unequivocally demonstrate the superiority of Co-Mixup over existing mixup methods. Evaluations conducted on standard benchmarks such as CIFAR-100, Tiny-ImageNet, and ImageNet reveal significant improvements in generalization performance. Additionally, Co-Mixup achieves better calibration and robustness, especially noticeable in weakly supervised localization tasks.

1. Classification Accuracy: Co-Mixup consistently outperforms traditional methods like CutMix and Puzzle Mix, reducing the Top-1 error by margins up to 0.75% on well-established datasets.
2. Enhanced Localization Performance: The method shows improved weakly supervised object localization accuracy, highlighting its strength in preserving essential features during the augmentation process.
3. Calibration and Robustness: Co-Mixup achieves superior expected calibration error and robustness to background corruption, demonstrating its effectiveness in producing well-generalized and reliable models.

Implications and Future Directions

Theoretical and Practical Implications:

Co-Mixup provides a theoretically robust framework for data augmentation that capitalizes on the optimization of saliency and diversity. Practically, it sets a new standard for constructing augmented datasets that help mitigate overfitting and improve model robustness.

Potential Future Applications:

The principles of Co-Mixup can potentially be extended beyond classification tasks to more complex scenarios, such as multi-label classification and object detection. Its application could also be fruitful in domains requiring high interpretability and robustness, such as medical imaging and autonomous driving.

In conclusion, Co-Mixup presents a significant advancement in the field of data augmentation for deep learning. By embedding a saliency-guided approach into the mixup paradigm, it offers both a theoretical framework and practical toolset for future research and applications, encouraging further exploration into optimization-driven data generation techniques.