Fast AutoAugment (1905.00397v2)

Abstract: Data augmentation is an essential technique for improving generalization ability of deep learning models. Recently, AutoAugment has been proposed as an algorithm to automatically search for augmentation policies from a dataset and has significantly enhanced performances on many image recognition tasks. However, its search method requires thousands of GPU hours even for a relatively small dataset. In this paper, we propose an algorithm called Fast AutoAugment that finds effective augmentation policies via a more efficient search strategy based on density matching. In comparison to AutoAugment, the proposed algorithm speeds up the search time by orders of magnitude while achieves comparable performances on image recognition tasks with various models and datasets including CIFAR-10, CIFAR-100, SVHN, and ImageNet.

• The paper introduces a novel density matching technique to search for augmentation policies that drastically reduce GPU-hours compared to traditional AutoAugment methods.
• The paper demonstrates comparable performance on datasets like CIFAR-10, CIFAR-100, SVHN, and ImageNet with significantly lower computational requirements.
• The paper leverages distributed frameworks and Bayesian optimization via TPE, paving the way for scalable and efficient data augmentation in computer vision.

Overview of "Fast AutoAugment"

This paper introduces Fast AutoAugment, an algorithm designed to address the computational inefficiency associated with traditional AutoAugment methods for data augmentation in deep learning models. Data augmentation is a crucial technique for enhancing the generalization abilities of machine learning models, particularly in the context of computer vision tasks. Traditional AutoAugment algorithms, although effective, require significant computational resources to discover optimal augmentation policies. Fast AutoAugment proposes a more efficient approach using a density matching technique, significantly reducing the computational burden while maintaining competitive performance.

Key Contributions

1. Efficient Search Strategy: Fast AutoAugment employs a novel search strategy based on density matching rather than the traditional reinforcement learning approach used by AutoAugment. This approach involves directly searching for augmentation policies that align the distribution of augmented and original datasets. By avoiding the repeated training of child models, the algorithm achieves a dramatic reduction in required GPU hours.
2. Comparative Performance: Fast AutoAugment is tested across multiple datasets, including CIFAR-10, CIFAR-100, SVHN, and ImageNet, where it demonstrates comparable performance to AutoAugment. The results show that Fast AutoAugment achieves similar or slightly better error rates with a fraction of the computational resources required by AutoAugment.
3. Implementation and Scalability: The proposed method leverages distributed computation frameworks like Ray, enhancing scalability and making it feasible to train and evaluate models efficiently. This approach highlights Fast AutoAugment's applicability to various model architectures and datasets without prohibitive computational costs.
4. Bayesian Optimization: The exploration of augmentation strategies is optimized using Bayesian techniques, specifically employing a Tree-structured Parzen Estimator (TPE) for practical implementation. This probabilistic approach allows for the effective tuning of augmentation policies by predicting promising regions in the search space.

Numerical Results

• GPU-Hour Reduction: Compared to AutoAugment, Fast AutoAugment reduces the GPU-hour requirement from 5000 to 3.5 for CIFAR-10, from 1000 to 1.5 for SVHN, and from 15000 to 450 for ImageNet.
• Performance Metrics: In the CIFAR-10 dataset, Fast AutoAugment achieves an error rate of 2.0% with Shake-Shake(26 2×96d) models, matching the best results from AutoAugment and surpassing the baseline performance.

Implications and Future Work

Fast AutoAugment provides a scalable and resource-efficient solution for data augmentation, contributing significantly to the field of automated machine learning (AutoML). This work paves the way for further exploration of joint optimization strategies combining neural architecture search (NAS) with efficient augmentation policies. Additionally, extending the application of Fast AutoAugment to broader computer vision tasks beyond image classification is a promising direction for future research.

In summary, Fast AutoAugment offers a compelling alternative to traditional augmentation strategies, significantly reducing computational overhead while maintaining performance, and thus broadening access to advanced data augmentation techniques.