RandAugment: Practical automated data augmentation with a reduced search space (1909.13719v2)

Abstract: Recent work has shown that data augmentation has the potential to significantly improve the generalization of deep learning models. Recently, automated augmentation strategies have led to state-of-the-art results in image classification and object detection. While these strategies were optimized for improving validation accuracy, they also led to state-of-the-art results in semi-supervised learning and improved robustness to common corruptions of images. An obstacle to a large-scale adoption of these methods is a separate search phase which increases the training complexity and may substantially increase the computational cost. Additionally, due to the separate search phase, these approaches are unable to adjust the regularization strength based on model or dataset size. Automated augmentation policies are often found by training small models on small datasets and subsequently applied to train larger models. In this work, we remove both of these obstacles. RandAugment has a significantly reduced search space which allows it to be trained on the target task with no need for a separate proxy task. Furthermore, due to the parameterization, the regularization strength may be tailored to different model and dataset sizes. RandAugment can be used uniformly across different tasks and datasets and works out of the box, matching or surpassing all previous automated augmentation approaches on CIFAR-10/100, SVHN, and ImageNet. On the ImageNet dataset we achieve 85.0% accuracy, a 0.6% increase over the previous state-of-the-art and 1.0% increase over baseline augmentation. On object detection, RandAugment leads to 1.0-1.3% improvement over baseline augmentation, and is within 0.3% mAP of AutoAugment on COCO. Finally, due to its interpretable hyperparameter, RandAugment may be used to investigate the role of data augmentation with varying model and dataset size. Code is available online.

An Analytical Overview of "RandAugment: Practical Automated Data Augmentation with a Reduced Search Space"

In the arena of deep learning, data augmentation has emerged as a pivotal technique to boost model generalization. The paper "RandAugment: Practical Automated Data Augmentation with a Reduced Search Space" by Cubuk et al. tackles some pressing challenges associated with automated data augmentation, introducing a novel method called RandAugment. This method aims to enhance model performance while simplifying the complexities associated with searching for optimal augmentation policies.

Key Contributions and Methodology

The paper begins by illuminating the intricacies and computational demands of traditional automated augmentation strategies. Previous methodologies often required a separate optimization phase on a smaller proxy task, which not only added to the computational burden but also led to sub-optimal generalization due to the disparity between proxy and target tasks. RandAugment adeptly addresses these issues through the following contributions:

1. Simplified Search Space: RandAugment introduces a drastically reduced search space characterized by only two interpretable hyperparameters: the number of augmentation transformations (N) and the global distortion magnitude (M). This simplification enables direct optimization on the target task without a separate proxy phase, significantly easing the computational overhead.
2. Uniform Operation Selection: The method eliminates the need for learned probabilities in selecting augmentation operations by uniformly applying transformations from a predefined set. This uniformity retains sufficient versatility in augmentations while eradicating the complexity of probability-based selection.
3. State-of-the-Art Performance: Extensive experimental results demonstrate that RandAugment achieves competitive or superior performance across various datasets, including CIFAR-10/100, SVHN, and ImageNet. Notably, it achieves 85.0% accuracy on ImageNet, marking a 0.6% improvement over previous state-of-the-art methods and a 1.0% gain over baseline augmentations.

Experimental Insights and Results

The experiments conducted reinforce the versatility and efficacy of RandAugment. A notable aspect of the approach is its applicability across a range of tasks and architectures. The simplified search space suffices to outperform previous automated augmentation methods that required exhaustive search processes.

• Performance Across Tasks:
  • On CIFAR-10, RandAugment matches or surpasses other methods, indicating robust performance across different model architectures, including Wide-ResNet and PyramidNet.
  • Similarly, significant gains are observed on CIFAR-100, with RandAugment performing consistently across varying model complexities.
  • For SVHN, the method demonstrates its generalization capability by achieving state-of-the-art results, highlighting its adaptability to different data domains.
• ImageNet and COCO:
  • The method's performance on larger scales is critically evaluated using ImageNet and COCO datasets.
  • On ImageNet, RandAugment outperforms both AutoAugment and Fast AutoAugment, particularly on larger models such as EfficientNet-B7, by achieving state-of-the-art accuracy with minimal computational augmentation.
  • In object detection tasks on COCO, RandAugment provides competitive results close to those produced by more computationally intensive methods, attesting to its efficiency and practical applicability.

Implications and Future Directions

RandAugment's contributions go beyond improving predictive performance by addressing computational challenges inherent in traditional automated augmentation methods. The reduced search space and elimination of a separate proxy search stage present significant implications:

• Scalability: The method's simplified approach ensures scalability to larger datasets and models, making it feasible to apply robust data augmentation across different domains with minimal overhead.
• Flexibility in Hyperparameters: The interpretability and flexibility of the hyperparameters N and M allow for straightforward tuning, making the method accessible for practical applications without demanding extensive computational resources.

Speculation on Future Developments

Anticipating future developments, the paper indicates potential exploration in several directions:

• Model Robustness: Investigating how RandAugment enhances model robustness against data corruptions and adversarial attacks can provide deeper insights into its strengths and limitations.
• Application in Semi-Supervised Learning: Extending the method's application to semi-supervised and unsupervised learning paradigms could unlock further performance gains, particularly in scenarios with limited labeled data.
• Expansion to Other Domains: Application of RandAugment beyond vision tasks, such as in speech recognition or 3D perception, represents a promising avenue for future work, potentially leading to advancements in a variety of machine learning fields.

In conclusion, RandAugment presents a practical, efficient solution to automated data augmentation, offering significant performance gains while addressing the computational complexity of existing methods. The paper establishes a foundation for scalable, interpretable augmentation strategies that can be seamlessly integrated into diverse machine learning workflows.