Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (1905.05393v1)

Abstract: A key challenge in leveraging data augmentation for neural network training is choosing an effective augmentation policy from a large search space of candidate operations. Properly chosen augmentation policies can lead to significant generalization improvements; however, state-of-the-art approaches such as AutoAugment are computationally infeasible to run for the ordinary user. In this paper, we introduce a new data augmentation algorithm, Population Based Augmentation (PBA), which generates nonstationary augmentation policy schedules instead of a fixed augmentation policy. We show that PBA can match the performance of AutoAugment on CIFAR-10, CIFAR-100, and SVHN, with three orders of magnitude less overall compute. On CIFAR-10 we achieve a mean test error of 1.46%, which is a slight improvement upon the current state-of-the-art. The code for PBA is open source and is available at https://github.com/arcelien/pba.

Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules

The paper "Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules" presents a novel approach to data augmentation in neural network training, addressing the computational inefficiency of existing methods like AutoAugment. The authors introduce Population Based Augmentation (PBA), a method that leverages dynamic, nonstationary augmentation policy schedules instead of a fixed policy. This approach significantly reduces the computational resources required, achieving performance metrics on par with AutoAugment while using substantially less compute power.

Key Contributions

1. Computational Efficiency:
   • PBA is demonstrated to achieve results comparable to AutoAugment on datasets such as CIFAR-10, CIFAR-100, and SVHN, with at least three orders of magnitude less computational effort. For instance, on CIFAR-10, PBA achieves a mean test error of 1.46%, slightly improving upon current state-of-the-art results.
2. Dynamic Scheduling:
   • Unlike AutoAugment, which uses a fixed augmentation policy, PBA employs nonstationary schedules that evolve throughout the training epochs. This flexibility allows for more efficient augmentation strategies tailored to different phases of training.
3. Open Source and Practicality:
   • The implementation of PBA is made available open source, facilitating experimentation and practical application on typical workstations.

Empirical Findings

PBA demonstrates comparable performance to AutoAugment across multiple model architectures and datasets. The paper provides a comprehensive numerical validation, such as achieving test errors of 2.03% on Shake-Shake (26 2x96d) models and 1.46% on PyramidNet with ShakeDrop. The experiments show that augmentation schedules learned by PBA help attain competitive results with significantly lower computational costs.

Comparison with AutoAugment

AutoAugment uses reinforcement learning to explore a large discrete space of augmentation policies, requiring extensive computational resources. Conversely, PBA efficiently searches for optimal schedules with Population Based Training (PBT), reducing pre-computation costs dramatically. For instance, generating an augmentation policy with PBA on CIFAR-10 costs approximately $7.5 compared to$7,500 with AutoAugment.

Mechanism and Implementation

PBA utilizes the Population Based Training framework, enabling simultaneous training of multiple model versions and exploration of various augmentation schedules. Unlike previous methods, PBA focuses on schedules adapted over epochs, leading to a more resource-efficient exploration. The PBT algorithm facilitates a balanced trade-off between exploitation of promising policies and exploration of new possibilities.

Implications and Future Directions

The introduction of PBA opens several promising avenues for future research in machine learning:

• Further Optimization: PBA can inspire further research into optimizing the balance between computational cost and augmentation effectiveness, especially for domain-specific datasets.
• Integration in Larger Frameworks: The integration of augmentation schedules within larger machine learning pipelines could be explored, potentially enhancing transfer learning applications.
• Parameter Tuning and Hybrid Approaches: Investigating hybrid approaches that combine PBA with fixed policy methods, or optimizing hyperparameters through meta-learning techniques, could yield further improvements.

Conclusion

Population Based Augmentation presents a significant advancement in the efficient learning of data augmentation policies by shifting from fixed to dynamic schedules. It not only offers a promising alternative to existing computationally heavy methods but also paves the way for future developments in augmentation techniques that adapt and evolve with training dynamics. The paper successfully outlines a clear path forward for practical implementation and continued innovation in data augmentation strategies.