Population Based Training of Neural Networks (1711.09846v2)

Abstract: Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm. In this work we present \emph{Population Based Training (PBT)}, a simple asynchronous optimisation algorithm which effectively utilises a fixed computational budget to jointly optimise a population of models and their hyperparameters to maximise performance. Importantly, PBT discovers a schedule of hyperparameter settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole course of training. With just a small modification to a typical distributed hyperparameter training framework, our method allows robust and reliable training of models. We demonstrate the effectiveness of PBT on deep reinforcement learning problems, showing faster wall-clock convergence and higher final performance of agents by optimising over a suite of hyperparameters. In addition, we show the same method can be applied to supervised learning for machine translation, where PBT is used to maximise the BLEU score directly, and also to training of Generative Adversarial Networks to maximise the Inception score of generated images. In all cases PBT results in the automatic discovery of hyperparameter schedules and model selection which results in stable training and better final performance.

• The paper introduces an adaptive hyperparameter optimization approach that evolves learning parameters during training to boost neural network performance.
• It combines exploitation of high-performing models with exploration of nearby hyperparameters to efficiently traverse the search space.
• Experimental results in reinforcement learning, translation, and GANs demonstrate that PBT outperforms traditional static tuning methods.

Population Based Training of Neural Networks

The paper Population Based Training of Neural Networks presents a novel approach to hyperparameter optimization and model training in neural networks through a methodology termed Population Based Training (PBT). PBT leverages a population of models to concurrently optimize neural network parameters and hyperparameters in an asynchronous manner. This approach deviates from the conventional practices in hyperparameter tuning, where a static set of hyperparameters is typically established before training begins and maintained throughout.

Key Contributions

1. Adaptivity in Hyperparameters: A primary innovation of PBT is the adaptive schedule for hyperparameters. Rather than sticking to a single set of hyperparameters, PBT evolves these hyperparameters over time, sharing the learnings across the population. This contrasts with traditional methods that tend to fix hyperparameters for the duration of the training, potentially leading to suboptimal behavior in dynamic, non-stationary learning conditions, such as those found in reinforcement learning environments.
2. Combination of Parallel and Sequential Search: PBT combines the strengths of parallel search methods (such as grid search or random search) and sequential optimization (like Bayesian optimization) to balance the computational cost and time efficiency. This coupling allows PBT to utilize fewer computational resources while avoiding multiple sequential training runs.
3. Effective Model Selection: The methodology employs an exploit-and-explore strategy, where poorly performing models in the population can adopt the weights and hyperparameters of better-performing models (exploit), and subsequently explore slight variations of these hyperparameters. This strategy ensures that resources are focused on promising areas of the hyperparameter space.

Experimental Validation

The efficacy of PBT is demonstrated across diverse domains, including deep reinforcement learning, supervised learning for machine translation, and the training of Generative Adversarial Networks (GANs).

Deep Reinforcement Learning

The paper reports substantial improvements in reinforcement learning tasks:

• DeepMind Lab: Training UNREAL agents using PBT achieved a significant increase in normalized human performance from 93% to 106%. PBT demonstrated automatic discovery of beneficial hyperparameter adaptations, such as the dynamic adjustment of unroll lengths and learning rates.
• Atari Learning Environment: Application of PBT to Feudal Networks on Atari games like Ms. Pacman and Gravitar achieved new state-of-the-art performance, benefiting from improved exploration-exploitation dynamics.
• StarCraft II: PBT-led training of A3C agents showed enhanced performance on several mini-games, with an average normalized human performance increase from 36% to 39%.

Machine Translation

In the field of supervised learning, PBT applied to Transformer networks for the WMT 2014 English-to-German translation task resulted in BLEU score improvements. PBT not only matched but also surpassed the performance of highly tuned traditional schedules, achieving a BLEU score enhancement from 22.30 to 22.65. The adaptive learning rate schedules discovered by PBT resembled those hand-tuned, however refined dynamically during training.

Generative Adversarial Networks

The paper also explores the application of PBT to the training of GANs:

• CIFAR-10 Dataset: PBT optimizations led to significant improvements in Inception scores, outperforming traditional baseline methods considerably (6.45 to 6.89). Importantly, PBT uncovered complex, non-monotonic learning rate schedules that were hitherto unconsidered by human experts or simpler heuristic methods.

Theoretical and Practical Implications

The theoretical implications of PBT are profound. By enabling hyperparameters to adapt dynamically, PBT addresses the intrinsic non-stationarity in complex learning problems. The automatic adaptation and tuning capabilities reduce the manual effort required in typically labor-intensive hyperparameter optimization.

Practically, PBT shows immense potential for automating the optimization process in new and unfamiliar models, thereby expediting the research and development process in AI. Future work may explore extensions of PBT to even broader domains, including more sophisticated neural architectures and hybrid meta-learning frameworks.

Conclusion

Population Based Training introduces an innovative paradigm for neural network optimization by harmonizing hyperparameter tuning and model training into a cohesive, adaptive process. The empirical results across various domains underscore the robustness and versatility of PBT, making it a compelling methodology for advancing the efficiency and performance of neural network-based systems. This bridging of hyperparameter optimization and model training promises to propel future research in machine learning methodology, enabling more sophisticated and capable AI systems.