Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2106.08962v2)

Abstract: Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

The field of deep learning has witnessed remarkable advancements, revolutionizing domains such as computer vision, natural language understanding, and speech recognition. However, with these advancements has come a proliferation of model complexity, parameter count, and computational requirements. This paper provides a detailed survey focused on enhancing the efficiency of deep learning models, ensuring they are not only accurate but also computationally feasible for deployment in real-world scenarios. The survey highlights key areas instrumental to improving model efficiency, emphasizing both theoretical insights and practical implications.

Core Areas of Model Efficiency

The paper categorizes model efficiency into five core areas, each tackling specific aspects of model optimization:

1. Compression Techniques: These involve reducing model size through methods like pruning and quantization. Notably, pruning techniques such as Optimal Brain Damage and Optimal Brain Surgeon have demonstrated significant parameter reduction without compromising model accuracy. Quantization, particularly post-training quantization and quantization-aware training, offers reductions in model size and inference latency, achieving a 4x decrease in model size with 8-bit quantization.
2. Learning Techniques: Approaches like distillation, where a smaller model learns from a pre-trained larger model, have significantly enhanced model performance while reducing size. Data augmentation and self-supervised learning provide avenues to improve model training and accuracy, thus achieving equivalent or improved performance with fewer resources.
3. Automation: Hyper-parameter optimization (HPO) and neural architecture search (NAS) represent automation techniques that optimize model parameters and architecture. Bayesian Optimization and methods like Population Based Training help automate the search for optimal model configurations, reducing manual intervention and time.
4. Efficient Architectures: The development of architectures like depthwise separable convolutions in MobileNet and attention mechanisms in Transformers exemplifies designing models with innate efficiency. These architectures achieve state-of-the-art performance while being computationally less demanding.
5. Infrastructure: Efficient model deployment hinges on robust software and hardware infrastructure. Tools within the Tensorflow and PyTorch ecosystems, coupled with hardware optimization libraries and accelerators like GPUs and TPUs, facilitate efficient model training and inference.

Practical Implications and Speculations

The implications of this survey extend into practical deployment and future research directions in AI. By leveraging efficient deep learning techniques, practitioners can deploy models on resource-limited devices, such as IoT devices and smartphones, while sustaining or even enhancing model performance. The insights provided equip researchers with strategies to balance model quality and computational demands, enabling scalable AI solutions.

Future developments in AI are likely to explore deeper the integration of efficiency-centric approaches in model design and training paradigms. As machine learning models expand into diverse applications, optimizing for computational efficiency will remain paramount. Continuing advancements in hardware acceleration and the evolution of self-supervised learning could redefine the boundaries of what is achievable with efficient deep learning.

In conclusion, this survey acts as a comprehensive guide for researchers and practitioners aiming to navigate the complex landscape of deep learning model efficiency. By integrating insights from each core area, the path towards smaller, faster, and better models becomes more accessible, fostering innovation in efficient AI technologies.