Energy and Policy Considerations for Deep Learning in NLP (1906.02243v1)

Abstract: Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.

• The paper presents quantitative measurements of energy use and carbon emissions in training models like BERT, GPT-2, and NAS.
• It highlights the substantial financial costs of hyperparameter tuning and model architecture searches, running into hundreds of thousands of dollars.
• The study proposes policy recommendations including standardized reporting and efficient algorithms to democratize access to high-performance computing.

Energy and Policy Considerations for Deep Learning in NLP

The paper "Energy and Policy Considerations for Deep Learning in NLP" by Emma Strubell, Ananya Ganesh, and Andrew McCallum presents a detailed analysis of the financial and environmental costs associated with training state-of-the-art NLP models. The authors underscore the substantial computational resources required to achieve the high accuracy observed in modern NLP tasks, and they quantify these costs in terms of energy consumption, carbon emissions, and monetary expenses.

Computational and Environmental Costs

The paper provides comprehensive quantifications of the energy and carbon footprint generated by training several prominent NLP models, including Transformer, ELMo, BERT, and GPT-2. The authors use empirical data to estimate the kilowatt-hours (kWh) consumed by GPU and CPU during the training processes. For example, training a BERT base model on 64 Tesla V100 GPUs emits approximately 1438 lbs of CO₂, while the training of the NAS model using 8 P100 GPUs emits around 626,155 lbs of CO₂. This staggering amount of carbon footprint underscores the critical issue of environmental sustainability in computational research.

Financial Implications

The financial costs associated with training these models are equally significant. The paper estimates that the cost of training multiple models, hyperparameter tuning, and performing architecture searches can run into hundreds of thousands of dollars. For instance, training the NAS model on P100 GPUs has an estimated cost between \$942,973 and \$3,201,722, while developing a new NLP model like LISA over six months has estimated expenses ranging from \$103,000 to \$350,000.

Policy Recommendations

1. Reporting Training Time and Sensitivity:

The authors propose that researchers should report both the training time and computational resources required for developing new NLP models. This is intended to facilitate comparative analysis and enable others to evaluate the reproducibility and practicality of adopting these models for different applications.

2. Equitable Access to Computational Resources:

Highlighting disparities in access to high-performance computing resources, the authors advocate for creating shared, centralized compute centers funded by government agencies. They argue that such facilities could democratize access to computational power, allowing a broader range of researchers to participate in cutting-edge NLP research.

3. Prioritizing Efficiency in Algorithms and Hardware:

There is a call for a concerted effort from both industry and academia to focus on developing more computationally efficient algorithms and energy-efficient hardware. Emphasizing the current inefficient practices in hyperparameter tuning, the authors suggest integrating more efficient search techniques such as Bayesian optimization into popular deep learning frameworks to reduce the energy consumption associated with model training and refinement.

Experimental Results

The paper presents an analysis of the energy use of common hardware configurations such as NVIDIA Titan X, GTX 1080 Ti, and P100 GPUs. The authors' case paper of the LISA model development reveals the extensive resources involved in tuning hyperparameters and model iterations, demonstrating the energy and cost-intensive nature of the R&D process.

Implications and Future Research

The implications of this research are manifold. Practically, it provides tangible metrics for the financial and environmental costs of current practices in NLP, promoting a more environmentally conscious approach in future studies. Theoretically, it challenges the community to innovate more efficient algorithms and hardware that can achieve high accuracy without incurring such high costs.

Future developments in this area could include the design of low-power accelerators, optimization of hyperparameter search methods, and advancements in cloud services that prioritize sustainability. Researchers might also explore decentralized approaches that leverage underutilized computing resources in more eco-friendly manners.

Conclusion

The paper by Strubell et al. serves as a crucial reminder of the hidden costs behind the significant strides made in NLP through deep learning. By presenting an in-depth analysis of the energy, financial, and environmental aspects of model training, it advocates for more sustainable practices in computational research. The proposed recommendations aim to foster a more equitable and environmentally responsible research environment in the NLP field.