Green AI (1907.10597v3)

Abstract: The computations required for deep learning research have been doubling every few months, resulting in an estimated 300,000x increase from 2012 to 2018 [2]. These computations have a surprisingly large carbon footprint [38]. Ironically, deep learning was inspired by the human brain, which is remarkably energy efficient. Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers, in particular those from emerging economies, to engage in deep learning research. This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. Our goal is to make AI both greener and more inclusive---enabling any inspired undergraduate with a laptop to write high-quality research papers. Green AI is an emerging focus at the Allen Institute for AI.

Comprehensive Review of "Green AI"

The paper "Green AI," authored by Roy Schwartz et al., presents a compelling case for rethinking how research in AI, particularly deep learning, is conducted with an emphasis on computational efficiency. The authors point out the significant environmental and financial costs associated with the computational demands of modern AI research, arguing for a shift in evaluation criteria to include efficiency alongside accuracy.

Computational Trends in Deep Learning

The paper brings attention to an alarming trend in deep learning: the computational costs associated with training state-of-the-art models have skyrocketed, increasing by approximately 300,000 times from 2012 to 2018. This growth far exceeds what would be predicted by Moore's Law. Factors such as model size, the amount of training data, and the number of hyperparameter experiments are noted for contributing to these increasing costs. For example, the paper cites that training XLNet required 512 TPU chips over 2.5 days, incurring an estimated cost of $250,000, emphasizing resource-intensive methodologies primarily accessible to well-funded organizations.

Environmental and Inclusion Concerns

The environmental concerns stemming from such computational intensiveness are stark. Training large models not only consumes significant electrical power, often from non-renewable sources, but also leaves a sizeable carbon footprint. Furthermore, the financial barriers erected by these computational demands create an exclusive landscape in AI research, disenfranchising researchers from less affluent institutions and emerging economies.

Efficiency as an Evaluation Criterion

To address these issues, the authors advocate for the recognition of efficiency as a critical metric in AI evaluation. They propose that AI research should report the financial and environmental costs of model training and inference, providing baselines for developing more efficient methods. Integrating efficiency measures would democratize AI research, enabling broader participation and fostering more sustainable scientific practices.

Measuring Computational Efficiency

The paper examines various metrics for evaluating computational efficiency:

• Carbon Emission: Direct but impractical due to dependence on local electricity infrastructure.
• Electricity Usage: Independent of geographic factors but hardware-dependent.
• Elapsed Real Time: Straightforward but influenced by hardware and workload factors.
• Number of Parameters: Hardware-agnostic but not an accurate reflection of efficiency due to different architectures' varying utilizations.
• Floating Point Operations (FPO): The preferred measure, providing a hardware-agnostic and implementationally neutral assessment of computational work.

FPO, while not perfect, offers a balanced approach to quantifying the computational effort of AI models. The authors recommend that AI research routinely report the FPO cost associated with generating results to motivate and benchmark efficiency improvements.

Empirical Analysis

The authors present data demonstrating diminishing returns from increased computational effort. For example, enhancements in deep learning models like ResNet to ResNext show marginal performance gains relative to the substantial increases in computational cost. They argue that improved reporting of the budget/accuracy trade-offs observed during model development can help researchers make more informed decisions and highlight efficient approaches' value.

Practical and Theoretical Implications

The implications of this research are manifold:

• Practical: Introducing efficiency as a standard evaluation criterion can foster development of AI methodologies that are accessible and sustainable. Reporting efficiency metrics can guide researchers in optimizing computationally intensive processes, making high-performance AI more attainable across diverse settings.
• Theoretical: The move towards Green AI might align more closely with how the human brain, an incredibly efficient system, operates. It encourages the exploration of innovative, less resource-intensive paradigms that could yield new theoretical insights and methodologies.

Future Directions

Looking ahead, the direction set forth by this paper opens several avenues for future research. There is a need for developing standardized tools for measuring FPO consistently across different platforms and improving models' efficiency without sacrificing accuracy. Further studies could explore the saturation point in computational expense versus performance gain, possibly discovering principles applicable to a broader range of computing disciplines.

Conclusion

The call for Green AI by Schwartz et al. is a critical step towards creating a more inclusive, sustainable, and environmentally conscious AI research community. By balancing computational efficiency with performance metrics, we can ensure that AI advancements are accessible and beneficial to a wider segment of the global population, while also conserving resources for future generations.