Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model (2211.02001v1)

Abstract: Progress in ML comes with a cost to the environment, given that training ML models requires significant computational resources, energy and materials. In the present article, we aim to quantify the carbon footprint of BLOOM, a 176-billion parameter LLM, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes of~\carboneq~if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also study the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of ML models and future research directions that can contribute towards improving carbon emissions reporting.

Estimating the Carbon Footprint of BLOOM, a 176B Parameter LLM

The paper "Estimating the Carbon Footprint of BLOOM, a 176B Parameter LLM" presents a comprehensive assessment of the environmental impact associated with the training and deployment of the BLOOM model. With an increasing focus on the ecological ramifications of developing large-scale machine learning applications, particularly LLMs like BLOOM, this paper provides critical data and analysis that underscore both direct and indirect environmental costs.

Human-induced climate change, exacerbated by industrial carbon emissions, is a well-documented challenge. Information and communications technology (ICT) contributes approximately 2% to global CO2 emissions—a figure difficult to pinpoint due to the distributed nature of ICT infrastructure. This paper provides empirical estimates for the carbon emissions of BLOOM across various phases of its lifecycle, including hardware manufacturing, model training, and deployment, using the Life Cycle Assessment (LCA) framework.

Key Findings

The paper reveals that the final training of BLOOM, a massive 176-billion parameter LLM, resulted in approximately 24.7 to 50.5 tonnes of CO2 emissions, depending on the inclusion of dynamic versus total consumption (including infrastructure manufacturing) estimates. This quantification incorporates several facets of its production and utilization:

1. Embodied Emissions: Approximately 11.2 tonnes of CO2 are emitted due to the material and energy inputs required for producing the computing infrastructure, highlighting the significant carbon footprint of hardware components alone.
2. Dynamic and Infrastructure Power Consumption: Dynamic power consumption—that is the electricity used directly in model training—resulted in an estimated 24.7 tonnes of CO2. When factoring in infrastructure and idle power usage, the total emissions from these sources reached 50.5 tonnes.
3. Inference and Deployment: The inference process for BLOOM, especially when managed as an API, accrues ongoing emissions. Preliminary analysis of GCP-hosted deployments showed an energy cost of 19 kgs of CO2 per day, selecting only one potential deployment mode among many.
4. Comparative Analysis: When juxtaposing BLOOM with similar models like GPT-3 and OPT-175B, it is evident that BLOOM's emissions are significantly lower. This difference is largely attributed to the low-carbon-intensity grid used in its training phase, which is powered by France's relatively clean energy sources.

The paper provides extensive examination of the BigScience workshop efforts, beyond just BLOOM's final model-training emissions. It also includes emissions from intermediate model experiments and other processes, totaling much higher with all accounted stages included.

Implications and Future Directions

This paper has substantial implications for the machine learning community, emphasizing the necessity for improved transparency and standardization in reporting the environmental impacts of ML models.

• Framework for Carbon Accounting: A thorough, comparable reporting framework that distinguishes between varied emission types—such as embodied versus dynamic power-related emissions—is critical. As machine learning models progressively influence other sectors, there should be an increased focus on systematic evaluations of system-level impacts.
• Deployment Efficiency: The deployment phase reflects substantial emissions, driven by continuous idle consumption of computational resources. Implementing efficiency improvements in deployment strategies, such as optimization and better resource usage, is imperative.
• Holistic Environmental Impact: Broader environmental impacts, including those beyond direct emissions, need to be considered, potentially including implications of ML models on societal and consumer behaviors, as ML technology usage raises global energy consumption indirectly.

This significant research directs attention to the evolving landscape of AI technology development with a plea for increased environmental considerations within fast-evolving ecosystems. By addressing these areas, future ML endeavors can aim to innovate more sustainably in their technological pursuits.