The rising costs of training frontier AI models (2405.21015v2)

Abstract: The costs of training frontier AI models have grown dramatically in recent years, but there is limited public data on the magnitude and growth of these expenses. This paper develops a detailed cost model to address this gap, estimating training costs using three approaches that account for hardware, energy, cloud rental, and staff expenses. The analysis reveals that the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4x per year since 2016 (90% CI: 2.0x to 2.9x). For key frontier models, such as GPT-4 and Gemini, the most significant expenses are AI accelerator chips and staff costs, each costing tens of millions of dollars. Other notable costs include server components (15-22%), cluster-level interconnect (9-13%), and energy consumption (2-6%). If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.

• The paper presents a detailed cost model that quantifies training expenses via hardware depreciation, cloud rental rates, and R&D labor costs.
• It reveals a 2.4x annual increase in costs since 2016, with major runs for GPT-4 and Gemini priced between $30M and $40M.
• The study underscores the economic and infrastructure challenges that may limit frontier AI research to well-funded organizations.

The Rising Costs of Training Frontier AI Models

This paper, authored by Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, and David Owen, addresses the escalating costs associated with training advanced AI models. The authors have developed a comprehensive cost model, leveraging three distinct approaches to estimate training expenses, namely hardware depreciation, cloud rental prices, and R&D labor costs, encompassing the multifaceted components of AI development.

Detailed Analysis of Frontier AI Model Training Costs

The authors have meticulously examined the growth in training costs for compute-intensive AI models, revealing a substantial increase at a rate of 2.4 times per year since 2016. The analysis suggests that the largest training runs, such as those for GPT-4 and Google's Gemini Ultra, cost between $30M and$40M. This upward trend implies that by 2027, the expenses could exceed a billion dollars, primarily dominated by high costs of AI accelerator chips and staff.

The study decomposes these costs into several key components:

• AI accelerator chips
• Server components
• Cluster-level interconnect
• Energy consumption

The authors report that the most significant cost contributors are AI accelerator chips and staff expenses, each constituting tens of millions of dollars. Server components and cluster-level interconnect follow, making up 15-22% and 9-13% of the costs, respectively, whereas energy consumption is relatively minor, accounting for only 2-6%.

Methodological Approaches

The paper introduces three complementary approaches to estimate the costs effectively:

1. Amortized Hardware CapEx + Energy Consumption: This approach considers the depreciation of hardware over the training period, providing granular cost estimates for individual hardware components and energy usage.
2. Cloud Rental Prices: By using historical cloud rental rates, this method offers a simpler but often overestimated cost view, highlighting the divergence in actual costs when hardware is owned or rented on a private basis.
3. Comprehensive Hardware, Energy, and R&D Staff Costs: This extensive approach includes all development aspects, from initial experiments to final training runs, incorporating R&D labor costs which can be substantial, between 29% and 49% of the total costs.

Implications and Future Directions

The results from this study have considerable implications for the future development and accessibility of frontier AI technologies. The growing financial barrier restricts large-scale AI training to well-funded entities, likely narrowing the diversity of research contributors predominantly to large corporations and government-backed institutions. This can limit innovative diversity and pose substantial governance challenges.

The authors speculate that future AI advancements will necessitate careful considerations around power capacity and data center infrastructure. Given the rapid increase in power requirements, securing sufficient power supplies will become increasingly critical. Regulatory and logistical challenges may further bottleneck the scalability of AI systems, necessitating proactive measures from both developers and policymakers.

Conclusion

This paper provides an in-depth evaluation of the cost structures associated with training frontier AI models, revealing a rapid and significant increase in necessary investments. The approaches outlined offer valuable methodologies for future cost estimations and highlight the economic and infrastructural challenges that lie ahead. As AI continues to scale, understanding and managing these costs will be crucial in shaping the trajectory and impact of this transformative technology.

Overall, the findings suggest that unless innovation in cost-reduction or efficiency improvements occurs, the concentration of AI capabilities among a few key players will persist, raising questions about the equitable development and deployment of AI technologies in the broader societal context.