Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI (2409.14160v2)

Abstract: With the growing attention and investment in recent AI approaches such as LLMs, the narrative that the larger the AI system the more valuable, powerful and interesting it is is increasingly seen as common sense. But what is this assumption based on, and how are we measuring value, power, and performance? And what are the collateral consequences of this race to ever-increasing scale? Here, we scrutinize the current scaling trends and trade-offs across multiple axes and refute two common assumptions underlying the 'bigger-is-better' AI paradigm: 1) that performance improvements are driven by increased scale, and 2) that all interesting problems addressed by AI require large-scale models. Rather, we argue that this approach is not only fragile scientifically, but comes with undesirable consequences. First, it is not sustainable, as, despite efficiency improvements, its compute demands increase faster than model performance, leading to unreasonable economic requirements and a disproportionate environmental footprint. Second, it implies focusing on certain problems at the expense of others, leaving aside important applications, e.g. health, education, or the climate. Finally, it exacerbates a concentration of power, which centralizes decision-making in the hands of a few actors while threatening to disempower others in the context of shaping both AI research and its applications throughout society.

• The paper challenges the 'bigger-is-better' assumption by showing that performance gains plateau beyond a certain model scale.
• It reveals that large-scale AI incurs high environmental and economic costs, including significant CO₂ emissions and steep compute expenses.
• The study advocates a balanced research agenda that emphasizes efficiency, transparent cost reporting, and diverse, accessible AI approaches.

Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI

The paper "Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI" by Gaël Varoquaux, Alexandra Sasha Luccioni, and Meredith Whittaker provides a critical examination of the prevailing assumption within AI research and development that larger models are inherently superior. This essay summarizes the authors' analysis, focusing on the scientific, environmental, and sociopolitical implications of this paradigm.

Inspection of the "Bigger-is-Better" Assumption

The paper scrutinizes two fundamental assumptions:

1. Improved performance is directly correlated with increased model scale.
2. All significant problems in AI require large-scale models.

The authors argue these assumptions are scientifically fragile and offer evidence showing improvements due to scale often saturate after a point. For example, in fields such as medical imaging and text embeddings, models of moderate size frequently achieve competitive performance (\autoref{fig:benchmarks}). Issues such as choice of model architecture, fine-tuning strategies, and application-specific requirements often play a more crucial role than sheer size.

Environmental and Economic Concerns

Unsustainable Compute Demands

The compute demands associated with large-scale models grow exponentially, outpacing even the advancements predicted by Moore's law. The paper highlights that while the size of state-of-the-art (SOTA) models is doubling approximately every five months, the cost of the underlying hardware, such as GPUs, remains relatively constant (\autoref{fig:doubling_time}). This leads to prohibitive economic costs and an alarming environmental footprint due to the energy required for training and inference (\autoref{fig:inference-cost}, \autoref{fig:inference-energy}).

For instance, training large models like LLMs can emit up to 550 tonnes of CO₂, and inference demands may consume as much energy as entire small countries (\autoref{app:historical_plots}). The rising cost of inference, even as hardware improves, underscores a troubling trajectory where increased efficiency results in higher overall energy usage due to Jevons Paradox.

Economic Viability and Accessibility

Economically, the paper reveals that the expense of training and deploying large models is becoming untenable for most research institutions and startups. As an illustrative case, OpenAI's ChatGPT incurs an estimated \$700,000 in daily compute costs. These expenses exacerbate the computational divide within the AI community, favoring large industrial labs with deep pockets over academia and smaller entities (\autoref{fig:compute_cost}).

Ethical and Sociopolitical Implications

Data Challenges and Surveillance Concerns

The pursuit of larger datasets leads to questionable data sourcing practices, encouraging widespread surveillance and the use of Internet-gathered data under dubious copyright justifications. Larger datasets are often plagued by issues of bias and questionable content, as shown in the LAION datasets' analysis, leading to problematic model outputs and opaque audit trails (\autoref{fig:training_size}).

Concentration of Power

The bigger-is-better paradigm fosters a concentration of AI capabilities among a few major players, predominantly large tech companies with the resources to sustain massive infrastructure and compute investments. This concentration of power has significant implications, including a narrowed research agenda focused on commercial interests and potential geopolitical tensions over AI capabilities. The authors argue this consolidation of AI power risks creating an oligarchy over technological advancements, impacting global diversity in AI research and application (\autoref{sec:narrowing}).

Recommendations for a Balanced Research Agenda

The paper proposes several norms to address these issues:

1. Valuing Research on Smaller Systems: By acknowledging the merit of smaller models, researchers can diversify the scientific inquiry beyond scaling and emphasize practical applications varying in complexity and resource constraints.
2. Transparency in Reporting Costs: Encouraging the reporting of computational costs, energy usage, and memory footprints alongside performance metrics will provide a balanced view of model efficiency and sustainability.
3. Reasonable Experiment Expectations: Avoiding excessive large-scale experiment demands will democratize participation in AI research, enabling a broader community to contribute with accessible resources.

Conclusion

In summary, the authors advocate for a paradigm shift away from the obsession with scale towards a more holistic approach considering efficiency, application specificity, and broader scientific questions. This shift would democratize AI research participation, promote environmental sustainability, and ensure a balanced distribution of power within the AI ecosystem. The insights provided in this paper challenge current practices and urge the research community to reflect on the broader impacts of their technological pursuits.

By fostering an environment where efficiency, context, and ethical considerations are as valued as performance benchmarks, the AI community can drive meaningful progress beneficial to society at large.