Characterizing the Relationship Between Training Data Mixtures and Emergent LLM Abilities

Characterize the complex, high-dimensional relationship between the composition of training datasets for large language models—specifically the proportions and properties of sources such as web text, books, code, and scientific papers—and the emergent capabilities exhibited by models trained via next-token prediction, with the aim of guiding optimal data-mixture design under resource constraints.

Background

The paper emphasizes that LLM capabilities are fundamentally determined by pre-training and fine-tuning data, making dataset composition a central design lever. While practitioners use heuristics (e.g., increasing the proportion of code to improve programming performance), the true mapping from data-mixture choices to emergent abilities is complex and high-dimensional.

Because LLMs are black-box systems with stochastic generation, the effects of different data sources and their proportions are difficult to deduce mechanistically. The authors highlight the need for statistical modeling (e.g., regression-based approaches) to investigate these dependencies and inform the optimization of data mixtures.