Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization

Abstract: A data mixture refers to how different data sources are combined to train LLMs, and selecting an effective mixture is crucial for optimal downstream performance. Existing methods either conduct costly searches directly on the target model or rely on mixture scaling laws that fail to extrapolate well to large model sizes. We address these limitations by introducing a compute-efficient pipeline for data mixture scaling. First, we propose CAMEL, a capacity-aware mixture law that models validation loss with the nonlinear interplay between model size and mixture. We also introduce a loss-to-benchmark prediction law that estimates benchmark accuracy from validation loss, enabling end-to-end performance prediction for the target model. Next, we study how to allocate a fixed compute budget across model scales to fit the law and reduce prediction error. Finally, we apply our method to Mixture-of-Experts models with up to 7B-A150M parameters to fit the law, and verify the optimal mixture derived from the law by extrapolating to a 55B-A1.2B target model. Compared to prior methods, we reduces mixture optimization costs by 50\% and improves downstream benchmark performance by up to 3\%.

• The paper proposes CAMEL, a capacity-aware data mixture scaling law that jointly models domain mixtures and model scale to optimize LLM training in the mid-training phase.
• It establishes a loss-to-benchmark mapping, predicting downstream task accuracy from validation losses across 14 benchmarks, ensuring robust and efficient performance.
• Empirical results demonstrate up to 3% improvement in weighted benchmark accuracy and 50% reduction in compute cost using an hourglass sampling strategy for scaling law fitting.

Capacity-Aware Mixture Laws for Efficient LLM Data Optimization

Overview

The paper "Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization" (2603.08022) introduces CAMEL, a novel capacity-aware data mixture scaling law designed for optimizing domain mixture ratios in LLM training, specifically targeting the mid-training phase. CAMEL jointly models the nonlinear interaction between domain mixtures and model scale, enabling efficient extrapolation of optimal training mixtures to large-scale architectures. The paper also establishes a loss-to-benchmark prediction law which maps validation losses to downstream benchmark accuracy, providing an end-to-end pipeline for mixture-based optimization under fixed computational budgets.

Motivation and Problem Formulation

Traditional approaches optimize data mixtures either by direct search on the target LLM or by fitting mixture scaling laws on proxy models. However, these methods are either computationally prohibitive or fail to generalize mixture recipes across model scales. Notably, mixtures optimized on smaller models do not reliably transfer to larger ones due to the nonlinear scaling behavior of capacity allocation among intrinsic domains. Efficient exploration of mixture ratios is thus crucial, especially in mid-training where domain-specific data quality significantly impacts downstream performance.

The paper recasts mixture optimization as a constrained capacity allocation problem, with the goal of minimizing validation loss across domain mixtures while explicitly accounting for model scale. This is formalized by constructing a joint mixture-to-loss scaling law, which enables principled prediction of optimal mixtures for arbitrary LLM sizes.

The Capacity-Aware Mixture Law (CAMEL)

CAMEL is derived by modeling the effective parameter allocation to intrinsic domains as a function of both mixture ratio and total model capacity. Each domain's training loss obeys an empirically validated power law, and the process of mixture optimization is captured via a capacity-constrained optimization objective. The resulting law for validation loss $L_{\text{val}}(r,M)$, parametrized by mixture $r$ and model size $M$, unifies mixture weights and model scale:

$$L_{\text{val}}(r, M) = C + \sum_{i=1}^k \left(t_{i}, r\right) \alpha_i M^{\beta_i} K_i$$

where $k$ is the number of intrinsic domains, $t_i$ is the domain-profile vector, and $\alpha_i$, $\beta_i$, $K_i$ are learned parameters. This formulation is theoretically motivated and empirically shown to achieve consistently lower prediction error than prior scaling laws (e.g., DML, SODM), especially under extrapolation to larger models.

Loss-to-Benchmark Mapping

The paper extends the scaling law framework by establishing a logistic-form mapping from a vector of validation losses to downstream benchmark accuracy for multi-domain tasks. For benchmark $b$, accuracy is predicted as:

$$\text{Acc}_b(L) = C_b + \frac{1}{1 + \exp(k_b L + B_b)}$$

where $L$ is the aggregate of individual validation losses and $C_b$, $k_b$, $B_b$ are benchmark-specific parameters. This mapping is validated across 14 benchmarks and achieves low mean absolute prediction error, demonstrating that validation losses serve as reliable predictors of downstream accuracy.

Compute-Efficient Experiment Design

The authors analyze sampling strategies for fitting the scaling law under fixed compute budgets. Unlike prior uniform allocation (rectangle), the hourglass strategy—prioritizing samples at the smallest and largest scales—consistently minimizes prediction error. This structured parameter estimation improves both the reliability and cost efficiency of end-to-end extrapolation.

Empirical Results

CAMEL is evaluated on Mixture-of-Experts models scaled from 590M-A12M to 7B-A150M parameters for law fitting, and extrapolated to a 55B-A1.2B target model for verification. Compared to model-size-agnostic approaches and prior scaling laws, CAMEL yields:

• 50% reduction in mixture optimization compute cost compared to baselines.
• Up to 3% improvement in weighted average benchmark accuracy.
• Robust generalization to held-out benchmarks, indicating strong cross-task mixture transferability.
• Stable identification of optimal mixtures across Balanced, Math Specialized, Code Specialized, and Knowledge Specialized objectives.

Strikingly, as model size increases, the optimal mixture allocates greater weight to knowledge domains, while math and code domains decrease—suggesting that large LLMs absorb general knowledge more efficiently and mixture design should evolve dynamically with scale.

Theoretical and Practical Implications

Theoretical

CAMEL offers a principled unification of scaling laws, mixture optimization, and benchmark alignment. By grounding mixture optimization in capacity allocation theory, it challenges the notion that mixture effects and model scale can be simply decoupled, and instead demonstrates the necessity of joint modeling for reliable extrapolation.

Practical

The proposed methodology enables rapid identification of high-quality training mixtures for arbitrary LLM scales, often requiring compute less than a single full pass on the target model. This facilitates efficient utilization of computational resources and accelerates LLM development cycles. The hourglass sampling policy further has broad applicability in hyperparameter estimation and scaling law fitting for other training domains.

Future Directions

The authors propose exploration of alternative parametric or non-parametric scaling laws, aiming for even more precise extrapolation. Adaptive compute allocation strategies across model scales are highlighted as a promising direction to further boost optimization efficacy in resource-constrained settings.

Conclusion

CAMEL represents an effective framework for capacity-aware mixture optimization in LLM training, integrating theoretical modeling and empirical validation. It delivers superior mixture recipes and performance with substantial compute savings, robust generalization, and practical guidance on data allocation strategies as model scale increases. The paper sets a new standard for mixture-based scaling law design in LLMs, with implications for future research on adaptive training, scaling law generalization, and automated corpus composition (2603.08022).