Bridging Compute- and Data-Optimal Pretraining

Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound. CD scaling extends classical scaling laws by introducing a token-effectiveness function, $η$, which quantifies the value of a derived token-produced, for example, through multi-epoch repetition or paraphrasing-relative to a fresh token, ranging from a perfect substitute to having no value. We fit $η$ for two data-expansion strategies, multi-epoch repetition and paraphrasing, across model sizes from 14M to 600M parameters using the Dolma-3 corpus. We find that token effectiveness is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and it saturates as the corpus is expanded. The functional form of $η$ implies diminishing returns when substituting compute for data as either model size or data availability increases. It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound---and shows that classical compute-optimal allocation is suboptimal across most practically relevant settings.

• The paper introduces a unified CD scaling law that bridges compute-optimal and data-optimal regimes by quantifying token effectiveness with a novel function.
• The paper empirically validates optimal resource allocation strategies and demonstrates diminishing returns from derived tokens, showing robust predictions with low RMSE.
• The paper prescribes precise compute-data Pareto frontiers to guide LLM developers in balancing model size, fresh data, and compute in data-scarce regimes.

Bridging Compute- and Data-Optimal Pretraining: An In-depth Analysis

Motivation and Problem Statement

The rapid scaling of LLM pretraining has historically assumed an unbounded supply of high-quality, fresh data. Classical scaling laws, notably those advanced by Kaplan et al. and Hoffmann et al., prescribe compute-optimal allocation between model parameters and unique tokens under this data-abundance assumption. However, as global compute budgets continue to grow exponentially while the stock of high-quality pretraining data plateaus, this paradigm is increasingly invalid. Recent projections indicate imminent exhaustion of public human-generated text data as early as 2026–2032 [villalobos2024will]. Thus, future LLM development must confront regimes where additional compute can only be allocated to derived tokens—repeated, paraphrased, or otherwise synthesized from existing corpora—rather than to more fresh data.

CD-Scaling Laws: Theoretical Foundation

To address this regime, the paper introduces Compute-Data (CD) scaling laws, providing a continuous interpolation between classical compute-optimal and data-optimal scaling regimes. The key innovation is the token effectiveness function, $\eta(N, \gamma, r)$, quantifying the value of a derived token (from multi-epoch repetition, paraphrasing, or distillation) relative to a fresh token, as a function of model size $N$, tokens-per-parameter ratio $\gamma = D/N$, and the derived-to-fresh token ratio $r = D'/D$.

The CD scaling law is expressed as:

$$L^{\mathrm{CD}}(N, D, D') = E + \frac{A}{N^\alpha} + \frac{B}{\left[D + \eta(N, \gamma, r) D'\right]^\beta}$$

When $D' = 0$, the original Chinchilla compute-optimal scaling law is recovered; as $D' \to \infty$ with fixed $D$, the data-optimal scaling law is approached, determined by the limiting behaviour of $\eta$.

This theoretical framing reveals three natural operational regimes:

1. Compute-bound: Additional compute (via derived tokens) reduces loss.
2. Data-bound: Loss can only be reduced by increasing fresh data; derived tokens have saturated.
3. Model-bound: Loss is floored by model capacity and cannot be reduced by further data or compute increases.

   Figure 1: CD-scaling law overview contextualizing the classical compute-optimal regime, data-optimal regime, and the interpolation enabled by token effectiveness $\eta$.

Empirical Characterization of Token Effectiveness

Empirical evaluation is conducted on OLMo3 models spanning $N$0M to $N$1M parameters, trained on Dolma-3 with a wide range of fresh ($N$2) and derived ($N$3) token budgets. Two data expansion strategies are assessed:

• Multi-epoch repetition: Standard data repetition.
• Paraphrasing: Diverse rephrasings leveraging a strong paraphraser (SmolLM2-1.7B-Instruct [allal2025smollm2smolgoesbig]).

$N$4 is measured for each run as the value that maps observed loss onto the Chinchilla scaling law with effective token count $N$5. Empirically, $N$6 is found to decay monotonically with both $N$7 and $N$8, saturating to a finite ceiling $N$9 that depends in a power-law fashion on model size and data budget.

Figure 2: Empirical dependence of $\gamma = D/N$0 on tokens-per-parameter, the expansion ratio, and evidence of saturation—demonstrating diminishing marginal value of derived tokens as $\gamma = D/N$1 increases, especially for large models.

The selected parametric form for $\gamma = D/N$2 is:

$\gamma = D/N$3

with

$\gamma = D/N$4

Thus, the maximum number of “fresh-equivalent” tokens achievable via data expansion is capped by $\gamma = D/N$5, which shrinks as models and/or datasets grow.

Fitting and Validation of CD-Scaling Laws

Joint optimization of law parameters $\gamma = D/N$6 confirms the fit across data regimes and model sizes. Crucially, RMSEs for the CD law stay low (0.02–0.05 in log-space), and cross-model-size validation shows robust transfer: laws fit on small models predict losses for models 20× larger within 0.05–0.08 log-loss RMSE. Ablation studies reject constant-$\gamma = D/N$7 baselines [muennighoff_scaling_2025], confirming the necessity of explicit $\gamma = D/N$8 and $\gamma = D/N$9 dependence for robust scaling extrapolation.

Figure 3: Fitted CD-scaling law: data-reducible loss versus fresh data (left) and versus effective data mapped by $r = D'/D$0 (right), confirming the mapping’s fidelity across all strategies.

Prescriptions for Optimal Resource Allocation

The CD-scaling law defines the joint compute-data Pareto frontier—the steepest-descent path on the empirical loss surface as a function of both compute and fresh data—guiding practitioners in optimal allocation. The empirical frontier, especially for data-scarce regimes, underscores that modest additional compute spent on derived data yields rapid initial loss reductions, after which returns diminish, especially for large models and high $r = D'/D$1.

Figure 5: Optimal compute-data allocation surfaces, showing dramatically increased efficacy of compute allocation shortly past the single-epoch regime and rotation toward joint scaling as compute budgets rise.

Regime Identification and Data Expansion Strategy Selection

Mapping out region boundaries, the CD law indicates when a given $r = D'/D$2 is compute-bound, data-bound, or model-bound—i.e., which resource constrains improvements. The law prescribes how many epochs (repetitions or paraphrasing passes) should be used prior to saturating returns, and—critically—which expansion strategy is optimal.

Key results include:

• Multi-epoch repetition is optimal for large models and large data budgets.
• Paraphrasing dominates for small models and low data budgets, but loses its marginal effectiveness rapidly with scale.

  Figure 7: Heatmap and boundary traces of saturation ceilings $r = D'/D$3 for repetition/paraphrase strategies, with optimal regime boundaries demarcated (top), and recommended epochs or passes to reach 90% of $r = D'/D$4 (bottom).

Quantitative Prescription

• The canonical “four-epoch” rule is only valid at intermediate scales; recommended epochs decrease with both model size and $r = D'/D$5.
• For $r = D'/D$6M and sub-Chinchilla budgets, paraphrasing is preferable; for $r = D'/D$7B or larger budgets, paraphrasing offers negligible additional benefit over repetition.

Loss-to-Downstream Transferability

Validation loss is confirmed as a sufficient statistic for downstream performance across all benchmarks and settings: all training runs, models, and strategies collapse onto a single curve when plotting downstream BPB or accuracy as a function of validation BPB.

Figure 4: Downstream benchmark scores (BPB and accuracy) versus validation BPB across all settings—demonstrating predictive validity of validation loss for downstream evaluation.

Implications and Outlook

The proposed CD-scaling law provides a robust, empirically validated framework for modeling LLM performance in an era where large-scale, unique data is exhausted. By characterizing the diminishing effectiveness of data expansion as a function of scale, it enables:

• Principled allocation of resources between compute, data, and model capacity.
• Quantitative decision-making in selecting between data expansion strategies, notably synthetic augmentation methods.
• Reliable extrapolation from small-scale experiments to production-scale deployments.
• Accurate prediction of downstream performance using only validation loss.

Theoretical Implications

• Strong evidence supports the hypothesis that substituting compute for data yields rapidly diminishing returns as models or datasets scale, a result that deviates sharply from prior scaling lore.
• The findings unify data-constrained [muennighoff_scaling_2025] and synthetic data [kang_demystifying_2025, maini_rephrasing_2024] scaling laws along a continuum controlled by $r = D'/D$8.
• The notion of a saturation ceiling $r = D'/D$9 formalizes prior empirical intuition and connects to recent theoretical analyses [yan2026largerdataset].

Practical Impact and Future Directions

Practically, the work guides LLM developers in future-scarce-data regimes: maximizing return on compute, knowing when to procure fresh data versus synthesize, and identifying when further investment in model scale will stop yielding proportional gains. Future extensions should:

• Attempt to predict $$L^{\mathrm{CD}}(N, D, D') = E + \frac{A}{N^\alpha} + \frac{B}{\left[D + \eta(N, \gamma, r) D'\right]^\beta}$$0 from corpus statistics (e.g., diversity, n-gram overlap, perplexity distributions) rather than fitting empirically, thus operationalizing the framework for new datasets and synthetic data modalities.
• Evaluate other data expansion strategies, including self-distillation [kim_pre-training_2025], and compositional approaches.
• Model mixed or adaptive strategies that dynamically allocate derived-token generation along the Pareto frontier.

Conclusion

The paper systematically constructs and empirically validates a unified scaling framework that bridges compute- and data-constrained LLM pretraining regimes. The introduction of a general token effectiveness function, $$L^{\mathrm{CD}}(N, D, D') = E + \frac{A}{N^\alpha} + \frac{B}{\left[D + \eta(N, \gamma, r) D'\right]^\beta}$$1, and the associated saturation ceiling $$L^{\mathrm{CD}}(N, D, D') = E + \frac{A}{N^\alpha} + \frac{B}{\left[D + \eta(N, \gamma, r) D'\right]^\beta}$$2, allows precise, quantitative predictions of performance and optimality across practical data-constrained and synthetic-data training settings. The implications are broad: they inform future LLM scaling, new corpus curation strategies, and downstream application performance modeling in the emerging era where data—not just compute or model size—becomes the primary bottleneck.

---

References

• (2607.25271) Bridging Compute- and Data-Optimal Pretraining
• (Huang et al., 2022) Huang et al., LLMs Can Self-Improve
• (Muennighoff et al., 2023) Muennighoff et al., Scaling Data-Constrained LLMs
• (Maini et al., 2024) Maini et al., Rephrasing the Web
• (Kang et al., 2 Oct 2025) Kang et al., Demystifying Synthetic Data
• [villalobos2024will] Position: Will we run out of data?
• (Kim et al., 18 Sep 2025) Kim et al., Pre-training under infinite compute
• [kaplan_scaling_2020] Scaling Laws for Neural LLMs
• [hoffmann_training_2022] Training Compute-Optimal LLMs
• [allal2025smollm2smolgoesbig] SmolLM2: When Smol Goes Big
• [synthetic_data_playbook] The Synthetic Data Playbook