Pre-training under infinite compute (2509.14786v1)

Abstract: Since compute grows much faster than web text available for LLM pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We first show that existing data-constrained approaches of increasing epoch count and parameter count eventually overfit, and we significantly improve upon such recipes by properly tuning regularization, finding that the optimal weight decay is $30\times$ larger than standard practice. Since our regularized recipe monotonically decreases loss following a simple power law in parameter count, we estimate its best possible performance via the asymptote of its scaling law rather than the performance at a fixed compute budget. We then identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe. Our best intervention combining epoching, regularization, parameter scaling, and ensemble scaling achieves an asymptote at 200M tokens using $5.17\times$ less data than our baseline, and our data scaling laws predict that this improvement persists at higher token budgets. We find that our data efficiency gains can be realized at much smaller parameter counts as we can distill an ensemble into a student model that is 8$\times$ smaller and retains $83\%$ of the ensembling benefit. Finally, our interventions designed for validation loss generalize to downstream benchmarks, achieving a $9\%$ improvement for pre-training evals and a $17.5\times$ data efficiency improvement over continued pre-training on math mid-training data. Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.

• The paper demonstrates that standard pre-training recipes overfit when compute exceeds data, as excessive epoching and increased parameter counts worsen validation loss.
• The study introduces a regularized scaling method that jointly tunes weight decay, learning rate, and epochs, achieving a monotonic, power-law decrease in loss.
• The research shows that ensembling and distillation yield lower loss asymptotes and enhanced data efficiency, outperforming traditional single-model approaches on benchmarks.

Pre-training under Infinite Compute: Scaling Laws, Regularization, and Ensembling

Motivation and Problem Setting

The paper "Pre-training under infinite compute" (2509.14786) addresses the regime where compute resources for LLM pre-training vastly outpace the growth of available web-scale data. This scenario is increasingly relevant as compute budgets grow at $4\times$ per year, while web data increases only $1.03\times$ per year. The central question is: How should pre-training be approached when data is the bottleneck and compute is unconstrained?

The authors formalize the problem by fixing a seed corpus (200M tokens) and systematically evaluating scaling recipes under unconstrained compute. The paper is grounded in the classical statistical learning framework, focusing on minimizing validation loss as a proxy for downstream generalization.

Limitations of Standard Scaling Recipes

Standard recipes for pre-training, such as those inspired by Chinchilla scaling, recommend increasing both model size and data jointly. In the data-constrained regime, practitioners often repeat data (epoching) and increase parameter count. However, the paper demonstrates that:

• Excessive epoching leads to overfitting: Validation loss increases after a threshold, contradicting monotonic decay assumptions in prior scaling law literature.
• Increasing parameter count does not guarantee improvement: Beyond a certain point, larger models overfit, and validation loss plateaus or worsens.

Figure 1: Increasing epochs or parameters in the standard recipe leads to overfitting and diminishing returns in validation loss.

These findings highlight the inadequacy of standard recipes when data is limited and compute is abundant.

Regularization Enables Monotonic Scaling

To address overfitting, the authors introduce a regularized scaling recipe. By jointly tuning weight decay, learning rate, and epoch count for each parameter count, they achieve monotonic improvement in validation loss, even for models with parameter-to-token ratios $140\times$ larger than Chinchilla.

Key observations:

• Optimal weight decay is much higher than standard practice: Up to $30\times$ larger for the most over-parameterized models.
• Loss follows a power law in parameter count: The best-fit law is $\hat{L}_{D,N} = \frac{A_D}{N^{\alpha_D}} + E_D$, with $\alpha_D \approx 1$ and $E_D$ (the asymptote) representing the best achievable loss as $N \to \infty$.

  Figure 2: Jointly tuned regularization yields monotonic power law scaling in parameter count, with a well-defined asymptote.

This approach aligns with theoretical results in over-parameterized regression, where optimal regularization mitigates double descent and enables predictable scaling.

Ensembling Outperforms Parameter Scaling

The paper further investigates ensembling as an alternative to scaling parameter count. Ensembles are constructed by independently training $K$ models and averaging their logits. The key findings are:

• Ensembling achieves a lower loss asymptote than parameter scaling: For fixed total parameter count, increasing the number of ensemble members $K$ yields better performance than increasing the size of a single model.
• Loss improvement scales as $1/K$: The asymptote for $K \to \infty$ is strictly lower than for $N \to \infty$ with $K=1$.

  Figure 3: Scaling ensemble member count yields a better asymptote than scaling parameter count alone.

Hyperparameter tuning for ensembles reveals that optimal settings differ from single-model recipes: more epochs and less weight decay per member are beneficial as $K$ increases.

Figure 4: Ensemble asymptotes benefit from more epochs and less weight decay per member as $K \to \infty$.

Joint Scaling: Composing Parameter and Ensemble Limits

The authors propose a joint scaling recipe: take both $N \to \infty$ and $K \to \infty$. This double limit yields the lowest achievable loss asymptote under infinite compute.

• The joint scaling asymptote is strictly lower than either parameter scaling or ensembling alone.
• Hyperparameter selection for the double limit is nontrivial: The best settings for ensembles differ from those for single models, and the order of limits affects practical tuning.

Figure 5: Composing regularized and ensembling recipes under the double limit yields the lowest loss asymptote.

Data Efficiency and Scaling Laws

The paper extends scaling law analysis to varying seed token counts. For each recipe (standard, regularized, joint scaling), the authors fit data-scaling laws and compare asymptotes.

• Regularized and joint scaling recipes are significantly more data-efficient: At 200M tokens, joint scaling achieves the same loss as the standard recipe with $5.17\times$ less data.
• Data efficiency improvements persist across token counts: Scaling exponents and asymptotes are similar, suggesting constant multiplicative gains.

Figure 6: Data-scaling laws for single model pre-training; regularized scaling achieves better asymptotes across token counts.

Figure 7: Data-scaling laws for ensembles; joint scaling yields $2\times$ to $5\times$ data efficiency improvements over regularized and standard recipes.

Distillation: Compressing Data Efficiency Gains

To make data-efficient recipes practical for deployment, the authors explore distillation:

• Ensemble distillation: Compresses an 8-member ensemble into a single 300M model, retaining $83\%$ of the ensemble's loss improvement.
• Self-distillation: Training a student model of the same size as the teacher (with mixed real and synthetic data) surprisingly improves loss, matching the regularized recipe's asymptote without increasing parameter count at training.

  Figure 8: Ensemble and self-distillation compress data efficiency gains into smaller models, outperforming regularized scaling asymptotes.

Downstream Task Performance

Validation loss improvements translate to downstream benchmarks (PIQA, SciQ, ARC Easy):

• Ensembles and distilled models outperform standard pre-training by $7\%$–$9\%$ on average.
• Regularization and ensembling yield smooth scaling in downstream accuracy, mirroring validation loss trends.

Figure 9: Lower validation loss correlates with lower error on downstream benchmarks.

Continued Pre-training and Immediate Applicability

The interventions generalize to continued pre-training (CPT) scenarios. On the MegaMath-Web-Pro dataset, ensembling epoched models with only 4B tokens outperforms default CPT on the full 73B tokens, yielding a $17.5\times$ data efficiency improvement.

Implementation Considerations

• Hyperparameter tuning is critical: Locally optimal settings for learning rate, epoch count, and weight decay must be found for each parameter and token count.
• Ensembling requires independent training runs: Varying data order or initialization seed suffices to induce diversity.
• Distillation leverages synthetic data: Mixing real and teacher-generated data avoids model collapse and enables self-distillation gains.
• Scaling law fitting: Power laws are fit to validation loss across parameter, ensemble, and data axes; asymptotes are used to estimate best achievable performance.

Theoretical and Practical Implications

• Asymptote estimation as a metric: The paper advocates evaluating scaling recipes by their loss asymptotes under infinite compute, rather than fixed compute budgets.
• Classical techniques remain effective: Regularization, ensembling, and distillation—longstanding methods in data-constrained deep learning—are shown to be highly effective in modern LLM pre-training under data constraints.
• Future directions: The results motivate revisiting other classical interventions (objective functions, architectures, data augmentation) and exploring their composition with ensembling and distillation for further data efficiency gains.

Conclusion

This work rigorously demonstrates that, in the regime of infinite compute and limited data, standard scaling recipes are suboptimal due to overfitting. Jointly tuned regularization restores monotonic scaling, and ensembling yields strictly better asymptotes. Composing these recipes and leveraging distillation compresses data efficiency gains into deployable models. The findings have immediate practical relevance for pre-training and continued pre-training, and suggest that future progress in data-constrained LLMs will require careful algorithmic design and evaluation via scaling law asymptotes.