Scaling Domain Data Repetition in LLM Pretraining

Abstract: As LLMs scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio ((\mathrm{TPP})). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed (\mathrm{TPP}), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same (\mathrm{TPP}) can provide a practical estimate for larger models.

• The paper finds that optimal repetition increases mildly with model size at fixed tokens per parameter, while domain validation loss is the strongest predictor, with a correlation of −0.944 compared with 0.400 for model size.
• The paper shows that repetition tolerance varies by domain—mathematics supports roughly five to six repeats, code four to five, and Wikipedia and medical data three to four—while the unique-data fraction has little effect across the tested range.
• The paper’s theory explains repetition through the balance between knowledge acquisition and noise fitting, showing that fixed token budgets can make larger models overfit sooner, whereas data budgets that scale with model size reverse this trend; learning-rate schedules also materially affect tolerance.

The paper studies a practical data-scaling problem in LLM pretraining: how to reuse a limited supply of high-quality domain data as model size and the total training-token budget increase. Its central setting fixes the tokens-per-parameter ratio, $\mathrm{TPP}=D/N$, so that a model with $N$ parameters is trained on $D=\mathrm{TPP}\cdot N$ tokens. This distinction is essential. Under a fixed total token budget, larger models encounter the same repeated examples with greater effective capacity and tend to overfit earlier. Under fixed TPP, however, the data budget grows with model size, changing the balance between knowledge acquisition and repetition-induced overfitting. The paper argues empirically and theoretically that this latter regime is the relevant one for practical scaling and yields a counterintuitive result: the optimal repetition count increases mildly rather than decreases with model size (2608.14071).

Problem formulation and experimental design

The authors consider four high-quality domains—code, mathematics, Wikipedia, and medical data—mixed with general web data. For each model size, they hold the total training-token budget fixed and vary two properties of the target domain. The unique-data fraction is $\alpha\in\{1/40,1/20,1/10\}$, while the repetition count is $e\in\{1,\ldots,7\}$. Thus, the domain contributes $\alpha D$ unique tokens and $e\alpha D$ total token presentations, with the remaining $(1-e\alpha)D$ tokens drawn from web data. This design separates the quantity of unique domain information from the number of times that information is presented, although the final domain mixture weight still depends on their product.

Each run is evaluated using in-domain held-out loss and out-of-domain pretraining loss. The primary optimum is the repetition count minimizing final in-domain validation loss. Since $e$ is evaluated on a discrete grid, the authors also fit a quadratic function of $e$ to estimate a continuous optimum. The experiments use model-size-dependent learning rates and batch sizes, Muon optimization, attention dropout, and a warmup followed by a constant learning rate in the main setup. The study therefore evaluates repetition under a controlled pretraining recipe rather than under an abstract fixed-dataset multi-epoch protocol.

The setup improves on cross-scale comparisons that keep $N$0 fixed, including prior work on repeated-data scaling [(Hernandez et al., 2022), 2310.???]. It directly targets the regime in which compute-optimal training increases the token budget with model size, as in standard neural scaling-law analyses (Kaplan et al., 2020, Hoffmann et al., 2022). The experiments nevertheless remain limited to one repeated domain per run, so they do not directly characterize simultaneous repetition of multiple domains.

Main empirical findings

The strongest empirical result is the relationship between a domain’s achievable validation loss and the amount of repetition it supports. Across domains, model sizes, and unique-data fractions, the estimated optimal repetition count has a Pearson correlation of $N$1 with the minimum validation loss. Domains with lower loss tolerate and benefit from more repetitions; domains with higher loss reach their optimum earlier and degrade more rapidly. The paper interprets minimum validation loss as a continuous proxy for domain-specific learnability or effective noise level rather than treating domain identity as the fundamental explanatory variable.

The domain ordering is substantial. Mathematics supports approximately five to six repetitions, Wikipedia approximately three to four, code approximately four to five, and medical data approximately three to four. The numerical pattern is consistent across the tested unique-data fractions. This result rejects a universal repetition threshold: whether repetition is useful is strongly domain dependent, and the relevant domain property is more closely associated with validation loss than with the raw number of unique tokens.

Model size has a weaker but consistent effect. The correlation between estimated optimal repetition and model size is $N$2, compared with $N$3 for minimum validation loss. At fixed TPP, larger models generally shift the optimum toward larger repetition counts. The implication is operationally important: repetition sweeps performed on smaller proxy models can be transferred conservatively to larger models when the TPP is preserved. In particular, a repetition count that does not induce overfitting in the smaller proxy is unlikely to induce it in the larger model under the experimental conditions.

By contrast, the unique-data fraction has almost no observable effect on the optimal repetition count. Its correlation with the estimated optimum is only $N$4. Increasing $N$5 lowers the absolute validation loss, as expected from the greater supply of unique high-quality data, but leaves the location of the loss minimum nearly unchanged. This suggests that proxy experiments need not reproduce the exact target-domain fraction, provided that the fraction lies within the tested range. The conclusion should not be generalized beyond $N$6 without additional evidence.

Repetition, unique data, and domain-specific degradation

A complementary experiment fixes the total fraction of high-quality domain tokens, $N$7, and trades unique data for repetition. This compares $N$8 repetitions of a smaller subset with a proportionally larger amount of unique data under the same total domain-token budget. The result is again domain dependent.

Mathematics is relatively robust: increasing repetition from one to four produces only a small increase in validation loss. In this domain, repeated examples can approximate the value of additional unique examples over a moderate range. Wikipedia behaves differently. Repetition by two has a modest cost, but heavier repetition causes a pronounced degradation. Code and medical data also degrade as unique examples are replaced with repeated ones, although the magnitude differs.

This distinction matters because the paper’s first experiment asks how much total domain exposure should be allocated to a fixed unique subset, whereas the fixed-$N$9 experiment asks whether repeated exposure can substitute for additional unique information. The two questions have different optima. A domain may benefit from increasing its overall training presence through repetition while still losing performance when repetition replaces unique samples at a fixed domain fraction. Consequently, repetition should not be interpreted as generally equivalent to data expansion.

Theoretical account of the optimum

The theoretical analysis uses a one-hot linear-regression model with infinitely many knowledge coordinates. Coordinate frequencies follow a power law, $D=\mathrm{TPP}\cdot N$0, and the target signal obeys a source condition with exponent $D=\mathrm{TPP}\cdot N$1. A finite model represents only the first $D=\mathrm{TPP}\cdot N$2 coordinates, while a training set contains $D=\mathrm{TPP}\cdot N$3 samples. Gradient descent is run from zero initialization.

Conditioning on the number of observations of each coordinate yields an exact decomposition of expected population risk into three terms:

1. Unrepresented knowledge from coordinates beyond the model capacity;
2. Knowledge-acquisition error, which decreases as optimization proceeds;
3. Noise-fitting error, which increases as the model fits sample-specific noise.

The optimum occurs where the marginal reduction in acquisition error is balanced by the marginal increase in noise fitting. This provides a mechanistic interpretation of the empirical validation-loss relationship. A low-loss domain corresponds, in the model, to a setting with a smaller effective noise penalty or a more favorable signal structure; it can therefore sustain more optimization steps or repeated presentations before the variance term dominates.

The noise-decay theorem formalizes this relationship. For fixed $D=\mathrm{TPP}\cdot N$4 and $D=\mathrm{TPP}\cdot N$5, increasing the observation noise variance cannot increase the optimal stopping time. In the small-noise limit,

$D=\mathrm{TPP}\cdot N$6

Thus, decreasing noise delays overfitting logarithmically in $D=\mathrm{TPP}\cdot N$7. The theorem supports the empirical use of minimum validation loss as a proxy for repetition tolerance, although the mapping is not directly calibrated into an LLM-specific quantitative law.

The model-size results distinguish the two scaling regimes. With fixed $D=\mathrm{TPP}\cdot N$8, once $D=\mathrm{TPP}\cdot N$9 exceeds a crossover scale

$\alpha\in\{1/40,1/20,1/10\}$0

the optimal stopping time is non-increasing with model size. The additional coordinates correspond to rarer knowledge units with insufficient observations, so optimization increasingly fits noise. This formalizes the conventional intuition that larger models overfit repeated data more readily when the data budget is held fixed.

Under linear data–model scaling, $\alpha\in\{1/40,1/20,1/10\}$1, the result reverses:

$\alpha\in\{1/40,1/20,1/10\}$2

As $\alpha\in\{1/40,1/20,1/10\}$3 grows, the proportional increase in data supplies additional observations and extends the period during which knowledge acquisition remains beneficial. This theoretical result does not reproduce the exact empirical repetition counts, but it establishes the qualitative sign change emphasized by the paper: the direction of the scaling trend depends on whether the token budget is fixed or scales with model size.

The theory rests on substantial assumptions. The power-law one-hot model abstracts domain learning into independent coordinates, uses Gaussian label noise, and analyzes full-batch gradient descent rather than a transformer trained with stochastic optimization and language-model cross-entropy. The source condition $\alpha\in\{1/40,1/20,1/10\}$4 is necessary for the stated asymptotics. Accordingly, the theory explains a plausible bias–variance mechanism and the regime reversal, but it does not establish that the exact exponents or crossover scales apply to LLMs.

Effects beyond in-domain validation loss

The paper separately examines out-of-domain pretraining performance while fixing the total high-quality-domain fraction and the web-data fraction. On ArXiv and News validation sets, replacing unique mathematical tokens with repeated mathematical tokens produces only small and inconsistent changes in OOD loss for repetition counts of one, two, and four. This contrasts with the clear in-domain degradation observed for Wikipedia and other domains.

The implication is that repetition primarily changes the allocation of learning capacity within the repeated domain when the global mixture proportions remain fixed. It does not necessarily produce a commensurate deterioration on unrelated pretraining distributions. However, the OOD evaluation is narrow: two validation corpora cannot establish general invariance across domains, and the experiment focuses on mathematics rather than all four target domains.

Learning-rate schedules materially affect repetition tolerance. Earlier decay causes validation loss to deteriorate after fewer repetitions, while delaying decay increases the tolerable repetition count; a constant learning rate supports the largest number of repetitions. The proposed explanation is that repeated samples encountered during low-learning-rate training are fitted more closely, increasing memorization, whereas sustained optimization noise provides stronger implicit regularization. This result qualifies the proxy-transfer recommendation: repetition optima are not determined solely by domain loss, model size, and TPP, but also depend on the optimization schedule.

Limitations and open questions

The empirical conclusions are based on four domains, three unique-data fractions, and repetition counts no larger than seven. The paper does not report a single quantitative scaling function that predicts the target-model optimum from proxy-model measurements; it establishes correlations and a conservative transfer heuristic. The claim that proxy-selected repetition remains safe at larger scale is supported within the tested model and TPP ranges, but its validity for substantially larger models, different TPP values, or changed optimizers remains open.

The experimental protocol repeats one domain at a time. Real pretraining mixtures may repeat code, mathematics, multilingual data, scientific text, and other domains simultaneously, creating interactions through shared representations, mixture reweighting, and competition for optimization steps. The fixed-$\alpha\in\{1/40,1/20,1/10\}$5 analyses also show that repeated and unique tokens are not interchangeable, so a multi-domain recipe cannot be obtained by independently applying single-domain repetition optima without validation.

Finally, the use of final validation loss as the optimization target leaves several questions unresolved. Early stopping, downstream task performance, memorization metrics, and calibration may induce different repetition optima. The OOD analysis suggests limited cross-domain impact, but it does not determine whether repeated data affects factual robustness, contamination-sensitive benchmarks, or representation quality. A quantitative account linking domain validation loss, learning-rate schedule, TPP, and repetition count remains an explicit open problem.

Conclusion

“Scaling Domain Data Repetition in LLM Pretraining” (2608.14071) shows that domain-data repetition should be analyzed under the scaling rule used for practical LLM training. At fixed TPP, the optimal repetition count increases mildly with model size, while its dominant determinant is domain-specific validation loss: lower-loss domains support substantially more repetition. The unique-data fraction has little effect on the optimum over the tested range, enabling proxy sweeps with representative data fractions. The accompanying theory attributes these patterns to the balance between knowledge acquisition and noise fitting and explains why fixed-data and fixed-TPP scaling produce opposite model-size trends. The results support repetition as a domain- and schedule-dependent component of data-mixture design rather than a universally harmful form of data reuse.