No-Repetition Scaling Law in ML Training
- No-Repetition Scaling Law is a framework of scaling relations that describes how model loss decays when each training example is seen only once.
- It unifies several formulations—such as compute frontier, InfoLaw, additive-overfitting, and one-pass SGD linear regression—to serve as a reference for repetition damage.
- The law functions both as a predictive model and a control surface for optimizing compute allocation and training strategies by quantifying the gap from repeated-data runs.
Searching arXiv for the cited papers and closely related scaling-law work. I’ll look up the specified arXiv records to ground the article in the current literature. No-Repetition Scaling Law denotes a class of reference scaling relations for model loss or test error in regimes where each training example is seen once, or where repetition variables are fixed to their no-repetition value. In recent work, the term appears in several closely related forms: as a fitted no-repetition compute frontier for LLMs, as the zero-repetition specialization of an information-based law for quality-weighted mixtures, as the limit of an additive overfitting law for data-constrained training, and as a one-pass SGD theorem in linear regression (Chudnovsky et al., 23 Jun 2026, Liu et al., 4 May 2026, Lovelace et al., 2 May 2026, Lin et al., 2024). Across these formulations, the no-repetition law functions both as a predictive model of baseline performance and as the reference curve against which the damage from repeated data is quantified.
1. Terminology and formal scope
The phrase “no repetition” is not tied to a single notation. In the exact-document repetition setting of Chudnovsky et al., the no-repetition baseline is
where is the number of times each document in a small “repeated pool” is replayed during training, and is the fraction of all training tokens drawn from that pool (Chudnovsky et al., 23 Jun 2026). In InfoLaw, the no-repetition regime is the specialization in the sense of “effectively unlimited source data,” , so that for every quality bucket and no bucket is ever repeated (Liu et al., 4 May 2026). In the data-constrained law of Ivgi et al., repetition is parameterized by , the number of extra epochs beyond the first, so the no-repetition case is (Lovelace et al., 2 May 2026). In the linear-regression theory of Lin et al., no repetition is implemented by one-pass SGD, where each example is seen once (Lin et al., 2024).
These notational differences matter because each formulation isolates a different object. The Chudnovsky law is a compute-indexed empirical frontier. InfoLaw is a data-aware model in which information density and diminishing returns under repetition enter explicitly. The additive-overfitting law starts from a Chinchilla-style decomposition and adds a repetition penalty. The linear-regression result is a theorem on reducible risk under one-pass SGD. A plausible implication is that “no repetition” should be understood as a boundary condition shared by several scaling frameworks, rather than as a single universal equation.
2. Canonical formulations
Several no-repetition laws now coexist in the literature.
| Setting | No-repetition condition | Resulting law |
|---|---|---|
| Exact-document repetition baseline | 0 | 1 |
| InfoLaw specialization | 2 | 3 |
| Additive-overfitting law | 4 | 5 |
| One-pass SGD linear regression | one pass over data | 6 |
In the fitted no-repetition frontier of Chudnovsky et al., compute is written as
7
with 8 the overtraining multiplier relative to Chinchilla-optimal. The fitted constants are 9 nats, 0, and 1 (Chudnovsky et al., 23 Jun 2026).
In InfoLaw, the full information model is
2
with
3
Under no repetition, 4 and 5, so
6
For large 7, the paper Taylor-expands 8, giving
9
and hence the zero-repetition law
0
The fitted coefficients are 1, 2, 3, and 4 (Liu et al., 4 May 2026).
In the additive-overfitting framework,
5
Setting 6 recovers the no-repetition form
7
which is precisely the Chinchilla-style law without a repetition penalty (Lovelace et al., 2 May 2026).
The linear-regression result gives a no-repetition theorem rather than a fitted empirical curve. Under a Gaussian sketch of size 8, one-pass SGD on 9 data, an isotropic Gaussian prior on the teacher, and a covariance spectrum 0 with 1, the reducible part of the test error is
2
with the variance term dominated by the other errors due to the implicit regularization of SGD (Lin et al., 2024).
3. Reference frontier for repetition damage
A central use of the no-repetition law is to provide the reference frontier against which repeated-data runs are evaluated. Chudnovsky et al. define Compute-Equivalent Gain (CEG) and Compute-Equivalent Loss (CEL) by inverting the fitted no-repetition curve: 3 Here 4 iff the repeated-data run matches the no-repetition frontier, and 5 indicates wasted compute (Chudnovsky et al., 23 Jun 2026).
In that controlled exact-document repetition setting, the repeated-token fraction is fixed to 6, and the eval loss 7 at fixed 8 peaks at an intermediate repeat count 9. The peak repeat count is fit as
0
while the corresponding repeated-pool size scales as
1
At 2, the peak compute-equivalent losses are
3
For the Qwen3-style 4M-parameter model on FineWeb-Edu-Dedup at 5, the worst-case 6 gives 7 and 8, meaning the run only “buys” two-thirds of the compute it spent. The paper notes that a raw eval-loss bump of 9–0 maps to an 1 compute gap because the fitted no-repetition law is shallow, with 2 (Chudnovsky et al., 23 Jun 2026).
This use of a no-repetition frontier changes the interpretation of repetition damage. Rather than treating loss deltas in isolation, the loss increase is translated into the amount of no-repetition compute that would have been required to reach the same loss. This suggests that no-repetition laws are as important diagnostically as they are predictively.
4. Relation to data mixture and data quality
In repetition-aware mixture scaling, the no-repetition regime is a special case rather than the full problem. The mixture law of Mhammedi et al. defines the target-pool repetition factor
3
the utility-decay factor
4
the effective target contribution
5
and the effective total data
6
Under no repetition, 7, hence 8, 9, and
0
The paper states that in this limit the law drops back to data-mixture-only scaling (Sedova et al., 12 May 2026).
The same boundary-case structure appears in InfoLaw. When 1, no bucket is ever repeated, 2, and the total information reduces to 3. Under this zero-repetition specialization, InfoLaw differs sharply from a “Kaplan-style” two-term law
4
The paper emphasizes three consequences: the 5-exponent is 6, “roughly half the usual 7–8”; 9 enters only inside a logarithm rather than as a direct power; and the prefactor 0 is correspondingly larger to match overall loss levels. Figures 3(a)–(f) are reported to show that under zero or low repetition the InfoLaw curve collapses all training configurations, including extrapolations to larger 1 and 2, onto a single straight line in log–log space, whereas the standard two-term fit mis-extrapolates once repetition or data-mixture effects become non-trivial (Liu et al., 4 May 2026).
Mixture pretraining under data constraints also changes the operational meaning of “safe” repetition. Mhammedi et al. report that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused 3–4 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale (Sedova et al., 12 May 2026). A plausible implication is that the no-repetition law is often best viewed as the lower-order limit of a broader repetition-aware law, not as the compute-optimal prescription in scarce-data settings.
5. Statistical interpretation and theoretical mechanisms
The no-repetition law in linear regression is derived under a specific set of assumptions: Gaussian covariates 5, a well-specified linear teacher, isotropic Gaussian prior on 6, covariance eigenvalues 7 with 8, a random Gaussian sketch 9, and one-pass SGD with a geometrically decaying step-size (Lin et al., 2024). In that setting, the population risk decomposes into irreducible risk, approximation error, and excess risk. The approximation error scales as 0, the leading bias term scales as 1, and the variance term is of strictly higher order. The paper attributes the suppression of the variance bump to the implicit regularization of SGD: one-pass SGD with small or decaying step size cuts off the effective spectrum at 2, so the classical growing-variance term becomes empirically unobservable.
This theorem provides one account of why a no-repetition law can exhibit monotonic improvement with model size. In the notation of Lin et al., the expected risk is
3
with no visible U-shaped variance term (Lin et al., 2024).
The complementary question is why repetition breaks that behavior. Chudnovsky et al. analyze a toy misspecified linear-regression model with 4 unique samples and 5 repeatable samples each duplicated 6 times, giving 7 total training examples. The block-diagonal covariance of duplicated noise is
8
and the conditional train and test risks have closed forms involving 9 and 00. The reported simulations confirm a non-monotonic test loss in 01 at fixed 02 and show that the location of maximal generalization error moves to larger 03 as 04 grows, qualitatively matching the empirical trend 05 (Chudnovsky et al., 23 Jun 2026).
Taken together, these two lines of analysis place no-repetition and repetition in direct theoretical opposition: one-pass SGD yields a clean two-term power law, while duplicated data introduces a memorization–generalization tradeoff that can generate an intermediate-06 loss peak.
6. Assumptions, limitations, and prescriptive use
The no-repetition laws in current use are fitted or derived under narrow assumptions. InfoLaw assumes a fixed bucketing 07 of data by a heuristic quality score, with a simple exponential quality-density 08; the number and boundaries of the buckets were not jointly optimized. Its normalization by 09 is empirical, because alternative forms failed to collapse or extrapolate. The “overtrain degree” 10 empirically shifts the intercept of the scaling curve but is not yet derived from first principles. The logarithmic form 11 was chosen for smooth extrapolation beyond 12B, and no attempt was made to re-fit 13 on very large models (Liu et al., 4 May 2026).
The repetition-aware mixture law of Mhammedi et al. is likewise bounded by its training setup. All runs use GPT-2-style decoder-only Transformers and constant LR schedules. The largest scale is 14M parameters, so extrapolation to 15B+ remains an open question. Quality-filtered data yielded slightly lower 16 than language-domain splits, which the paper interprets as evidence that the “distance” of generic versus target data is not fully captured by the effective-data formula (Sedova et al., 12 May 2026).
The additive-overfitting law makes a different simplification. Its one-parameter form isolates repetition damage in a single coefficient 17,
18
This yields 19 on the FineWeb sweep, compared with 20 for Chinchilla plus effective-data and 21 for the four-parameter variant. The fitted one-parameter coefficient under standard weight decay is 22. The same framework is used prescriptively: with 23M and 24 FLOPs, the law prescribes 25B and 26, achieving perplexity 27 versus 28 for the prior effective-data law. In a weight-decay case study, strong weight decay 29 reduces the fitted penalty from 30 to 31, approximately a 32 reduction, while increasing the single-epoch floor by 33 nats (Lovelace et al., 2 May 2026).
Across these papers, the recommended use is consistent. One first fits a no-repetition or low-repetition reference law on a manageable suite of experiments, then uses that law to search over compute allocations, repetition levels, or mixture weights before running large-scale training. InfoLaw recommends fitting 34 and 35 on a small suite of experiments and then computing 36 to predict loss for large candidate recipes, enabling search over hundreds of thousands of mixture weights (Liu et al., 4 May 2026). Chudnovsky et al. recommend quantifying repetition damage via 37, fixing the repeated-token fraction to match the corpus, and sweeping 38 to locate worst-case structures (Chudnovsky et al., 23 Jun 2026). Ivgi et al. recommend solving
39
and scanning small integers 40 to minimize the loss, with the explicit guideline that if the optimum is 41–42, additional compute should be allocated to increasing 43 rather than repeating data (Lovelace et al., 2 May 2026).
In that sense, the no-repetition scaling law has become both a baseline and a control surface: a baseline because it defines the frontier absent duplication, and a control surface because repetition-aware optimization procedures are constructed by measuring departures from it.