---
title: No-Repetition Scaling Law in ML Training
url: https://www.emergentmind.com/topics/no-repetition-scaling-law
type: topic
---

# No-Repetition Scaling Law in ML Training

Searching arXiv for the cited papers and closely related scaling-law work.
I’ll look up the specified arXiv records to ground the article in the current literature.
No-Repetition Scaling Law denotes a class of reference scaling relations for model loss or test error in regimes where each training example is seen once, or where repetition variables are fixed to their no-repetition value. In recent work, the term appears in several closely related forms: as a fitted no-repetition compute frontier for language models, as the zero-repetition specialization of an information-based law for quality-weighted mixtures, as the \(R_D=0\) limit of an additive overfitting law for data-constrained training, and as a one-pass SGD theorem in linear regression [2606.24998] [2605.02364] [2605.01640] [2406.08466]. Across these formulations, the no-repetition law functions both as a predictive model of baseline performance and as the reference curve against which the damage from repeated data is quantified.

## 1. Terminology and formal scope

The phrase “no repetition” is not tied to a single notation. In the exact-document repetition setting of Chudnovsky et al., the no-repetition baseline is
\[
L_0(F,N)=L(F,p\!=\!0,R\!=\!1,N),
\]
where \(R\) is the number of times each document in a small “repeated pool” is replayed during training, and \(p\equiv f\) is the fraction of all training tokens drawn from that pool [2606.24998]. In InfoLaw, the no-repetition regime is the specialization \(r=0\) in the sense of “effectively unlimited source data,” \(S\gg K\), so that \(R_d\to1\) for every quality bucket \(d\) and no bucket is ever repeated [2605.02364]. In the data-constrained law of Ivgi et al., repetition is parameterized by \(R_D\), the number of extra epochs beyond the first, so the no-repetition case is \(R_D=0\) [2605.01640]. In the linear-regression theory of Lin et al., no repetition is implemented by one-pass SGD, where each example is seen once [2406.08466].

These notational differences matter because each formulation isolates a different object. The Chudnovsky law is a compute-indexed empirical frontier. InfoLaw is a data-aware model in which information density and diminishing returns under repetition enter explicitly. The additive-overfitting law starts from a Chinchilla-style decomposition and adds a repetition penalty. The linear-regression result is a theorem on reducible risk under one-pass SGD. A plausible implication is that “no repetition” should be understood as a boundary condition shared by several scaling frameworks, rather than as a single universal equation.

## 2. Canonical formulations

Several no-repetition laws now coexist in the literature.

| Setting | No-repetition condition | Resulting law |
|---|---|---|
| Exact-document repetition baseline | \(p=0,\;R=1\) | \(L_0(C)=E+K\,C^{-\gamma}=2.365+6.647\times10^{5}\,C^{-0.317}\) |
| InfoLaw specialization | \(S\gg K,\;R_d\to1\) | \(L_0(K,N,w)=3.7373\Bigl[K\bigl(0.140\ln N+0.018\bigr)\sum_d e^{-0.922\,d}w_d\Bigr]^{-0.0441}\) |
| Additive-overfitting law | \(R_D=0\) | \(L(N,U_D,0)=E+\frac{A}{N^\alpha}+\frac{B}{U_D^\beta}\) |
| One-pass SGD linear regression | one pass over data | \(\mathbb E\,\mathcal R(M,N)=\sigma^2+\Theta\!\bigl(M^{-(a-1)}+N^{-(a-1)/a}\bigr)\) |

In the fitted no-repetition frontier of Chudnovsky et al., compute is written as
\[
F\equiv C\approx 6\,N\,T = 120\cdot OT\cdot N^2,
\]
with \(OT\) the overtraining multiplier relative to Chinchilla-optimal. The fitted constants are \(E=2.365\) nats, \(K=6.647\times10^5\), and \(\gamma=0.317\) [2606.24998].

In InfoLaw, the full information model is
\[
{\rm Info}(w,K,S;N)
=\sum_{d=0}^{D-1}
f_d\,M_d\,\ln K
\Bigl[1-\exp\bigl(-X(N)R_d/\ln K\bigr)\Bigr],
\]
with
\[
L=a\,[{\rm Info}(w,K,S;N)]^{-\beta}.
\]
Under no repetition, \(M_d=w_dK\) and \(R_d=1\), so
\[
{\rm Info}_0(w,K;N)
=\sum_d f_d\,(w_dK)\,\ln K\,\bigl[1-e^{-X(N)/\ln K}\bigr].
\]
For large \(K\), the paper Taylor-expands \(1-e^{-X(N)/\ln K}\approx X(N)/\ln K\), giving
\[
{\rm Info}_0\approx K\,X(N)\sum_d f_d\,w_d,
\]
and hence the zero-repetition law
\[
L_0(K,N,w)
=3.7373\Bigl[K\bigl(0.140\ln N+0.018\bigr)\sum_d e^{-0.922\,d}w_d\Bigr]^{-0.0441}.
\]
The fitted coefficients are \(a=3.7373\), \(\beta=0.0441\), \(f_d=e^{-0.922d}\), and \(X(N)=0.140\ln N+0.018\) [2605.02364].

In the additive-overfitting framework,
\[
L(N,U_D,R_D)
=
E+\frac{A}{N^\alpha}
+\frac{B}{\bigl(U_D(1+R_D)\bigr)^\beta}
+P\,R_D\,\frac{N}{U_D}.
\]
Setting \(R_D=0\) recovers the no-repetition form
\[
L(N,U_D,0)=E+\frac{A}{N^\alpha}+\frac{B}{U_D^\beta},
\]
which is precisely the Chinchilla-style law without a repetition penalty [2605.01640].

The linear-regression result gives a no-repetition theorem rather than a fitted empirical curve. Under a Gaussian sketch of size \(M\), one-pass SGD on \(N\) data, an isotropic Gaussian prior on the teacher, and a covariance spectrum \(\lambda_i\asymp i^{-a}\) with \(a>1\), the reducible part of the test error is
\[
\Theta\!\bigl(M^{-(a-1)}+N^{-(a-1)/a}\bigr),
\]
with the variance term dominated by the other errors due to the implicit regularization of SGD [2406.08466].

## 3. Reference frontier for repetition damage

A central use of the no-repetition law is to provide the reference frontier against which repeated-data runs are evaluated. Chudnovsky et al. define Compute-Equivalent Gain (CEG) and Compute-Equivalent Loss (CEL) by inverting the fitted no-repetition curve:
\[
C^\star(L)=\Bigl(\frac{K}{L-E}\Bigr)^{1/\gamma},
\qquad
\mathrm{CEG}=\frac{C^\star(L)}{C_{\rm actual}},
\qquad
\mathrm{CEL}=1-\mathrm{CEG}.
\]
Here \(\mathrm{CEG}=1\) iff the repeated-data run matches the no-repetition frontier, and \(\mathrm{CEG}<1\) indicates wasted compute [2606.24998].

In that controlled exact-document repetition setting, the repeated-token fraction is fixed to \(f=0.1\), and the eval loss \(L\) at fixed \((N,OT,f)\) peaks at an intermediate repeat count \(R=R^{\mathrm{peak}}\). The peak repeat count is fit as
\[
R^{\mathrm{peak}}=2.31\times10^{10}\,N^{-0.96},
\]
while the corresponding repeated-pool size scales as
\[
D_r^{\mathrm{peak}}=7.58\times10^{-10}\,N^{1.84}.
\]
At \(OT=1\), the peak compute-equivalent losses are
\[
0.19,\,0.19,\,0.21,\,0.21,\,0.26,\,0.33
\quad\text{for}\quad
N=34,48,63,93,153,344\rm\,M.
\]
For the Qwen3-style \(344\)M-parameter model on FineWeb-Edu-Dedup at \(OT=1\), the worst-case \(R^*\approx155\) gives \(\mathrm{CEG}\approx0.67\) and \(\mathrm{CEL}\approx0.33\), meaning the run only “buys” two-thirds of the compute it spent. The paper notes that a raw eval-loss bump of \(2\)–\(4\%\) maps to an \(\mathcal O(1)\) compute gap because the fitted no-repetition law is shallow, with \(\gamma\approx0.32\) [2606.24998].

This use of a no-repetition frontier changes the interpretation of repetition damage. Rather than treating loss deltas in isolation, the loss increase is translated into the amount of no-repetition compute that would have been required to reach the same loss. This suggests that no-repetition laws are as important diagnostically as they are predictively.

## 4. Relation to data mixture and data quality

In repetition-aware mixture scaling, the no-repetition regime is a special case rather than the full problem. The mixture law of Mhammedi et al. defines the target-pool repetition factor
\[
r=\frac{h\,D_{\rm total}}{D_{\rm target}},
\]
the utility-decay factor
\[
\rho(r)=r_1\bigl(1-e^{-(r-1)/r_1}\bigr),
\]
the effective target contribution
\[
D_T=D_{\rm target}[1+\rho(r)],
\]
and the effective total data
\[
D_{\rm eff}=(1-h)\,D_{\rm total}+\tau D_T.
\]
Under no repetition, \(r=1\), hence \(\rho(1)=0\), \(D_T=D_{\rm target}\), and
\[
D_{\rm eff}=(1-h)D_{\rm total}+\tau D_{\rm target}.
\]
The paper states that in this limit the law drops back to data-mixture-only scaling [2605.12715].

The same boundary-case structure appears in InfoLaw. When \(S\gg K\), no bucket is ever repeated, \(R_d=1\), and the total information reduces to \({\rm Info}_0\). Under this zero-repetition specialization, InfoLaw differs sharply from a “Kaplan-style” two-term law
\[
L(T,N)\approx A\,N^{-\alpha}+B\,T^{-\gamma}.
\]
The paper emphasizes three consequences: the \(T\)-exponent is \(0.0441\), “roughly half the usual \(0.08\)–\(0.1\)”; \(N\) enters only inside a logarithm rather than as a direct power; and the prefactor \(a=3.7373\) is correspondingly larger to match overall loss levels. Figures 3(a)–(f) are reported to show that under zero or low repetition the InfoLaw curve collapses all training configurations, including extrapolations to larger \(N\) and \(T\), onto a single straight line in log–log space, whereas the standard two-term fit mis-extrapolates once repetition or data-mixture effects become non-trivial [2605.02364].

Mixture pretraining under data constraints also changes the operational meaning of “safe” repetition. Mhammedi et al. report that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused \(15\)–\(20\) times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale [2605.12715]. A plausible implication is that the no-repetition law is often best viewed as the lower-order limit of a broader repetition-aware law, not as the compute-optimal prescription in scarce-data settings.

## 5. Statistical interpretation and theoretical mechanisms

The no-repetition law in linear regression is derived under a specific set of assumptions: Gaussian covariates \(x\sim\mathcal N(0,H)\), a well-specified linear teacher, isotropic Gaussian prior on \(w^*\), covariance eigenvalues \(\lambda_i\asymp i^{-a}\) with \(a>1\), a random Gaussian sketch \(S\), and one-pass SGD with a geometrically decaying step-size [2406.08466]. In that setting, the population risk decomposes into irreducible risk, approximation error, and excess risk. The approximation error scales as \(M^{1-a}\), the leading bias term scales as \(N^{-(a-1)/a}\), and the variance term is of strictly higher order. The paper attributes the suppression of the variance bump to the implicit regularization of SGD: one-pass SGD with small or decaying step size cuts off the effective spectrum at \(\lambda\approx1/(N_{\rm eff}\gamma)\), so the classical growing-variance term becomes empirically unobservable.

This theorem provides one account of why a no-repetition law can exhibit monotonic improvement with model size. In the notation of Lin et al., the expected risk is
\[
\mathbb E\,\mathcal R(M,N)=\sigma^2+\Theta\!\bigl(M^{-(a-1)}+N^{-(a-1)/a}\bigr),
\]
with no visible U-shaped variance term [2406.08466].

The complementary question is why repetition breaks that behavior. Chudnovsky et al. analyze a toy misspecified linear-regression model with \(n\) unique samples and \(d\) repeatable samples each duplicated \(r\) times, giving \(N=n+dr\) total training examples. The block-diagonal covariance of duplicated noise is
\[
\Sigma_r
=
I_n\oplus\bigoplus_{i=1}^d \mathbf 1_r\mathbf 1_r^\top,
\qquad
X_{\rm in}^\top\Sigma_r X_{\rm in}=C_u+r^2C_d,
\]
and the conditional train and test risks have closed forms involving \((C_u+rC_d)^{-1}\) and \(C_u+r^2C_d\). The reported simulations confirm a non-monotonic test loss in \(r\) at fixed \((m,d,n)\) and show that the location of maximal generalization error moves to larger \(d\) as \(m\) grows, qualitatively matching the empirical trend \(R^*\propto N^{-0.96}\) [2606.24998].

Taken together, these two lines of analysis place no-repetition and repetition in direct theoretical opposition: one-pass SGD yields a clean two-term power law, while duplicated data introduces a memorization–generalization tradeoff that can generate an intermediate-\(R\) loss peak.

## 6. Assumptions, limitations, and prescriptive use

The no-repetition laws in current use are fitted or derived under narrow assumptions. InfoLaw assumes a fixed bucketing \(D=6\) of data by a heuristic quality score, with a simple exponential quality-density \(f_d\); the number and boundaries of the buckets were not jointly optimized. Its normalization by \(\ln K\) is empirical, because alternative forms failed to collapse or extrapolate. The “overtrain degree” \(m\) empirically shifts the intercept of the scaling curve but is not yet derived from first principles. The logarithmic form \(X(N)=0.140\ln N+0.018\) was chosen for smooth extrapolation beyond \(N=1.2\)B, and no attempt was made to re-fit \(X(N)\) on very large models [2605.02364].

The repetition-aware mixture law of Mhammedi et al. is likewise bounded by its training setup. All runs use GPT-2-style decoder-only Transformers and constant LR schedules. The largest scale is \(\sim800\)M parameters, so extrapolation to \(10\)B+ remains an open question. Quality-filtered data yielded slightly lower \(R^2\) than language-domain splits, which the paper interprets as evidence that the “distance” of generic versus target data is not fully captured by the effective-data formula [2605.12715].

The additive-overfitting law makes a different simplification. Its one-parameter form isolates repetition damage in a single coefficient \(P\),
\[
L(N,U_D,R_D)
=
E+\frac{A}{N^\alpha}
+\frac{B}{\bigl(U_D(1+R_D)\bigr)^\beta}
+P\,R_D\,\frac{N}{U_D}.
\]
This yields \(R^2_{\rm multi}\approx0.95\) on the FineWeb sweep, compared with \(R^2_{\rm multi}\approx0.58\) for Chinchilla plus effective-data and \(R^2_{\rm multi}\approx0.994\) for the four-parameter variant. The fitted one-parameter coefficient under standard weight decay is \(P\approx0.02305\). The same framework is used prescriptively: with \(U_D=500\)M and \(C=2\times10^{19}\) FLOPs, the law prescribes \(N=2.2\)B and \(R_D=2\), achieving perplexity \(17.73\) versus \(19.34\) for the prior effective-data law. In a weight-decay case study, strong weight decay \((\lambda=1.0)\) reduces the fitted penalty from \(P_{\rm std}\approx0.02305\) to \(P_{\rm strong}\approx0.00681\), approximately a \(70\%\) reduction, while increasing the single-epoch floor by \(+0.2\) nats [2605.01640].

Across these papers, the recommended use is consistent. One first fits a no-repetition or low-repetition reference law on a manageable suite of experiments, then uses that law to search over compute allocations, repetition levels, or mixture weights before running large-scale training. InfoLaw recommends fitting \(f_d\) and \(X(N)\) on a small suite of experiments and then computing \({\rm Info}(w,K;N)\) to predict loss for large candidate recipes, enabling search over hundreds of thousands of mixture weights [2605.02364]. Chudnovsky et al. recommend quantifying repetition damage via \(\mathrm{CEG}/\mathrm{CEL}\), fixing the repeated-token fraction to match the corpus, and sweeping \(R\) to locate worst-case structures [2606.24998]. Ivgi et al. recommend solving
\[
N=\frac{C}{6\,U_D(1+R_D)}
\]
and scanning small integers \(R_D\) to minimize the loss, with the explicit guideline that if the optimum is \(R_D=0\)–\(1\), additional compute should be allocated to increasing \(N\) rather than repeating data [2605.01640].

In that sense, the no-repetition scaling law has become both a baseline and a control surface: a baseline because it defines the frontier absent duplication, and a control surface because repetition-aware optimization procedures are constructed by measuring departures from it.

Source: https://www.emergentmind.com/topics/no-repetition-scaling-law