Repetition-Aware Mixture Scaling Law

• The paper introduces a loss law that extends classical Chinchilla scaling by incorporating data repetition and mixture composition.
• It models repetition using factors like target-domain repetition, spectral redundancy, and effective token counts to guide training design.
• The approach optimizes computing resources by balancing repeated data penalties with generic data regularization for improved target performance.

Searching arXiv for the cited papers and closely related work on repetition-aware mixture scaling laws. A repetition-aware mixture scaling law is a class of scaling-law formulations in which loss depends not only on model size and consumed tokens, but also on how data are mixed across domains, qualities, or components, and on how often limited subsets are replayed. In this literature, repetition is modeled in several mathematically distinct ways: as a target-domain repetition factor such as $r = h D_\text{total} / D_\text{target}$, as a spectral redundancy index $\rho_{\mathrm{red}} = 1/\beta$, as a bucketwise repetition factor $R_d$ inside an information-equivalent token count, or as an additive overfitting penalty on top of a Chinchilla-style base law. The common departure from classical scaling laws is that repeated tokens are not treated as equivalent to fresh tokens, so mixture composition and repetition jointly determine effective data, scaling exponents, and compute-optimal training configurations (Bi et al., 25 Sep 2025, Sedova et al., 12 May 2026, Liu et al., 4 May 2026).

1. Scope and historical setting

Classical Chinchilla-style laws model final validation loss as

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$

with $N$ the number of parameters and $D$ the number of training tokens. In the repetition-aware literature, this form is regarded as inadequate whenever high-quality or target-domain data are scarce and must be reused, because it assumes that every training token is unique or, equivalently, that raw token count is a sufficient proxy for usable information (Lovelace et al., 2 May 2026).

Mixture-aware scaling work already generalized the Chinchilla form to target-domain loss under a training mixture $h \in \Delta_k$. One representative formulation predicts

$$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$

with mixture-dependent coefficients $A^h$ and $B^h$, and uses mirror descent on the simplex to derive optimal domain weights for a target domain under a given training budget $\rho_{\mathrm{red}} = 1/\beta$0 (Shukor et al., 12 Jul 2025). That framework is mixture-aware but not repetition-aware: it treats the data term through raw $\rho_{\mathrm{red}} = 1/\beta$1, assumes i.i.d. draws from $\rho_{\mathrm{red}} = 1/\beta$2, and does not explicitly model repeated or duplicated data.

The explicit repetition-aware turn arises in settings where a finite target corpus is mixed with an effectively unlimited generic corpus, or where a fixed unique-token budget $\rho_{\mathrm{red}} = 1/\beta$3 is traversed for multiple epochs. In the target-plus-generic setting, training uses a target fraction $\rho_{\mathrm{red}} = 1/\beta$4, total training tokens $\rho_{\mathrm{red}} = 1/\beta$5, and target pool size $\rho_{\mathrm{red}} = 1/\beta$6, which induces the repetition factor

$\rho_{\mathrm{red}} = 1/\beta$7

In the data-constrained setting, the same structure is written as $\rho_{\mathrm{red}} = 1/\beta$8, where $\rho_{\mathrm{red}} = 1/\beta$9 counts repetitions beyond the first pass (Sedova et al., 12 May 2026, Lovelace et al., 2 May 2026). The central problem is then no longer only compute allocation across $R_d$0 and $R_d$1, but allocation across fresh tokens, repeated tokens, and model capacity.

2. Spectral redundancy laws and the mixture bottleneck

A theoretical route to repetition-aware mixture laws is given by the redundancy interpretation of scaling laws in kernel ridge regression. In this setting, the covariance or kernel integral operator

$R_d$2

has eigenpairs $R_d$3, and the effective dimension is

$R_d$4

Under a polynomial spectral tail

$R_d$5

and a source condition

$R_d$6

the bias-variance decomposition yields

$R_d$7

Optimizing over $R_d$8 gives

$R_d$9

The redundancy index is $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$0, so the exponent can also be written as

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$1

This identifies the scaling exponent with redundancy in the representation: a steeper spectrum, corresponding to larger $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$2 and smaller $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$3, yields larger $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$4; a flatter spectrum yields smaller $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$5 (Bi et al., 25 Sep 2025).

The mixture result is the key step that turns this redundancy law into a repetition-aware mixture law. If a mixture distribution has component operators

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$6

and each component satisfies

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$7

then the mixture operator has tail index

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$8

Consequently,

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$9

The heaviest tail, equivalently the smallest $N$0, dominates the global exponent. In the paper’s interpretation, smaller $N$1 corresponds to higher redundancy, so the most repetitive component that has non-zero weight bottlenecks the asymptotic learning curve (Bi et al., 25 Sep 2025).

This framework also provides a spectral reading of repetition. A highly repetitive corpus is associated with a flatter covariance tail, hence smaller $N$2, larger effective dimension $N$3, and slower risk decay. A more diverse or deduplicated corpus is associated with a steeper tail and a larger $N$4. The proposition that “Redundancy reduction improves $N$5” formalizes this monotonicity: if a transformation produces a steeper tail $N$6, then

$N$7

3. Scarce-target pretraining with generic regularization

The most explicit use of the term “repetition-aware mixture scaling law” appears in a pretraining setting with two data sources: a scarce target dataset of size $N$8 tokens and an effectively unlimited generic dataset. Training consumes $N$9 tokens, draws a target token with probability $D$0, and therefore induces the repetition factor

$D$1

The empirical program comprises more than 2,000 language-model training runs with GPT-2–style decoder-only Transformers of size 101M, 143M, 192M, 340M, 539M, and 805M non-embedding parameters, spanning multilingual, domain-specific, and quality-filtered mixtures. Across all settings, the paper reports that repetition is a central driver of target-domain performance, that mixture training tolerates much higher repetition than single-source training, and that scarce target corpora can be reused $D$2–$D$3 times, with the optimal number of repetitions depending on target data size, compute budget, and model scale (Sedova et al., 12 May 2026).

The law begins by defining an effective target data size

$D$4

For small repetition, $D$5, so early passes count almost fully; for large repetition, $D$6, so the target contribution saturates. Total effective data is then

$D$7

where $D$8 is the generic-token count, $D$9 is the saturating target contribution, and $h \in \Delta_k$0 weights the relative value of target tokens versus generic tokens for target loss.

At fixed model size, the proposed loss law is

$h \in \Delta_k$1

Across model sizes, it is generalized to

$h \in \Delta_k$2

The $h \in \Delta_k$3 term is a weight penalty, motivated by the empirical observation that high target fractions tend to hurt target loss at fixed $h \in \Delta_k$4 because they increase repetition and reduce generic regularization. Parameters are fit by minimizing a weighted Huber loss

$h \in \Delta_k$5

using basin-hopping with 100 random restarts.

The principal empirical claim is that generic data acts as an implicit regularizer. Because only target tokens repeat while generic tokens stay fresh, mixture pretraining can sustain much higher target repetition than single-source training. The paper reports that, for realistic budgets, the best target performance often occurs at $h \in \Delta_k$6–$h \in \Delta_k$7, and for some smaller pools even higher, up to $h \in \Delta_k$8 in a multi-domain setting with very small pools. Held-out weighted $h \in \Delta_k$9 values are $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$0 for German, $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$1 for Maths, $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$2 for Quality-filtered mixtures, and $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$3 for Wiki/peS2o under the fixed-size law, while the multi-size law gives test $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$4 for German at 539M and $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$5 for Maths at 936M. For German mixture optimization, the median wasted token fraction is $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$6 for the repetition-aware law, compared with $$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$7 for the repetition-agnostic baseline (Sedova et al., 12 May 2026).

4. Information-based and penalty-based variants

Two later strands modify the token-count axis itself. One retains a base scaling law but adds an overfitting term for repetition; the other replaces raw tokens with an information-equivalent variable.

The mixture-only precursor remains the optimal-data-mixtures law

$$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$8

where

$$\mathcal{L}(N,D,h) = E + \left(\sum_{i=1}^k C_i\,h_i^{\gamma_i}\right)^{-1} + \frac{A^h}{N^\alpha} + \frac{B^h}{D^\beta},$$9

This framework is mixture-aware and prescriptive, but it leaves repetition outside the model and therefore treats raw $A^h$0 as the effective data scale (Shukor et al., 12 Jul 2025).

InfoLaw instead formulates pretraining as information accumulation with bucketwise repetition. Documents are grouped into quality buckets $A^h$1, with mixture weights $A^h$2, source proportions $A^h$3, source tokens $A^h$4, and training tokens $A^h$5. For each bucket,

$A^h$6

Quality density is parameterized as

$A^h$7

so total intrinsic information in bucket $A^h$8 is $A^h$9. The total learned information is then

$B^h$0

and loss follows the power law

$B^h$1

The fitted parameters reported in the main experiments are $B^h$2, $B^h$3, and

$B^h$4

InfoLaw predicts unseen recipes and larger-scale runs up to $B^h$5B and $B^h$6B tokens with $B^h$7 mean and $B^h$8 max absolute error in loss, and it extrapolates across overtraining levels (Liu et al., 4 May 2026).

A distinct line of work starts from a Chinchilla base fit on single-epoch runs and then models repetition as an additive overfitting penalty. Using a unique-token budget $B^h$9, repetitions $\rho_{\mathrm{red}} = 1/\beta$00, and $\rho_{\mathrm{red}} = 1/\beta$01, the most general proposed form is

$\rho_{\mathrm{red}} = 1/\beta$02

Its simplest one-parameter version is

$\rho_{\mathrm{red}} = 1/\beta$03

The additive term is designed to let validation loss increase with further training tokens in high-repetition regimes, which effective-data saturation laws cannot do. The paper reports that the one-parameter penalty already fits multi-epoch data much better than effective-data baselines, and that strong weight decay $\rho_{\mathrm{red}} = 1/\beta$04 reduces the one-parameter coefficient $\rho_{\mathrm{red}} = 1/\beta$05 by approximately $\rho_{\mathrm{red}} = 1/\beta$06, providing a scaling-law explanation for why optimal weight decay in data-constrained regimes can be much larger than standard practice (Lovelace et al., 2 May 2026).

5. Architectural generalizations: Transformers, MoE, and upcycling

Mixture-of-Experts theory introduces a different notion of repetition-awareness by separating active capacity from routing combinatorics. For a sparse MoE Transformer with hard top-$\rho_{\mathrm{red}} = 1/\beta$07 routing, the active parameter count is

$\rho_{\mathrm{red}} = 1/\beta$08

and the worst-case generalization bound under a $\rho_{\mathrm{red}} = 1/\beta$09-dimensional manifold data model and $\rho_{\mathrm{red}} = 1/\beta$10 targets is

$\rho_{\mathrm{red}} = 1/\beta$11

The covering-number decomposition shows a clean separation between an active-capacity term and a MoE-specific routing overhead

$\rho_{\mathrm{red}} = 1/\beta$12

The paper explicitly identifies this routing term as conservative and argues that any improvement beyond the worst-case routing overhead must come from data-dependent routing stability and expert specialization. In repetition-aware reinterpretations, that suggests replacing worst-case $\rho_{\mathrm{red}} = 1/\beta$13 by an effective routing entropy based on expert usage frequencies or realized routing patterns (Mayaki, 10 Apr 2026).

Upcycling laws introduce repetition-awareness through prior training rather than corpus replay. In MoE upcycling, a dense model pretrained on $\rho_{\mathrm{red}} = 1/\beta$14 tokens is converted into an MoE by replicating dense MLP weights into multiple experts and then further trained for $\rho_{\mathrm{red}} = 1/\beta$15 tokens. The empirical two-stage law is

$\rho_{\mathrm{red}} = 1/\beta$16

and the joint law for a fixed architecture is

$\rho_{\mathrm{red}} = 1/\beta$17

Because $\rho_{\mathrm{red}} = 1/\beta$18, the effective exponent for additional upcycling data,

$\rho_{\mathrm{red}} = 1/\beta$19

decreases as dense sunk cost grows: larger $\rho_{\mathrm{red}} = 1/\beta$20 gives a better head start but slower progress with further $\rho_{\mathrm{red}} = 1/\beta$21. The paper reports that using disjoint subsets for dense pretraining and upcycled training makes little difference, so the interaction is not primarily due to literal sample overlap. It also derives a threshold

$\rho_{\mathrm{red}} = 1/\beta$22

below which upcycling is more efficient than from-scratch MoE training and above which from-scratch becomes preferable (Liew et al., 5 Feb 2025).

6. Empirical pathologies, misconceptions, and open questions

A controlled study of exact document repetition shows that repetition damage is not monotone in repeat count. Using Qwen3-style decoder-only transformers of size $\rho_{\mathrm{red}} = 1/\beta$23 and $\rho_{\mathrm{red}} = 1/\beta$24 million parameters, a fixed repeated-token fraction $\rho_{\mathrm{red}} = 1/\beta$25, and a no-repetition reference frontier

$\rho_{\mathrm{red}} = 1/\beta$26

the paper finds that evaluation loss peaks at an intermediate repeat count $\rho_{\mathrm{red}} = 1/\beta$27, not at maximal repetition. Peak locations obey

$\rho_{\mathrm{red}} = 1/\beta$28

and the compute-equivalent loss can be large: for the most damaging repeat count on FineWeb-Edu-Dedup, a $\rho_{\mathrm{red}} = 1/\beta$29M-parameter model at $\rho_{\mathrm{red}} = 1/\beta$30 has $\rho_{\mathrm{red}} = 1/\beta$31, hence $\rho_{\mathrm{red}} = 1/\beta$32. This directly contradicts the common intuition that the harmfulness of repetition should simply increase with repeat count (Chudnovsky et al., 23 Jun 2026).

A second misconception is that single-source heuristics transfer unchanged to mixture pretraining. Earlier single-source work supplied a rule-of-thumb that “up to ~4 repetitions is safe; benefits to ~16; overfitting by ~40,” but the target-plus-generic mixture experiments report a distinct regime in which scarce target corpora can be reused $\rho_{\mathrm{red}} = 1/\beta$33–$\rho_{\mathrm{red}} = 1/\beta$34 times and sometimes $\rho_{\mathrm{red}} = 1/\beta$35, because generic data remains fresh and acts as an implicit regularizer. This means that a repetition-aware mixture law is not merely a correction to single-source repetition laws; it is a different scaling object in which mixture composition changes the tolerated repetition frontier (Sedova et al., 12 May 2026).

The current literature also has pronounced scope conditions. The redundancy-theoretic work assumes clean polynomial tails $\rho_{\mathrm{red}} = 1/\beta$36, a source condition, and finite convex mixtures with fixed $\rho_{\mathrm{red}} = 1/\beta$37; it leaves non-polynomial tails, explicit duplication models, and practical estimation of $\rho_{\mathrm{red}} = 1/\beta$38 open. The empirical pretraining laws are descriptive rather than mechanistic, are fitted on finite architecture families, and report weaker fits in some regimes, such as quality-filtered mixtures. Open directions therefore include more explicit models of duplication and near-duplicate clusters, data-dependent routing analyses for MoE, joint optimization of data mixture, model size, and learning schedule, and extension to other modalities such as vision or code (Bi et al., 25 Sep 2025, Liu et al., 4 May 2026, Sedova et al., 12 May 2026).

Taken together, these works define a coherent research area rather than a single canonical equation. The unifying principle is that repetition, redundancy, and mixture composition alter the effective data axis itself. In theoretical formulations, the decisive quantity is the tail index or redundancy index. In empirical pretraining laws, it is an effective target size, an information-equivalent token count, or an additive overfitting penalty. In architectural variants, it can be the entropy of reused routing patterns or the diminishing marginal value of further training after large sunk cost. A repetition-aware mixture scaling law is therefore best understood as a family of scaling relations that endogenize repeated data, instead of absorbing it into raw token count.