Data-Constrained Scaling in Machine Learning

• Data-Constrained Scaling refers to optimizing machine learning models with limited data or when data resources are not directly related to data volume. Data features such as quantity, diversity, quality, and observability are essential factors in this approach, influencing the performance of decision systems, optimization procedures, and model behaviors.
• Successful scaling in machine learning models in the face of limited data enables the effective extraction of useful data for training models, adaptation, and reliability, especially in scenarios such as medical imaging, supervised learning, or scenarios when large volumes of data are unfeasible.
• Data Constrained Scaling solutions across models often rely on sophisticated algorithms for data selection, replication, training, and learning to maximize the use of data and provide a predictive model that is generalized and adaptable to new data scenarios.
• bulleted_list_following_shiloh_gman_order_be_solutons_5_questions_and_elements_specific_to_the_topic_of__data _constrained_scaling_18__Scaling laws that relate unique data, repeating data, data aquality, selection method to preprocessing the data, prediction accuracy, and more?

Data-Constrained Scaling is the study of how models, optimization procedures, decision systems, and data-generation pipelines behave when the quantity, diversity, quality, or observability of useful data is limited while computation, model capacity, reliability requirements, or deployment scale continues to increase. The term encompasses several related regimes: repeated use of a finite corpus; allocation of scarce domain data within mixtures; scaling under limited downstream labels; constructing reliable decisions from rare-event samples; reducing communication or constraint-processing costs; and estimating scaling laws when exhaustive experiments are unaffordable. Across these settings, raw data volume is not interchangeable with unique, informative, usable, or statistically resolutive data.

1. Conceptual scope and resource axes

Data-constrained scaling differs from conventional scaling because the number of processed examples is not necessarily the number of independent or informative examples. In language-model pretraining, a corpus may contain $U$ unique tokens while training processes $D=UE$ tokens over $E$ epochs. In mixture pretraining, a target corpus of size $D_{\mathrm{target}}$ may be sampled with repetition factor

$r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$

where $h$ is the target-mixture fraction. In downstream transfer, the relevant limitation may instead be the number of labeled fine-tuning examples, while in chance-constrained optimization it may be the inability of $N$ observations to resolve violation probabilities substantially smaller than $1/N$.

The principal resource axes are therefore distinct:

• Unique data: independent or non-repeated examples available for learning.
• Processed data: total examples or tokens presented to the optimizer, including repetitions.
• Data quality and information content: diversity, relevance, syntheticity, resolution, and task-relevant information per example.
• Model capacity: parameter count or representational dimension.
• Optimization exposure: training steps, epochs, or repeated stochastic updates.
• Measurement compute: resources required to estimate a scaling law or select a training subset.
• Communication and constraint-processing capacity: bandwidth, memory, and the number of pairwise, scenario, or multi-hop relationships that can be processed.

Conventional Chinchilla-style language-model scaling uses

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$

where $N$ is model size and $D=UE$0 is training-token count. Its implicit assumption is that additional tokens are sufficiently novel and useful. Data-constrained regimes violate that assumption. Repetition may initially remain useful, but its marginal value can decay, while overfitting, memorization, or domain imbalance can increase.

Several papers extend scaling laws by adding interaction or saturation terms. “Scaling Data-Constrained LLMs” replaces raw data and model size by effective quantities that saturate under repetition and excess parameters, obtaining fitted repetition and excess-capacity scales of approximately $D=UE$1 and $D=UE$2 in its C4 experiments (Muennighoff et al., 2023). “Prescriptive Scaling Laws for Data Constrained Training” instead adds an explicit repetition-induced overfitting penalty,

$D=UE$3

or more flexible variants with exponents on repetition and capacity (Lovelace et al., 2 May 2026). “Practical Scaling Laws” introduces unique data $D=UE$4, total processed data $D=UE$5, and a saturating loss wrapper:

$D=UE$6

The three terms represent undercapacity, undertraining, and overfitting, respectively (Bryant et al., 9 May 2026).

2. Repetition, regularization, and model–data interaction

Repeated training is not uniformly harmful. In controlled language-model experiments, up to approximately four epochs of repeated data produced negligible changes in held-out loss relative to unique data at fixed total compute. At greater repetition, the value of additional compute decayed rapidly; the fitted repetition scale was approximately 15 repetitions beyond the first epoch, corresponding to roughly 16 total epochs (Muennighoff et al., 2023).

The implication is not that repeated data is equivalent to fresh data indefinitely. Early repetitions can support additional optimization and allow the model to revisit examples with improved representations. Later repetitions increasingly expose capacity to memorization and specialization. The relative usefulness of repetition depends on model size, unique-data size, optimization, regularization, and data composition.

The additive-overfitting formulation makes this dependence explicit. Its penalty

$D=UE$7

increases with the number of additional epochs $D=UE$8, model size $D=UE$9, and inverse unique-data size $E$0. More flexible versions use

$E$1

with fitted $E$2 and $E$3 indicating superlinear sensitivity to repetition and capacity (Lovelace et al., 2 May 2026). In experiments with more than 300 decoder-only models, the proposed law tracked validation loss better than effective-data baselines, including on an external C4/deduplicated-data sweep. Its four-parameter form achieved $E$4 on the principal multi-epoch sweep and $E$5 on the external sweep (Lovelace et al., 2 May 2026).

The resulting allocation rule differs from ordinary Chinchilla scaling. At fixed compute and unique-data size, increasing epochs reduces the undertraining term but also forces a smaller model and increases the overfitting penalty. Once the marginal cost of repetition exceeds its training benefit, additional compute is better spent on model capacity. “Practical Scaling Laws” derives a finite model-size optimum at fixed unique data and unlimited training:

$E$6

Thus, the optimal model grows with available unique data but remains finite for every finite corpus (Bryant et al., 9 May 2026).

Regularization changes this boundary. In the prescriptive-law experiments, increasing weight decay from $E$7 to $E$8 reduced the one-parameter overfitting coefficient from $E$9 to $D_{\mathrm{target}}$0, approximately a 70% reduction. Strong weight decay imposed a single-epoch loss cost but became preferable at higher compute, where standard regularization required excessive repetition (Lovelace et al., 2 May 2026). Data-constrained pretraining experiments similarly found that strong weight decay, with selected values reaching $D_{\mathrm{target}}$1 and $D_{\mathrm{target}}$2, was often necessary for repeated-data training (Xu et al., 5 Jun 2026).

The SoftQ law models the interaction between model size and unique data directly:

$D_{\mathrm{target}}$3

Unlike an additive law, SoftQ predicts that the value of additional unique data depends on model size. It fit a 20-point grid of 72M–1.4B parameter models trained on 100M–400M unique tokens with lower error than Chinchilla, Quanta, and Muennighoff-style alternatives, and achieved a held-out RMSE of $D_{\mathrm{target}}$4 when extrapolating to the 400M-token data budget (Xu et al., 5 Jun 2026).

3. Data quality, mixtures, selection, and synthetic supervision

Raw token count can be adjusted for quality. “Scaling Parameter-Constrained LLMs with Quality Data” defines effective training tokens as

$D_{\mathrm{target}}$5

where $D_{\mathrm{target}}$6 is compression-derived diversity and $D_{\mathrm{target}}$7 is inverse perplexity under Llama-2 7B Chat, used as a teacher-model syntheticity measure (Chang et al., 2024). The fitted model combines this quantity with a Chinchilla-like law and reports a Pearson correlation of approximately $D_{\mathrm{target}}$8 between predicted and observed average accuracy across 210 model–data points. The coefficient signs were $D_{\mathrm{target}}$9 and $r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$0 in the reported fit, yielding a higher effective-token estimate for less diverse but more teacher-predictable synthetic data. This is an empirical result of the selected proxies and benchmark regime, not a general conclusion that diversity harms generalization.

Data selection also has a compute cost. “Compute-Constrained Data Selection” formalizes total expenditure as selection compute plus training compute:

$r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$1

where $r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$2 is the number of selected examples and the second term scores the entire candidate corpus. BM25 selection was estimated at approximately $r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$3 FLOPs, embedding retrieval at $r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$4 FLOPs, perplexity selection at $r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$5 FLOPs, and LESS gradient selection at $r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$6 FLOPs (Yin et al., 2024). Methods that look strongest at a fixed token budget can therefore be suboptimal at fixed total compute. BM25 and embedding retrieval were generally preferred at 7B and 13B training scale, whereas perplexity and gradient-based selection became more attractive for substantially larger training models. The estimated break-even ratios were approximately

$r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$7

for perplexity selection and approximately

$r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$8

for gradient selection.

Mixture pretraining introduces a regularizing role for abundant generic data. “Scaling Laws for Mixture Pretraining Under Data Constraints” found that target-domain performance was governed more directly by target repetition $r=h\frac{D_{\mathrm{total}}}{D_{\mathrm{target}}},$9 than by target-mixture fraction $h$0. Target loss was often U-shaped: additional target exposure initially improved performance, while excessive repetition caused overfitting (Sedova et al., 12 May 2026). Mixture training tolerated approximately 15–20 repetitions of scarce target data in the reported regimes, substantially more than single-source repetition, because generic tokens remained fresh and provided a regularizing signal. The proposed effective target contribution was

$h$1

while generic data continued contributing without the same saturation. The resulting repetition-aware mixture law predicted optimal target fractions more accurately than repetition-agnostic alternatives.

Synthetic-supervision pipelines exhibit another form of data constraint: the source corpus may be large, but only a small fraction of candidate examples may be usable. “Scaling Multi-Hop Training Data via Graph-Constrained Path Selection” constructed multi-hop supervision from legal contracts by separating path discovery from teacher verbalization (Chen et al., 29 May 2026). On CUAD, random chains passed the teacher quality gate 22.0% of the time and saturated at roughly 18K usable examples. Graph-Constrained Semantic Chain Synthesis increased the pass rate to 94.5% and produced approximately 80K usable examples, a 4.4-fold expansion. A matched-size ablation yielded 22.51% Token F1 for random SFT and 22.66% for constrained SFT at 18K examples, while full-scale constrained SFT reached 38.58% at 80K examples. The evidence therefore supports increased teacher synthesizability and usable-data yield rather than a clear per-example quality advantage.

4. Alternative objectives and reuse-efficient optimization

Data-constrained scaling can change which training objective is preferable. “Diffusion Beats Autoregressive in Data-Constrained Settings” compared autoregressive models with masked diffusion models at fixed unique-token budgets of 25M, 50M, and 100M, with training durations up to 800 epochs (Prabhudesai et al., 21 Jul 2025). Autoregressive training initially achieved lower validation loss because it provided dense supervision under a fixed left-to-right factorization. Masked diffusion was initially more compute-expensive but exposed the same sequence to varying masking patterns, contexts, prediction subsets, and effective token orderings.

The fitted effective data-reuse scales were approximately

$h$2

for autoregressive training and

$h$3

for diffusion. For 100M unique tokens, the best autoregressive result occurred near approximately 50 epochs, while diffusion continued improving toward approximately 500 epochs. The reported best validation losses were approximately 3.71 for autoregressive training and 3.55 for diffusion. The diffusion advantage appeared only after a compute-dependent crossover, with the fitted critical compute scaling approximately as

$h$4

The paper interprets masked diffusion as implicit data augmentation, although alternative explanations include differences in gradient noise, supervision density, optimization geometry, and regularization.

A related theoretical result concerns multi-pass SGD in linear regression. “Improved Scaling Laws in Linear Regression via Data Reuse” considers a covariance spectrum $h$5 and a target-energy spectrum proportional to $h$6 with $h$7 (Lin et al., 10 Jun 2025). For a sketched $h$8-dimensional model trained on $h$9 distinct examples for $N$0 stochastic-gradient steps, the excess risk scales as

$N$1

up to the effective-step and logarithmic conventions of the theorem. One-pass SGD instead obtains

$N$2

The result shows that data reuse can continue reducing optimization bias beyond $N$3 under the stated spectral assumptions. It does not imply that repeated examples contain the statistical information of $N$4 fresh samples, nor does it establish the same rate for neural networks or LLMs.

Masked-input regularization provides a different approach. “Data-Constrained LLM Pretraining: Improved Regularization and Scaling Laws” adds an auxiliary causal next-token objective on randomly masked inputs:

$N$5

Across 72M–1.4B parameter models, MIR improved validation loss over strongly weight-decayed autoregressive baselines and improved six of eight reported downstream metrics at 1.4B (Xu et al., 5 Jun 2026). Using SoftQ asymptotes, the paper estimated MIR as equivalent to approximately 1.3 times as much unique data over the 200M–400M range. This estimate is conditional on the fitted baseline and is not a universal information-equivalence constant.

Visual transfer learning exhibits an analogous regime dependence. “Scaling Laws for Data-Efficient Visual Transfer Learning” models downstream error as a function of upstream data $N$6, model size $N$7, and downstream fine-tuning data $N$8 (Yang et al., 17 Apr 2025). Its distillation boundary theory predicts that distilled models are advantageous under scarce data, while direct training of the larger model eventually overtakes distillation as pretraining data becomes sufficient. The reported experiments span approximately 2.5M–38M parameters and downstream data regimes from roughly 1K to 1M samples. The crossover is task- and model-dependent; the paper does not provide universal numerical thresholds.

5. Data-constrained reliability, metric learning, and edge systems

In chance-constrained optimization, data constraints arise because rare-event probabilities cannot be resolved by ordinary samples. If $N$9 independent observations are available, a violation probability $1/N$0 will generally produce no observed violations. “The Scaling Behaviors in Achieving High Reliability via Chance-Constrained Optimization” characterizes an $1/N$1 barrier and studies asymptotic extrapolation beyond it (Deo et al., 10 Apr 2025). Under scaling and tail assumptions, the optimal cost satisfies

$1/N$2

where $1/N$3 is the relevant marginal extreme scale. Light-tailed distributions yield approximately polylogarithmic growth in $1/N$4, whereas heavy-tailed distributions yield polynomial growth.

The paper distinguishes extrapolated Pareto efficiency from finite-sample certification. A base solution at $1/N$5 satisfying $1/N$6 and $1/N$7 can be scaled along a trajectory

$1/N$8

Under the stated structural assumptions, points on this trajectory can be approximately Pareto-efficient even when their violation probabilities are far below $1/N$9. This does not provide a distribution-free high-confidence certificate for an unseen rare event.

“Chance constrained sets approximation” uses a related but distinct strategy (Mammarella et al., 2021). It constructs a low-complexity scalable simple approximating set

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$0

and determines $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$1 from an independent sample using an order statistic. If $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$2 is the maximum scale admissible under sample $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$3, the final scale is

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$4

with binomial condition

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$5

With confidence at least $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$6, every point in the returned set has violation probability at most $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$7. The method separates geometric complexity, selected by the shape $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$8, from probabilistic certification, selected by the one-dimensional scale statistic.

A newer decision-scaled scenario approach modifies sampled constraints as

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$9

Using the same original samples, it replaces the scenario risk level $N$0 by $N$1, leading to a sample requirement of order

$N$2

rather than $N$3 (Choi et al., 18 Mar 2026). The reduction is polynomial in the rare-event parameter, but the transfer from the scaled scenario guarantee to the original chance constraint is asymptotic and depends on regular variation, tail index $N$4, and constraint-scaling exponent $N$5. Increasing $N$6 reduces sample size but produces more conservative decisions.

Metric learning presents a combinatorial constraint-processing version of the same problem. With $N$7 observations, possible pairs number $N$8 and triplets number $N$9. “Constraint Selection in Metric Learning” introduces Loss-dependent Weighted Instance Selection (LWIS), which updates sampling weights according to current constraint losses (Capitaine, 2016). Difficult constraints involving overlapping class regions receive greater probability, while easy observations retain nonzero probability. The method avoids full distance-matrix construction and processes only sampled constraints. On eight UCI datasets, LWIS achieved higher reported accuracy than random, Liu–Vemuri, and Mei et al. selection methods while remaining closer in runtime to inexpensive random selection. On the Forest Cover Type dataset with 581,012 observations, it avoided the full quadratic constraint-processing burden.

Edge video analytics provides a systems-level form of data-constrained scaling. FilterForward places semantic filtering on constrained edge nodes and uploads only selected high-quality event frames (Canel et al., 2019). A shared MobileNet feature extractor feeds multiple application-specific microclassifiers, allowing computational reuse. The system addresses camera uplinks of a few hundred kilobits per second, while raw or compressed high-resolution streams require substantially more bandwidth. Reported bandwidth reductions were 6.3-fold for a full-frame object-detector microclassifier and 13-fold for a localized classifier. With many concurrent classifiers, shared feature extraction achieved up to 6.1 times the throughput of pixel-level discrete classifiers. The system reduces WAN traffic but still requires decoding and base-DNN processing for every input frame.

6. Estimating and transferring scaling laws under limited measurement

A final data constraint concerns the experiments needed to fit a scaling law itself. “Amortizing Scaling Law Construction Costs” formulates scaling-law measurement as Bayesian optimization over model size, token budget, and hyperparameters (Jha et al., 4 Sep 2026). A dense grid trains every configuration, although the final compute-optimal law uses primarily the best-loss frontier. The proposed method combines Gaussian-process Bayesian optimization, compute slicing, and surrogate-fantasized evaluations.

Compute slicing restricts acquisition to an expanding window of compute scales, beginning with lower-cost configurations and progressively including higher-cost slices. Fantasization supplies predicted losses for unobserved grid points, allowing provisional dense-grid frontier fitting. In the reported experiments, the combination of expanding-window acquisition and fantasization matched dense-grid fits with computational savings of approximately 10–100 times. At 1% of the dense-grid compute, extrapolated losses remained within 1.6% of the full-data fit at compute values six to seven orders of magnitude beyond the observed grid. These results are empirical and depend on the finite-grid reference, surrogate adequacy, and the studied scaling-law parameterizations.

Cross-domain transfer provides another way to reduce measurement requirements. “On the Invariance and Generality of Neural Scaling Laws” argues that bijective transformations preserve task-relevant information and therefore preserve scaling exponents, whereas information-reducing transformations modify statistical efficiency and the irreducible-loss floor (Han et al., 8 May 2026). It defines information resolution as

$D=UE$00

where $D=UE$01. The proposed transformed law is

$D=UE$02

The factor $D=UE$03 represents variance inflation, while $D=UE$04 represents an information-loss-induced floor shift. Under the associated compute model, lower information resolution shifts the compute-optimal allocation toward a smaller model and a larger data budget:

$D=UE$05

$D=UE$06

In an OpenWebText-to-MIMIC-IV discharge-summary transfer, compression-based resolution $D=UE$07 yielded a predicted data-scaling exponent close to the empirical target exponent, with the reported exponent recovered within approximately 3%. The framework’s estimates of $D=UE$08 through compression, entropy, or signal-to-noise ratios are proxies rather than direct measurements of task-relevant mutual information.

Across these methods, the central methodological distinction is between interpolation and extrapolation. A scaling law fitted on a dense grid may describe the observed regime without predicting a data-constrained regime. Conversely, a compact law can extrapolate only if its interaction terms, saturation behavior, or information-resolution assumptions are supported by measurements spanning the relevant bottlenecks. Claims about universal exponents, fixed repetition thresholds, or transferable quality coefficients therefore require qualification.

The common conclusion is that data-constrained scaling is not a single law or algorithm. It is a family of resource-allocation problems in which unique data, repeated data, data quality, model capacity, regularization, selection cost, reliability targets, and usable-data yield must be modeled jointly. Depending on the setting, the appropriate response may be to reduce model size, increase regularization, change the learning objective, mix scarce data with fresh generic data, select data using a compute-aware utility, scale decisions rather than samples, filter constraints by loss, move semantic processing to the edge, or reduce the cost of fitting the scaling law itself. In every case, the decisive quantity is not raw data volume alone, but the amount of useful information that can be extracted under the available computational, statistical, and operational constraints.