Scaling Law by Difficulty

• Scaling Law by Difficulty is a framework characterizing how explicit difficulty measures—such as token learning time, difficulty gaps, and task separability—influence performance metrics.
• It spans diverse settings including language-model pretraining, sparse evaluation matrices, and physical neural networks, each formalizing difficulty with specific observables.
• The scaling laws provide actionable insights for optimizing training strategies and evaluation designs, improving loss reduction and overall prediction accuracy.

Searching arXiv for the specified papers to ground the article in current preprints. “Scaling Law by Difficulty” denotes a family of empirical and theoretical relations in which performance, loss, ranking error, or optimization time varies systematically with a quantified notion of difficulty. In recent arXiv usage, difficulty has been formalized at several levels: token learning time in language-model pretraining, item-difficulty gaps in sparse evaluation matrices, task-conditioned separability in physical neural networks, logic-preserving variation difficulty for LLM reasoning, the Zipf-law exponent governing token frequencies, and the average difficulty of fine-tuning data in structured-reasoning curricula (Wang et al., 29 Jun 2026, Kang, 11 May 2026, Ermolaev et al., 30 Jun 2026, Mondorf et al., 14 May 2026, Kunstner et al., 25 May 2025, Yang et al., 26 Aug 2025). Across these settings, the common claim is that aggregate curves are often governed by a heterogeneous spectrum of hard and easy cases rather than by uniform averages alone.

1. Conceptual scope and formalizations

The phrase has no single canonical equation. Instead, it refers to a recurrent pattern: once difficulty is made explicit, global behavior can often be written as a low-dimensional law in that difficulty variable, or in an interaction between difficulty and another resource such as data coverage, model scale, or representation quality. In the literature summarized here, “difficulty” is not treated as a metaphor; it is attached to specific observables or latent variables and then linked to measurable outcomes (Wang et al., 29 Jun 2026, Kang, 11 May 2026, Ermolaev et al., 30 Jun 2026, Mondorf et al., 14 May 2026, Kunstner et al., 25 May 2025, Yang et al., 26 Aug 2025).

Setting	Difficulty variable	Measured outcome
Language-model pretraining	Token learning time $\tau_i$	Validation loss derivative and training trajectory
Sparse benchmark evaluation	Difficulty gap $D$ with sparsity $S$	Ranking error $E = 1-\rho$
Physical neural networks	Task-conditioned exponent $\alpha$ in $L(\mathrm{HTC})$	Classification loss
Logic-preserving robustness	$\mathrm{LD}_{\min}$ and $\mathrm{MD}_{\mathcal H}$	Accuracy, failure severity, DRS
Zipfian optimization	Power-law exponent $\alpha$ in $\pi_k \propto 1/k^\alpha$	Iteration complexity of GD and SD
Difficulty-aware fine-tuning	Average training difficulty $D$0	Test accuracy $D$1

A central implication is that “difficulty” can denote at least three distinct objects: a property of examples, a property of tasks, or a property of the data distribution itself. The resulting laws therefore differ in semantics even when they share power-law or interaction-law form. A common misconception is to treat all such laws as interchangeable; the cited work instead shows that the meaning of the difficulty variable determines what can be predicted and what can be controlled.

2. Token-level learning times in language-model scaling

In large-scale language modeling, the aggregate validation loss is often written as

$D$2

Wang et al. decompose this smooth law over the set $D$3 of contextualized validation tokens,

$D$4

where the horizontal axis $D$5 may be training step $D$6, data scale $D$7, or model scale $D$8. Empirically, each token loss $D$9 stays on a high plateau, undergoes one main drop, and then flattens again, so the trajectory is fitted with a sigmoid whose center $S$0 is the token’s “learning time” and whose sharpness parameter is $S$1 (Wang et al., 29 Jun 2026).

The derivative-level picture is more consequential than the loss fit itself. Writing the token derivative as a localized “learning pulse,” and replacing individual pulses by a shared template $S$2, the global loss derivative becomes

$S$3

where

$S$4

is the empirical distribution of learning times. Across more than one hundred pre-training runs on large and diverse real-language corpora with modern LLM architectures, scaling up to 6B parameters and 300B training tokens, the measured spectrum $S$5 follows an approximate power law over its main regime,

$S$6

with $S$7 or $S$8 depending on the axis. On the training-step axis, for example, $S$9 implies $E = 1-\rho$0, and the measured $E = 1-\rho$1 matches that tail index.

This formulation directly challenges the view that smooth scaling laws require smooth tokenwise progress. The reported result is the opposite: smooth global power laws can hide stepwise token learning. The same signal is also actionable. By estimating $E = 1-\rho$2, scoring samples by the fraction of tokens whose learning times fall in a target interval $E = 1-\rho$3, bucketing by score, and reweighting the mixture toward tokens that are about to become learnable, the optimization trajectory can be altered. The reported effect is an approximately $E = 1-\rho$4 faster reduction in validation loss over the target interval, meaning that the same loss level is reached in about $E = 1-\rho$5 fewer steps (Wang et al., 29 Jun 2026).

3. Difficulty gaps and the scaling law of evaluation failure

A distinct use of “Scaling Law by Difficulty” arises in benchmark evaluation when the evaluation matrix is both sparse and heterogeneous in item difficulty. In the two-parameter logistic Item Response Theory setting, the evaluation matrix is

$E = 1-\rho$6

coverage is

$E = 1-\rho$7

and the difficulty gap is

$E = 1-\rho$8

Performance is measured by Spearman rank correlation $E = 1-\rho$9 between an estimated ranking $\alpha$0 and ground truth $\alpha$1, with ranking error

$\alpha$2

The failure surface for simple-average ranking error is modeled as

$\alpha$3

with centered variables $\alpha$4 and $\alpha$5 (Kang, 11 May 2026).

Under difficulty-biased missingness, the fitted coefficients are

$\alpha$6

with $\alpha$7. The positive interaction term implies that the slope with respect to difficulty,

$\alpha$8

increases with sparsity. In the paper’s concise form, the evaluation-failure scaling law is

$\alpha$9

or, to leading order,

$L(\mathrm{HTC})$0

The empirical findings quantify a collapse regime for simple averaging. Spearman rank correlation degrades from $L(\mathrm{HTC})$1 at $L(\mathrm{HTC})$2 coverage to $L(\mathrm{HTC})$3 at $L(\mathrm{HTC})$4 coverage with high difficulty heterogeneity, and the 150-condition grid sweep over sparsity $L(\mathrm{HTC})$5 and difficulty gap $L(\mathrm{HTC})$6 shows a strong $L(\mathrm{HTC})$7 interaction with $L(\mathrm{HTC})$8, $L(\mathrm{HTC})$9. The authors report that once $\mathrm{LD}_{\min}$0 and $\mathrm{LD}_{\min}$1, simple-average $\mathrm{LD}_{\min}$2 drops below $\mathrm{LD}_{\min}$3, and at the extreme grid corner $\mathrm{LD}_{\min}$4 it falls to $\mathrm{LD}_{\min}$5. By contrast, a standard 2PL IRT model maintains $\mathrm{LD}_{\min}$6 across all conditions and $\mathrm{LD}_{\min}$7 throughout the grid sweep (Kang, 11 May 2026).

The significance is methodological rather than architectural. Difficulty here is not a property of model internals, but of the item pool. The law therefore explains when benchmark aggregation collapses, especially in settings where evaluation matrices are incomplete and difficulty gaps are extreme.

4. Task difficulty and substrate-agnostic power laws in physical neural networks

In physical neural networks, task difficulty is encoded through the exponent of a power law relating classification loss to the Hotelling Trace Criterion (HTC), a task-conditioned measure of state separability that can be evaluated without training. Given class means, global mean, between-class scatter $\mathrm{LD}_{\min}$8, and within-class scatter $\mathrm{LD}_{\min}$9, the criterion is

$\mathrm{MD}_{\mathcal H}$0

where $\mathrm{MD}_{\mathcal H}$1 are the nonzero eigenvalues of $\mathrm{MD}_{\mathcal H}$2 (Ermolaev et al., 30 Jun 2026).

Across highly nonlinear optical fibres, vertical-cavity surface-emitting lasers, and coupled nonlinear oscillator networks, the reported empirical law is

$\mathrm{MD}_{\mathcal H}$3

When fit globally across four PNN implementations, the benchmark-specific forms are

$\mathrm{MD}_{\mathcal H}$4

with Pearson correlation $\mathrm{MD}_{\mathcal H}$5, and

$\mathrm{MD}_{\mathcal H}$6

with $\mathrm{MD}_{\mathcal H}$7. The exponent is larger for MNIST than for Fashion-MNIST, and the paper interprets this as a quantitative encoding of task difficulty: easier tasks have a steeper power-law slope, so a given fractional increase in HTC yields a larger gain in accuracy.

The key claim is a universal collapse onto a task-specific curve. Despite major differences among substrates, once loss and HTC are computed, the data fall onto the same line in the $\mathrm{MD}_{\mathcal H}$8 plane for a given dataset. The prefactor $\mathrm{MD}_{\mathcal H}$9 is also task-dependent, setting the baseline loss at $\alpha$0, and harder tasks have a larger $\alpha$1. This makes the law task-specific but substrate-agnostic.

The framework also supports calibration and prediction. A small calibration set of trained systems is used to determine $\alpha$2 and $\alpha$3, after which new systems need only an HTC measurement to predict expected loss. In the reported coupled nonlinear oscillator network example, only a handful of training runs suffice to fix the exponent, and subsequent architectures and parameter settings need no further offline training to predict performance with better than $\alpha$4 accuracy. Layer-wise HTC further acts as a diagnostic: in a three-layer CNON, Layers 2–3 adhere to the same power-law line, while Layer 1 deviates, indicating under-utilization that is invisible to standard loss monitoring (Ermolaev et al., 30 Jun 2026).

5. Logic-preserving difficulty scaling in LLM robustness evaluation

In robustness work on LLM reasoning, difficulty is attached not to the original problem statement alone but to logic-preserving variations of a symbolic template. The LPDS framework defines two distance-based metrics. The first is the minimum normalized Levenshtein distance,

$\alpha$5

which measures lexical deviation from correct reasoning traces. The second is Mahalanobis distance in hidden-state space,

$\alpha$6

which measures how far a response lies from the manifold of correct solution traces (Mondorf et al., 14 May 2026).

LPDS then searches the variation space of a symbolic template $\alpha$7 with a discrete beam-search procedure. Starting from a template variation $\alpha$8, the algorithm alternates candidate generation, cheap scoring with an input-embedding Mahalanobis proxy, pruning, exact scoring with the expensive output-based metric, beam update, and best-so-far tracking. The reported hyperparameters are $\alpha$9, $\pi_k \propto 1/k^\alpha$0, $\pi_k \propto 1/k^\alpha$1, $\pi_k \propto 1/k^\alpha$2, and $\pi_k \propto 1/k^\alpha$3. This design is intended to find a variation $\pi_k \propto 1/k^\alpha$4 that maximizes difficulty while controlling cost.

The empirical law is monotonic rather than power-law. Over 100 templates with 1,000 variations each, both Levenshtein and Mahalanobis distances separate correct from incorrect responses much better than perplexity or self-certainty, with $\pi_k \propto 1/k^\alpha$5–$\pi_k \propto 1/k^\alpha$6 versus $\pi_k \propto 1/k^\alpha$7–$\pi_k \propto 1/k^\alpha$8 for those baselines. Logistic mixed-effects models of the form

$\pi_k \propto 1/k^\alpha$9

yield per-standard-deviation odds ratios as low as $D$00–$D$01, indicating a strong negative association between difficulty and correctness. After beam search, explored variations are binned into 20 quantiles by $D$02, and accuracy decays monotonically with quantile. Smaller models begin to fail at lower quantiles, whereas Llama-70B maintains high accuracy until about quantile $D$03 and then drops. The paper summarizes these curves by a Difficulty–Robustness Score, the normalized area under the accuracy–quantile curve.

The framework is also comparative. On GSM-Symbolic, the relative accuracy drop from canonical GSM8K to logic-preserving search samples is reported as $D$04 percentage points for Llama-3.2-3B versus $D$05 points for random sampling, and $D$06 points for Qwen-32B versus $D$07 points for random sampling. Fine-tuning on the top-difficulty third of errors improves performance on all quantiles, whereas training on the easiest third helps only the low-difficulty region. The resulting claim is that random sampling can overstate robustness because it misses the brittle tail of high-difficulty, logic-preserving variations (Mondorf et al., 14 May 2026).

6. Optimization difficulty under Zipf’s law

A more theoretical variant of the topic concerns optimization difficulty induced by heavy-tailed token frequencies. In the linear bigram model for next-token prediction, one observes one-hot token pairs $D$08 and fits $D$09 by minimizing

$D$10

The heavy-tailed assumption is

$D$11

with $D$12. Here the difficulty variable is the Zipf exponent itself: smaller $D$13 corresponds to heavier tails, while $D$14 matches text data (Kunstner et al., 25 May 2025).

Under constant-step-size gradient descent with $D$15, the iteration complexity to reach relative error $D$16 obeys three regimes: $D$17

$D$18

$D$19

For sign descent, used as a proxy for Adam, the corresponding laws are

$D$20

$D$21

$D$22

The paper identifies $D$23 as “worst-case” for gradient descent in the sense that the number of iterations required to reach a small relative error scales almost linearly with dimension. The derivation proceeds through a closed-form GD dynamics in the Hessian eigenbasis and an integral approximation showing that, for $D$24, the correct rescaling is $D$25, with normalized loss converging to $D$26. Setting $D$27 yields $D$28.

This yields a sharp optimizer comparison. For light tails $D$29, gradient descent is dimension-free and sign descent is worse. For the medium-tail regime $D$30, sign descent scales only as $D$31, giving a $D$32 speed-up relative to the Zipfian or near-Zipfian GD behavior. The stated intuition is that heavy-tailed data create many tiny eigenvalues; GD stalls because progress in low-frequency directions requires waiting on the order of $D$33 steps, while sign descent effectively normalizes per-coordinate step lengths and mitigates that penalty (Kunstner et al., 25 May 2025).

7. Training-data difficulty, U-shaped performance, and structured solution templates

A further meaning of “Scaling Law by Difficulty” concerns the composition of fine-tuning data for procedural reasoning. In this setting, the paper operationalizes example difficulty with three proxies: solution-trace length,

$D$34

model-based judgment using a GPT-4 rating in $D$35, and the empirical error distribution based on absolute prediction error $D$36. Datasets are partitioned into low, medium, and high difficulty bins, and the average difficulty of a fine-tuning set $D$37 is

$D$38

Test accuracy is then treated as a function $D$39 (Yang et al., 26 Aug 2025).

The reported law is U-shaped. There exist two critical points $D$40 such that

$D$41

$D$42

with a minimum at some $D$43. The qualitative statement is that performance first drops as fine-tuning is flooded with low-difficulty data, then recovers and eventually surpasses earlier levels when the set is composed of sufficiently challenging instances. The paper’s illustrative AIME24 table reports accuracy falling from $D$44 at $D$45k synthetic low-difficulty points to $D$46 at $D$47k. On MATH500, average absolute error across chain-length bins is $D$48 for $D$49, $D$50 for $D$51, $D$52 for $D$53, and $D$54 for $D$55.

This finding motivates the Structured Solution Template framework. Stage 1 applies weighted template fine-tuning with structured solution-template chains and dynamically weighted loss,

$D$56

with

$D$57

and uses XML tags $D$58. Stage 2 performs prompt-time chain injection with a small LoRA-adapted Qwen-1.5B. Stage 3 uses curriculum fine-tuning with Group Relative Policy Optimization, hard-mining the backbone’s weakest $D$59k problems and generating “plan–then–execute–self-correct” traces.

The paper’s interpretation is that too many easy, short, one-pattern chains promote shallow pattern matching, whereas complex, multi-step templates force the model to learn reusable abstractions. This directly contradicts the monotonic assumption that adding easier synthetic data is necessarily beneficial. The reported combined effect of SST is to reshape the empirical U-curve and move training into the high-difficulty regime, with gains of $D$60 points on GSM8K, $D$61 points on AIME24, and improvements on the Dynamic En benchmark (Yang et al., 26 Aug 2025).