Phantom Gains: Auditing Self-Improvement Against a Measured Null

Abstract: Whether a LLM has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ($p &lt; 10<sup>{-8}$). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.

• The paper introduces a frozen-control audit using repeated sampling, per-problem transitions, Fisher tests, and false-discovery control to separate genuine self-improvement from measurement noise.
• The paper finds that standard metrics can report phantom effects, including a 0.280 expansion rate for an untrained model, while self-training corrupts 11–13 times more problems than the measured null.
• The paper shows that gpt-oss-120b distillation improves rarely solved problems more reliably than three self-training methods, while cautioning that the results cover limited training schedules, benchmarks, and model families.

Overview

"Phantom Gains: Auditing Self-Improvement Against a Measured Null" (2608.20290) is a methodological audit of transition-level evaluation in the self-improving LLM literature. The authors — Xu, Yan, Chen, and Kechadi — train Qwen3-8B with rank-32 LoRA under three rounds of self-training (STaR-style ground-truth filtering, majority-vote SFT in the manner of TTRL, and a policy-gradient arm using the vote as reward), alongside distillation from gpt-oss-120b as a positive control. Their central instrument is a frozen control: an untrained copy of the backbone pushed through the identical sampling and grading pipeline, so that every reported statistic can be read against a null measured on the same problems rather than assumed. From this design they isolate seven measurement failures, each demonstrated on a run where it reverses a published-style conclusion, and they show that two of them are standard practice while one arises inside the correction for another.

The paper's headline numbers are stark. A ledger built on a single greedy decode assigns an untrained model a corruption-to-learning ratio of 1.5; the expansion statistic in current use assigns that same model an "expansion" rate of 0.280; and the natural threshold repair leaves a non-zero null of 0.058 rather than zero. Against these artifacts, the controlled comparison finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not, and that both methods corrupt problems solved at baseline at rates roughly eleven to thirteen times above the measured floor.

Audit design

The unit of analysis is the per-problem transition. For each problem $p$ and checkpoint $t$, the authors draw $k{=}128$ samples and record the solve rate $\hat\pi_t(p)$, discretizing into a binary state via hysteresis bands set at $\tfrac12 \pm z\sqrt{1/4k}$ with $z=2$. Each problem falls into one of seven mutually exclusive categories (stable-correct, stable-incorrect, learned, corrupted, recovered, transient, oscillating), and the headline quantity is the corruption-to-learning ratio $clr$: problems destroyed per problem gained, where $clr > 1$ means net capability loss regardless of mean accuracy.

Three benchmarks partition the measurable phenomena. MATH-500 (200 stratified problems) is heavily contaminated and hosts corruption; AIME 2025/2026 (60 problems, the latter post-dating the backbone's cutoff) hosts expansion claims; and a constructed difficulty band of 1,163 problems, profiled from held-out MATH training data at $k{=}8$ and enriched near the decision boundary, makes corruption statistically observable at all. The authors are explicit that the band's counts are not a forgetting rate for a natural distribution, and that $clr$ is not portable between sets with different difficulty profiles.

A key observation makes measured nulls nearly free: every arm's checkpoint 0 is an independent $t$0 evaluation of the untrained model. With eleven such baselines on AIME, the study owns 110 ordered frozen pairs at no marginal cost — though, as discussed below, eleven turns out to be the minimum for stability.

Seven ways to mismeasure self-improvement

F1: a single decode is not a state. Temperature-zero decoding is nondeterministic under batching. Re-evaluating the frozen model twice on MATH-500 under single greedy decoding yields 9 apparent corruptions against 6 learnings ($t$1), while the solve-rate estimator on the same samples reports 1 and 0. The authors test the mechanism directly rather than citing it: serializing requests cuts verdict flips from 16/200 to 4/200 (Fisher $t$2), confirming batching as the dominant cause — but even fully serialized, the frozen model flips 2% of greedy verdicts, enough to fabricate transitions on small sets. On the difficulty band the gap widens to 18.9% versus 0.8%. The implication is that a single-decode ledger cannot be repaired by more careful decoding; it must be replaced by an estimator.

F2: no threshold repairs the expansion statistic. Following MATH-Beyond, a problem counts as expanded when the base model produced zero correct samples in $t$3 draws and the trained model produces at least $t$4. At $t$5, the criterion in use, the frozen control "expands" 7 of 25 unreached AIME problems — a rate of 0.280, higher than majority-vote self-training itself scores. All seven artifacts are exactly one lucky sample, which appears to license requiring $t$6. It does not: over all 110 frozen pairs, the $t$7 null pools to 0.058 [0.038, 0.078], larger than the 0.048 the statistic was meant to certify as nil. Only $t$8 approaches zero. The threshold repair was fitted to the noise it was fitted on.

F3: a fixed token cap scores style change as capability loss. Distillation adopts its teacher's verbose style; truncation rises from 16.5% to 65.3%, making it nominally the most destructive arm ($t$9). Conditioned on completions that finish, it improves from 0.656 to 0.845 — the most effective method tested. Its corruption counts are excluded from the capability analysis.

F4–F6. Transition metrics carry roughly ten times less power than accuracy: detecting the observed between-method difference requires ~249 events per arm, which a 112-problem pilot cannot supply. Single training seeds invert method comparisons (F5): STaR holds $k{=}128$0 within 0.45–0.74 across three seeds while majority-vote SFT spans 0.55 to 1.53, crossing the destruction threshold on two of three. And a 10-prompt refusal probe yields a phantom 10-point safety drop that dissolves at 50 prompts (F6).

F7: a null measured once is a point estimate. This failure sits inside the correction for F2. The $k{=}128$1 null of 0/25 on a single frozen pair becomes 0.058 over 110 pairs; exhaustively re-estimating from subsets shows four replicates still admit estimates from 0.022 to 0.098, and only nine replicates hold within ±0.02 of the full value. Notably, an analytic binomial calculation returns 0.044 at $k{=}128$2 — optimistic because the base-unreached set is selected on a noisy zero — so an author who derived the null instead of measuring it would certify the false conclusion anyway.

The replacement test

Rather than thresholding, the authors pool every independent evaluation of the untrained model into a single baseline (eleven evaluations, 1,408 draws per problem) and apply a per-problem one-sided Fisher exact test under Benjamini–Hochberg FDR control at 0.05. Held out one baseline at a time and run identically, this test returns zero detections on all eleven replicates. Robustness checks show attained cutoffs of 0.0095 (distillation) versus 0.0094 (majority-vote SFT), so the adaptive procedure holds both arms to effectively the same standard, and results are unchanged under Bonferroni, joint procedures, and fixed thresholds. Exchangeability of the pooled baselines is verified empirically: Pearson dispersion 0.966 on 500 degrees of freedom, implying near-zero intraclass correlation despite the known batching nondeterminism.

What the controlled audit finds

On a matched ladder — one stream, one retained volume, one evaluation, only the correctness filter varying — the dissociation is sharp. On the 22 low-base AIME problems (pooled rate ≤ 5/1,408), distillation detects 8–11 per seed against 0–2 for every self-training arm, with the frozen null at zero throughout. A logistic model rejects the reading that this merely follows from the teacher's larger overall gain: the teacher carries a further $k{=}128$3 on low-base problems beyond its own lift ($k{=}128$4), where both self-training arms are indistinguishable from zero. Combining per-seed comparisons gives $k{=}128$5 against majority-vote SFT.

The paper is careful about scope. On the ten problems the base model never reaches at all, the split is 5 against 2 and not significant ($k{=}128$6); the claim is about rarely-reached problems, not about expansion into genuinely new territory. Across all arms, every problem newly solved at pass@1 was already reachable at pass@$k{=}128$7 — consistent with the sharpening account of Huang et al. and Yue et al., and a replication of concurrent work by Strozzi on a different domain, here with a positive control that prior work lacks.

Destruction is large, not threshold flicker. On the band, STaR corrupts 106 and majority-vote SFT 88 of 1,163 problems against a design-matched floor median of 8 — an 11–13× clearance once the floor is given as many checkpoints as the arms. More than half of each arm's corruption events exceed solve-rate changes of 0.234, the maximum any frozen model produced across twenty independent comparisons. Aggregate improvement and substantial destruction coexist in the same run: STaR gains +5.6 accuracy points while destroying 106 problems.

The policy-gradient arm adds a cautionary diagnostic. On identical data in identical order, all three seeds collapse into unbounded repetition — but at different rounds (one, two, or three updates), with round-one training loss spanning 169.9 to 8.5 across seeds. On seed 2, between the damaging update and the fatal one, the majority vote became more accurate and more unanimous (77.9% → 85.1%) while evaluation truncation went from 47% to 96%: a supervision signal reading healthy while the model it supervises dies.

Limitations and open questions

The paper states its bounds plainly. Three rounds and ~270 optimizer steps cannot speak to accumulation over longer schedules; what is established is that this regime does not expand, not that self-training cannot. Expansion rests on 22 low-base problems on one benchmark, and the never-reached subset yields no significant result. Corruption rests on a band drawn from MATH training problems almost certainly in the backbone's pretraining data, so some events may reflect perturbation of memorized answers rather than reasoning. One backbone family carries every substantive claim; a second backbone was characterized but deliberately left unrun because an affordable band would have been underpowered by the paper's own power analysis. The refusal probe supports only a bound (any change smaller than ~11 points), not a clearance. Whether curriculum-generating methods, which exist precisely to escape the easy-stream limitation, behave differently remains unanswered by this design.

Conclusion

This paper treats transition-level auditing as a measurement problem in its own right and supplies the controls it demands. Its strongest contributions are empirical: a frozen model scored at $k{=}128$8 under standard practice, an expansion statistic whose null is 0.280 at the threshold in use and non-zero at any threshold below five, and a demonstration that the replicates needed to bound a null already exist in any multi-arm study — provided there are at least nine of them, more than most studies possess. The substantive finding, that distillation reaches rarely-solved problems while self-training sharpens and destroys, is delivered with a positive control and an explicit statement of what the evidence does not cover. The reporting standard the paper proposes — a measured, interval-bounded null for every statistic, matched in design to the arm being audited — is inexpensive relative to the audits it protects, costing nothing for five of the seven failure modes it catches.