Quantify the sources of apparent progress under the KEEP ratchet

Determine how much of the observed progress or drift in AutoLR reflects true model improvement, changing candidate quality, diminishing returns, or accumulated selection noise when accepted offline candidates rewrite the active experimental trunk.

Background

AutoLR promotes some offline candidates to the active trunk through a KEEP decision. Because each KEEP changes the baseline against which subsequent candidates are evaluated, a non-reproducible acceptance can affect the entire subsequent experimental trajectory rather than merely adding one noisy observation.

The paper identifies this stateful amplification mechanism as the KEEP ratchet. Although the archive establishes that the baseline is mutable, the available evidence does not disentangle genuine model improvements from changes in candidate quality, diminishing returns, or accumulated selection noise. Resolving these contributions is necessary for assessing whether apparent longitudinal progress reflects real improvement or selection-induced drift.

References

Statistically reliable accumulation across generations under comparable resources remains open.

— The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement  (2609.11873 - Duan et al., 10 Sep 2026) in Section 5, subsection “Evaluating L5 Recursive Improvement”

Because KEEP rewrites the active trunk (Equation~eq:trunk-update), a non-reproducible acceptance does more than add one noisy result: it changes the reference used by subsequent experiments. The archive establishes this mutable-baseline mechanism, but it cannot identify how much observed progress or drift reflects true model improvement, changing candidate quality, diminishing returns, or accumulated selection noise.

— AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems  (2609.04871 - Zhang et al., 4 Sep 2026) in Section 5.3, subsection “RQ3: Reliability of Offline Promotion,” paragraph “Calibration gap and mutable-baseline risk”