Whether a closed-loop training budget resolves late retrieval onset

Determine whether extending training with a closed-loop budget that continues until retrieval onset and then runs for a fixed post-onset duration enables the $8$B gated delta-rule student, the $8$B RWKV-7 student, and the randomized-pool model to learn retrieval, rather than treating their failures under fixed step budgets as method failures.

Background

Retrieval learning begins abruptly at seed-dependent onset steps. At $8$B, two gated delta-rule seeds reached onset at 6.5k and 11k steps, while a third had not reached onset by the 16k-step budget. The RWKV-7 arm and the pool-annealing arm likewise failed to show onset within their allotted budgets.

The paper argues that these negative results may reflect truncation of late learning rather than failure of the retrieval method. It proposes a closed-loop budget that trains until the avalanche and then continues for a fixed number of additional steps, but this intervention had not yet been run.