Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

Abstract: An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost performance, but it requires several complete training runs and discards all but the winner. Repeated Optimizer Resampling (ROR) instead searches during one evolving run. Every $b$ epochs, each candidate optimizer scouts from the current model weights for $s$ epochs. The best scout continues for the remaining $b-s$ epochs, and that completed segment becomes the new incumbent if it improves the validation objective. This design allows the preferred optimizer to change as training progresses. We compare two variants of ROR on MNIST, Fashion-MNIST, and two motor insurance claim-count models. Nine fixed optimizers and both ROR variants are evaluated with the same ten seeds. One-epoch ROR uses 24\% to 35\% of the aggregate training needed to identify the best fixed optimizer exhaustively and remains close to that optimizer on all four tasks. These results support short scouting as a practical way to search over optimizers without completing every candidate run.

• The paper introduces Repeated Optimizer Resampling (ROR), which periodically tests nine optimizer and learning-rate pairs from the current model and continues with the best candidate in a single training path.
• One-epoch scouting used 24–35% of exhaustive-search training cost and generally matched the hindsight-best fixed optimizer, while preserving optimizer state offered no consistent advantage over cold starts.
• ROR’s value depends on task dynamics: repeated selection helps when the preferred optimizer changes during training, but one-shot selection can be cheaper and competitive on stable tasks.

Motivation and problem statement

Optimizer selection in deep learning is conventionally resolved before training: a candidate optimizer is fixed, and the choice is either heuristic or settled by an expensive hyperparameter search in which all but the winning run are discarded. Richman and Wüthrich treat the optimizer instead as a decision that can be revisited during a single evolving training run. Their method, Repeated Optimizer Resampling (ROR), opens a short tournament at regular intervals: every candidate optimizer "scouts" from the current incumbent weights for $s$ epochs, the best scout continues for the remaining $b-s$ epochs of the segment, and the completed segment becomes the new incumbent only if it improves the validation objective by more than a threshold $\delta$; otherwise training stops. The result is one model trajectory together with a data-dependent optimizer schedule, in contrast to Hyperband-style successive halving, which permanently eliminates losing configurations (2608.18810).

A key design question is optimizer state. An optimizer is more than its update rule: momentum buffers, adaptive moment estimates, iteration counters, and schedule-free variables carry information from earlier gradients. The authors therefore compare two state policies: State-Preserving ROR (SP-ROR), which restores the incumbent's accumulated state when the same optimizer wins again, and Cold-Start ROR (CS-ROR), which reinitializes every optimizer at every tournament. Preserving state for losing optimizers is deliberately rejected, since that state was learned along discarded weights and exact preservation would require maintaining nine parallel trajectories—recreating the exhaustive search ROR is designed to avoid.

Method and cost accounting

A tournament costs $b + (K-1)s$ epoch-equivalents for $K$ candidates; with $K=9$ and $b=3$, this is 11, 19, and 27 epoch-equivalents for $s = 1, 2, 3$. The exhaustive benchmark cost is the sum of all nine fixed-optimizer runs to their stopping points. ROR's stopping rule is strict: a rejected tournament ends training immediately, so both policies stop at their first rejection. The paper also introduces a one-shot baseline that queries the optimizer set once after a single scout epoch, retains the winning branch's weights, and continues with a freshly initialized instance of the winner.

Experimental design

Four tasks are evaluated with ten shared seeds: MNIST and Fashion-MNIST with a 300,938-parameter dense GELU MLP (cross-entropy loss), and the French motor third-party liability claim-count dataset (678,007 policies) with a 4,791-parameter embedding MLP and a 4,433-parameter tabular CLS Transformer (Poisson deviance, exposure as offset). The candidate set comprises nine optimizer/learning-rate pairs: Adam, AdamW, Lion, Muon (with hybrid AdamW branch for non-hidden kernels), SGD-Nesterov, Nadam, RMSprop, LAMB, and ScheduleFreeAdamW. Learning rates are fixed in advance and not tuned per task. All comparisons use paired seed-level differences with $t_9$ confidence intervals.

Results

Short scouting dominates the cost–performance trade-off. With $b=3$, one-epoch scouting ($b-s$0) uses only 24% to 35% of the aggregate training required to complete all nine fixed-optimizer runs and costs 51% to 68% less than $b-s$1 designs. Predictive differences across scout lengths are mostly not significant: fourteen of sixteen paired short-scout comparisons include zero, with the exceptions on MNIST, where $b-s$2 is favored (e.g., CS-ROR $b-s$3 minus $b-s$4: $b-s$5 percentage points, CI $b-s$6). The authors attribute this to validation losses flattening after the earliest gradient steps.

ROR matches the hindsight-best fixed optimizer without guaranteeing superiority. Across all four tasks, no paired comparison between any ROR variant and the best fixed optimizer excludes zero. On the Insurance Transformer, one-epoch CS-ROR and SP-ROR achieve the lowest observed deviances in the study (0.237627 and 0.237646 versus fixed Lion's 0.237817) at roughly 29% of the full-search cost (42.9 and 40.7 versus 140.8 epoch-equivalents). On the MLP, two-epoch SP-ROR attains the lowest observed deviance (0.238362) but only at 199.5 epoch-equivalents against 442.4 for the full search. On MNIST, fixed Muon (98.095%) remains slightly ahead of one-epoch ROR (97.948%–97.988%), while $b-s$7 ROR edges it (98.132%) at nearly the full-search cost.

The one-shot baseline is a strong cheap competitor. One-shot selection is cheaper than one-epoch ROR everywhere and beats it on MNIST (98.116% at 20.3 epoch-equivalents), but trails ROR's observed means on Fashion-MNIST and both insurance tasks at roughly half ROR's cost. The practical implication the authors draw is that repeated tournaments earn their extra cost only where the preferred optimizer changes during training.

Schedules reveal task-dependent switching behavior. Image tasks are stable: MNIST averages 2.6 accepted tournaments and 0.5 switches, with all ten SP-ROR paths starting in ScheduleFreeAdamW. The Insurance MLP is the outlier, averaging 9.0 accepted tournaments and 5.6 switches, with final optimizers spread across five families. This directly supports the paper's premise that the early-favored optimizer need not remain favored, while also showing ROR does not switch mechanically.

A contradictory finding on Muon. Fixed Muon ranks 15 of 16 on the insurance MLP, in direct contrast to the tabular benchmark of Gorishniy et al. (Gorishniy et al., 16 Apr 2026) where Muon consistently outperformed AdamW. The authors attribute this to the fixed untuned learning rate, the very small model size, the embedding-heavy parameterization routing most parameters to Muon's AdamW branch, and the Poisson deviance objective.

Caveats and open questions

The study covers four tasks, ten seeds, and small networks, with learning rates fixed in advance; a different tuning could change both the fixed-optimizer ranking and the ROR schedule. The same validation set is queried at every tournament, so repeated selection may overfit that sample—a larger search would require a reusable holdout or nested validation. Epoch-equivalents treat all epochs as equally expensive, ignoring per-update cost differences such as Muon's matrix operations. The stopping rules are asymmetric: fixed baselines receive five epochs of patience while ROR stops at its first rejected tournament, which works against ROR's predictive results but flatters its cost accounting. Finally, no multiple-comparison adjustment is applied, and the paper explicitly leaves open whether ROR's optimizer schedules can reliably outperform a well-tuned fixed optimizer on larger problems.

Conclusion

The paper demonstrates that optimizer selection can be folded into a single training run at a fraction of exhaustive-search cost. One-epoch scouting captures most of ROR's benefit, using 24% to 35% of the training needed to identify the best fixed optimizer while finishing statistically indistinguishable from it on all four tasks. Preserving incumbent optimizer state offers no consistent advantage over cold starts. The residual case for repeated rather than one-shot selection rests on tasks whose preferred optimizer changes mid-training, at roughly twice the one-shot cost.