Reliability of ROR optimizer schedules against fixed optimizers

Determine whether optimizer changes selected by Repeated Optimizer Resampling (ROR) can reliably outperform a well-tuned fixed optimizer across a larger and more diverse set of training tasks.

Background

The paper evaluates ROR as a method for selecting among nine fixed optimizer and learning-rate pairs during a single evolving training trajectory. Although ROR sometimes achieves lower observed losses than the best fixed optimizer, none of the paired comparisons across the four experiments excludes zero, so the experiments do not establish reliable superiority. The authors specifically identify the need for a larger study to assess whether optimizer changes during training can generalize beyond the tested image-classification and insurance-pricing tasks and consistently outperform a well-tuned fixed choice.