Determine whether optimizer rankings exhibit systematic scaling behavior

Determine whether the relative rankings of constant-learning-rate EMA, learning-rate decay, and Schedule-Free optimization strategies follow a systematic pattern as model size, dataset size, and optimization budget vary, while independently retuning each strategy.

Background

The paper compares constant learning rate with exponential moving average, learning-rate decay with EMA, and Schedule-Free optimization in a 675-million-parameter SiT-XL/2 diffusion-transformer setting trained on ImageNet-1K. Although Schedule-Free performs favorably in the main experiments, the authors contrast these findings with a separate controlled benchmark in which its ranking differs across tasks and training regimes.

The paper notes that the available studies differ simultaneously in model and dataset scale, diffusion formulation, architecture, training objective, learning-rate protocol, training budget, and data domain. Consequently, the observed reversal in optimizer rankings cannot be attributed to any single factor. The unresolved problem is to conduct a controlled study that varies model and data scale while independently retuning each optimization strategy, thereby determining whether optimizer rankings follow a systematic scaling law or pattern.