Universality and theory of the 0.32 data-scaling exponent for optimal learning rate

Investigate and theoretically explain the observed power-law exponent of approximately 0.32 relating the optimal learning rate to the number of training tokens under MuonH Frobenius-sphere optimization, and ascertain whether this exponent is universal across optimizers and architectures through rigorous empirical verification.

Background

Empirical sweeps show that the optimal learning rate scales with training tokens as η* ∝ T^{-0.32} for models trained with MuonH under Frobenius-sphere constraints. This matches an exponent previously reported for AdamW, suggesting a possible universal behavior.

The authors highlight the need for both rigorous empirical validation of this universality and a theoretical account for why this exponent emerges across different optimizers.