Characterize equal-compute speedups for under-parameterized networks with batch size exceeding parameter count

Determine whether increasing width can yield approximately equal-compute speedups for under-parameterized networks in which the batch-size ratio exceeds the parameter-count ratio, namely \alpha>\nu, a regime not covered by the analysis of normalized SGD.

Background

The paper analyzes normalized SGD in a power-law random-feature model and studies whether increasing batch size or model width can reduce the number of optimization steps enough to offset the corresponding increase in computational cost. In the deterministic edge-of-stability phase, the authors derive bounds showing that width provides at most square-root effective-learning-rate gains under conditions including \nu\geq1 or, for \nu<1, \nu>\alpha (equivalently, the total parameter count exceeds the batch size).

The complementary regime of an under-parameterized network with \alpha>\nu is not analyzed. The authors explicitly state that they do not rule out speedups there, leaving unresolved whether increasing width can produce approximately equal-compute speedups in this setting. This is a concrete limitation of the phase-3 speedup characterization rather than a general suggestion for future work.

References

We do not treat here the case of an under-parameterized network where the batch size $\alpha$ is larger than the total parameter count $\nu$, and we do not exclude speedups in this case; instead we note that this case is far removed from modern practice.

— A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling  (2610.06701 - Lavie et al., 5 Oct 2026) in Appendix, Section “Absence of approximately equal-compute speedup,” subsection “Batch size and width in Phase 3” (labelled Section \ref{subapp:no_speedup})