Convergence guarantees for inner-iteration stepsize updates in OSGM-SVRG

Establish convergence guarantees for the OSGM-SVRG framework when the stepsize is updated at each stochastic-gradient inner iteration rather than only at the outer-loop epoch level.

Background

The paper introduces Stochastic Online Scaled Gradient Methods (SOSGM), including an OSGM-SVRG framework that adapts the preconditioner at outer iterations using the deterministic full gradient at each snapshot point. This design permits the convergence analysis to avoid stochastic feedback in the stepsize update.

A practical variant instead updates the stepsize and momentum during every inner stochastic iteration. The paper notes that such inner-loop adaptation can be empirically effective, but its convergence is not established; the conclusion explicitly identifies guarantees for this setting as an unresolved question.