Sharpness of the lower bound on the convergence-rate exponent

Determine whether the lower bound on the exponent parameter \(\eta\) in Theorem \ref{tempobyq} is sharp, particularly through further investigation of the convergence rate when the step-size exponent \(\gamma\) is close to its lower admissible bound.

Background

The paper establishes convergence rates for stochastic gradient descent with step sizes $\gamma_n=\Theta(n^{-\gamma})$ and noise whose conditional variance has polynomial growth or decay. For the case $\beta\in(1/2,1]$, locally Lipschitz-continuous gradients, and iterates remaining in a compact set, Theorem \ref{tempobyq} proves that the objective-function error is $o(n^{-1+\eta})$ almost surely and in expectation for $\eta$ strictly above a specified lower bound.

The authors report that their numerical experiments raise doubts about whether this lower bound on $\eta$ is optimal. They specifically identify the regime in which the step-size exponent $\gamma$ is close to its lower admissible limit as requiring further investigation, because the simulations suggest that the noise may have negligible influence on the observed convergence rate.