Anytime-valid envelopes for adaptive step sizes and Markovian noise

Extend the anytime-valid envelope recipe for stochastic gradient descent to adaptive methods such as AdaGrad and Adam whose step sizes are current-gradient-dependent and therefore F_t-adapted, and to iterative algorithms driven by Markovian noise.

Background

The paper's framework handles predictable step sizes, meaning that the step size at iteration t is measurable with respect to the past. Many practical adaptive optimization methods, including AdaGrad and Adam, instead choose the step size using the current stochastic gradient. This destroys the martingale-difference structure of the weighted noise sum on which the paper's anytime-valid envelope construction relies. Extending the framework to this setting would require new probabilistic control for the resulting non-martingale noise terms.

The paper also identifies Markovian noise as an unresolved direction. Its main constructions are developed for noise processes satisfying strongly adapted sub-psi conditions, whereas Markovian dependence introduces temporal dependence that is not directly covered by the presented supermartingale arguments. Addressing both extensions would broaden the method to important adaptive and dependent stochastic algorithms.