Develop finite-time theory for time-varying contractive Markovian stochastic approximation

Develop a finite-time theory for time-varying contractive Markovian stochastic approximation that controls the variation of the generalized Moreau-envelope Lyapunov functions associated with changing contraction factors, thereby obtaining local residual-preconditioning factors for general decreasing residual-preconditioning schedules.

Background

For general decreasing residual-preconditioning schedules, the paper uses a conservative common analysis scale based on the largest residual-preconditioning parameter. This yields a bound involving the worst-case factor rather than the local factor associated with the current parameter. The authors explain that a sharper result would require controlling a time-varying Lyapunov function and explicitly leave this extension unresolved.

References

Obtaining a bound with the local factor $L_{\beta_k}$ at each iteration would require a finite-time theory for time-varying contractive Markovian SA, or equivalently a way to control the variation of the associated Moreau envelopes. We leave such an extension for future work.

— V-SHARE: Provable Convergence Guarantees for Advantage-based Algorithms  (2610.21448 - Saha et al., 15 Oct 2026) in Appendix, Section Decreasing Residual-Preconditioning Schedules, subsection General decreasing schedules, Remark [Conservatism of the general-schedule bound]