Shrinking-tube guarantees for multi-timescale stochastic approximation

Extend shrinking-tube guarantees to multi-timescale stochastic approximation with coupled recursions operating at different step-size scales, including actor–critic reinforcement learning and stochastic bilevel optimization, by quantifying how tracking errors in the faster recursion affect the shrinking tolerances for the slower recursion.

Background

The paper establishes shrinking-tube concentration for a single projected stochastic-approximation recursion driven by adaptive Markovian data, including extensions with martingale-difference noise and predictable bias. The conclusion identifies multi-timescale stochastic approximation as an unresolved extension in which several coupled recursions evolve simultaneously at different rates.

The unresolved technical issue is to quantify the influence of the faster recursion's tracking error on the admissible shrinking-tube tolerances and trajectory guarantees for the slower recursion. The authors specifically identify actor–critic reinforcement learning and stochastic bilevel optimization as motivating settings.