Vanishing learning rate regime for stochastic FTRL

Determine the long-run behavior of stochastic follow-the-regularized-leader (S-FTRL) dynamics when run with a vanishing learning rate. In particular, ascertain whether the closedness–stochastic asymptotic stability equivalence persists, and whether the boundary-drift phenomenon in harmonic games fails, potentially yielding convergence to a constant Bregman-distance neighborhood of a strategic center.

Background

The paper develops results for S-FTRL under constant learning rates, including convergence and stability characterizations (e.g., equivalence with closed faces) and a disruption of recurrence in harmonic games, where trajectories drift on average toward the boundary.

The authors identify understanding the vanishing learning rate regime as a key open direction. They hypothesize that stability via closedness may remain valid, but the harmonic-game drift-to-boundary effect might break down, leading instead to convergence toward a fixed Bregman-distance level set around a central equilibrium.

References

Several important directions remain open in the general context of stochastic \ac{FTRL} dynamics. The first is what happens if the dynamics are run with a vanishing learning rate, as in . In this case, we conjecture that an analogue of \cref{thm:club} continues to hold, but \cref{thm:harmonic} fails: in games where the deterministic dynamics are recurrent (e.g., harmonic games), eq:FTRL-stoch will most likely result in the sequence of play converging to some constant (Bregman) distance from a ``central'' equilibrium of the game (the game's strategic center in the case of harmonic games).

— The impact of uncertainty on regularized learning in games (2506.13286 - Cauvin et al., 16 Jun 2025) in Section 6, Concluding remarks