Two-Timescale Stochastic Approximation

• TTSA is a stochastic optimization method that updates two interdependent sequences on distinct timescales to tackle hierarchical learning and bilevel optimization problems.
• It leverages differential inclusions and set-valued maps to model dynamics under both martingale and Markovian noise, offering finite-time and asymptotic guarantees.
• The framework underpins practical algorithms in reinforcement learning and distributed control, providing insights into sample complexity, convergence rates and operator averaging.

Two-Timescale Stochastic Approximation (TTSA) is a foundational paradigm in stochastic algorithms for optimization, control, and machine learning, enabling the analysis and design of schemes in which two coupled sequences of iterates evolve under distinct step-size (learning rate) schedules and may interact through general, potentially non-smooth or set-valued operators. This dual-rate structure is crucial in a variety of contexts, especially in hierarchical learning architectures (e.g., actor-critic RL), bilevel optimization, primal-dual and minimax problems, and distributed learning under Markovian noise. TTSA algorithms are characterized by distinct fast and slow timescales: typically, the "fast" variable tracks a time-varying equilibrium dictated by the "slower" variable, which itself adapts more conservatively. Recent advances provide precise finite-time, asymptotic, and distributional performance guarantees under broad noise models—including controlled Markov noise and set-valued mean fields—and under contractive, nonexpansive, or even arbitrary-norm contraction mappings. These theoretical results underpin the sample complexity analyses and statistical efficiency of a range of practical algorithms.

1. Fundamental Theory and Differential Inclusion Approach

The canonical TTSA algorithm updates two interdependent sequences $(x_n)$ and $(y_n)$ (or $(\theta_n, w_n)$) via

$$x_{n+1} = x_n + a(n)\big[ u_n + M_{n+1}^{(1)} \big], \qquad u_n \in h(x_n, y_n),$$

$$y_{n+1} = y_n + b(n)\big[ v_n + M_{n+1}^{(2)} \big], \qquad v_n \in g(x_n, y_n).$$

Key conditions are

• $\sum_n a(n) = \infty$, $\sum_n a(n)^2 < \infty$, $\sum_n b(n) = \infty$, $\sum_n b(n)^2 < \infty$.
• Asymptotic separation: $\lim_{n\to\infty} b(n)/a(n) = 0$; the $y$-updates are genuinely slower than the $x$-updates.

By modeling $h$ and $g$ as Marchaud maps (set-valued, upper semi-continuous, convex-compact values, and pointwise bounded), the TTSA recursions are interpreted as noisy discretizations of coupled differential inclusions: $\dot{x} \in h(x, y), \quad \dot{y} \in g(x, y).$ On fast timescales, $y$ appears nearly static, so $x$ tracks the attractor set $A_y$ of $\dot{x} \in h(x, y)$. The slow timescale evolves under a convexified mean field: $$G(y) = \overline{\text{co}} \left( \bigcup_{x \in A_y} g(x, y) \right).$$ Under global attractor and Lyapunov stability assumptions for both scales, the interpolated process clusters to

$\{(x, y): y \in A_0, x \in \lambda(y) \},$

where $A_0$ is the slow attractor and $\lambda(y) = A_y$ is upper semi-continuous.

2. Extensions to Markovian and Set-Valued Noise

Classic TTSA analyses assume martingale difference noise. More general results allow for:

• Controlled Markov noise: When the driving noise is a controlled Markov process whose kernel depends on the parameters, occupation measures and solutions to Poisson equations are used to express the effect of noise and bias (Karmakar et al., 2015, Chandak et al., 24 Mar 2025).
• Set-valued mean fields: Differential inclusion theory enables handling of scenarios with multi-valued equilibria for the fast system, as in stochastic Lagrangian dual problems (Ramaswamy et al., 2015).

For the general Markov noise setup, the averaged mean field for a given parameter is integrated against ergodic occupation measures, yielding

$$\dot{x}(t) \in \hat{g}(y, x(t)), \quad \hat{g}(y, x) = \{ \tilde{g}(y, x, \nu): \nu \in D^{(2)}(y, x) \}$$

and, once equilibrated, the slow variable $y$ evolves by

$$\dot{y}(t) \in \hat{h}(y(t)), \quad \hat{h}(y) = \{\tilde{h}(y, \lambda(y), \nu): \nu \in D^{(1)}(y, \lambda(y))\}.$$

Thus, convergence is controlled by the geometry of invariant measures and the structure of noise-induced bias terms.

3. Finite-Time Analyses: Concentration, Rates, and Sample Complexity

Early analyses for TTSA focused on asymptotics; recent works have established explicit high probability and mean-square error bounds for both linear and nonlinear cases. Key developments include:

• Lock-in probability and sparse projection (sparse exponential grid projection) (Dalal et al., 2017, Dalal et al., 2019): For $a(n) = (n+1)^{-\alpha}$, $b(n) = (n+1)^{-\beta}$, $1 > \alpha > \beta > 0$, with infrequent projection, the error satisfies

$$\max\{ \|\theta_n' - \theta^*\|, \|z_n'\| \} \leq C \max\{ n^{-\beta/2}\sqrt{\ln(n/\delta)}, n^{-(\alpha-\beta)} \}$$

with high probability, yielding tight convergence rate bounds. No square summability on step sizes is required.

• Singular perturbation Lyapunov function and precise drift analysis yield

$$\mathbb{E}\| \Theta_k \|^2 \leq K_1 (1 - c\mu^\lambda)^k + \frac{K_2 \mu^{2-\lambda}}{\gamma_{\max} c}$$

where $\mu = \epsilon^\beta$, $\lambda = \alpha/\beta$; explicitly separating transient and steady-state error (Gupta et al., 2019).

• Concentration bounds via Alekseev’s formula and martingale inequalities enable explicit finite-time error tolerance guarantees (Borkar et al., 2018). For appropriate constants $C_1, C_2$,

$$P(|c_n - z_n| < \varepsilon V_{n_0+T+1}) \geq 1 - C_1 e^{-C_2 \varepsilon^2 V_{n_0}}$$

The best known mean-squared error rates under minimal conditions for nonlinear TTSA are $O(1/k^{2/3})$ (Doan, 2020, Doan, 2021, Chandak et al., 24 Mar 2025). Notably, the introduction of Ruppert-Polyak operator averaging allows improvement to the optimal $O(1/k)$ rate under strong monotonicity and standard Lipschitz assumptions (Doan, 23 Jan 2024). In cases where the slow timescale is noiseless a $O(1/n)$ rate can also be achieved (Chandak et al., 24 Mar 2025).

4. Functional Limit Theorems and Central Limit Structure

Functional central limit theorems (FCLT) and explicit Gaussian approximation bounds have been established for TTSA:

• Fluctuation theory (Faizal et al., 2023, Butyrin et al., 11 Aug 2025): Interpolated and appropriately scaled trajectories of the fast variable converge to a Gauss-Markov process (linear SDE), and the slow variable’s fluctuations are characterized by an ODE with deterministic drift but no direct diffusion. For example,

$$du^*(t) = [\nabla_x h(\lambda(y^*(t)), y^*(t)) + (\varphi/2)I]u^*(t)dt + G(\lambda(y^*(t)), y^*(t))dB(t)$$

The slow component evolves as

$$w^*(t) = \int_0^t [\nabla f(y^*(s))w^*(s) + \nabla_x g(\lambda(y^*(s)), y^*(s))u^*(s)]ds$$

• Asymptotic normality under Markovian noise: For general nonlinear systems, TTSA iterates $(x_n, y_n)$ satisfy

$$\begin{pmatrix} \beta_n^{-1/2}(x_n-x^*) \ \gamma_n^{-1/2}(y_n-y^*) \end{pmatrix} \xrightarrow{d} N\left(0, \begin{pmatrix} U_x & 0 \ 0 & U_y \end{pmatrix}\right)$$

with limiting covariance matrices given by integral representations involving the linearizations and noise covariance (Hu et al., 17 Jan 2024, Butyrin et al., 11 Aug 2025).

• Non-asymptotic Berry-Esseen-type bounds (Butyrin et al., 11 Aug 2025): For Polyak–Ruppert averaged and last-iterate estimators, convex distance (distributional error) decays as $n^{-1/4}$ (martingale noise) and $n^{-1/6}$ (Markov), under technical assumptions and with detailed dependence on timescale separation.

5. Practical Applications in Reinforcement Learning, Optimization, and Control

TTSA underpins a wide spectrum of practical algorithms:

• Actor–critic and policy evaluation algorithms: The structure of TTSA directly models decoupled parameter updates in gradient temporal difference (GTD, GTD2, TDC) learning (Karmakar et al., 2015, Dalal et al., 2017, Dalal et al., 2019, Haque et al., 2023). Explicit sample complexity rates are established, e.g., for TDC
  • $\mathbb{E}\| \theta_n - \theta^* \|^2 = O(1/n)$ (with tightly matched CLT covariance)
  • Asymptotic equivalence in statistical efficiency of GTD2 and TDC under Markovian sampling is shown (Hu et al., 17 Jan 2024)
• Bilevel and Lagrangian dual optimization: TTSA supports rigorous analysis of bilevel problems and stochastic Lagrangian dual ascent, including cases with set-valued mean fields and non-singleton attractors (Ramaswamy et al., 2015, Hong et al., 2020, Sharrock, 2022). In bilevel optimization, the fast variable approximates a solution to the inner (strongly convex) problem, and the slow variable performs projected optimization over the outer function, with rates such as $O(K^{-2/3})$ (strongly convex), $O(K^{-2/5})$ (weakly convex), and $O(K^{-1/4})$ (convex objective gap).
• Distributed and networked learning: In distributed TTSA, agents maintain local state and update over networks characterized by distinct mixing rates, with explicit finite-time MSE bounds depending on network topology and two communication graphs (Doan et al., 2019).

6. Advanced Topics: Arbitrary Norms, Non-Expansive Mappings, Constant Stepsizes

Recent advances address TTSA under nuanced operator settings and nonstandard update policies:

• Arbitrary norm contraction and generalized Moreau envelopes: When mappings are contractive only under an arbitrary norm (e.g., max-norm in RL), generalized Moreau envelopes provide smooth surrogate Lyapunov functions, yielding finite-time bounds $O(1/n^{2/3})$ (general) and $O(1/n)$ (noiseless slow timescale), and $O(1/n)$ for Q-learning with Polyak averaging (Chandak et al., 24 Mar 2025).
• Non-expansive mappings and inexact Krasnosel’skii–Mann iterations: In settings where the slow timescale operator is merely non-expansive, the convergence is slower, decaying as $O(1/k^{1/4-\epsilon})$, with almost sure convergence to fixed-point sets (Chandak, 18 Jan 2025).
• Constant stepsizes and stationary distribution: For constant stepsize TTSA, the joint process converges in $\mathcal{W}_2$-Wasserstein metric to a unique stationary distribution, with geometric rates and explicit decomposition of bias ($O(\alpha) + O(\beta)$) and variance ($O(\alpha)$ for slow, $O(\beta)$ for fast), without requirements such as $\beta^2 \ll \alpha$. Tail-averaging and Richardson–Romberg extrapolation further reduce MSE to $O(\beta^4 + 1/t)$ (Kwon et al., 16 Oct 2024).

7. Methodological Innovations and Future Directions

Innovations in the mathematical analysis include:

• Operator Ruppert–Polyak averaging: Application of pre-averaging to operator samples decouples noise and operator coupling, yielding $O(1/k)$ optimal convergence for nonlinear TTSA under standard regularity (Doan, 23 Jan 2024).
• Singular perturbation techniques and Lyapunov function construction: Application of joint Lyapunov functions modeled after singular perturbation theory enables clean separation of error dynamics for multi-timescale systems (Gupta et al., 2019).
• Poisson equation and occupation measure methods: Poisson equations are essential for de-biasing Markovian noise, ensuring performance bounds match CLT-statistics (Hu et al., 17 Jan 2024, Chandak et al., 24 Mar 2025).

Further research directions include tighter characterization of rates in Markovian and non-expansive cases, improved finite-time concentration beyond mean-square error, and extension to adaptive and fully online, non-Euclidean, or high-dimensional settings.

---

In summary, TTSA constitutes a comprehensive theoretical and algorithmic toolkit for addressing stochastic algorithms with interacting timescales, providing a precise understanding of convergence, fluctuation, and sample efficiency under minimal regularity. The detailed theory spans from general set-valued and nonlinear inclusions to optimal rates under constant stepsize, arbitrary norm contraction, and general noise models, and it informs a wide array of contemporary methods in reinforcement learning, distributed control, and optimization.