Nonlinear Two-Time-Scale Stochastic Approximation

• Nonlinear two-time-scale stochastic approximation is a method that employs two interlocked stochastic recursions with distinct step sizes to solve coupled nonlinear equations based on noisy observations.
• It achieves faster convergence of one variable while the other evolves slowly, enabling almost-sure convergence proofs and finite-time mean-square error rates.
• The approach applies to reinforcement learning, optimization, and distributed control, utilizing techniques such as Lyapunov functions, martingale methods, and ODE analysis.

Nonlinear two-time-scale stochastic approximation is a class of stochastic recursive methods for solving coupled nonlinear equations or fixed-point problems when only noisy observations of the underlying operators are available. In its standard form, two interlocked iterates evolve with distinct step-size sequences, so that one component tracks a rapidly changing quasi-equilibrium while the other evolves on a slower effective dynamics. The framework appears prominently in stochastic control, optimization, machine learning, and, most notably, certain Actor–Critic schemes for Reinforcement Learning (Vidyasagar, 15 Mar 2026). Modern theory treats the subject through several complementary lenses: singular perturbation and ODE methods, Lyapunov and martingale techniques, finite-time mean-square analysis, controlled Markov noise, concentration bounds, and asymptotic distribution theory (Doan, 2020).

1. Canonical formulation and time-scale separation

A common nonlinear root-finding formulation seeks a pair $(\theta^*,\phi^*)$ satisfying

$f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$

with updates driven by noisy measurements of $f$ and $g$: $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$ In the formulation analyzed by "Convergence of Two Time-Scale Stochastic Approximation: A Martingale Approach" (Vidyasagar, 15 Mar 2026), the step sizes satisfy the two-time-scale Robbins–Monro conditions

$$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$

Equivalent formulations appear as coupled stochastic fixed-point recursions,

$$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$

or as gradient-type recursions

$$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$

depending on whether the problem is posed as a fixed-point equation, a root-finding problem, or a stochastic optimization scheme (Chandak, 27 Apr 2025, Doan, 2020).

The central structural feature is timescale separation. On the faster scale, one iterate approximately tracks the equilibrium defined by the other variable held quasi-static; on the slower scale, the second iterate evolves according to a reduced dynamics in which the fast variable has already relaxed. In the distributed linear setting, this is described explicitly as the fast iterate tracking the solution of $A_{11}x+A_{12}y=b_1$ while the slow iterate sees an almost steady fast component (Doan et al., 2019). In nonlinear analyses, the same idea is encoded through equilibrium maps such as $H(y)$, $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$0, or $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$1, which solve the fast subsystem for frozen slow coordinates (Doan, 2020, Chandak, 27 Apr 2025).

The notation is not uniform across the literature. Some papers label the larger step size as the fast scale, while others attach the fast/slow terminology to the associated subsystem rather than to the symbol itself. This suggests that the mathematically essential object is the asymptotic ratio of the two step-size sequences, not the specific naming convention.

2. Stability assumptions and analytical machinery

Most nonlinear TTSA results impose a structural decomposition through a fast-equilibrium map. In the strong-monotonicity formulation of "Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performance" (Doan, 2020), one assumes a mapping $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$2 such that $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$3 uniquely, together with Lipschitz continuity of $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$4 and $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$5, strong monotonicity of $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$6, and one-point strong monotonicity of $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$7 around the slow equilibrium. In the contractive formulation of "O(1/k) Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximation" (Chandak, 27 Apr 2025), the fast operator is a Banach contraction in $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$8, which yields a unique fixed point $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$9, while the reduced map $f$0 is contractive in $f$1.

A more global singular-perturbation viewpoint is developed in (Vidyasagar, 15 Mar 2026). There the limiting ODE system

$f$2

is assumed globally exponentially stable for all small $f$3. The assumptions include global Lipschitzness of $f$4 and $f$5, existence of a globally Lipschitz and continuously differentiable map $f$6 with $f$7, and separate slow and fast Lyapunov functions $f$8 and $f$9. These are combined into

$g$0

which permits a unified drift estimate for the coupled recursion (Vidyasagar, 15 Mar 2026).

The literature also extends beyond Euclidean quadratic Lyapunov arguments. For arbitrary-norm contractions, "Finite-Time Bounds for Two-Time-Scale Stochastic Approximation with Arbitrary Norm Contractions and Markovian Noise" constructs smooth surrogates via the generalized Moreau envelope, replacing nondifferentiable norm squares by smooth equivalent norms suitable for recursive descent analysis (Chandak et al., 24 Mar 2025). For non-expansive reduced dynamics, "Non-Expansive Mappings in Two-Time-Scale Stochastic Approximation: Finite-Time Analysis" interprets the slow recursion as a stochastic inexact Krasnoselskii-Mann iteration, with the reduced map $g$1 merely non-expansive rather than contractive (Chandak, 18 Jan 2025).

At the broadest level, "Stochastic Approximation with Two Time Scales: The General Case" treats settings in which iterates on either or both time scales do not necessarily converge. There the fast subsystem is described through invariant probability measures of an averaged ODE, and the slow limit is expressed as a differential inclusion

$g$2

whose asymptotic behavior is characterized through internally chain-transitive invariant sets rather than a single equilibrium point (Borkar, 2024).

3. Almost-sure convergence, boundedness, and asymptotic regimes

The classical nonlinear convergence theorem in (Doan, 2020) introduces residuals

$g$3

and proves that $g$4 is an almost supermartingale. Under step sizes satisfying

$g$5

the iterates converge almost surely to $g$6 (Doan, 2020).

The martingale-Lyapunov treatment in (Vidyasagar, 15 Mar 2026) strengthens this line of analysis in several directions. The noisy measurements are decomposed into conditional-mean bias terms and martingale-difference noises: $g$7 with an analogous decomposition for the second recursion. This permits nonzero conditional mean and conditional variances that may grow with time. Under summability conditions on step sizes, bias envelopes, and conditional variance envelopes, the paper proves that $g$8 is almost surely bounded and convergent; moreover $g$9 and $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$0 almost surely, hence $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$1 and $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$2 almost surely (Vidyasagar, 15 Mar 2026). A notable feature is that boundedness is obtained as an immediate by-product of the supermartingale argument rather than as an external assumption.

This almost-sure program coexists with more general asymptotic regimes. In the set-valued framework of (Borkar, 2024), the fast subsystem can admit multiple invariant measures, so the slow dynamics is no longer a single reduced ODE but a convexified differential inclusion. In that regime, the slow iterate need not converge to a point; instead, every limit point of the shifted paths is contained in an internally chain-transitive invariant set. That result formalizes the possibility of convergence to invariant sets, oscillatory attractors, or multi-stable slow dynamics (Borkar, 2024).

Controlled Markov noise leads to a related inclusion-based description. In the two-time-scale framework with controlled Markov noise, invariant occupation measures define set-valued averaged drifts, the fast inclusion has equilibrium $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$3, and the slow iterate converges almost surely to an internally chain-transitive invariant set of the reduced inclusion $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$4 under boundedness or asymptotic tightness assumptions (Karmakar, 2020). The earlier off-policy temporal-difference formulation reaches an analogous conclusion under compactness, continuity, Lipschitzness, martingale $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$5 bounds, and almost-sure boundedness (Karmakar et al., 2015).

4. Finite-time mean-square rates and decoupling phenomena

Finite-time analysis of nonlinear TTSA initially centered on a $$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$6 regime. Under Lipschitz and strong-monotonicity assumptions, (Doan, 2020) chooses

$$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$7

and proves a Lyapunov recursion of the form

$$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$8

which yields

$$\theta_{t+1}=\theta_t+\alpha_t y_{t+1},\qquad \phi_{t+1}=\phi_t+\beta_t z_{t+1}.$$9

In particular, the slow mean-square error decays as $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$0 (Doan, 2020). The Markovian counterpart introduces geometric mixing and obtains $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$1, so the i.i.d. nonlinear rate survives up to a logarithmic factor under dependent data (Doan, 2021). In distributed TTSA over two communication graphs, the dominant rate remains $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$2, with the spectral gap penalty $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$3 quantifying the price of consensus (Doan et al., 2019).

Several later works sharpen this picture. "Fast Nonlinear Two-Time-Scale Stochastic Approximation: Achieving $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$4 Finite-Sample Complexity" modifies the algorithm by inserting Ruppert–Polyak averaging into the operator evaluations and proves

$$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$5

under Lipschitz and strong-monotonicity assumptions (Doan, 2024). "O(1/k) Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximation" obtains the same $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$6 order for the original, unmodified nonlinear TTSA in the contractive setting. Its key device is an averaged slow-scale noise process $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$7, a denoised iterate $$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$8, and a combined recursion

$$0<\beta_t\ll \alpha_t\to0,\qquad \sum_t \alpha_t=\sum_t\beta_t=\infty,\qquad \sum_t \alpha_t^2<\infty,\qquad \sum_t \beta_t^2<\infty.$$9

which leads to

$$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$0

in the $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$1 regime (Chandak, 27 Apr 2025).

The martingale approach in (Vidyasagar, 15 Mar 2026) proves a rate of a different form. In the zero-bias, bounded-variance case, with $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$2 and $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$3 for any small $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$4, it shows

$$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$5

for every $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$6. The same paper states that this improves upon the $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$7 bound proved in Doan (2023), and is virtually the same as the $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$8 rate proved in Doan (2024) for a Polyak-Ruppert averaged version of TTSSA, but is achieved directly on the raw iterates (Vidyasagar, 15 Mar 2026).

A central issue is whether the two mean-square errors can decouple, each scaling solely with its own step size. "Finite-Time Decoupled Convergence in Nonlinear Two-Time-Scale Stochastic Approximation" shows that such decoupling is available under a nested local linearity assumption and fourth-moment control. With $$x_{k+1}=x_k+\alpha_k\bigl(f(x_k,y_k)-x_k+M_{k+1}\bigr),\qquad y_{k+1}=y_k+\beta_k\bigl(g(x_k,y_k)-y_k+N_{k+1}\bigr),$$9 and $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$0, it proves

$$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$1

when $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$2 and $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$3 (Han et al., 2024). The same work also provides a numerical example indicating that global Lipschitzness alone does not generally suffice for decoupled finite-time rates.

That limitation is sharpened in "Nonlinear Two-Time-Scale Stochastic Approximation: A Sharp Phase Transition and How to Beat It" (Sarkar et al., 12 Jun 2026). In a normal form with fast error $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$4, slow error $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$5, and slow drift remainder $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$6 of order $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$7, the paper proves

$$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$8

and a matching scalar Gaussian lower bound shows that the slower term is unavoidable for the naive recursion. Thus the decoupled $$x_{k+1}=x_k-\alpha_k\bigl(F(x_k,y_k)+\xi_k\bigr),\qquad y_{k+1}=y_k-\beta_k\bigl(G(x_k,y_k)+\psi_k\bigr),$$9 slow rate is guaranteed exactly when $A_{11}x+A_{12}y=b_1$0. The paper then introduces an auxiliary online bias estimator on an intermediate timescale and proves that the corrected recursion achieves $A_{11}x+A_{12}y=b_1$1 for every $A_{11}x+A_{12}y=b_1$2 (Sarkar et al., 12 Jun 2026). A plausible implication is that the obstacle is not information-theoretic, but algorithmic: it is tied to the untreated predictable bias in the nonlinear coupling term.

5. Noise models, dependence structures, and statistical refinements

A major development in recent TTSA theory is the relaxation of the noise model beyond martingale differences with uniformly bounded conditional variance. The martingale framework of (Vidyasagar, 15 Mar 2026) allows measurement errors with nonzero conditional mean and conditional variances that grow without bound, modeled through deterministic envelopes $A_{11}x+A_{12}y=b_1$3 for the biases and $A_{11}x+A_{12}y=b_1$4 for the conditional second moments. In the general case, the paper proves

$A_{11}x+A_{12}y=b_1$5

for every $A_{11}x+A_{12}y=b_1$6, where $A_{11}x+A_{12}y=b_1$7 describe bias decay and $A_{11}x+A_{12}y=b_1$8 describe variance growth (Vidyasagar, 15 Mar 2026).

Dependent data introduce an additional layer of complexity because the observations are biased at finite times even when their stationary means are correct. In (Doan, 2021), the Markov chain has a geometric mixing time $A_{11}x+A_{12}y=b_1$9, and the proof controls the resulting bias by splitting the noise into a nearly unbiased delayed term and a remainder caused by slow iterate motion over the mixing window. This yields a finite-time rate $H(y)$0 under Markovian sampling (Doan, 2021). For arbitrary norm contractions and Markovian noise, (Chandak et al., 24 Mar 2025) combines generalized Moreau envelopes with Poisson-equation corrections and proves a general bound

$H(y)$1

which gives $H(y)$2 in the general case and $H(y)$3 when the slow timescale is noiseless (Chandak et al., 24 Mar 2025).

The controlled Markov framework generalizes further by allowing iterate-dependent transition kernels and multiple invariant measures. There the fast and slow averaged drifts are defined through ergodic occupation measures, and the limiting dynamics is expressed via differential inclusions rather than single averaged ODEs (Karmakar, 2020, Karmakar et al., 2015). This is especially relevant for reinforcement learning, where the sampling process often depends on the evolving policy or critic parameters.

Beyond mean-square bounds, the literature includes high-probability and asymptotic distributional results. "Concentration bounds for two time scale stochastic approximation" derives uniform-in-time concentration inequalities by viewing TTSA as a noisy discretization of a singularly perturbed ODE and applying Alekseev’s nonlinear variation-of-constants formula together with martingale concentration (Borkar et al., 2018). At the asymptotic-distribution level, "Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications" establishes

$H(y)$4

under controlled Markovian noise, with the limiting covariances characterized through Poisson-equation-based long-run covariance matrices (Hu et al., 2024). This identifies the asymptotic fluctuation scale that finite-time analyses approximate only indirectly.

6. Variants, applications, and broader scope

Nonlinear TTSA is a unifying template across several algorithmic domains. The motivating example emphasized in (Vidyasagar, 15 Mar 2026) is Actor–Critic reinforcement learning. More specific reinforcement-learning instantiations include off-policy TDC with importance weighting under controlled Markov noise (Karmakar et al., 2015), GTD2 and TDC with nonlinear function approximation and identical asymptotic covariance under a Markovian CLT (Hu et al., 2024), SSP Q-Learning and discounted Q-Learning with Polyak averaging under arbitrary-norm contraction analysis (Chandak et al., 24 Mar 2025), and broader policy-evaluation and policy-gradient style two-timescale recursions (Karmakar, 2020).

Optimization and game-theoretic applications are equally prominent. The contractive $H(y)$5 theory in (Chandak, 27 Apr 2025) applies to stochastic gradient descent-ascent for smooth strongly-convex–concave minimax problems and to two-time-scale Lagrangian optimization. The non-expansive theory in (Chandak, 18 Jan 2025) covers minimax optimization, linear stochastic approximation, and Lagrangian optimization when the reduced slow map is only non-expansive. The arbitrary-norm Markovian framework in (Chandak et al., 24 Mar 2025) yields an $H(y)$6 rate for learning strongly monotone generalized Nash equilibrium and an $H(y)$7 rate in the noiseless-slow-scale setting.

Distributed variants add consensus and communication structure to the two-timescale mechanism. "Finite-Time Performance of Distributed Two-Time-Scale Stochastic Approximation" studies a network of agents using two doubly stochastic communication matrices $H(y)$8 and $H(y)$9, one for each timescale, and proves a finite-time mean-square bound with explicit dependence on the graph spectral gap through $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$00 (Doan et al., 2019). "Distributed Stochastic Approximation with Local Projections" uses a fast distributed projection process and a slow SA step, establishing consensus, asymptotic feasibility, and convergence to the equilibrium set of a projected dynamical system (Shah et al., 2017).

A recurrent misconception is that TTSA theory is exhausted by globally convergent, contractive, point-to-point dynamics. The broader literature does not support that view. Non-expansive slow maps lead to convergence to a fixed-point set rather than a unique point (Chandak, 18 Jan 2025); general nonlinear dynamics may converge only to internally chain-transitive invariant sets of a differential inclusion (Borkar, 2024); and controlled Markov noise may require occupation-measure averaging instead of a single deterministic reduced vector field (Karmakar, 2020). Conversely, another misconception is that $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$01 is an intrinsic nonlinear barrier. The collected results show that this bound is a baseline under broad conditions, but it can be improved to $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$02, $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$03 for all $f(\theta,\phi)=0,\qquad g(\theta,\phi)=0,$04, or decoupled per-timescale rates when stronger contractivity, averaging, regularity, or bias-correction mechanisms are available (Chandak, 27 Apr 2025, Vidyasagar, 15 Mar 2026, Sarkar et al., 12 Jun 2026).

Taken together, the subject has evolved from asymptotic ODE heuristics into a technically differentiated theory spanning almost-sure convergence, finite-time MSE, high-probability concentration, CLTs, Markov dependence, arbitrary norms, non-expansive reduced dynamics, distributed implementations, and regularity-dependent rate transitions. The common thread is the same singularly perturbed architecture: one stochastic recursion rapidly stabilizes relative to another, and the quality of this tracking determines both the asymptotic limit set and the attainable finite-time rate.