Two-Time-Scale Stochastic Optimization

• The paper introduces a two-time-scale framework that decouples fast gradient estimation from slow decision updates to address biased, dependent samples in time-varying MDPs.
• It establishes finite-time convergence rates under various structural conditions such as strong convexity, the PL condition, and general non-convexity.
• The framework underpins novel actor-critic methods in reinforcement learning and control, demonstrating improved convergence in practical applications.

Searching arXiv for the main paper and closely related two-time-scale stochastic approximation literature. The two-time-scale stochastic optimization framework is a class of stochastic approximation methods for optimization problems in which the update direction for a primary decision variable depends on an auxiliary variable that must itself be estimated online. In the formulation stated in "A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning" (Zeng et al., 2021), the setting is one where gradients are computed with the aid of an auxiliary variable under samples generated by time-varying MDPs controlled by the underlying optimization variable. This induces gradient estimates that are biased and dependent, because the samples are neither i.i.d. nor generated under a fixed data distribution. The framework addresses this difficulty by running two coupled recursions simultaneously: a faster recursion estimates the true gradient, and a slower recursion updates the candidate optimizer. The paper states finite-time complexity results under strong convexity, the PL condition, and general non-convexity, and applies the framework to online actor-critic methods for average-reward MDPs, the linear-quadratic regulator, entropy-regularized MDPs, and gradient-based policy evaluation (Zeng et al., 2021).

1. Definition and problem class

The framework in (Zeng et al., 2021) studies optimization problems in which the optimizer cannot access a direct unbiased gradient oracle. Instead, the gradient is computed with the aid of an auxiliary variable, while the data stream is generated by a controlled stochastic process whose law changes with the optimization variable itself. In the paper’s abstract, these data are described as samples generated by time-varying MDPs controlled by the underlying optimization variable (Zeng et al., 2021).

This problem class is structurally distinct from classical stochastic gradient methods with i.i.d. sampling. The supplied material states that in such settings the sample-based gradient is typically biased because the chain is not at stationarity when the parameter changes, dependent across time because consecutive samples come from a Markov process, and often coupled to an auxiliary variable such as a critic, dual variable, or tracking variable. This coupling creates a natural two-time-scale recursion. The same material also states that these time-varying samples can potentially lead to the divergence of the iterates, which is precisely the instability the framework is designed to control (Zeng et al., 2021).

A closely related abstraction appears in later two-time-scale optimization work that casts the problem as an upper-level objective whose gradient depends on the solution of a lower-level problem. In "Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learning" (Zeng et al., 2024), the lower-level problem is to find the root of a strongly monotone operator, and the upper-level gradient becomes available only after solving that lower-level equation approximately. This suggests a unifying view of two-time-scale methods as stochastic single-loop surrogates for coupled root-finding or bilevel systems, particularly when the auxiliary variable cannot be eliminated in closed form (Zeng et al., 2024).

2. Two-time-scale mechanism

In the formulation given in (Zeng et al., 2021), one time scale is used to estimate the true gradient from these samples, and the other is used to update the estimate of the optimal solution. The essential design principle is that the gradient-estimation iterate is updated "faster" than the optimization iterate, even though the two are implemented simultaneously (Zeng et al., 2021).

The supplied details characterize why this separation is useful. In a two-time-scale scheme, one iterate updates faster to approximately solve an auxiliary subproblem, while the slower iterate moves the main decision variable. The fast iterate therefore reduces the bias in the outer gradient estimate. The examples explicitly listed in the supplied material are a critic tracking a value function, a dual variable tracking a constraint, and an auxiliary fixed point tracking a stationary distribution or Poisson equation solution (Zeng et al., 2021).

Later analyses make this mechanism mathematically explicit. In "O(1/k) Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximation" (Chandak, 27 Apr 2025), the faster variable is denoted $x_k$, the slower variable $y_k$, and the goal is to solve the coupled fixed-point system

$f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$

The iteration is

$$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$

with $a_k$ the fast step size and $b_k$ the slow step size, chosen so that $b_k$ decays faster than $a_k$. That paper identifies the main analytical difficulty as the interaction between coupling error and noise across the two scales, and it introduces an averaged-noise decomposition for the slow iterate as a proof device (Chandak, 27 Apr 2025).

A conceptually related but algorithmically distinct idea is developed in "Fast Nonlinear Two-Time-Scale Stochastic Approximation: Achieving $O(1/k)$ Finite-Sample Complexity" (Doan, 2024), where Ruppert–Polyak averaging is applied to the operator samples themselves rather than to the iterates. There, the algorithm maintains smoothed operator estimates $f_k$ and $y_k$0 and then updates the main variables using those denoised quantities. This suggests that one can interpret the framework of (Zeng et al., 2021) as part of a broader line of work in which the faster time scale serves not merely as a second state variable, but as an online estimator that regularizes the stochastic oracle seen by the slower recursion (Doan, 2024).

3. Finite-time complexity regimes

The first contribution stated in (Zeng et al., 2021) is to characterize the finite-time complexity of the proposed two-time-scale stochastic gradient method. The abstract specifies that the paper provides explicit formulas for the convergence rates under three structural assumptions: strong convexity, the PL condition, and general non-convexity (Zeng et al., 2021).

The supplied material does not include the main theorem statements, the exact objective function, the coupled update rules, assumption lists, or proof lemmas from (Zeng et al., 2021). Accordingly, exact constants and theorem forms beyond the abstract are not recoverable from the provided text. What is recoverable is the structural scope of the theory: the framework is presented as one finite-time method encompassing multiple optimization geometries, rather than a result specialized to a single convex or linear setting (Zeng et al., 2021).

The significance of this positioning becomes clearer in comparison with adjacent literature. "A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic" (Hong et al., 2020) analyzes a TTSA algorithm for bilevel optimization and states that when the outer problem is strongly convex the method finds an $y_k$1-optimal solution, while under weak convexity it finds an $y_k$2-stationary solution. The later paper "Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learning" (Zeng et al., 2024) states improved rates of $y_k$3 in the strongly convex and PL regimes, $y_k$4 in the convex regime, and $y_k$5 for general nonconvexity, attributing the improvement to additional averaging steps that eliminate the direct coupling between the main variables (Zeng et al., 2024).

Other variants reveal how sensitive the achievable rate is to the structural assumptions. "Finite-Time Bounds for Two-Time-Scale Stochastic Approximation with Arbitrary Norm Contractions and Markovian Noise" (Chandak et al., 24 Mar 2025) gives a general $y_k$6 mean-square rate and an improved $y_k$7 rate when the slower timescale is noiseless. By contrast, "Non-Expansive Mappings in Two-Time-Scale Stochastic Approximation: Finite-Time Analysis" (Chandak, 18 Jan 2025) studies the case where the slow mapping is only non-expansive, obtaining a mean-square decay rate of $y_k$8. This suggests that the complexity statement in (Zeng et al., 2021) belongs to a finite-time program in which geometry of the reduced slow dynamics—contractive, strongly monotone, PL, or merely non-expansive—directly governs the best rate one can expect.

4. Reinforcement learning and control applications

The abstract of (Zeng et al., 2021) presents the framework primarily through policy optimization problems. The first application is the infinite-horizon average-reward MDP with finite state and action spaces, for which the paper derives a convergence rate of

$y_k$9

for the online actor-critic algorithm under function approximation, and states that this recovers the best known rate derived specifically for this problem (Zeng et al., 2021).

The second application is the linear-quadratic regulator, where the paper states that an online actor-critic method converges with rate

$f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$0

The third application is policy optimization in an entropy regularized Markov decision process, again via an actor-critic algorithm, with convergence

$f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$1

The abstract further states that the results for the second and third problems are novel and previously unknown in the literature (Zeng et al., 2021).

The same abstract also notes an application to gradient-based policy evaluation algorithms in reinforcement learning, although the supplied material does not include the specific algorithmic forms or guarantees for that part (Zeng et al., 2021).

These applications align with later papers that place RL squarely inside the two-time-scale template. "Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learning" (Zeng et al., 2024) identifies TDC, online actor-critic for LQR, and entropy-regularized MDP policy optimization as specializations of a unified two-time-scale optimization problem. It reports $f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$2 for linear-function-approximation TDC, $f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$3 for LQR actor-critic under a PL structure, and $f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$4 for entropy-regularized actor-critic, explicitly contrasting those results with older $f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$5-type analyses (Zeng et al., 2024).

A different RL perspective appears in "Two Timescale Stochastic Approximation with Controlled Markov noise and Off-policy temporal difference learning" (Karmakar et al., 2015), which gives asymptotic convergence for coupled recursions driven by controlled Markov noise and applies the framework to off-policy TDC with linear function approximation. That paper is asymptotic rather than finite-time, but it makes explicit a feature already central to (Zeng et al., 2021): the data-generating Markov process is controlled by the iterates, so averaging must be performed over ergodic occupation measures rather than over a fixed sample distribution (Karmakar et al., 2015).

5. Relation to bilevel, stochastic approximation, and fluctuation theory

The two-time-scale framework is closely related to bilevel optimization, but it is not limited to classical deterministic bilevel problems. In "A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic" (Hong et al., 2020), the inner problem is unconstrained and strongly convex, the outer problem is smooth and constrained, and the algorithm uses a stochastic gradient update with a larger step size for the inner problem and a projected stochastic gradient update with a smaller step size for the outer problem. That formulation emphasizes a hypergradient viewpoint, in which the outer gradient depends on the inner minimizer $f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$6 (Hong et al., 2020).

A continuous-time analogue appears in "Two-Timescale Stochastic Approximation for Bilevel Optimisation Problems in Continuous-Time Models" (Sharrock, 2022). There the bilevel problem is

$f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$7

and the stochastic dynamics are modeled by SDEs with learning rates

$f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$8

where $f(x^*,y^*)=x^*,\qquad g(x^*,y^*)=y^*.$9, so the $$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$0-recursion runs on the faster time scale. The paper proves almost sure convergence and a central limit theorem with explicit asymptotic covariance, thereby linking two-time-scale optimization to singular perturbation and weak-convergence analysis in continuous time (Sharrock, 2022).

At the level of fluctuation theory, "Functional Central Limit Theorem for Two Timescale Stochastic Approximation" (Faizal et al., 2023) shows that the fast fluctuation limit is a linear diffusion, while the slow fluctuation limit is an ordinary differential equation driven by the fast fluctuation. A central point of that paper is that the slow iterate’s fluctuations are influenced by the fast iterate and must be normalized by the square root of the faster step size rather than the slower one. This suggests that finite-time bias and variance tradeoffs in frameworks such as (Zeng et al., 2021) should not be interpreted as independent noise accumulation on two separate scales; rather, the slow scale inherits its leading fluctuation structure from the fast one (Faizal et al., 2023).

6. Later developments, variants, and limitations

Subsequent work extends the two-time-scale framework in several directions. "O(1/k) Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximation" (Chandak, 27 Apr 2025) improves the previous best known $$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$1 bound in the nonlinear contractive setting to $$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$2 without modifying the original coupled iteration. "Fast Nonlinear Two-Time-Scale Stochastic Approximation: Achieving $$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$3 Finite-Sample Complexity" (Doan, 2024) achieves the same rate by modifying the algorithm through averaged operator estimates. "Finite-Time Bounds for Two-Time-Scale Stochastic Approximation with Arbitrary Norm Contractions and Markovian Noise" (Chandak et al., 24 Mar 2025) generalizes the analysis to arbitrary norm contractions and Markovian noise, using the generalized Moreau envelope and solutions of Poisson equation (Chandak, 27 Apr 2025, Doan, 2024, Chandak et al., 24 Mar 2025).

There are also settings in which the two-time-scale idea is present but not expressed as classical stochastic approximation. "Time Blocks Decomposition of Multistage Stochastic Optimization Problems" (1804.01711) and "Decomposition Methods for Dynamically Monotone Two-Time-Scale Stochastic Optimization Problems" (Rigaut et al., 2023) treat problems with slow strategic decisions and fast operational decisions using slow-scale Bellman recursions and within-block stochastic programs. Those works use the phrase two time scales in a decomposition sense rather than in the Borkar-style SA sense, but they share the same organizing principle: fast decisions adapt within a slow block, while the slow variable sees only a reduced state transition across blocks (1804.01711, Rigaut et al., 2023).

An important limitation must nevertheless be stated for (Zeng et al., 2021). The supplied document is described as only a Supplement/appendix shell with placeholder text, dummy equations, and a bibliography, and it explicitly states that the actual technical body is absent. Consequently, the exact objective function, the two coupled update rules, theorem statements, assumption lists, proof lemmas, and application-specific derivations are not recoverable from the provided text. The abstract supports the existence of the finite-time results and the stated rates $$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$4 and $$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$5, but it does not support reconstruction of the full algorithm or proof architecture from that paper alone (Zeng et al., 2021).

This suggests two complementary interpretations. First, (Zeng et al., 2021) occupies an important position in the development of finite-time two-time-scale optimization for RL and control, because it explicitly unifies strong convexity, PL, and general non-convexity under time-varying MDP sampling and reports concrete actor-critic rates. Second, later papers make the framework technically more transparent by spelling out the contractive root-finding structure, the role of averaged operators, the interaction between timescale separation and noise, and the specific circumstances under which the classical $$x_{k+1}=x_k+a_k\big(f(x_k,y_k)-x_k+M_{k+1}\big), \qquad y_{k+1}=y_k+b_k\big(g(x_k,y_k)-y_k+M'_{k+1}\big),$$6 barrier can be improved (Zeng et al., 2024, Chandak, 27 Apr 2025).