Timescale-Based Update Method

• Timescale-based update methods are schemes that use variable step sizes to let some subsystems react fast while others update slowly, ensuring a quasi-stationary environment.
• They are implemented across diverse areas such as reinforcement learning, stochastic approximation, decentralized multi-agent learning, and numerical simulation through tailored recursions and time-dilation techniques.
• These methods improve convergence, reduce non-stationarity, and enhance computational efficiency by strategically separating fast and slow dynamics within complex systems.

Searching arXiv for recent and foundational papers on timescale-based update methods across reinforcement learning, stochastic approximation, and numerical simulation. In the literature represented here, a timescale-based update method is an update scheme in which coupled variables, agents, or spatial regions evolve at different effective speeds rather than under a single uniform schedule. In stochastic approximation and reinforcement learning, this usually appears as fast and slow iterates with distinct step sizes; in decentralized multi-agent learning, as agent-specific learning rates or phase lengths; and in numerical simulation, as local timesteps, time-step rescaling, or a space-time dilation factor $a(x,t)\in(0,1]$ (Hong et al., 2020, Nekoei et al., 2023, Hopkins et al., 10 Oct 2025). Taken together, these works suggest a common design objective: to let one part of the system react quickly while another part changes slowly enough that the fast dynamics see a quasi-stationary environment.

1. Terminological scope and major variants

The phrase “timescale-based” is not tied to a single formalism. In the cited literature it denotes at least four related constructions: two-timescale stochastic approximation, multi-timescale learning in multi-agent reinforcement learning, time-step rescaling or time dilation in numerical integration, and local or componentwise adaptive timestepping for multiscale dynamics. A further development is the single-timescale alternative, where actor and critic or upper- and lower-level variables are updated on the same order of step size, but the analysis still hinges on how quickly one subsystem tracks another (Chen et al., 2021, Luo et al., 2022, Olshevsky et al., 2022).

Variant	Representative update law	Representative setting
Two-timescale stochastic approximation	$x_{n+1}=x_n+\alpha_n H(\cdot)$, $y_{n+1}=y_n+\beta_n G(\cdot)$	Bilevel optimization, actor-critic, TDC($\lambda$)
Multi-timescale learning	One “fast” agent uses rate $\alpha$, the others use $\beta$	Decentralized cooperative MARL
Time dilation / rescaling	$$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$ or $\Delta t\to b\,\Delta t$	Multiphysics simulation, Langevin integration
Local adaptive timestepping	Componentwise $k_{ij}$ or clusterwise $\Delta t(P_i)$	ODE/PDE integration, rigid-body DEM

This taxonomy is directly reflected in the cited papers. In two-timescale bilevel optimization, the inner problem is updated with a larger step size and the outer problem with a smaller one (Hong et al., 2020). In decentralized cooperative MARL, all agents update concurrently but one designated agent is “fast” while the others are “slow,” with the fast role cycling every $x_{n+1}=x_n+\alpha_n H(\cdot)$0 steps (Nekoei et al., 2023). In numerical simulation, the governing equation itself may be modified by a dilation field $x_{n+1}=x_n+\alpha_n H(\cdot)$1, or the deterministic substeps of an integrator may be rescaled by a factor $x_{n+1}=x_n+\alpha_n H(\cdot)$2 chosen to recover continuous-time dynamical properties (Hopkins et al., 10 Oct 2025, Sivak et al., 2013).

2. Canonical mathematical forms

A standard two-timescale stochastic approximation recursion maintains a fast iterate and a slow iterate. In the Markovian-noise framework, the updates are

$x_{n+1}=x_n+\alpha_n H(\cdot)$3

with $x_{n+1}=x_n+\alpha_n H(\cdot)$4, $x_{n+1}=x_n+\alpha_n H(\cdot)$5, $x_{n+1}=x_n+\alpha_n H(\cdot)$6, and $x_{n+1}=x_n+\alpha_n H(\cdot)$7 or, equivalently under the paper’s naming convention, $x_{n+1}=x_n+\alpha_n H(\cdot)$8 when $x_{n+1}=x_n+\alpha_n H(\cdot)$9 is designated the fast timescale (Mahadevan et al., 29 May 2026). The same paper formulates the limiting ODE picture as $y_{n+1}=y_n+\beta_n G(\cdot)$0 with $y_{n+1}=y_n+\beta_n G(\cdot)$1 approximately constant, and $y_{n+1}=y_n+\beta_n G(\cdot)$2 with $y_{n+1}=y_n+\beta_n G(\cdot)$3.

In bilevel optimization, the same structure appears as

$y_{n+1}=y_n+\beta_n G(\cdot)$4

where $y_{n+1}=y_n+\beta_n G(\cdot)$5 estimates $y_{n+1}=y_n+\beta_n G(\cdot)$6 and $y_{n+1}=y_n+\beta_n G(\cdot)$7 estimates the surrogate gradient

$y_{n+1}=y_n+\beta_n G(\cdot)$8

(Hong et al., 2020). The requirement $y_{n+1}=y_n+\beta_n G(\cdot)$9 ensures that $\lambda$0 tracks $\lambda$1.

In decentralized cooperative MARL, the multi-timescale update is agent-indexed rather than variable-indexed. If $\lambda$2 is the “fast” agent at training iteration $\lambda$3, then

$\lambda$4

for $\lambda$5, while for every $\lambda$6,

$\lambda$7

with $\lambda$8 (Nekoei et al., 2023). For policy-gradient methods this becomes a PPO-style update using $\lambda$9 for the fast agent and $\alpha$0 for the slow agents.

In numerical simulation, timescale-based updates alter the time variable itself. The time-dilation method rewrites

$\alpha$1

as

$\alpha$2

or equivalently

$\alpha$3

(Hopkins et al., 10 Oct 2025). In Langevin integration, the OVRVO scheme rescales the deterministic substeps by

$\alpha$4

with $\alpha$5 for the Ornstein–Uhlenbeck step (Sivak et al., 2013). In multi-adaptive Galerkin integration, each component $\alpha$6 receives its own local intervals $\alpha$7 and local time step $\alpha$8, selected from an a posteriori error estimate of the global error (Logg, 2012).

3. Bilevel optimization and actor-critic formulations

The two-timescale stochastic approximation framework for bilevel optimization analyzes

$\alpha$9

under the assumptions that $\beta$0 is $\beta$1-strongly convex and that the stochastic gradients have bounded variance (Hong et al., 2020). For a strongly convex outer problem, the paper uses

$\beta$2

and proves

$\beta$3

For weakly convex outer objectives, the constant-step schedule

$\beta$4

yields an $\beta$5 stationarity guarantee, and in the convex-outer case the paper further obtains $\beta$6 (Hong et al., 2020). The same framework specializes to a two-timescale natural actor-critic proximal policy optimization method with critic step size larger than actor step size, giving an $\beta$7 policy-performance gap.

Single-timescale bilevel optimization was later developed as an explicit alternative to two-timescale and double-loop schemes. The STABLE method uses a single loop, batch size $\beta$8, and matched decay

$\beta$9

while tracking $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$0 and $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$1 by STORM-style recursions (Chen et al., 2021). The resulting sample complexity is $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$2 to achieve an $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$3-stationary point in the nonconvex case and $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$4 to achieve an $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$5-optimal solution in the strongly convex case. The paper contrasts this with TTSA’s $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$6 sample complexity for nonconvex $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$7.

A comparable shift away from explicit timescale separation appears in actor-critic. In fully decentralized single-timescale actor-critic, actor, critic, and reward-estimator step sizes are all chosen as $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$8, with the main theorem using $$L[a]\{U\}\equiv \frac{1}{a}\partial_t U+\nabla\cdot F(U)-S(U)=0$$9 and obtaining

$\Delta t\to b\,\Delta t$0

which implies $\Delta t\to b\,\Delta t$1 sample complexity under Markovian sampling (Luo et al., 2022). The analysis relies on the “hidden smoothness of the optimal critic variable” $\Delta t\to b\,\Delta t$2. A related single-timescale actor-critic analysis based on the small-gain theorem ties the actor and critic step sizes together as $\Delta t\to b\,\Delta t$3, $\Delta t\to b\,\Delta t$4, and proves $\Delta t\to b\,\Delta t$5 sample complexity for finding an $\Delta t\to b\,\Delta t$6-approximate stationary point (Olshevsky et al., 2022).

Two-timescale analysis remains central when the noise is Markovian and projection-free stability is desired. A 2026 convergence result establishes, without any projection or truncation, almost sure boundedness and convergence of the iterates,

$\Delta t\to b\,\Delta t$7

and applies this theorem to TDC($\Delta t\to b\,\Delta t$8) with eligibility traces, yielding the first almost sure convergence result for off-policy learning with linear function approximation (Mahadevan et al., 29 May 2026).

4. Decentralized and multi-agent learning

In decentralized cooperative MARL, the central problem is non-stationarity induced by concurrent policy updates. Independent learning (IIBR) lets all $\Delta t\to b\,\Delta t$9 agents update simultaneously by best response, while sequential learning (SIBR) updates only one agent at a time and is guaranteed, under bounded-reward assumptions, to converge to an agent-by-agent optimum (Nekoei et al., 2023). Multi-timescale learning (MTL) interpolates between these extremes: one agent updates with learning rate $k_{ij}$0, the other $k_{ij}$1 agents with $k_{ij}$2, and the identity of the fast agent is switched every $k_{ij}$3 steps. The paper states explicitly that $k_{ij}$4 recovers SIBR and $k_{ij}$5 recovers IIBR.

The empirical comparison on 12 cooperative tasks in MPE, LBF, RWARE, and SMAC reports the following aggregate normalized scores (Nekoei et al., 2023):

Method	Mean	Median
IQL	0.363	0.464
MTQL	0.404	0.507
IPPO	0.534	0.578
MTPPO	0.599	0.714

The same paper reports that MTPPO improves over IPPO in all tasks, with gains up to $k_{ij}$6 in “MMM2” on SMAC, and that MTQL outperforms IQL in most tasks, notably RWARE tiny-4ag and Foraging-$k_{ij}$7. It also reports “gap closed” numbers with respect to the best CTDE methods, including $k_{ij}$8 for MTPPO on SMAC and $k_{ij}$9 for MTQL on MPE. Hyperparameter guidance is likewise explicit: the $\Delta t(P_i)$0 ratio $\Delta t(P_i)$1–$\Delta t(P_i)$2 works well, $\Delta t(P_i)$3, and very large $\Delta t(P_i)$4 recovers SIBR while very small $\Delta t(P_i)$5 approximates IIBR (Nekoei et al., 2023).

Unsynchronized decentralized Q-learning extends timescale separation to settings in which agents do not synchronize policy-revision clocks. The fast timescale is a constant-step-size Q update inside an exploration phase, and the slow timescale is a phase-end policy revision with effective step size $\Delta t(P_i)$6 (Yongacoglu et al., 2023). Under weak acyclicity, small constant $\Delta t(P_i)$7, persistent forced exploration $\Delta t(P_i)$8, and phase lengths $\Delta t(P_i)$9, the paper proves that for any $x_{n+1}=x_n+\alpha_n H(\cdot)$00 there exists $x_{n+1}=x_n+\alpha_n H(\cdot)$01 such that for all $x_{n+1}=x_n+\alpha_n H(\cdot)$02,

$x_{n+1}=x_n+\alpha_n H(\cdot)$03

A distinct two-timescale construction appears in zero-sum stochastic games with function approximation. There the fast iterate $x_{n+1}=x_n+\alpha_n H(\cdot)$04 fits a local $x_{n+1}=x_n+\alpha_n H(\cdot)$05-function, while the slow iterate $x_{n+1}=x_n+\alpha_n H(\cdot)$06 forms the policy via softmax, updated by the convex combination

$x_{n+1}=x_n+\alpha_n H(\cdot)$07

(Chen et al., 2023). The paper interprets this as a parameter-space analogue of smoothed best-response dynamics, constructs a Lyapunov function via a generalized Moreau envelope of the regularized Nash gap, and derives an explicit last-iterate finite-sample bound implying total sample complexity $x_{n+1}=x_n+\alpha_n H(\cdot)$08.

Multi-timescale design can also refer to temporal horizons rather than step-size separation. In a multi-timescale PPO variant inspired by multi-discount critics, exposing a routing mechanism to policy gradients causes “surrogate objective hacking,” and gradient-free inverse-variance weighting causes the “Paradox of Temporal Uncertainty” (Sun, 15 Apr 2026). The proposed Target Decoupling architecture keeps multiple discount heads $x_{n+1}=x_n+\alpha_n H(\cdot)$09 on the critic side but updates the actor only with the longest-horizon advantage $x_{n+1}=x_n+\alpha_n H(\cdot)$10. On LunarLander-v2, the paper reports that Target-Decoupling breaks the 200-point “solved” threshold at approximately 1,500 episodes, peaks around 240 points, and exhibits a tight variance band across 5 seeds, whereas the single-timescale PPO baseline remains stuck around 150 points (Sun, 15 Apr 2026).

5. Numerical integration, local timestepping, and engineering systems

In multiphysics simulation, the time-dilation method introduces a continuous field $x_{n+1}=x_n+\alpha_n H(\cdot)$11 that locally stretches time so that stiff subdomains do not dominate the global timestep (Hopkins et al., 10 Oct 2025). The method requires $x_{n+1}=x_n+\alpha_n H(\cdot)$12, $x_{n+1}=x_n+\alpha_n H(\cdot)$13 outside the fast subdomains, and smoothness conditions

$x_{n+1}=x_n+\alpha_n H(\cdot)$14

together with CFL-like constraints

$x_{n+1}=x_n+\alpha_n H(\cdot)$15

The reported speedups are $x_{n+1}=x_n+\alpha_n H(\cdot)$16–$x_{n+1}=x_n+\alpha_n H(\cdot)$17 for Bondi accretion, $x_{n+1}=x_n+\alpha_n H(\cdot)$18 for Evrard collapse, and $x_{n+1}=x_n+\alpha_n H(\cdot)$19–$x_{n+1}=x_n+\alpha_n H(\cdot)$20 for a full AGN disk simulation; the abstract further reports effective speedup factors exceeding $x_{n+1}=x_n+\alpha_n H(\cdot)$21 (Hopkins et al., 10 Oct 2025).

A closely related but more specialized construction is time-step rescaling for Langevin integration. The OVRVO splitting rescales the deterministic velocity and position updates by $x_{n+1}=x_n+\alpha_n H(\cdot)$22 and uses $x_{n+1}=x_n+\alpha_n H(\cdot)$23 in the Ornstein–Uhlenbeck thermalization step (Sivak et al., 2013). With

$x_{n+1}=x_n+\alpha_n H(\cdot)$24

the integrator exactly recovers the continuous-time zero-force mean-squared displacement, velocity autocorrelation, mean-squared velocity, and terminal drift for arbitrary $x_{n+1}=x_n+\alpha_n H(\cdot)$25. The paper further emphasizes explicit Hamiltonian updates, thermodynamic bookkeeping into heat, protocol work, and shadow work, and a simple path action for nonequilibrium path sampling.

Local timestepping in rigid-body DEM provides another form of timescale-based updating. Each particle stores its recent time-stamped states and proposes

$x_{n+1}=x_n+\alpha_n H(\cdot)$26

with $x_{n+1}=x_n+\alpha_n H(\cdot)$27 in the reported implementation (Noble et al., 2023). Particles are clustered by broad-phase collision predicates, and each cluster advances to the earliest predicted space-time collision time

$x_{n+1}=x_n+\alpha_n H(\cdot)$28

The reported overall speedups are between $x_{n+1}=x_n+\alpha_n H(\cdot)$29 and $x_{n+1}=x_n+\alpha_n H(\cdot)$30 compared to a state-of-the-art global adaptive stepper, with the advantage disappearing during “shock” phases when the cluster graph collapses to one large component (Noble et al., 2023).

Multi-adaptive Galerkin methods generalize local timestepping to ODEs and time-dependent PDEs by assigning each component $x_{n+1}=x_n+\alpha_n H(\cdot)$31 its own local intervals $x_{n+1}=x_n+\alpha_n H(\cdot)$32, time steps $x_{n+1}=x_n+\alpha_n H(\cdot)$33, and local polynomial degrees $x_{n+1}=x_n+\alpha_n H(\cdot)$34 (Logg, 2012). The next local step can be selected from

$x_{n+1}=x_n+\alpha_n H(\cdot)$35

where $x_{n+1}=x_n+\alpha_n H(\cdot)$36 is a residual magnitude, $x_{n+1}=x_n+\alpha_n H(\cdot)$37 is a stability factor derived from the dual problem, and $x_{n+1}=x_n+\alpha_n H(\cdot)$38 or $x_{n+1}=x_n+\alpha_n H(\cdot)$39 depending on whether mcG or mdG is used. When fixed-point iteration fails because of stiffness, the method inserts a small number of stabilizing small explicit steps (Logg, 2012).

Engineering control systems also adopt explicit large- and small-timescale splits. In joint power and admission control based on channel distribution information, admission control is performed on a large timescale using sampled CDI and a convex-approximation-plus-deflation algorithm, while power is adapted on a small timescale by the Foschini–Miljanic-type update

$x_{n+1}=x_n+\alpha_n H(\cdot)$40

(Chen et al., 2017). In cascading-outage risk assessment, short-timescale protection-triggered outages, mid-timescale random outages and redispatch, and long-timescale load variation are combined in a quasi-dynamic simulation reformulated as a Markovian tree search with a forward search and backward update algorithm (Yao et al., 2016).

6. Design principles, limitations, and conceptual boundaries

The most stable formulations impose an explicit separation condition. In bilevel TTSA this is $x_{n+1}=x_n+\alpha_n H(\cdot)$41, so the inner variable tracks $x_{n+1}=x_n+\alpha_n H(\cdot)$42 (Hong et al., 2020). In Markovian two-timescale SA the notation is reversed—$x_{n+1}=x_n+\alpha_n H(\cdot)$43 is the fast iterate, $x_{n+1}=x_n+\alpha_n H(\cdot)$44 the slow one, and $x_{n+1}=x_n+\alpha_n H(\cdot)$45 (Mahadevan et al., 29 May 2026). In multi-timescale MARL the same principle appears as a large $x_{n+1}=x_n+\alpha_n H(\cdot)$46 ratio; in unsynchronized decentralized Q-learning it appears as long exploration phases relative to a constant Q-learning step size (Nekoei et al., 2023, Yongacoglu et al., 2023). This suggests that the essential requirement is not a particular symbol convention but a gap between the adaptation rates of the coupled subsystems.

The same literature also shows that timescale separation is not the only viable route. STABLE keeps $x_{n+1}=x_n+\alpha_n H(\cdot)$47, $x_{n+1}=x_n+\alpha_n H(\cdot)$48, and Hessian-block trackers on the same order of step size (Chen et al., 2021). Decentralized single-timescale actor-critic updates actor and critic with the same $x_{n+1}=x_n+\alpha_n H(\cdot)$49 order and controls the coupling through the hidden smoothness of $x_{n+1}=x_n+\alpha_n H(\cdot)$50 (Luo et al., 2022). The small-gain analysis of single-timescale actor-critic similarly replaces explicit two-timescale arguments by coupled subsystem bounds (Olshevsky et al., 2022). A plausible implication is that “timescale-based” reasoning can survive even when only one nominal timescale is used, provided another structural property plays the role ordinarily taken by rate separation.

Several papers also state explicit failure modes. In multi-timescale PPO with multiple discount factors, naively fusing advantages across timescales can yield “surrogate objective hacking” or the “Paradox of Temporal Uncertainty,” so adding more timescales does not automatically improve optimization (Sun, 15 Apr 2026). In time-dilation methods, too-strong $x_{n+1}=x_n+\alpha_n H(\cdot)$51 induces spurious source terms, and in non-equilibrium rapidly changing regions dilation distorts transient dynamics (Hopkins et al., 10 Oct 2025). In decentralized cooperative MARL, very large switch periods recover slow sequential learning and very small switch periods approximate simultaneous independent learning, so the fast/slow split must still be tuned (Nekoei et al., 2023).

A common misconception is that timescale-based methods are solely about convergence acceleration. The cited work presents a broader picture. In some cases the method is introduced to recover a mathematical property that a naive discretization loses, such as exact free-particle diffusion and terminal drift in Langevin integration (Sivak et al., 2013). In others, the objective is to preserve correct local steady states while avoiding a prohibitively small global timestep (Hopkins et al., 10 Oct 2025), or to make state-space search tractable by discretizing long, mid, and short operational timescales (Yao et al., 2016). The technical role of the timescale split therefore depends on the application: quasi-stationary tracking, non-stationarity reduction, geometric collision prediction, thermodynamic consistency, or computational tractability.