Dual-Timescale Eligibility Traces in RL

• Dual-timescale eligibility traces are mechanisms that combine a fast transient memory with slower consolidation to bridge temporal credit assignment gaps.
• They are implemented using explicit dual-trace architectures or hierarchical adaptations that balance rapid responsiveness with long-term stability.
• These traces enhance learning efficiency in applications like brain-computer interfaces and deep reinforcement learning by effectively linking actions to delayed rewards.

Searching arXiv for recent and foundational papers on dual-timescale or related eligibility trace mechanisms. {"query":"eligibility traces dual timescale arXiv eligibility trace multiple timescale cascading expected traces two-timescale reinforcement learning", "max_results": 10, "sort_by": "relevance"} Unable to access a dedicated tool interface in this message format, but proceeding using the provided arXiv records as the source corpus for the article and citing their arXiv IDs directly. Dual-timescale eligibility traces denote eligibility-trace mechanisms in which temporal credit assignment depends on at least two coupled temporal processes, usually a fast transient memory of recent events and a slower process that stabilizes, consolidates, or reads out that memory. In the cited literature, the phrase has both a literal and a broader use. Literal uses appear in algorithms with explicit fast and slow trace variables, whereas broader uses include systems in which a short-lived eligibility window interacts with slower value learning, neuromodulatory gating, refractory recovery, replay-like reconstruction, or compressed sequence dynamics (Nallani et al., 17 Sep 2025, Lehmann et al., 2017, Gerstner et al., 2018). The common objective is to bridge a temporal gap between causal activity and delayed error, reward, or supervisory signals without reverting to purely one-step bootstrapping.

1. Canonical eligibility traces and the origin of timescale separation

In standard reinforcement learning, an eligibility trace is a decaying memory of recently visited state-action pairs. A discrete-time formulation used in sequential decision making sets

$$e_{n}(s,a) = \left\{ \begin{array}{ll} 1 & \text{if } s=s_n, a=a_n \ \gamma \lambda e_{n-1}(s,a) & \text{otherwise} \end{array} \right.$$

and updates values by

$$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$

The current choice receives eligibility $1$, earlier choices decay by $\gamma \lambda$, and $\lambda=0$ recovers the no-trace case (Lehmann et al., 2017).

A continuous-time formulation makes the temporal interpretation explicit: $$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$ for $t>B(s,a)$, and zero otherwise, where $B(s,a)$ is the last time the state-action pair was selected and $\tau$ is the time constant of trace decay (Lehmann et al., 2017). This establishes a fast memory process that can persist over seconds, while the learned values $Q$ and $$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$0 accumulate more slowly across repeated experience. The same paper interprets this division as a fast eligibility window versus a slower value-learning process, even though it does not define two separate trace constants (Lehmann et al., 2017).

In neoHebbian three-factor formulations, the separation is even clearer. The synapse is described by an internal eligibility variable $$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$1 and an observable weight $$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$2, with

$$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$3

and

$$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$4

Here pre- and postsynaptic activity write a transient synaptic flag, and a delayed third factor $$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$5 converts that flag into durable plasticity (Gerstner et al., 2018). In this formulation, dual timescales are intrinsic: milliseconds to tens of milliseconds for local coincidence, and seconds to minutes for modulatory readout.

2. Explicit fast/slow eligibility-trace architectures

Some recent algorithms instantiate dual timescales directly by maintaining two eligibility traces per synapse. In an online spiking neural network decoder for brain-computer interfaces, the instantaneous three-factor Hebbian term is

$$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$6

and it is accumulated into a fast trace and a slow trace: $$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$7

$$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$8

with representative time constants $$Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).$$9 and $1$0. The two traces are mixed as

$1$1

used for an immediate fast update, and then passed through a momentum-smoothed consolidation stream applied every $1$2 timesteps (Nallani et al., 17 Sep 2025). The stated role of the fast trace is rapid reaction to abrupt changes, whereas the slow trace preserves longer-term structure and stabilizes learning. Empirically, the method achieved comparable decoding accuracy with Pearson $1$3 on Zenodo Indy and $1$4 on MC Maze, with 28–35% memory reduction relative to BPTT-trained SNNs (Nallani et al., 17 Sep 2025).

A different explicit construction appears in online deep reinforcement learning with adaptive and multiple time-scale traces. Standard accumulating traces are generalized to $1$5 traces $1$6, each associated with a different memory timescale, and updated by

$1$7

with an adaptive decay factor

$1$8

The divergence state $1$9 is driven by policy and value-output divergences, so traces are decayed more aggressively when parameter drift makes old gradients unreliable (Kobayashi, 2020). The $\gamma \lambda$0 case yields an explicit dual-timescale interpretation: short-term memory can selectively replace long-term memory, interpolating between standard accumulating traces and replacing traces. On four PyBullet Gym tasks, the proposed setting $\gamma \lambda$1 outperformed no-trace, standard-trace, replacing-trace, and adaptive single-trace baselines (Kobayashi, 2020).

These two lines of work share a common design principle: a fast component improves responsiveness, while a slow component counteracts instability or forgetting. The difference is that the BCI formulation couples the traces to explicit consolidation, whereas the online DRL formulation emphasizes adaptive forgetting under parameter-dependent gradient divergence.

3. Biological and neurocomputational realizations

Behavioral evidence for a fast eligibility process interacting with slower value formation is provided by one-shot sequential learning in humans. In a multi-step task with six states plus a goal state, rewarded episode-1 experience changed episode-2 choices not only at $\gamma \lambda$2, one step from reward, but also at $\gamma \lambda$3, two steps from reward. The repeated action at $\gamma \lambda$4 occurred in 85% of cases, and a continuous-time trace fit yielded a characteristic time constant of roughly $\gamma \lambda$5 seconds, corresponding to about 2–3 inter-stimulus intervals (Lehmann et al., 2017). The paper explicitly distinguishes this short-lived behavioral trace from the slower accumulation of long-term values.

Experimental synaptic plasticity studies reviewed in the neoHebbian framework describe a similar fast/slow split. Co-activation writes a metastable eligibility trace, and dopamine, norepinephrine, serotonin, acetylcholine, or plateau potentials convert that trace into weight change if they arrive before it decays (Gerstner et al., 2018). The reviewed experiments support traces of around 1 second in striatum, about 3 seconds for LTD and 5–10 seconds for LTP in cortex, around 2 seconds in a hippocampal place-field paradigm, and minutes in a hippocampal tagging-like regime that the review interprets as more akin to synaptic tagging or consolidation than to fast reinforcement learning (Gerstner et al., 2018). This literature treats dual timescales as a biological necessity: fast local coincidence is insufficient unless a slower broadcast factor can still access the synaptic flag.

In spiking-network learning, precise timing can require a second internal state beyond a simple decaying trace. For e-prop with the Izhikevich neuron, the hidden state

$\gamma \lambda$6

induces a two-component eligibility vector in which the recovery variable $\gamma \lambda$7 acts as an exponential filter of voltage eligibility. When the postsynaptic neuron spikes, the voltage eligibility is reset by a negative term involving the recovery eligibility, allowing negative eligibility for presynaptic spikes that arrive late relative to the postsynaptic spike (Traub et al., 2020). The paper interprets this as the missing ingredient for full STDP-like timing dependence. A simplified STDP-LIF construction achieves a similar effect by imposing a negative pseudo-derivative during refractoriness (Traub et al., 2020). In both cases, fast spike interactions coexist with a slower refractory or recovery process that determines whether later inputs potentiate or depress the synapse.

Theta-sequence models propose a different biological route to effective dual timescales. A short neuronal trace of order $\gamma \lambda$8 ms is not lengthened directly; instead, behavior is compressed into theta cycles so that the effective eligibility window becomes

$\gamma \lambda$9

where $\lambda=0$0 is the ratio of sequence speed to behavioral speed (George, 2023). The task-level horizon can therefore be seconds even when the biological trace remains around $\lambda=0$1 s. This suggests that dual-timescale credit assignment need not require two explicit trace variables; it can also emerge from the interaction of a fast local trace with a slower behavioral trajectory represented at compressed neural timescale.

4. Generalizations beyond two traces

A major extension replaces a single exponential trace with a cascade of internal states. Cascading eligibility traces are defined by

$\lambda=0$2

yielding the closed-form kernel

$\lambda=0$3

The impulse response is gamma-like, $\lambda=0$4, and peaks at a desired delay $\lambda=0$5 when $\lambda=0$6 (Ralambomihanta et al., 17 Jun 2025). A one-state system recovers the classical exponential trace, a two-state system is described as conceptually similar to dual-trace ideas, and higher-order cascades yield sharper temporal selectivity. On MNIST, standard traces remained effective up to about 2 seconds but degraded for 4s+, whereas higher-order CETs maintained high performance up to 10 seconds (Ralambomihanta et al., 17 Jun 2025). The paper therefore generalizes dual-timescale designs into a multi-timescale state-space framework.

Expected eligibility traces generalize the trace along a different axis. Instead of updating with the sampled backward-view trace $\lambda=0$7, the method learns the state-conditional expectation

$\lambda=0$8

and updates values with

$\lambda=0$9

A mixed trace

$$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$0

interpolates between pure expected traces and standard TD$$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$1, which is exactly recovered when $$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$2 (Hasselt et al., 2020). The paper does not present this as a dual-timescale trace method in name, but it has two coupled learning dynamics: a value learner with stepsize $$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$3 and a trace-predictor learner with stepsize $$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$4. It also adds counterfactual credit assignment, because the learned trace can update plausible predecessor sequences that were not sampled on the current visit (Hasselt et al., 2020).

Not every extension that resembles fast/slow learning is literally dual-timescale. Enhanced-FQL$$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$5 introduces Fuzzified Eligibility Traces,

$$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$6

together with segmented experience replay over contiguous segments, but it explicitly does not define two separate trace processes or two decay rates (Jalaeian-Farimani, 7 Jan 2026). Its closest connection to dual-timescale thinking is conceptual: an online decaying trace is paired with a slower replay-based reconstruction process.

5. Boundary cases, non-equivalences, and recurring misconceptions

A recurrent source of confusion is the difference between eligibility-trace-like backward credit assignment and actual dual-timescale traces. Eligibility Propagation for Time Hopping is a structural replacement for traces in a non-sequential simulation framework. It records transitions in a directed state-transition graph and propagates Q-value changes backward through predecessor transitions using

$$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$7

continuing only when

$$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$8

The method is explicitly not dual-timescale: it introduces no multiple decay rates, no separate fast and slow traces, and no mixture of trace components (0904.0546).

A second misconception is to equate any $$e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)$$9-controlled bias-variance tradeoff with dual timescales. In LSTD$t>B(s,a)$0-RP, the trace

$t>B(s,a)$1

and the multi-step Bellman operator

$t>B(s,a)$2

produce a tradeoff between estimation and approximation error, but the method remains single-timescale. The paper explicitly states that its “dual” aspect is not a coupled two-timescale update scheme (Li et al., 2018).

A third distinction concerns two-timescale stochastic approximation with eligibility traces. In off-policy TDC$t>B(s,a)$3,

$t>B(s,a)$4

$t>B(s,a)$5

$t>B(s,a)$6

the fast/slow distinction applies to the auxiliary variable $t>B(s,a)$7 and the value parameter $t>B(s,a)$8, not to two eligibility traces (Mahadevan et al., 29 May 2026). The contribution is the first almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation under Markovian noise, not the introduction of fast and slow trace variables (Mahadevan et al., 29 May 2026).

These boundary cases show that “dual-timescale eligibility traces” should not be used as a blanket label for any trace-based, multi-step, or coupled-timescale RL method. The temporal locus of the separation matters: it may reside in the trace variables themselves, in the weight-update pipeline, in the neuromodulatory readout, or in the stochastic-approximation parameters.

6. Applications, empirical behavior, and unresolved design questions

Across applications, dual-timescale or generalized multi-timescale eligibility mechanisms are introduced to improve delayed credit assignment without sacrificing online adaptability. In online deep RL, multiple time-scale traces were designed specifically because standard accumulating traces retain outdated gradients under parameter drift, whereas replacing traces discard too much useful history; the proposed hierarchical method improved both sample efficiency and trained-policy return on InvertedPendulumBulletEnv-v0, InvertedPendulumSwingupBulletEnv-v0, HalfCheetahBulletEnv-v0, and AntBulletEnv-v0 (Kobayashi, 2020). In BCIs, explicit fast/slow traces support continual adaptation to neural instability while keeping memory $t>B(s,a)$9 in sequence length $B(s,a)$0, rather than $B(s,a)$1 as in BPTT (Nallani et al., 17 Sep 2025).

In biological and behavioral domains, the same design logic appears under different names. Human one-shot reinforcement at $B(s,a)$2 after a single reward implies that reward information reached decisions two steps away on a roughly 10-second timescale (Lehmann et al., 2017). NeoHebbian experiments indicate that eligibility flags must persist long enough for delayed modulatory signals, but not so long that irrelevant events are blended together (Gerstner et al., 2018). Cascading eligibility traces make this precision issue explicit: standard exponential traces “mix together any events that happen during the delay,” whereas higher-order cascades can tune their peak to behavioral or retrograde delays from milliseconds to minutes (Ralambomihanta et al., 17 Jun 2025).

Several unresolved issues recur across the literature. One is representational validity under parameter change: this is central in deep RL, where adaptive decay was introduced as a heuristic response to gradient divergence, and the paper explicitly notes that adaptive decaying had a minor impact and needs further theoretical development (Kobayashi, 2020). Another is the choice between exactly two timescales and richer families. CETs argue that two states are often too broad and that more states are needed for temporal precision (Ralambomihanta et al., 17 Jun 2025). A third issue is the relation between explicit and implicit dual-timescale mechanisms. Theta-sequence compression, expected traces, and three-factor plasticity all exhibit fast/slow structure without necessarily maintaining two trace variables (George, 2023, Hasselt et al., 2020, Gerstner et al., 2018). A plausible implication is that dual-timescale eligibility tracing is best understood as a design pattern for credit assignment rather than a single canonical algorithm.

Taken together, the literature supports a layered view. At the narrowest level are explicit fast/slow traces with separate decay constants and mixing or consolidation rules (Nallani et al., 17 Sep 2025). At an intermediate level are hierarchical or adaptive multi-timescale trace systems (Kobayashi, 2020, Ralambomihanta et al., 17 Jun 2025). At the broadest level are biologically and algorithmically related mechanisms in which a fast eligibility process is coupled to a slower readout, predictor, or consolidation dynamic (Lehmann et al., 2017, Gerstner et al., 2018, Hasselt et al., 2020). The unifying concern is the same throughout: preserving enough temporal specificity to solve delayed credit assignment while retaining enough memory depth to propagate useful learning signals backward in time.