---
title: Dual-Timescale Eligibility Traces in RL
url: https://www.emergentmind.com/topics/dual-timescale-eligibility-traces
type: topic
---

# Dual-Timescale Eligibility Traces in RL

Searching arXiv for recent and foundational papers on dual-timescale or related eligibility trace mechanisms.
{"query":"eligibility traces dual timescale arXiv eligibility trace multiple timescale cascading expected traces two-timescale reinforcement learning", "max_results": 10, "sort_by": "relevance"}
Unable to access a dedicated tool interface in this message format, but proceeding using the provided arXiv records as the source corpus for the article and citing their arXiv IDs directly.
Dual-timescale eligibility traces denote eligibility-trace mechanisms in which temporal credit assignment depends on at least two coupled temporal processes, usually a fast transient memory of recent events and a slower process that stabilizes, consolidates, or reads out that memory. In the cited literature, the phrase has both a literal and a broader use. Literal uses appear in algorithms with explicit fast and slow trace variables, whereas broader uses include systems in which a short-lived eligibility window interacts with slower value learning, neuromodulatory gating, refractory recovery, replay-like reconstruction, or compressed sequence dynamics [2509.14447][1707.04192][1801.05219]. The common objective is to bridge a temporal gap between causal activity and delayed error, reward, or supervisory signals without reverting to purely one-step bootstrapping.

## 1. Canonical eligibility traces and the origin of timescale separation

In standard reinforcement learning, an eligibility trace is a decaying memory of recently visited state-action pairs. A discrete-time formulation used in sequential decision making sets
\[
e_{n}(s,a) = \left\{ \begin{array}{ll}
1 & \text{if } s=s_n, a=a_n \\
\gamma \lambda e_{n-1}(s,a) & \text{otherwise}
\end{array} \right.
\]
and updates values by
\[
Q(s, a) \leftarrow Q(s, a) + \alpha \cdot RPE(n) \cdot e_n(s, a).
\]
The current choice receives eligibility \(1\), earlier choices decay by \(\gamma \lambda\), and \(\lambda=0\) recovers the no-trace case [1707.04192].

A continuous-time formulation makes the temporal interpretation explicit:
\[
e_t(s,a) = \exp\left(-\frac{t - B(s,a)}{\tau}\right)
\]
for \(t>B(s,a)\), and zero otherwise, where \(B(s,a)\) is the last time the state-action pair was selected and \(\tau\) is the time constant of trace decay [1707.04192]. This establishes a fast memory process that can persist over seconds, while the learned values \(Q\) and \(V\) accumulate more slowly across repeated experience. The same paper interprets this division as a fast eligibility window versus a slower value-learning process, even though it does not define two separate trace constants [1707.04192].

In neoHebbian three-factor formulations, the separation is even clearer. The synapse is described by an internal eligibility variable \(e_{ij}\) and an observable weight \(w_{ij}\), with
\[
\frac{d}{dt} e_{ij} = \eta \, x_j \, g(y_i) - e_{ij}/\tau_e
\]
and
\[
\frac{d}{dt}{w_{ij} = e_{ij} \, M_{3rd}(t).
\]
Here pre- and postsynaptic activity write a transient synaptic flag, and a delayed third factor \(M_{3rd}(t)\) converts that flag into durable plasticity [1801.05219]. In this formulation, dual timescales are intrinsic: milliseconds to tens of milliseconds for local coincidence, and seconds to minutes for modulatory readout.

## 2. Explicit fast/slow eligibility-trace architectures

Some recent algorithms instantiate dual timescales directly by maintaining two eligibility traces per synapse. In an online spiking neural network decoder for brain-computer interfaces, the instantaneous three-factor Hebbian term is
\[
\Delta W^{(\ell)}_{\mathrm{hebb}(t) = \big(\tilde{\mathbf e}^{(\ell)}_t \odot d^{(\ell)}_t\big) \big(\mathrm{pre}^{(\ell)}_t\big)^\top,
\]
and it is accumulated into a fast trace and a slow trace:
\[
E^{(\ell)}_{\mathrm{fast}(t) = \lambda_{\mathrm{fast} E^{(\ell)}_{\mathrm{fast}(t-1) + \Delta W^{(\ell)}_{\mathrm{hebb}(t),
\]
\[
E^{(\ell)}_{\mathrm{slow}(t) = \lambda_{\mathrm{slow} E^{(\ell)}_{\mathrm{slow}(t-1) + \Delta W^{(\ell)}_{\mathrm{hebb}(t),
\]
with representative time constants \(\tau_{\mathrm{fast} \approx 120\,\text{ms}\) and \(\tau_{\mathrm{slow} \approx 700\,\text{ms}\). The two traces are mixed as
\[
E^{(\ell)}_{\mathrm{comb}(t) = \alpha_{\mathrm{mix} E^{(\ell)}_{\mathrm{fast}(t) + (1-\alpha_{\mathrm{mix}) E^{(\ell)}_{\mathrm{slow}(t),
\]
used for an immediate fast update, and then passed through a momentum-smoothed consolidation stream applied every \(K\) timesteps [2509.14447]. The stated role of the fast trace is rapid reaction to abrupt changes, whereas the slow trace preserves longer-term structure and stabilizes learning. Empirically, the method achieved comparable decoding accuracy with Pearson \(R \ge 0.63\) on Zenodo Indy and \(R \ge 0.81\) on MC Maze, with 28–35% memory reduction relative to BPTT-trained SNNs [2509.14447].

A different explicit construction appears in online deep reinforcement learning with adaptive and multiple time-scale traces. Standard accumulating traces are generalized to \(K\) traces \(e^i\), each associated with a different memory timescale, and updated by
\[
e_t^i = \begin{cases}
e_t^{i-1} & i \neq 1 \land \Delta e^i e_t^{i-1} > 0\\
\gamma \lambda_\mathrm{max}^i \lambda^d_{t,n} e_{t-1}^i + \beta^i g_t & \mathrm{otherwise}
\end{cases}
\]
with an adaptive decay factor
\[
\lambda^d_{t,n} = \exp\left(-\kappa d^s_{t,n}\right).
\]
The divergence state \(d^s_{t,n}\) is driven by policy and value-output divergences, so traces are decayed more aggressively when parameter drift makes old gradients unreliable [2008.10040]. The \(K=2\) case yields an explicit dual-timescale interpretation: short-term memory can selectively replace long-term memory, interpolating between standard accumulating traces and replacing traces. On four PyBullet Gym tasks, the proposed setting \((0.5, 0.9, 1.0)\) outperformed no-trace, standard-trace, replacing-trace, and adaptive single-trace baselines [2008.10040].

These two lines of work share a common design principle: a fast component improves responsiveness, while a slow component counteracts instability or forgetting. The difference is that the BCI formulation couples the traces to explicit consolidation, whereas the online DRL formulation emphasizes adaptive forgetting under parameter-dependent gradient divergence.

## 3. Biological and neurocomputational realizations

Behavioral evidence for a fast eligibility process interacting with slower value formation is provided by one-shot sequential learning in humans. In a multi-step task with six states plus a goal state, rewarded episode-1 experience changed episode-2 choices not only at \(D1\), one step from reward, but also at \(D2\), two steps from reward. The repeated action at \(D2\) occurred in 85% of cases, and a continuous-time trace fit yielded a characteristic time constant of roughly \(\tau \approx 10\) seconds, corresponding to about 2–3 inter-stimulus intervals [1707.04192]. The paper explicitly distinguishes this short-lived behavioral trace from the slower accumulation of long-term values.

Experimental synaptic plasticity studies reviewed in the neoHebbian framework describe a similar fast/slow split. Co-activation writes a metastable eligibility trace, and dopamine, norepinephrine, serotonin, acetylcholine, or plateau potentials convert that trace into weight change if they arrive before it decays [1801.05219]. The reviewed experiments support traces of around 1 second in striatum, about 3 seconds for LTD and 5–10 seconds for LTP in cortex, around 2 seconds in a hippocampal place-field paradigm, and minutes in a hippocampal tagging-like regime that the review interprets as more akin to synaptic tagging or consolidation than to fast reinforcement learning [1801.05219]. This literature treats dual timescales as a biological necessity: fast local coincidence is insufficient unless a slower broadcast factor can still access the synaptic flag.

In spiking-network learning, precise timing can require a second internal state beyond a simple decaying trace. For e-prop with the Izhikevich neuron, the hidden state
\[
s_j^t = \begin{pmatrix} v_j^t \\ u_j^t \end{pmatrix}
\]
induces a two-component eligibility vector in which the recovery variable \(u\) acts as an exponential filter of voltage eligibility. When the postsynaptic neuron spikes, the voltage eligibility is reset by a negative term involving the recovery eligibility, allowing negative eligibility for presynaptic spikes that arrive late relative to the postsynaptic spike [2006.09988]. The paper interprets this as the missing ingredient for full STDP-like timing dependence. A simplified STDP-LIF construction achieves a similar effect by imposing a negative pseudo-derivative during refractoriness [2006.09988]. In both cases, fast spike interactions coexist with a slower refractory or recovery process that determines whether later inputs potentiate or depress the synapse.

Theta-sequence models propose a different biological route to effective dual timescales. A short neuronal trace of order \(\mathsf{O}(10)\) ms is not lengthened directly; instead, behavior is compressed into theta cycles so that the effective eligibility window becomes
\[
\tau_z^{\mathrm{eff} = \kappa \tau_z,
\]
where \(\kappa = v_E/v_T\) is the ratio of sequence speed to behavioral speed [2305.08124]. The task-level horizon can therefore be seconds even when the biological trace remains around \(0.01\) s. This suggests that dual-timescale credit assignment need not require two explicit trace variables; it can also emerge from the interaction of a fast local trace with a slower behavioral trajectory represented at compressed neural timescale.

## 4. Generalizations beyond two traces

A major extension replaces a single exponential trace with a cascade of internal states. Cascading eligibility traces are defined by
\[
\dot h^{1}_t = -\alpha\, h^{1}_t + h_t,\qquad
\dot h^{k}_t = -\alpha\, h^{k}_t + h_t^{k-1},\qquad
\dot h^{\mathrm{CET}_t = -\alpha\, h^{\mathrm{CET}_t + h_t^{n-1},
\]
yielding the closed-form kernel
\[
h^{\mathrm{CET}_t = \frac{1}{(n-1)!}\int_{0}^t (t-s)^{n-1}e^{-\alpha\,(t-s)}\,h_s\, ds.
\]
The impulse response is gamma-like, \(g(t)\propto t^{n-1}e^{-\alpha t}\), and peaks at a desired delay \(T\) when \(\alpha=(n-1)/T\) [2506.14598]. A one-state system recovers the classical exponential trace, a two-state system is described as conceptually similar to dual-trace ideas, and higher-order cascades yield sharper temporal selectivity. On MNIST, standard traces remained effective up to about 2 seconds but degraded for 4s+, whereas higher-order CETs maintained high performance up to 10 seconds [2506.14598]. The paper therefore generalizes dual-timescale designs into a multi-timescale state-space framework.

Expected eligibility traces generalize the trace along a different axis. Instead of updating with the sampled backward-view trace \(e_t\), the method learns the state-conditional expectation
\[
z(s) \equiv \mathbb{E}\!\left[e_t \mid S_t=s\right],
\]
and updates values with
\[
\Delta w_t \equiv \alpha_t \,\delta_t\, z(S_t).
\]
A mixed trace
\[
y_t = (1 - \eta) z(S_t) + \eta \left(\gamma_t \lambda y_{t-1} + \nabla_w v_w(S_t)\right)
\]
interpolates between pure expected traces and standard TD\((\lambda)\), which is exactly recovered when \(\eta=1\) [2007.01839]. The paper does not present this as a dual-timescale trace method in name, but it has two coupled learning dynamics: a value learner with stepsize \(\alpha\) and a trace-predictor learner with stepsize \(\beta\). It also adds counterfactual credit assignment, because the learned trace can update plausible predecessor sequences that were not sampled on the current visit [2007.01839].

Not every extension that resembles fast/slow learning is literally dual-timescale. Enhanced-FQL\((\lambda)\) introduces Fuzzified Eligibility Traces,
\[
E_{i,j}(t) = \min \{ \gamma\lambda E_{i,j}(t-1) + \zeta_{i,j}(s_t,a_t) , 1\},
\]
together with segmented experience replay over contiguous segments, but it explicitly does not define two separate trace processes or two decay rates [2601.04392]. Its closest connection to dual-timescale thinking is conceptual: an online decaying trace is paired with a slower replay-based reconstruction process.

## 5. Boundary cases, non-equivalences, and recurring misconceptions

A recurrent source of confusion is the difference between eligibility-trace-like backward credit assignment and actual dual-timescale traces. Eligibility Propagation for Time Hopping is a structural replacement for traces in a non-sequential simulation framework. It records transitions in a directed state-transition graph and propagates Q-value changes backward through predecessor transitions using
\[
Q_{S_t,A} = R_{S_t,A} + \gamma \max \{ Q_{S_{t+1},A'} \},
\]
continuing only when
\[
|Q'_{\max} - Q_{\max}| > \varepsilon.
\]
The method is explicitly not dual-timescale: it introduces no multiple decay rates, no separate fast and slow traces, and no mixture of trace components [0904.0546].

A second misconception is to equate any \(\lambda\)-controlled bias-variance tradeoff with dual timescales. In LSTD\((\lambda)\)-RP, the trace
\[
z_i=\sum\nolimits_{k=1}^{i}(\lambda\gamma)^{i-k}\psi(X_k)
\]
and the multi-step Bellman operator
\[
T^\lambda =(1-\lambda)\sum\nolimits_{i=0}^{\infty}\lambda^iT^{i+1}
\]
produce a tradeoff between estimation and approximation error, but the method remains single-timescale. The paper explicitly states that its “dual” aspect is not a coupled two-timescale update scheme [1805.10005].

A third distinction concerns two-timescale stochastic approximation with eligibility traces. In off-policy TDC\((\lambda)\),
\[
e_t = \lambda \gamma \rho_{t-1} e_{t-1} + \phi_t,
\]
\[
\nu_{t+1} = \nu_t + \alpha_t \left(\rho_t \delta_t e_t - \phi_t \phi_t^\top \nu_t\right),
\]
\[
\theta_{t+1} = \theta_t + \beta_t\left(\rho_t \delta_t e_t - \rho_t(1-\lambda)\gamma \phi_{t+1} e_t^\top \nu_t\right),
\]
the fast/slow distinction applies to the auxiliary variable \(\nu_t\) and the value parameter \(\theta_t\), not to two eligibility traces [2605.31172]. The contribution is the first almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation under Markovian noise, not the introduction of fast and slow trace variables [2605.31172].

These boundary cases show that “dual-timescale eligibility traces” should not be used as a blanket label for any trace-based, multi-step, or coupled-timescale RL method. The temporal locus of the separation matters: it may reside in the trace variables themselves, in the weight-update pipeline, in the neuromodulatory readout, or in the stochastic-approximation parameters.

## 6. Applications, empirical behavior, and unresolved design questions

Across applications, dual-timescale or generalized multi-timescale eligibility mechanisms are introduced to improve delayed credit assignment without sacrificing online adaptability. In online deep RL, multiple time-scale traces were designed specifically because standard accumulating traces retain outdated gradients under parameter drift, whereas replacing traces discard too much useful history; the proposed hierarchical method improved both sample efficiency and trained-policy return on InvertedPendulumBulletEnv-v0, InvertedPendulumSwingupBulletEnv-v0, HalfCheetahBulletEnv-v0, and AntBulletEnv-v0 [2008.10040]. In BCIs, explicit fast/slow traces support continual adaptation to neural instability while keeping memory \(O(1)\) in sequence length \(T\), rather than \(O(T)\) as in BPTT [2509.14447].

In biological and behavioral domains, the same design logic appears under different names. Human one-shot reinforcement at \(D2\) after a single reward implies that reward information reached decisions two steps away on a roughly 10-second timescale [1707.04192]. NeoHebbian experiments indicate that eligibility flags must persist long enough for delayed modulatory signals, but not so long that irrelevant events are blended together [1801.05219]. Cascading eligibility traces make this precision issue explicit: standard exponential traces “mix together any events that happen during the delay,” whereas higher-order cascades can tune their peak to behavioral or retrograde delays from milliseconds to minutes [2506.14598].

Several unresolved issues recur across the literature. One is representational validity under parameter change: this is central in deep RL, where adaptive decay was introduced as a heuristic response to gradient divergence, and the paper explicitly notes that adaptive decaying had a minor impact and needs further theoretical development [2008.10040]. Another is the choice between exactly two timescales and richer families. CETs argue that two states are often too broad and that more states are needed for temporal precision [2506.14598]. A third issue is the relation between explicit and implicit dual-timescale mechanisms. Theta-sequence compression, expected traces, and three-factor plasticity all exhibit fast/slow structure without necessarily maintaining two trace variables [2305.08124][2007.01839][1801.05219]. A plausible implication is that dual-timescale eligibility tracing is best understood as a design pattern for credit assignment rather than a single canonical algorithm.

Taken together, the literature supports a layered view. At the narrowest level are explicit fast/slow traces with separate decay constants and mixing or consolidation rules [2509.14447]. At an intermediate level are hierarchical or adaptive multi-timescale trace systems [2008.10040][2506.14598]. At the broadest level are biologically and algorithmically related mechanisms in which a fast eligibility process is coupled to a slower readout, predictor, or consolidation dynamic [1707.04192][1801.05219][2007.01839]. The unifying concern is the same throughout: preserving enough temporal specificity to solve delayed credit assignment while retaining enough memory depth to propagate useful learning signals backward in time.

Source: https://www.emergentmind.com/topics/dual-timescale-eligibility-traces