Elastic Step DQN (ES-DQN)

• Elastic Step DQN is an adaptive multi-step reinforcement learning algorithm that adjusts the bootstrapping horizon based on state similarity or value stability.
• It employs unsupervised clustering and state-value differences to determine optimal n-step returns, thereby mitigating overestimation bias.
• Empirical results demonstrate that ES-DQN enhances sample efficiency and training stability across various environments compared to traditional DQN variants.

Elastic Step DQN (ES-DQN) is an adaptive multi-step extension of Deep Q-Networks that dynamically determines the bootstrapping horizon based on state similarity or value stability, with the aim of reducing overestimation bias and improving training stability in deep reinforcement learning. In contrast to conventional n-step DQN, which propagates value updates over a fixed number of steps, ES-DQN automatically adjusts the effective n-step horizon for each transition, leveraging either unsupervised clustering of neural activations or running value differences. This adaptive approach results in empirically superior sample efficiency, reduced overestimation bias, and improved training performance relative to standard DQN baselines and heuristic multi-step variants (Ly et al., 2022, Ly et al., 6 Jun 2025, Badran et al., 2020).

1. Core Update Mechanism and Relation to Multi-step DQN

ES-DQN generalizes the standard 1-step DQN and fixed n-step DQN by employing a dynamic, transition-specific bootstrapping horizon $n_t$ determined by state similarity measures. For each sampled transition $(s_t, a_t)$, the target is constructed as an $n_t$-step return: $$R_t^{(n_t)} = \sum_{k=0}^{n_t-1} \gamma^k r_{t+k+1} + \gamma^{n_t} \max_{a'} Q_T(s_{t+n_t}, a'; \theta^-)$$ The loss function is the squared TD-error: $$L(\theta) = \frac{1}{2} \big(y_t - Q_P(s_t, a_t; \theta)\big)^2$$ where $y_t = R_t^{(n_t)}$. If $n_t \equiv 1$, the rule reduces to standard DQN; if $n_t$ is fixed, the rule is equivalent to n-step DQN (Ly et al., 2022).

2. Adaptive Horizon Selection via State Similarity

ES-DQN uses unsupervised clustering (HDBSCAN) on a state representation $\phi(s)$—typically the last hidden layer of the Q-network—to assign each state to a cluster $C(\phi(s_t))$. For a rollout starting at $(s_t, a_t)$0, the elastic horizon $(s_t, a_t)$1 is extended as long as the agent remains in states assigned to the same cluster: $(s_t, a_t)$2 If a transition leads to a state in a different cluster or episode termination, the elastic segment is truncated. This method allows longer, lower-variance multi-step returns in locally coherent regions of state space and restricts bootstrapping depth when the agent transitions to a new dynamical regime, reducing risk of compounding approximation errors (Ly et al., 2022).

An alternative mechanism employed in Ensemble Elastic Step DQN (EEDQN) (Ly et al., 6 Jun 2025) uses a state-value difference threshold to decide when to cut the segment: the difference $(s_t, a_t)$3 must remain below an adaptive buffer-based threshold. This achieves similar adaptivity by segmenting on local value stability.

3. Algorithmic Implementation and Pseudocode

The ES-DQN learning loop comprises several components:

• Transition sampling and neural feature extraction
• Online clustering of state representations
• Elastic horizon determination via cluster membership (or state-value difference)
• Storage of elastic-segment transitions in experience replay
• Bootstrapped target computation and Q-network optimization
• Periodic target network synchronization

A compact pseudocode for the cluster-based ES-DQN update is as follows:

$$R_t^{(n_t)} = \sum_{k=0}^{n_t-1} \gamma^k r_{t+k+1} + \gamma^{n_t} \max_{a'} Q_T(s_{t+n_t}, a'; \theta^-)$$5 (Ly et al., 2022)

In EEDQN, the elastic-segmentation is based on buffered state-value differences and ensemble-aggregation in the target (Ly et al., 6 Jun 2025).

4. Overestimation Bias and Theoretical Intuitions

Conventional DQN is prone to overestimation bias due to the repeated use of the max-operator in bootstrapped updates. Multi-step returns can reduce this bias but increase sensitivity to mis-specified $(s_t, a_t)$4 and state mismatch, as error can compound across transitions. ES-DQN’s elastic horizon exploits state-local consistency—detected through clustering or value stability—to allow safe extension of bootstrapping depth, limiting updates to transitions within “coherent” state regions. This adaptivity mitigates error accumulation from abrupt state changes and reduces the tendency for Q-values to become optimistically biased, as empirically evidenced by lower observed $(s_t, a_t)$5 and negligible positive return–$(s_t, a_t)$6 correlations (Ly et al., 2022).

5. Experimental Methodology

Experiments evaluate ES-DQN and comparators in tabular and function-approximation regimes:

• Environments: CartPole-v1, Acrobot-v1, MountainCar-v0 (OpenAI Gym) (Ly et al., 2022); MinAtar (Breakout, Freeway, Asterix, SpaceInvaders, Seaquest) (Ly et al., 6 Jun 2025).
• Baselines: 1-step DQN, fixed n-step DQN ($(s_t, a_t)$7 ∈ {2,3,4,6,8}), Double DQN, Average DQN, MaxMin DQN, Ensemble Elastic Step DQN.
• Metrics: mean episodic return (final $(s_t, a_t)$8 epochs), episodic Q-value estimates ($(s_t, a_t)$9), overestimation threshold crossing ($n_t$0), Spearman $n_t$1, and statistical significance (t- or permutation tests).
• Hyperparameters: replay buffer $n_t$2, batch 64, $n_t$3, clusterer HDBSCAN (min_cluster_size=30, leaf_size=10), re-cluster every 1000 steps, $n_t$4-greedy annealed to minimum 0.1, learning rates in $n_t$5 (Ly et al., 2022, Ly et al., 6 Jun 2025).

6. Empirical Findings

Key results demonstrate:

• Superior final returns: ES-DQN achieves the best or statistically on-par mean returns in CartPole, Acrobot, and MountainCar, with significant margin over DQN, Double DQN, Average DQN, and all fixed-$n_t$6 baselines in several settings (Ly et al., 2022).
• Reduced overestimation: In MountainCar, average/max $n_t$7 for ES-DQN is $n_t$8 versus $n_t$9 for standard DQN. Spearman correlation $$R_t^{(n_t)} = \sum_{k=0}^{n_t-1} \gamma^k r_{t+k+1} + \gamma^{n_t} \max_{a'} Q_T(s_{t+n_t}, a'; \theta^-)$$0 is $$R_t^{(n_t)} = \sum_{k=0}^{n_t-1} \gamma^k r_{t+k+1} + \gamma^{n_t} \max_{a'} Q_T(s_{t+n_t}, a'; \theta^-)$$1 for ES-DQN (no positive link), but positive for all other methods.
• Stability: Training curves exhibit steady, low-variance convergence, avoiding $$R_t^{(n_t)} = \sum_{k=0}^{n_t-1} \gamma^k r_{t+k+1} + \gamma^{n_t} \max_{a'} Q_T(s_{t+n_t}, a'; \theta^-)$$2 spikes or divergence at high $$R_t^{(n_t)} = \sum_{k=0}^{n_t-1} \gamma^k r_{t+k+1} + \gamma^{n_t} \max_{a'} Q_T(s_{t+n_t}, a'; \theta^-)$$3, with tight confidence bands across 30 random seeds.
• Ablation and robustness: In EEDQN on MinAtar, combining elastic-step with an ensemble (min-aggregation for multi-step, mean for single-step) yields robust, state-of-the-art performance across four out of five games, with quantitative superiority over DQN, DDQN, (Average DQN), MaxMin, and fixed n-step methods (Ly et al., 6 Jun 2025).
• Clustering validity: Cluster visualization confirms that HDBSCAN produces semantically meaningful state partitions (e.g., distinct dynamical regimes in MountainCar), validating the adaptivity rationale.

Summary of final mean returns for ES-DQN vs. major baselines (absolute value, ± standard deviation):

Environment	ES-DQN	DQN	Double DQN	Average DQN
CartPole	413.4 ± 112.2	100.7 ± 29.6	225.7 ± 166.6	183.6 ± 38.2
Acrobot	–94.5 ± 9.9	–107.6 ± 89.4	–103.3	–100.8 ± 31.4
MountainCar	–123.4 ± 19.8	–156.8 ± 26.5	–162.8 ± 55.6	–153.9 ± 21.7

(Ly et al., 2022)

7. Extensions: Ensemble ES-DQN and Adaptive Synchronization

Ensemble Elastic Step DQN (EEDQN) (Ly et al., 6 Jun 2025) integrates the elastic-step mechanism with a small ensemble of Q-networks, leveraging min-aggregation over the ensemble on multi-step targets and mean aggregation on single-step targets. This combination further depresses overoptimistic multi-step updates while preventing excessive pessimism in one-step regions, yielding the highest or second-highest mean returns in MinAtar environments with robust control of overestimation (normalized max-$$R_t^{(n_t)} = \sum_{k=0}^{n_t-1} \gamma^k r_{t+k+1} + \gamma^{n_t} \max_{a'} Q_T(s_{t+n_t}, a'; \theta^-)$$4 always below 1.0).

Separately, an orthogonal adaptive synchronization rule (Badran et al., 2020) replaces rigid, periodic target-network updates with a simple criterion based on weighted reward averages in two sliding windows. The target network is only synchronized when agent performance meaningfully degrades, validated in Atari-2600 experiments where final scores increased by ~40% on average over standard DQN and Rainbow.

• "Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks" (Ly et al., 2022)
• "Ensemble Elastic DQN: A novel multi-step ensemble approach to address overestimation in deep value-based reinforcement learning" (Ly et al., 6 Jun 2025)
• "An adaptive synchronization approach for weights of deep reinforcement learning" (Badran et al., 2020)