---
title: 'Horizon-DQN: Long-Horizon RL for 2048'
url: https://www.emergentmind.com/topics/horizon-dqn-h-dqn
type: topic
---

# Horizon-DQN: Long-Horizon RL for 2048

Searching arXiv for recent papers on “Horizon-DQN” / “H-DQN” and adjacent uses of the acronym.
I’ll look up arXiv entries directly for “Horizon-DQN”, “h-DQN”, and “Deep Abstract Q-Networks”.
Horizon-DQN (H-DQN) is a long-horizon, value-based reinforcement-learning agent introduced for the game 2048 as a response to delayed and misleading reward signals. In this formulation, the central difficulty is not merely reward sparsity, but the fact that immediate merge rewards are often a poor proxy for the real objective of preserving board structure for future large merges. H-DQN is presented as a composite architecture that extends Rainbow-style design with categorical distributional learning, dueling networks, Double Q-learning, multi-step targets, NoisyNet exploration, prioritized replay, sequence-level prioritized replay, and a recurrent LSTM encoder with burn-in. Within the reported experiments, it is the strongest method among the compared DQN, PPO, and QR-DQN variants, reaching the 2048 tile at 5,000 episodes and the 4096 tile when trained longer [2507.05465].

## 1. Definition and problem setting

In the cited 2048 study, H-DQN is proposed specifically for **long-horizon credit assignment**. The motivating claim is that one-step temporal-difference learning is too myopic for 2048 because moves that produce small immediate merges can damage the board organization needed for much larger future returns. The method is therefore designed to model **long-horizon returns**, **rare but high-value trajectories**, and **temporal structure across action sequences**, rather than treating experience as isolated one-step transitions [2507.05465].

The paper does not present H-DQN as a fundamentally new RL primitive. Instead, it explicitly frames the method as a **“new composite architecture, Horizon DQN (H-DQN)”** targeted at long-horizon planning and built by integrating multiple established ingredients. This matters conceptually: the distinctive contribution lies less in a single novel Bellman operator than in the way recurrent state, distributional targets, prioritized sequence replay, and multi-step learning are assembled into a single agent for a delayed-reward combinatorial environment.

A further point of scope is that H-DQN in this sense is specific to the 2025 2048 paper. The acronym overlaps with other literatures that use similar names for substantially different ideas. That ambiguity is important because “H-DQN” can otherwise be mistaken for hierarchical DQN or for adaptive-horizon DQN variants, even though the 2048 method is neither of those in the standard sense.

## 2. Architectural composition

H-DQN begins from a board state \(s_t\), computes an embedding with an MLP torso, and then passes that embedding through an LSTM:

\[
e_t = \phi(s_t;\theta_{\mathrm{torso}}),\quad \phi:\mathbb{R}^{16}\to\mathbb{R}^{64},
\]

\[
h_t = \mathrm{LSTM}(e_t,\,h_{t-1};\theta_{\mathrm{RNN}}),
\]

with hidden state \(h_t \in \mathbb{R}^{64}\) [2507.05465]. The recurrent state then feeds a dueling decomposition,

\[
Q(h_t,a) = V(h_t) + \Bigl(A(h_t,a) - \tfrac{1}{|\mathcal A|}\!\sum_{a'}A(h_t,a')\Bigr),
\]

together with an action-conditioned categorical distribution,

\[
p_\theta(s_t,a) = [\,p^i_\theta(h_t,a)\,]_{i=1}^N,\;\sum_i p^i=1.
\]

This architecture combines three distinct inductive biases. The dueling split separates state value from action-specific advantage. The categorical head models a full return distribution rather than a scalar expectation. The recurrent encoder extends temporal context beyond the nominal \(n\)-step target, which the paper interprets as helpful for delayed credit assignment.

The paper is explicit that H-DQN uses **categorical distributional RL (C51)** rather than quantile regression. This distinguishes it from the paper’s QR-DQN baseline, which uses quantile regression with a convolutional encoder. H-DQN is therefore not “QR-DQN plus extras”; it is a different distributional construction with a categorical support [2507.05465].

Several implementation details remain unspecified. The paper does **not report** the exact number of atoms \(N\), the support bounds \((v_{\min},v_{\max})\), the exact projection implementation beyond mentioning an \(L^2\)-projection \(\Phi\), the number of MLP layers in the torso, the number of LSTM layers, the burn-in length, the exact \(n\)-step horizon, or the final values of PER exponents and NoisyNet initialization. These omissions constrain exact reproducibility.

## 3. Learning objective and replay mechanism

The paper defines the multi-step Double-Q target for each timestep \(t\) in a sampled subsequence of length \(L\) as

\[
R_t^{(n)} = \sum_{k=0}^{n-1}\gamma^k\,r_{t+k} + \gamma^n\,Q^-\bigl(s_{t+n}, \arg\max_{a'}Q(s_{t+n},a')\bigr).
\]

This target is projected onto a fixed categorical support via \(\Phi\), and the online network is trained with categorical cross-entropy:

\[
\mathcal{L}_t = -\sum_{i=1}^N \Phi\bigl(R_t^{(n)}\bigr)_i \,\log p^i_\theta(h_t,a_t).
\]

Thus, H-DQN combines three standard stabilizing devices in a single target: multi-step return propagation, Double-Q action selection/evaluation separation, and distributional prediction over fixed atoms [2507.05465].

Replay is also structurally richer than ordinary DQN replay. H-DQN uses **sequence-level prioritized replay**, storing and sampling subsequences rather than isolated transitions. The reported prioritization equations are

\[
p_i \propto |\delta_i|^\alpha,\quad \delta_i = R_t^{(n)} - \mathbb{E}[Z(s_t,a_t)],
\]

and importance weights of the usual prioritized-replay form,

\[
w_i \propto (N p_i)^{-\beta},
\]

with normalization by the maximum weight, although the exact denominator is not cleanly typeset in the paper. What is clear is that priority depends on sequence TD-error magnitude, not on single-step transition error.

Exploration uses NoisyNet rather than \(\varepsilon\)-greedy in H-DQN. The paper gives the factorized Gaussian parameterization

\[
W = \mu_W + \sigma_W \odot \epsilon_W,\quad \epsilon_W\sim\mathcal{N}(0,1),
\]

and states that every linear layer, including the torso, RNN gates, and dueling heads, uses factorized Gaussian noise [2507.05465]. This is intended to replace manual exploration scheduling with parameter-space stochasticity that adapts as learning proceeds.

## 4. Environment, inputs, and training protocol

The environment is Gym-2048 with the standard four actions: **up**, **down**, **left**, and **right**, so \(|\mathcal A|=4\). Each step yields score-based reward only when merges occur. The experimental setup also includes a **domain-independent monotonicity bonus** over rows and columns with coefficient

\[
\lambda_{\text{mono}} = 0.01,
\]

which is an important caveat because it means the training protocol is not purely free of auxiliary shaping [2507.05465].

A notable inconsistency concerns input representation. The general experimental section states that all agents consume a binary tensor \([16 \times 4 \times 4]\), where each channel indicates the presence of a \(2^i\) tile. By contrast, the H-DQN method section begins from the “raw \(4 \times 4\) board state” and defines an MLP torso \(\phi:\mathbb{R}^{16}\to\mathbb{R}^{64}\), which suggests a flattened 16-dimensional input. The paper does not resolve this discrepancy. The most careful reading is therefore that the exact H-DQN state encoding is reported inconsistently [2507.05465].

The globally reported training protocol includes:

- **Main training duration**: 5,000 episodes  
- **Scaled H-DQN**: 9,000 episodes  
- **Maximum episode length**: 10,000 steps  
- **Evaluation**: 1,000 episodes with minimal exploration  
- **Replay buffer size**: \(10^6\)  
- **Update frequency**: every 4 steps  

Hyperparameters were tuned with Optuna over 20 trials for learning rate, batch size, discount factor, target update rate, and \(\epsilon\)-decay rate, but the final selected values for H-DQN are not reported. Likewise, the paper does not report the number of random seeds for the main results [2507.05465].

## 5. Reported empirical results and strategic behavior

The paper reports a clear performance ordering among the four studied agents in 2048. At 5,000 episodes, H-DQN achieves the best average score, the best maximum score, and the highest tile reached [2507.05465].

| Agent | Avg score | Max score |
|---|---:|---:|
| DQN | 1,442.64 | 3,988 |
| PPO | 1,830.52 | 5,756 |
| QR-DQN | 3,478.51 | 8,660 |
| H-DQN | 5,693.67 | 18,210 |

At 5,000 episodes, the maximum tiles are 512 for DQN, 512 for PPO, 1024 for QR-DQN, and 2048 for H-DQN. When the same H-DQN is trained for 9,000 rather than 5,000 episodes, the reported average score rises to **6,536.43**, the maximum score to **41,828**, and the maximum tile to **4096** [2507.05465].

The tile-reaching statistics over 1,000 evaluation episodes further show a shift toward stronger outcomes with longer training. At 5,000 episodes, H-DQN reaches the 2048 tile in 89 episodes and never reaches 4096. At 9,000 episodes, it reaches the 2048 tile in 107 episodes and the 4096 tile in 24 episodes. This suggests that the scaled run improves not only the single best trajectory but also the overall tail of the outcome distribution.

The paper also emphasizes qualitative strategic changes. DQN is described as nearly uniform in action usage and prone to immediate reward chasing, whereas strong H-DQN agents develop a pronounced directional bias consistent with **corner-locking**. Reported move distributions are:

- **H-DQN, 5,000 episodes**: Left 9.47%, Right 52.15%, Up 36.27%, Down 2.11%
- **H-DQN, 9,000 episodes**: Left 54.73%, Right 2.56%, Up 2.60%, Down 40.11%

The preferred corner differs across runs, but the paper interprets the consistent emergence of strong directional asymmetry as evidence of a learned long-term board-organization strategy rather than local merge maximization [2507.05465].

## 6. Relation to other “H-DQN” usages, limitations, and open directions

The name H-DQN can be misleading because adjacent literature uses similar labels for substantially different ideas. In hierarchical RL, “h-DQN” usually denotes a meta-controller/controller architecture with intrinsic rewards and temporal abstraction. A related paper on Hierarchical Q-Networks explicitly states that its own HQN is **not** the canonical h-DQN structure and lacks subgoals, intrinsic motivation, options, and goal-conditioned subpolicies [1801.06689]. Horizon-DQN in 2048 is likewise not a hierarchical goal-selection method of that kind.

It is also distinct from DAQN, which addresses sparse rewards and long horizons through expert-provided state abstraction, abstract-action discovery, and a model-based high-level planner using R-Max over an abstract graph. DAQN shares the sparse-reward motivation but differs architecturally in using explicit symbolic abstraction and a hybrid model-based/model-free hierarchy rather than recurrent distributional sequence replay over raw task states [1710.00459].

A third nearby line is adaptive-horizon DQN. “Elastic Step DQN” adapts the effective multi-step backup horizon online by clustering hidden-state representations and extending a transition while cluster labels remain equal. That is a horizon-adaptive DQN in the sense of backup length selection, but it is not Horizon-DQN as defined in the 2048 work, and it is not hierarchical DQN either [2210.03325]. A plausible implication is that “H-DQN” should always be disambiguated by paper and domain, because acronym overlap masks deep differences in objective, architecture, and temporal semantics.

The 2048 paper is candid about H-DQN’s limitations. It reports a tightly coupled hyperparameter space, noting that small changes in the PER exponent \(\alpha\) or NoisyNet \(\sigma\) initialization could change peak performance by about \(\pm 15\%\). It also reports substantial compute overhead: sequence-level replay and recurrence required an **8-env batched R2D2 rollout**, step throughput was about **1.7x slower** than standard Rainbow, and memory usage scaled linearly with subsequence length \(L\). Generalization beyond 2048 remains open, and the paper acknowledges that auxiliary heuristics such as monotonicity bonuses, corner-building biases, wall/gap formation metrics, empty-tile counts, and score normalization can help but also add more hyperparameters and may destabilize the recurrent encoder [2507.05465].

The authors propose several future directions: **model-based planning / Dyna-style rollouts**, **curriculum learning**, **meta-gradient hyperparameter adaptation**, **distributed actor-learner scaling** such as Ape-X or IMPALA, **PopArt normalization**, **transfer / hierarchical methods**, and more efficient simulation via the Madrona engine [2507.05465]. This suggests that H-DQN is best understood not as a final resolution of long-horizon credit assignment, but as a strong model-free recurrent distributional baseline for delayed-reward combinatorial domains.

Source: https://www.emergentmind.com/topics/horizon-dqn-h-dqn