Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities

Abstract: Off-policy, value-based reinforcement learning methods such as Q-learning are appealing because they can learn from arbitrary experience, including data collected by older policies or other agents. In practice, however, bootstrapping makes long-horizon learning brittle: estimation errors at later states propagate backward through temporal-difference (TD) updates and can compound over time. We propose long-horizon Q-learning (LQL), which introduces a principled backstop against compounding error when learning the optimal action-value function. LQL builds on a prior optimality tightening observation: any realized action sequence lower-bounds what the optimal policy can achieve in expectation, so acting optimally earlier should not be worse than following the observed actions for several steps before switching to optimal behavior. Our contribution is to turn this inequality into a practical stabilization mechanism for Q-learning by using a hinge loss to penalize violations of these bounds. Importantly, LQL computes these penalties using network outputs already produced for the TD error, requiring no auxiliary networks and no additional forward passes relative to Q-learning. When combined with multiple state-of-the-art methods on a range of online and offline-to-online benchmarks, LQL consistently outperforms both 1-step TD and n-step TD learning at similar runtime.

• The paper introduces Long-Horizon Q-Learning (LQL) which uses trajectory-wise optimality inequalities as hinge penalties to constrain value overestimation in off-policy RL.
• Experimental results show LQL consistently outperforms 1-step and n-step TD methods, achieving up to 76% success on challenging long-horizon tasks.
• The approach offers computational efficiency with minimal overhead while providing stable Q-value estimation in sparse-reward and high-dimensional environments.

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities

Introduction and Motivation

Off-policy value-based reinforcement learning (RL) methods, especially Q-learning and its deep variants, face significant challenges when scaling to long-horizon tasks. The fundamental problem arises from the reliance on bootstrapping, where estimation errors in later states are propagated backward and can amplify across temporal-difference (TD) updates. While conventional strategies, such as using $n$-step TD and $\lambda$-returns, seek to mitigate compounding error by leveraging multi-step targets, these approaches inherit an innate tension: in off-policy contexts, the target value necessarily follows the (possibly suboptimal) actions logged in the replay buffer, resulting in pessimistic estimates when suboptimal actions are present. Furthermore, increasing $n$ introduces substantial off-policy bias and sample inefficiency.

This work introduces Long-Horizon Q-Learning (LQL), a method that integrates trajectory-based optimality inequalities as a practical and computationally efficient backstop for off-policy RL. LQL supplements the standard TD error with hinge penalties based on a trajectory-wise optimality tightening principle, providing a direct and low-overhead mechanism to constrain value overestimation and propagate reward signal robustly across extended time horizons.

Algorithmic Formulation

LQL relies on the following insight: for any given state, acting optimally from the current moment must always yield performance at least as good as committing to any arbitrary future action sequence before switching to optimal behavior. This leads to a set of trajectory-wise optimality inequalities that can be instantiated as two-sided hinge penalties in the Q-function objective:

• Lower-Bound (LB) Penalty: For any time indices $k, \ell$ with $\ell > k$, LQL penalizes cases where $$Q(s_k, a_k) < G_{k:\ell} + \gamma^{\ell-k} Q(s_\ell, a_\ell^*)$$, pushing value estimates upward if the cumulative reward along the observed trajectory plus the bootstrapped value at step $\ell$ exceeds the estimate at step $k$.
• Upper-Bound (UB) Penalty: For any $i \leq k$, LQL enforces $$Q(s_i, a_i^*) \geq G_{i:k} + \gamma^{k-i} Q(s_k, a_k)$$, penalizing over-optimistic Q-values that would suggest exceeding the performance of the greedy policy from any previous point.

A key strength of LQL is its implementation efficiency: unlike prior optimality tightening approaches, no auxiliary networks or additional forward passes are required. All penalties utilize the same critic and target network computations as standard TD learning.

(Figure 1)

Figure 1: LQL establishes a principled backstop; compounding TD error in conventional methods is bounded by cross-temporal constraints.

The combined objective for each replayed trajectory is: $\lambda$0 where $\lambda$1 is the 1-step TD loss, and $\lambda$2, $\lambda$3 are the mean squared hinge penalties over all suitable $\lambda$4 or $\lambda$5 index pairs. Empirically, default penalty coefficients ($\lambda$6) are used without per-task tuning.

Empirical Evaluation

LQL is comprehensively evaluated on the OGBench and RoboMimic benchmarks, with experiments spanning manipulation, navigation, and long-horizon sparse-reward domains, including difficult settings like humanoid locomotion through giant mazes (episode lengths $\lambda$7 steps) and offline-to-online RL under policy behavioral heterogeneity.

Policy Extraction Protocols

To ensure that LQL's benefit is not coupled to a particular actor or extraction mechanism, four diverse policy extraction approaches are employed:

• Gaussian actor (SAC-style)
• Best-of-$\lambda$8 with flow-matching behavioral cloning
• Flow Q-learning (FQL)
• Action chunking (QC-FQL)

For each, the underlying critic is trained with either 1-step TD, $\lambda$9-step TD, or LQL.

Results Overview

Across all policy architectures and environments, LQL consistently outperforms both 1-step TD and $n$0-step TD. Particularly on long-horizon sparse-reward tasks, LQL produces substantial gains, e.g., on the humanoidmaze-giant task, LQL achieves ~$n$1 average success where 1-step TD fails completely, and $n$2-step TD peaks at 38.4% for $n$3 before degrading due to off-policy bias at larger $n$4.

The runtime overhead of LQL is minimal: empirically measured at only 4-5% per update across policy classes, with further amortization as network size increases.

Value Stability and Error Attribution

Ablation studies disentangle the source of LQL's benefit, showing that the improvement is not explained by trajectory minibatching alone—removing the hinge losses annihilates the gains.

Figure 2: LQL's gains are not explained by trajectory sampling alone; hinge penalties yield statistically significant improvements in success rates across environments and batch sizes.

Additionally, LQL stabilizes the learned Q-function, preventing the classic value blowup often observed in function approximation settings under TD learning. Hinge penalties enforce analytic bounds on Q-values, suppressing overestimation.

Analysis and Theoretical Considerations

From a theoretical viewpoint, while the optimality inequalities used by LQL are strict in-expectation results, per-sample hinge penalties in stochastic domains introduce some bias in the presence of stochastic dynamics. The paper establishes horizon-independent upper bounds on this bias (Appendix, Section~\ref{app:false_penalties}) and demonstrates empirically that LQL retains its advantage in both deterministic and stochastic settings, with robustness to penalty magnitude hyperparameters.

Furthermore, the trajectory length $n$5 is revealed as a practical scaling axis. Unlike batch size (where larger batches may harm performance due to overfitting to inaccurate targets), increasing $n$6 in LQL admits strictly better final performance across tasks, provided sufficient computational budget.

Comparison to Prior Optimality Tightening

LQL extends the optimality tightening work of He et al. (He et al., 2016) by directly incorporating actor policy actions in the penalty computation, avoiding redundant critic evaluations and allowing for same-state upper bounds. Empirically, this design outperforms prior OT methods across continuous control tasks, pointing to more efficient and stable optimization.

Implications and Future Directions

LQL provides a viable, computationally attractive mechanism for leveraging long-horizon structure in off-policy RL. Its ability to propagate learning signal across thousands of timesteps without suffering from the batch scaling pathologies of TD enables new performance regimes in sparse-reward and offline–to-online settings, and its design is broadly compatible with modern actor architectures, including diffusion and chunked policies.

The main limitation remains on the theoretical side: per-sample hinge penalties induce an irreducible bias under strong environment stochasticity. Potential advances could include hybridizing LQL with $n$7-step TD learning to combine accelerated reward propagation with cross-temporal value regularization, and further dissecting the relative contributions of LB and UB penalties to value stability.

Conclusion

Long-Horizon Q-Learning offers a robust, scalable approach for addressing compounding error in off-policy, value-based RL. By operationalizing trajectory-wise Bellman inequalities as hinge penalties, LQL advances the stability and efficacy of RL in long-horizon and heterogeneous-data regimes. These results clarify the bottlenecks of conventional multi-step learning and chart promising directions for scaling off-policy RL to real-world domains with challenging temporal credit assignment structure.