Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizon (2009.13503v2)

Abstract: Episodic reinforcement learning and contextual bandits are two widely studied sequential decision-making problems. Episodic reinforcement learning generalizes contextual bandits and is often perceived to be more difficult due to long planning horizon and unknown state-dependent transitions. The current paper shows that the long planning horizon and the unknown state-dependent transitions (at most) pose little additional difficulty on sample complexity. We consider the episodic reinforcement learning with $S$ states, $A$ actions, planning horizon $H$, total reward bounded by $1$, and the agent plays for $K$ episodes. We propose a new algorithm, \textbf{M}onotonic \textbf{V}alue \textbf{P}ropagation (MVP), which relies on a new Bernstein-type bonus. Compared to existing bonus constructions, the new bonus is tighter since it is based on a well-designed monotonic value function. In particular, the \emph{constants} in the bonus should be subtly setting to ensure optimism and monotonicity. We show MVP enjoys an $O\left(\left(\sqrt{SAK} + S^2A\right) \poly\log \left(SAHK\right)\right)$ regret, approaching the $\Omega\left(\sqrt{SAK}\right)$ lower bound of \emph{contextual bandits} up to logarithmic terms. Notably, this result 1) \emph{exponentially} improves the state-of-the-art polynomial-time algorithms by Dann et al. [2019] and Zanette et al. [2019] in terms of the dependency on $H$, and 2) \emph{exponentially} improves the running time in [Wang et al. 2020] and significantly improves the dependency on $S$, $A$ and $K$ in sample complexity.

• The paper introduces the Monotonic Value Propagation (MVP) algorithm, demonstrating that episodic reinforcement learning can achieve sample complexity comparable to contextual bandits.
• MVP achieves exponential improvements in sample complexity regarding the planning horizon H dependency compared to state-of-the-art algorithms.
• The findings challenge the belief that episodic RL is inherently harder than bandits, showing it can achieve similar sample efficiency with the MVP algorithm.

Overview of "Is Reinforcement Learning More Difficult Than Bandits?" by Zhang, Ji, and Du

In the paper titled "Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizon," the authors Zihan Zhang, Xiangyang Ji, and Simon S. Du address a fundamental question in the domain of sequential decision-making: Does episodic reinforcement learning (RL) inherently require more samples than contextual bandits (CB) due to the additional complexities of long planning horizons and state-dependent transitions?

Context and Contribution

Episodic reinforcement learning involves decision-making over a sequence of steps, characterized by an uncertain environment with the dual challenges of a planning horizon and unknown transitions. In contrast, contextual bandits can be seen as a particular case of RL where the planning horizon is reduced to a single step. While RL is generally perceived to be more complex, the authors argue that the long horizon and unknown transitions may not significantly increase the sample complexity in terms of learning efficiency.

The primary contribution of this work is the introduction of a new algorithm called Monotonic Value Propagation (MVP). This algorithm significantly improves existing solutions by achieving an $$O\left(\left(\sqrt{SAK} + S^2A\right) \text{polylog}\left(SAHK\right)\right)$$ regret bound. Remarkably, this performance approaches the lower bound of contextual bandits, $\Omega\left(\sqrt{SAK}\right)$, barring logarithmic terms. This represents a substantial advancement over previous algorithms concerning their dependency on the planning horizon $H$.

Methodology

MVP employs a novel approach through a Bernstein-type bonus integrated into its exploration-exploitation strategy. The algorithm distinguishes itself from prior work by ensuring that the exploration bonus incorporates a strong monotonic property that propagates optimism effectively across the planning horizon. This is accomplished with a more refined set of constants in the bonus function, which includes only necessary terms, thereby simplifying the optimism analysis and achieving computational efficiency throughout the learning process.

Results and Implications

MVP's performance demonstrates two significant advances:

1. It achieves exponential improvements in sample complexity, especially in terms of the dependency on the planning horizon $H$, compared to state-of-the-art polynomial-time algorithms by Dann et al. (2019) and Zanette et al. (2019).
2. It exponentially enhances the running time of the method described by Wang et al. (2020) while also significantly improving dependencies on $S$, $A$, and $K$.

These improvements indicate that episodic reinforcement learning can be as sample-efficient as contextual bandits, challenging the prevailing conjectures about the inherent complexity due to planning horizons and state-dependent transitions. This positions the MVP algorithm as a critical tool for both theoretical exploration and practical application of RL in environments previously considered challenging due to extensive planning requirements.

Future Directions

One of the unresolved questions remains the removal of the additive $S^2A$ term in the sample complexity bounds. Addressing this could further harmonize the complexity of RL with that of CB, potentially aligning their lower bounds precisely.

Furthermore, potential applications of the techniques developed for MVP, such as monotonic value propagation and novel Bernstein-type bonuses, could extend to other reinforcement learning frameworks, including those without stationary rewards or those involving non-tabular environments.

Conclusion

This paper presents a pivotal step toward understanding the fundamental comparative complexities of RL and CB and offers a near-optimal algorithm that potentially redefines sample efficiency expectations in reinforcement learning. By challenging entrenched beliefs in the field, the work encourages a re-evaluation of algorithmic design for long-horizon decision-making tasks.