Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning? (2005.00527v2)

Abstract: Learning to plan for long horizons is a central challenge in episodic reinforcement learning problems. A fundamental question is to understand how the difficulty of the problem scales as the horizon increases. Here the natural measure of sample complexity is a normalized one: we are interested in the number of episodes it takes to provably discover a policy whose value is $\varepsilon$ near to that of the optimal value, where the value is measured by the normalized cumulative reward in each episode. In a COLT 2018 open problem, Jiang and Agarwal conjectured that, for tabular, episodic reinforcement learning problems, there exists a sample complexity lower bound which exhibits a polynomial dependence on the horizon -- a conjecture which is consistent with all known sample complexity upper bounds. This work refutes this conjecture, proving that tabular, episodic reinforcement learning is possible with a sample complexity that scales only logarithmically with the planning horizon. In other words, when the values are appropriately normalized (to lie in the unit interval), this results shows that long horizon RL is no more difficult than short horizon RL, at least in a minimax sense. Our analysis introduces two ideas: (i) the construction of an $\varepsilon$-net for optimal policies whose log-covering number scales only logarithmically with the planning horizon, and (ii) the Online Trajectory Synthesis algorithm, which adaptively evaluates all policies in a given policy class using sample complexity that scales with the log-covering number of the given policy class. Both may be of independent interest.

An Analysis of Horizon Length in Reinforcement Learning

This paper addresses a significant question within the field of reinforcement learning (RL): What effect does the length of the planning horizon have on the sample complexity necessary to achieve near-optimal policies? The authors challenge the conjecture proposed by Jiang and Agarwal, which suggested that longer horizons should inherently increase sample complexity by a polynomial factor. Instead, they present findings that reveal a sample complexity scaling logarithmically with the horizon length, effectively asserting that long horizon RL poses no greater difficulty than short horizon RL, at least in the minimax sense.

Key Contributions

1. Sample Complexity Insight: The paper introduces a rigorous examination of sample complexity, emphasizing episodic, tabular RL scenarios where reward functions need only be normalized within the unit interval. The groundbreaking claim here is that sample complexity scales only logarithmically with respect to the horizon length $H$, contrary to prior conjectures suggesting polynomial scaling according to established upper bounds.
2. Algorithmic Innovations:

The authors advance two algorithmic concepts: - The formation of an $\varepsilon$-net for optimal policies that surprisingly scales logarithmically with $H$. - The introduction of the Online Trajectory Synthesis algorithm facilitating adaptive evaluations across a comprehensive policy class, closely tied to the log-covering number instead of direct horizon length.

Implications and Future Directions

• Theoretical Implications:

These results have profound theoretical implications as they suggest the complexity of achieving optimal policies is not strictly dependent on horizon length. The authors conjecture that their sample complexity findings could represent a minimax optimal scenario, paving the way for future explorations into whether similar scaling might be achievable for broader RL settings, including those with continuous states or actions.

• Practical Applications:

Practically, this opens up possibilities for RL applications where long horizon planning is crucial but where sample efficiency has traditionally been a bottleneck. Improved sample efficiency denotes potential reductions in computational requirements and energy expenditures in real-world applications, notably impacting domains such as robotics and autonomous systems.

• Further Research:

The paper prompts further inquiries into sample complexity bounds concerning other RL configurations, particularly those involving more sophisticated domains or reward structures. Future research may verify or refute the authors' predictions regarding an optimal logarithmic sample complexity across varied RL settings and algorithmic strategies.

Conclusion

The findings elucidated in this paper signify a pivotal step toward a nuanced understanding of horizon length’s effect on reinforcement learning's sample complexity. By refuting established conjectures, the authors challenge prevailing assumptions, urging the AI research community to reevaluate and explore new horizons in policy optimization strategies.