Optimistic Q-learning for average reward and episodic reinforcement learning (2407.13743v3)

Abstract: We present an optimistic Q-learning algorithm for regret minimization in average reward reinforcement learning under an additional assumption on the underlying MDP that for all policies, the time to visit some frequent state $s_0$ is finite and upper bounded by $H$, either in expectation or with constant probability. Our setting strictly generalizes the episodic setting and is significantly less restrictive than the assumption of bounded hitting time \textit{for all states} made by most previous literature on model-free algorithms in average reward settings. We demonstrate a regret bound of $\tilde{O}(H^5 S\sqrt{AT})$, where $S$ and $A$ are the numbers of states and actions, and $T$ is the horizon. A key technical novelty of our work is the introduction of an $\overline{L}$ operator defined as $\overline{L} v = \frac{1}{H} \sum_{h=1}^H L^h v$ where $L$ denotes the BeLLMan operator. Under the given assumption, we show that the $\overline{L}$ operator has a strict contraction (in span) even in the average-reward setting where the discount factor is $1$. Our algorithm design uses ideas from episodic Q-learning to estimate and apply this operator iteratively. Thus, we provide a unified view of regret minimization in episodic and non-episodic settings, which may be of independent interest.

• The paper introduces an optimistic Q-learning algorithm that achieves a regret bound of Ō(H^5 S√(AT)) under average reward settings.
• It develops a novel operator with strict span contraction, unifying episodic and average reward reinforcement learning frameworks.
• The findings enhance practical applications in robotics and gaming by relaxing state-hit time assumptions in model-free RL.

Optimistic Q-learning for Average Reward and Episodic Reinforcement Learning

The paper "Optimistic Q-learning for average reward and episodic reinforcement learning" by Priyank Agrawal and Shipra Agrawal introduces an optimistic Q-learning algorithm tailored for the average reward reinforcement learning while also providing applicability to episodic settings. The paper assumes an underlying MDP where for all policies, the expected time to visit some frequent state $s_0$ is finite and upper bounded by $H$, thus generalizing episodic settings but imposing a less restrictive assumption than bounded hitting times for all states.

The authors present significant theoretical advancements and numerical results in the model-free reinforcement learning (RL) domain. Specifically, they propose a regret bound of $\tilde{O}(H^5 S \sqrt{AT})$ for average reward settings, where $S$ is the number of states, $A$ is the number of actions, and $T$ is the time horizon. Notably, this regret bound does not require uniformly bounding the hitting times or mixing times for all states under any policy, as assumed in much prior literature. Instead, it relies on the bounded expected return time to a frequently visited state. This transition reflects a pragmatic shift by considering conditions more likely met in practical applications such as repeated robotic tasks or game runs.

Key Contributions and Technical Innovation

The core technical contribution is the introduction of a novel operator $\overline{L}$ defined as: $\overline{L} v = \frac{1}{H} \sum_{h=1}^H L^h v$ where $L$ denotes the BeLLMan operator with a discount factor of 1. The authors demonstrate that this operator exhibits strict contraction properties (in span) in average reward settings under the given assumption. This insight represents a pivotal point in bridging the characteristics of episodic and average reward settings.

The algorithm builds upon the optimistic Q-learning paradigm commonly applied in episodic RL. However, it extends this concept by incorporating iterative estimations of the $\overline{L}$ operator to adhere to an average reward framework. Importantly, the algorithm includes an update mechanism for $V^{H+1}$ ensuring iterative convergence to $V^*$ (the optimal bias vector), leveraging the span contraction property.

Implications and Future Directions

Practical Implications

The findings imply substantial improvements for RL applications that need to operate without explicit knowledge of the MDP's transition model. For instance:

1. Robotics: Robotic systems with varying task lengths can now better accommodate tasks that do not conform to uniform episode durations.
2. Gaming: Algorithms for AI players in games can adapt to scenarios where game lengths vary significantly, optimizing for average performance over extended play periods.

Theoretical Implications

1. Unified View: The algorithm provides a comprehensive framework that unifies episodic and average reward settings, potentially simplifying the analysis and development of RL algorithms applicable across these settings.
2. Improved Bounds: The regret bound of $\tilde{O}(H^5 S\sqrt{AT})$ in average reward settings signifies a notable leap, especially since it bypasses more stringent assumptions made in previous works, thus broadening the applicability of model-free RL methods.

Future Research Directions

The results prompt several exciting research avenues:

1. Further Reducing Complexity: Improved methods to further reduce the dependency on $H$ in the regret bounds, leveraging recent advancements in episodic Q-learning algorithms.
2. Adaptative Parameters: Investigations into dynamic adjustment of parameters such as $H$ and $p$ based on observed state visitation frequencies could yield more efficient algorithms.
3. Deep RL Implementation: Extending these theoretical foundations into deep RL frameworks, potentially impacting hierarchical learning architectures and meta-learning models that operate over heterogeneous task distributions.

Overall, this paper underscores a pivotal stride in model-free reinforcement learning by harmonizing the treatment of episodic and average reward settings. It positions the research community to revisit several standard assumptions and encourages adoption in more naturally occurring variable task-length environments, driving practical advancements alongside theoretical ones in the reinforcement learning domain.