A Natural Extension To Online Algorithms For Hybrid RL With Limited Coverage (2403.09701v2)
Abstract: Hybrid Reinforcement Learning (RL), leveraging both online and offline data, has garnered recent interest, yet research on its provable benefits remains sparse. Additionally, many existing hybrid RL algorithms (Song et al., 2023; Nakamoto et al., 2023; Amortila et al., 2024) impose coverage assumptions on the offline dataset, but we show that this is unnecessary. A well-designed online algorithm should "fill in the gaps" in the offline dataset, exploring states and actions that the behavior policy did not explore. Unlike previous approaches that focus on estimating the offline data distribution to guide online exploration (Li et al., 2023b), we show that a natural extension to standard optimistic online algorithms -- warm-starting them by including the offline dataset in the experience replay buffer -- achieves similar provable gains from hybrid data even when the offline dataset does not have single-policy concentrability. We accomplish this by partitioning the state-action space into two, bounding the regret on each partition through an offline and an online complexity measure, and showing that the regret of this hybrid RL algorithm can be characterized by the best partition -- despite the algorithm not knowing the partition itself. As an example, we propose DISC-GOLF, a modification of an existing optimistic online algorithm with general function approximation called GOLF used in Jin et al. (2021); Xie et al. (2022a), and show that it demonstrates provable gains over both online-only and offline-only reinforcement learning, with competitive bounds when specialized to the tabular, linear and block MDP cases. Numerical simulations further validate our theory that hybrid data facilitates more efficient exploration, supporting the potential of hybrid RL in various scenarios.
- Harnessing density ratios for online reinforcement learning.
- Auer, P. (2003). Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res., 3(null):397–422.
- Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21.
- Minimax regret bounds for reinforcement learning.
- Robust fitted-q-evaluation and iteration under sequentially exogenous unobserved confounders.
- Adversarially trained actor critic for offline reinforcement learning.
- Better exploration with optimistic actor-critic.
- pymdptoolbox. https://github.com/sawcordwell/pymdptoolbox.
- On oracle-efficient pac rl with rich observations.
- Provably efficient rl with rich observations via latent state decoding.
- Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms.
- Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137–2143. PMLR.
- Offline policy evaluation and optimization under confounding.
- Conservative q-learning for offline reinforcement learning.
- Is q-learning minimax optimal? a tight sample complexity analysis.
- Reward-agnostic fine-tuning: Provable statistical benefits of hybrid reinforcement learning. arXiv preprint arXiv:2305.10282.
- Provably good batch reinforcement learning without great exploration.
- Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes.
- Kinematic state abstraction and provably efficient rich-observation reinforcement learning.
- Tactical optimism and pessimism for deep reinforcement learning.
- Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.
- Toward the fundamental limits of imitation learning.
- Bridging offline reinforcement learning and imitation learning: A tale of pessimism.
- Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity.
- Hybrid rl: Using both offline and online data can make rl efficient.
- Pessimistic model-based offline reinforcement learning under partial coverage.
- Leveraging offline data in online reinforcement learning.
- Provably efficient causal reinforcement learning with confounded observational data.
- Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683–6694.
- The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157.
- Policy finetuning: Bridging sample-efficient offline and online reinforcement learning.
- Zanette, A. (2023). When is realizability sufficient for off-policy reinforcement learning?
- Learning near optimal policies with low inherent bellman error.
- Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730–2775. PMLR.
- Nearly minimax optimal reinforcement learning for linear mixture markov decision processes.