Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias (2310.08558v1)

Abstract: It is desirable for policies to optimistically explore new states and behaviors during online reinforcement learning (RL) or fine-tuning, especially when prior offline data does not provide enough state coverage. However, exploration bonuses can bias the learned policy, and our experiments find that naive, yet standard use of such bonuses can fail to recover a performant policy. Concurrently, pessimistic training in offline RL has enabled recovery of performant policies from static datasets. Can we leverage offline RL to recover better policies from online interaction? We make a simple observation that a policy can be trained from scratch on all interaction data with pessimistic objectives, thereby decoupling the policies used for data collection and for evaluation. Specifically, we propose offline retraining, a policy extraction step at the end of online fine-tuning in our Offline-to-Online-to-Offline (OOO) framework for reinforcement learning (RL). An optimistic (exploration) policy is used to interact with the environment, and a separate pessimistic (exploitation) policy is trained on all the observed data for evaluation. Such decoupling can reduce any bias from online interaction (intrinsic rewards, primacy bias) in the evaluation policy, and can allow more exploratory behaviors during online interaction which in turn can generate better data for exploitation. OOO is complementary to several offline-to-online RL and online RL methods, and improves their average performance by 14% to 26% in our fine-tuning experiments, achieves state-of-the-art performance on several environments in the D4RL benchmarks, and improves online RL performance by 165% on two OpenAI gym environments. Further, OOO can enable fine-tuning from incomplete offline datasets where prior methods can fail to recover a performant policy. Implementation: https://github.com/MaxSobolMark/OOO

An Analysis of Offline Retraining in Online Reinforcement Learning: The OOO Framework

The paper presents a comprehensive paper on the interaction of offline data with online reinforcement learning (RL), introducing a novel framework termed Offline-to-Online-to-Offline (OOO) reinforcement learning. The central aim of this research is to address the biases introduced by exploration bonuses during online RL, particularly when the available offline data do not offer adequate state coverage, compelling the need for aggressive exploration.

Decoupling Exploration and Exploitation

The core insight of the OOO framework is to decouple the policies used during the data collection phase from those used during evaluation. Conventionally, RL systems utilize exploration bonuses to encourage agents to visit novel states, which indeed enhances coverage but can also bias the learned policies. Often, such exploration-driven policies fail to optimize for the task reward. By contrast, OOO introduces a dual-policy mechanism where a distinct policy is optimized post-interaction using a pessimistic offline RL approach on the accumulated data, thereby mitigating biases from exploration-focused policies.

Methodology and Implementation

The paper employs a detailed two-step process within the OOO framework:

1. Exploration Phase: An optimistic exploration policy interacts with the environment, driven by rewards that combine task-specific goals and exploration bonuses. This phase aims to broaden the state exploration and maximize the novelty-seeking behavior of the agent.
2. Exploitation Phase & Offline Retraining: Following the data collection, a separate exploitation policy is trained on all observed data using a pessimistic, exploitation-centric objective. This allows the policy to focus purely on task-specific rewards, potentially recovering a policy that achieves higher task performance than one continually optimized on both intrinsic and extrinsic rewards.

Empirical Contributions

The research extensively evaluates the OOO framework across a diverse set of benchmarks, including tasks requiring significant state coverage and hard exploration, such as robotic manipulation tasks from the D4RL suite and sparse-reward locomotion in OpenAI gym environments. The empirical results demonstrate substantial improvements, with marked performance gains over traditional offline-to-online algorithms, notably boosting the performance of base methods like Implicit Q-Learning (IQL) and Calibrated Q-Learning (Cal-QL).

Strong numerical endorsements are underscored by improvements in performance, such as a 165% enhancement in goal-reaching tasks over specific baselines. The exploitation policy derived through offline retraining frequently outperforms even the most exploration-optimized policies, underscoring the efficacy of decoupling exploration from exploitation.

Practical and Theoretical Implications

The findings presented in the paper have profound implications for enhancing RL systems, particularly in scenarios with limited offline data coverage and expensive data acquisition environments like healthcare and robotics. The OOO framework provides a powerful tool to refine policies leveraging both exploration and exploitation, setting a precedent for future RL algorithm designs that should strategically consider policy decoupling mechanisms.

Theoretically, the framework challenges prevailing paradigms in RL by advocating for separate policy optimization tracks, raising potential future inquiries into exploration-exploitation trade-offs and offline policy evaluation strategies.

Conclusion

The paper positions itself as a critical paper in the field of RL, emphasizing offline retraining's role in correcting biases introduced during exploration. The OOO framework's adoption could spearhead more robust, efficient RL models that navigate the complexities intrinsic to environments demanding both extensive exploration and precise exploitation. Future research might explore further synergy and integration of more sophisticated exploration bonuses within the OOO structure, as well as analyze its application to broader RL challenges.