Is Value Learning Really the Main Bottleneck in Offline RL? (2406.09329v2)

Abstract: While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance of offline RL. Motivated by this observation, we aim to understand the bottlenecks in current offline RL algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: is the main bottleneck of offline RL indeed in learning the value function, or something else? To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, we find that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL, often more so than the value learning objective. For instance, we show that common value-weighted behavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test-time policy improvement methods and show that these methods lead to better performance.

Is Value Learning Really the Main Bottleneck in Offline RL?

The paper "Is Value Learning Really the Main Bottleneck in Offline RL?" by Park et al. investigates the key factors limiting the performance of offline reinforcement learning (RL) algorithms. While the common consensus has been that the primary bottleneck in offline RL is due to the challenges associated with accurately learning the value function from suboptimal data, this paper aims to challenge this conventional view by conducting a comprehensive analysis of the bottlenecks in offline RL systems.

Objectives and Scope

The primary objective of this paper is to determine whether the main limitation in offline RL algorithms is indeed the value learning process or if other factors contribute more significantly to their underperformance. To this end, the authors conduct a systematic empirical analysis focusing on three components of offline RL:

1. Value Learning: The accuracy of value function estimation.
2. Policy Extraction: The effectiveness of extracting a policy from the learned value function.
3. Policy Generalization: The ability of the policy to generalize to states encountered during deployment but not seen during training.

Key Observations

The authors' analysis leads to two key observations that challenge the conventional belief focusing solely on value learning improvements:

1. Policy Extraction Algorithm: The choice of a policy extraction algorithm significantly impacts the performance of offline RL, often more than the value learning objective. For instance, value-weighted behavioral cloning methods such as Advantage-Weighted Regression (AWR) fail to fully leverage the learned value function compared to behavior-constrained policy gradient methods such as DDPG+BC. It was observed that switching to behavior-constrained policy gradients leads to substantial improvements in both performance and scalability.
2. Policy Generalization: Imperfect policy generalization on out-of-support states during test time is often a more substantial bottleneck than policy learning on in-distribution states. The paper shows that policy generalization issues can be mitigated in practice by using suboptimal but high-coverage data or by employing test-time policy training techniques.

Empirical Setup and Results

The authors evaluated various value learning and policy extraction methods using multiple datasets across diverse environments. This extensive empirical paper provides robust evidence for their claims:

1. Decoupled Value Learning Algorithms: SARSA, IQL, and CRL were examined to isolate the components of value function learning from policy extraction.
2. Policy Extraction Techniques: The authors compared the performance of AWR, DDPG+BC, and Sampling-based Action Selection (SfBC) across several tasks.

The data-scaling matrices in the experiments indicated that policy extraction mechanisms, notably DDPG+BC, often had a more significant impact on performance than the specific value learning algorithm used. Furthermore, analysis of policy generalization revealed that offline RL methods are effective on in-distribution states but struggle to generalize to out-of-distribution test-time states.

Practical Implications and Recommendations

The findings suggest actionable recommendations for improving offline RL:

1. Policy Extraction:
   • Avoid value-weighted behavioral cloning like AWR; instead, use behavior-constrained policy gradient methods like DDPG+BC for better performance and scalability.
2. Data Collection:
   • Prioritize collecting high-coverage datasets even if they are suboptimal, as this improves test-time policy accuracy.
3. Test-Time Policy Improvement:
   • Utilize simple test-time policy improvement strategies such as on-the-fly policy extraction (OPEX) and test-time training (TTT) to further distill value function information into the policy.

Future Directions

This research emphasizes two critical avenues for future work in offline RL:

1. Improved Policy Extraction Algorithms: Developing methods that better leverage learned value functions while ensuring effective policy updates during learning.
2. Policy Generalization: Focusing on strategies to enhance the ability of policies to generalize to states encountered at test time, which differs from the existing emphasis on value function pessimism.

The paper represents a significant step in understanding the intrinsic bottlenecks in offline RL and provides a roadmap for both researchers and practitioners to enhance the performance and applicability of offline RL algorithms.