Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View (2401.11237v2)

Abstract: Some reinforcement learning (RL) algorithms can stitch pieces of experience to solve a task never seen before during training. This oft-sought property is one of the few ways in which RL methods based on dynamic-programming differ from RL methods based on supervised-learning (SL). Yet, certain RL methods based on off-the-shelf SL algorithms achieve excellent results without an explicit mechanism for stitching; it remains unclear whether those methods forgo this important stitching property. This paper studies this question for the problems of achieving a target goal state and achieving a target return value. Our main result is to show that the stitching property corresponds to a form of combinatorial generalization: after training on a distribution of (state, goal) pairs, one would like to evaluate on (state, goal) pairs not seen together in the training data. Our analysis shows that this sort of generalization is different from i.i.d. generalization. This connection between stitching and generalisation reveals why we should not expect SL-based RL methods to perform stitching, even in the limit of large datasets and models. Based on this analysis, we construct new datasets to explicitly test for this property, revealing that SL-based methods lack this stitching property and hence fail to perform combinatorial generalization. Nonetheless, the connection between stitching and combinatorial generalisation also suggests a simple remedy for improving generalisation in SL: data augmentation. We propose a temporal data augmentation and demonstrate that adding it to SL-based methods enables them to successfully complete tasks not seen together during training. On a high level, this connection illustrates the importance of combinatorial generalization for data efficiency in time-series data beyond tasks beyond RL, like audio, video, or text.

• The paper demonstrates that SL-based RL methods struggle with inherent combinatorial generalization, limiting their stitching capabilities.
• Empirical results reveal that simply scaling data or models fails to achieve the necessary temporal relationship encoding.
• Temporal data augmentation is introduced as an effective strategy to boost generalization in both state-based and image-based tasks.

Introduction

Reinforcement learning (RL) algorithms have the remarkable ability to stitch together experiences to tackle new problems. This capability enables RL-based solutions to handle tasks never explicitly encountered during training, arguably a distinguishing feature when comparing RL to supervised learning (SL). The literature includes RL algorithms based on dynamic programming that have long utilized this stitching property, enabling superior data efficiency and off-policy reasoning. However, the use of SL-based methods in RL has blurred the lines, as certain outcome conditional behavioral cloning (OCBC) methods have shown impressive results in benchmarks without an apparent mechanism facilitating stitching. This paper critically examines the generalization capabilities of such SL-based RL algorithms within the contexts of reaching a target goal state and attaining a specified return value.

Combinatorial Generalization and Stitching

Central to the concept of stitching is the idea of combinatorial generalization, which characterizes an algorithm's ability to combine previously learned experiences to handle new (state, goal) pairs not jointly observed during training. This is akin to a person navigating to a new location by combining knowledge of how to reach a familiar intermediary point, such as a taxi stand, and from there traveling to the ultimate destination. Traditional dynamic programming-based RL methods exhibit this ability through their inherent structure. In contrast, the paper posits that OCBC methods' reliance on SL principles hinders them from inherently performing combinatorial generalization. This deficiency is analytically demonstrated, casting doubt on the ability of SL-based algorithms to match RL methods requiring such combinatorial reasoning, especially when operating with temporal sequences.

Empirical Validation

The authors create datasets to explicitly test combinatorial generalization, and the empirical results show that strategies like Decision Transformers (DT) and Reward-Value Surrogate (RvS) fail to exhibit the stitching property. The standard test suites, such as D4RL, were found inadequate for this purpose as they inadvertently included (state, goal) pairs within the training distribution, thus not requiring genuine stitching. Experiments with newly created environments confirm the theory that merely scaling up data volume or model architecture will not endow SL-based methods with combinatorial generalization abilities.

Temporal Data Augmentation

Considering the lack of combinatorial generalization in OCBC algorithms, the authors suggest a simple yet effective remedy: temporal data augmentation. This strategy augments existing training data to explicitly encode temporal relationships between states across trajectories. Using such data augmentation, OCBC algorithms can learn to navigate between unseen (state, goal) pairs. Theoretical insights accompany empirical demonstrations that this form of augmentation enhances the generalization capabilities of SL-based approaches substantially in both state-based and image-based tasks.

Conclusion and Future Work

This paper contributes significantly by redefining the narrative around the capability of SL-based RL algorithms to stitch experiences and by providing a practical solution to imbue them with this vital asset. The temporal data augmentation technique demonstrates a valuable stride toward achieving combinatorial generalization. However, a more natural inclusion of combinatorial generalization in SL-based algorithms, without the need for explicit temporal augmentation, remains an open and intriguing avenue for further research. The broader implications for time-series data processing tasks hint at the potential for significant advances in data efficiency across a variety of domains beyond RL.