Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning (2310.20587v5)

Abstract: Offline reinforcement learning (RL) aims to find a near-optimal policy using pre-collected datasets. In real-world scenarios, data collection could be costly and risky; therefore, offline RL becomes particularly challenging when the in-domain data is limited. Given recent advances in LLMs and their few-shot learning prowess, this paper introduces $\textbf{La}$nguage Models for $\textbf{Mo}$tion Control ($\textbf{LaMo}$), a general framework based on Decision Transformers to effectively use pre-trained LLMs (LMs) for offline RL. Our framework highlights four crucial components: (1) Initializing Decision Transformers with sequentially pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to full-weight fine-tuning, to combine the pre-trained knowledge from LMs and in-domain knowledge effectively, (3) using the non-linear MLP transformation instead of linear projections, to generate embeddings, and (4) integrating an auxiliary language prediction loss during fine-tuning to stabilize the LMs and retain their original abilities on languages. Empirical results indicate $\textbf{LaMo}$ achieves excellent performance in sparse-reward tasks and closes the gap between value-based offline RL methods and decision transformers in dense-reward tasks. In particular, our method demonstrates superior performance in scenarios with limited data samples.

Unleashing the Power of Pre-trained LLMs for Offline Reinforcement Learning

This paper addresses the challenging task of Offline Reinforcement Learning (RL), which involves developing an optimal policy using pre-collected datasets without the option of further data collection. The current methodology faces limitations, especially when in-domain data is scarce. To tackle these challenges, the authors propose a framework named LLMs for Motion Control (LaMo), which leverages pre-trained LLMs (LMs) in combination with Decision Transformers to enhance the performance in offline RL tasks.

The LaMo framework consists of four key components. First, it initializes Decision Transformers with sequentially pre-trained LMs. Second, it employs the Low-Rank Adaptation (LoRA) fine-tuning method. LoRA is highlighted as a less resource-intensive alternative in contrast to full-weight fine-tuning, allowing effective integration of pre-trained LM knowledge with domain-specific data. Third, it uses non-linear Multi-Layer Perceptrons (MLPs) instead of linear projections for embeddings, enhancing the representation learning capacity. Fourth, an auxiliary language prediction loss is integrated during fine-tuning, which stabilizes LMs and maintains their inherent language understanding abilities.

Empirical evaluations reveal that the LaMo framework achieves state-of-the-art performance in various sparse-reward tasks and narrows the gap between value-based offline RL methods and decision transformers in dense-reward tasks. Notably, LaMo displays superior performance under limited data scenarios, a testament to the few-shot learning capabilities inherited from the LMs. For instance, LaMo outperforms other approaches in the Kitchen and Atari domains, where rewards are either sparse or dense, making it particularly adept at handling diverse reward structures.

In terms of implications, the research indicates a promising direction for leveraging pre-trained LLMs beyond traditional NLP tasks and applying them to motion control problems in reinforcement learning. This could potentially ease the computational burden and data requirements by utilizing the few-shot learning capabilities of LMs, thereby broadening the applicability of RL techniques in real-world settings where data acquisition is costly or risky. The paper suggests that future work could explore utilizing larger LLMs or integrating more sophisticated prompt engineering techniques to further harness the language reasoning ability of these models.

The findings demonstrate the feasibility and benefits of cross-domain pre-training and adaptation, suggesting that the integration of LMs could be a fruitful line of inquiry in advancing RL methodologies. Furthermore, this cross-pollination of techniques from NLP to RL could drive new innovations and insights essential for tackling the complexities inherent in offline RL tasks.