RTFM: Generalising to Novel Environment Dynamics via Reading (1910.08210v6)

Abstract: Obtaining policies that can generalise to new environments in reinforcement learning is challenging. In this work, we demonstrate that language understanding via a reading policy learner is a promising vehicle for generalisation to new environments. We propose a grounded policy learning problem, Read to Fight Monsters (RTFM), in which the agent must jointly reason over a language goal, relevant dynamics described in a document, and environment observations. We procedurally generate environment dynamics and corresponding language descriptions of the dynamics, such that agents must read to understand new environment dynamics instead of memorising any particular information. In addition, we propose txt2$\pi$, a model that captures three-way interactions between the goal, document, and observations. On RTFM, txt2$\pi$ generalises to new environments with dynamics not seen during training via reading. Furthermore, our model outperforms baselines such as FiLM and language-conditioned CNNs on RTFM. Through curriculum learning, txt2$\pi$ produces policies that excel on complex RTFM tasks requiring several reasoning and coreference steps.

• The paper introduces the RTFM problem and a novel FiLM^2 model designed to improve RL generalization to novel environment dynamics using language understanding.
• Experiments show the FiLM^2 model outperforms baselines on the RTFM problem, achieving strong generalization to novel environment dynamics via deeper text and visual integration.
• These findings highlight language comprehension as a powerful tool for improving RL policy generalization and suggest future research directions in integrating external documentation and goals.

Generalising to Novel Environment Dynamics via Reading

In this paper, the authors present a novel approach to address the challenges of generalizing reinforcement learning (RL) policies to novel environments through language understanding. They introduce a new framework called "Read to Fight Monsters" (RTFM), which underscores the importance of language comprehension in reinforcement learning. The paper proposes both a problem and a model, which aim to enhance the ability of RL agents to adapt to changes in environment dynamics—a common impediment in the deployment of RL solutions to real-world problems.

Problem Definition: RTFM

The RTFM problem is meticulously designed to test the capacity of RL agents to generalize using language. It involves environments with procedurally generated dynamics and corresponding textual descriptions that agents must understand in order to devise appropriate strategies. Agents are tasked with achieving goals in these environments by deciphering relevant information from documents, language goals, and observations—each of which changes dynamically. The procedural generation ensures a vast combinatorial space, making memorization infeasible and requiring genuine comprehension and generalization.

Proposed Model:

To tackle the challenges posed by the RTFM problem, the authors introduce a model that utilizes Bidirectional Feature-wise Linear Modulation (FiLM$^2$). This model constructs representations capturing interactions between the language goals, documents describing environment dynamics, and observations. FiLM$^2$ extends conventional FiLM techniques by enabling bi-directional modulations, thereby fostering deeper integration of text and visual inputs. This is a crucial enhancement for dynamically understanding the environment.

Experimental Evaluation

The paper reports strong numerical results showcasing the superiority of the proposed model over existing baselines, such as FiLM and language-conditioned convolutional neural networks (CNNs). The authors conduct rigorous testing across different variants of the RTFM problem, emphasizing adaptability to unseen environment dynamics. The model shows exemplary generalization capabilities, maintaining high performance in evaluative settings even when faced with environment dynamics not seen during training.

Through curriculum learning—a strategy where models are initially trained on simpler tasks before progressing to complex ones—the model achieves competent performance on complex RTFM tasks. These tasks often require multiple reasoning and coreference steps, underscoring the model's robustness and capacity to scale to intricate language understanding requirements.

Implications and Future Work

The findings highlight language comprehension as a formidable tool for enhancing RL policy generalization. The ability of the model to outperform baselines while generalizing to novel dynamics underlines its potential applications in scenarios where RL must adaptively process information from diverse sources. As suggested by the authors, future research could explore hierarchical decision-making or plan induction using language. Such directions promise improvements in language-grounded policy learning and underscore the need for better models that incorporate external documentation and language goals.

In summary, this paper introduces a competitive framework for studying language-conditioned reinforcement learning, providing compelling evidence for language's role in enabling adaptive policies in the face of changing environments.