An investigation of model-free planning (1901.03559v2)

Abstract: The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More recently, a new family of methods have been proposed that learn how to plan, by providing the structure for planning via an inductive bias in the function approximator (such as a tree structured neural network), trained end-to-end by a model-free RL algorithm. In this paper, we go even further, and demonstrate empirically that an entirely model-free approach, without special structure beyond standard neural network components such as convolutional networks and LSTMs, can learn to exhibit many of the characteristics typically associated with a model-based planner. We measure our agent's effectiveness at planning in terms of its ability to generalize across a combinatorial and irreversible state space, its data efficiency, and its ability to utilize additional thinking time. We find that our agent has many of the characteristics that one might expect to find in a planning algorithm. Furthermore, it exceeds the state-of-the-art in challenging combinatorial domains such as Sokoban and outperforms other model-free approaches that utilize strong inductive biases toward planning.

• The paper introduces the Deep Repeated ConvLSTM architecture that enables model-free RL agents to exhibit effective planning behavior.
• Experiments in combinatorial environments like Sokoban and Boxworld show that the model-free approach can match or surpass traditional planning methods.
• The study highlights notable data efficiency and performance improvements with increased computation, offering scalable planning without explicit models.

An Examination of Model-Free Planning Approaches

The paper "An Investigation of Model-Free Planning" by Guez et al. presents an empirical exploration into the capabilities of model-free reinforcement learning (RL) agents to demonstrate planning behavior without relying on explicit environmental models or specialized planning architectures. This paper extends the contemporary understanding of planning in RL, traditionally considered a domain demanding model-based techniques, into the field of model-free methodologies that utilize standard neural network components such as convolutional layers and LSTMs.

Core Contributions and Findings

The authors propose a model-free RL approach that achieves significant planning capabilities through a deep neural architecture, referred to as the Deep Repeated ConvLSTM (DRC) network. This architecture is designed to learn implicitly, without prescribing any specific planning inductive bias beyond those inherent in convolutional networks and recurrence mechanisms. Through extensive experimentation across various combinatorial domains, the paper demonstrates that these model-free architectures can match or surpass the performance of model-based and strongly planning-biased model-free approaches.

Key facets of the evaluation include:

• Generalization Across Combinatorial Domains: The presented DRC network is designed to handle massively combinatorial spaces such as Sokoban and Boxworld, showcasing its ability to generalize across different procedural environments. The network surpasses prior specialist approaches like VINs and model-free planners with additional structural biases towards planning.
• Data Efficiency: The DRC architecture demonstrates data efficiency, maintaining competitive performance with limited training data. This positions it as a flexible, yet robust alternative to model-based strategies that often rely on extensive training datasets for model learning.
• Scalability with Increased Computation: One of the distinguishing features of the DRC network is its capability to improve performance with additional computation time, a haLLMark of effective planning algorithms. This aspect is evidenced by enhanced performance with prolonged deliberation periods during testing.

Methodological Insights

The paper's methodological advancements are grounded in the rich capacities of neural network function approximators. The DRC architecture is premised on convolutional LSTMs, arranged in a stack-and-repeat configuration, allowing the network to amortize planning computations iteratively. Key enhancements like pooling-and-injection mechanisms and top-down skip connections further augment the network's planning proficiency.

Implications for AI Development

This investigation underscores the potential of model-free strategies to incorporate implicit planning capabilities. The findings suggest pathways for designing RL systems that:

• Offer greater flexibility by avoiding rigid model assumptions, thereby broadening the applicability to diverse, dynamic environments.
• Facilitate efficient learning and decision-making when provided with only the environment interactions, aligning with the demands of increasingly complex interactive systems.

Future Developments

Looking forward, the trajectory of this research area may encompass exploring hybrid approaches that further blend model-free intuition with model-based reliability, enhancing explainability and predictability in agent actions. Additionally, the investigation into more general representations that allow RL agents to operate effectively across a broader spectrum of tasks, including those resembling real-world complexities, represents a significant avenue for continued exploration.

In conclusion, this paper challenges and refines the boundaries of model-free RL models' capabilities, advocating for a reconsideration of the definitions and requirements of planning within artificial agents, and illuminating the potential roads for future advancements in generalized AI planning systems.