On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (1511.09249v1)

Abstract: This paper addresses the general problem of reinforcement learning (RL) in partially observable environments. In 2013, our large RL recurrent neural networks (RNNs) learned from scratch to drive simulated cars from high-dimensional video input. However, real brains are more powerful in many ways. In particular, they learn a predictive model of their initially unknown environment, and somehow use it for abstract (e.g., hierarchical) planning and reasoning. Guided by algorithmic information theory, we describe RNN-based AIs (RNNAIs) designed to do the same. Such an RNNAI can be trained on never-ending sequences of tasks, some of them provided by the user, others invented by the RNNAI itself in a curious, playful fashion, to improve its RNN-based world model. Unlike our previous model-building RNN-based RL machines dating back to 1990, the RNNAI learns to actively query its model for abstract reasoning and planning and decision making, essentially "learning to think." The basic ideas of this report can be applied to many other cases where one RNN-like system exploits the algorithmic information content of another. They are taken from a grant proposal submitted in Fall 2014, and also explain concepts such as "mirror neurons." Experimental results will be described in separate papers.

• The paper introduces RNNAI, a framework that integrates RL controllers and RNN world models using principles from algorithmic information theory.
• It leverages predictive coding and intrinsic curiosity to continuously compress and improve sensory-motor interaction histories.
• The iterative training method of co-evolving the controller and model shows potential for advanced decision-making in complex, real-world tasks.

An Expert Overview of "On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models"

Author: Jürgen Schmidhuber

Introduction

The paper addresses the reinforcement learning (RL) problem in partially observable environments, where real-world conditions often necessitate general computational capabilities beyond traditional models. Schmidhuber et al. explore advanced combinations of reinforcement learning controllers (C) and recurrent neural network (RNN)-based world models (M), guided by principles from Algorithmic Information Theory (AIT).

Reinforcement Learning and Recurrent Neural Networks

Traditional RL agents must interact with dynamic environments to maximize cumulative reward signals without prior knowledge of the environment's specifics. Earlier frameworks utilizing artificial recurrent neural networks (RNNs) have advanced considerably. For instance, evolutionary algorithms have evolved to search neural network (NN) weights with success in certain benchmarks, such as driving simulated cars from high-dimensional video input. Nonetheless, this falls short in complexity as the RL and input space grow.

The Novel Framework: RNNAI

The paper presents an architecture termed RNNAI (Recurrent Neural Network Artificial Intelligence), where hybridized reinforcement learning controllers work with RNN-based world models to mimic how biological brains model environments for reasoning and planning. Here, RNNAI continuously improves its RNN-based world model through intrinsic curiosity and exploratory behaviour, signifying "learning to think."

Theoretical Foundation: Algorithmic Information Theory

Algorithmic Information Theory, which focuses on the complexity and compressibility of data, supports the methodology underlying RNNAI. The paper draws on principles such as mutual information between programs and their AIT-based optimality searches. These principles suggest that significant performance and learning efficiency gains can be achieved by compressing and exploiting shared information between the world model and the controller.

Detailed Components: Controller (C) and World Model (M)

The World Model (M):

• Predictive Coding: M is designed to learn a predictive model of the environment. This model is continuously trained by replaying experiences to minimize a combination of prediction error and information cost.
• Storage of Interaction History: All sensory-motor interaction history is stored to retrain and improve the world model continually. This mirrors AIT's compression-driven approach, emphasizing the importance of keeping comprehensive data for long-term learning efficacy.
• Compression Performance: M's performance is measured by how well it compresses the history through predictive coding. The model's size and complexity are dynamically adjusted to balance prediction accuracy and simplicity.

The Controller (C):

• Algorithmic Transfer Learning: C is tuned to leverage the information encapsulated in M. The controller's learning task is simplified by querying and interpreting M's accumulated knowledge.
• Learning Mechanisms: Several mechanisms for training C are discussed, from direct supervised training to more sophisticated evolutionary and policy gradient methods. It ensures that C can effectively utilize the dynamic and comprehensive input from M.

Implementation and Training

The training proceeds through alternating phases, where the controller is trained while the world model's weights are fixed, followed by updating the world model based on the latest experiences. This iterative method ensures both components co-evolve synergistically.

Practical Implications

The potential applications are vast, ranging from autonomous vehicles to robotic control systems and beyond. By continuously refining the model to predict and encode environmental interactions better, the system can develop superior problem-solving methods. Furthermore, incorporating intrinsic motivation for exploring and improving predictions aligns with how human learning and curiosity-driven exploration work.

Future Directions

Future AI systems could significantly benefit from adopting this framework, especially in environments where real-time predictions and decision-making are crucial. The integration of hierarchical planning and abstract reasoning could open new possibilities in autonomous systems capable of more sophisticated and adaptive behaviors.

Conclusion

This paper's contributions lie in marrying RL with sophisticated RNN-based world models under AIT principles to form a more generalized and efficient AI framework. By allowing controllers to exploit model-encoded knowledge meaningfully, the system mimics types of abstract thinking and planning observed in human cognition. Future developments in this domain hold the promise of creating AI capable of more nuanced and robust decision-making in complex real-world settings.