Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (1509.08731v1)

Abstract: The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents. Most learning algorithms that involve optimisation of the mutual information rely on the Blahut-Arimoto algorithm --- an enumerative algorithm with exponential complexity that is not suitable for modern machine learning applications. This paper provides a new approach for scalable optimisation of the mutual information by merging techniques from variational inference and deep learning. We develop our approach by focusing on the problem of intrinsically-motivated learning, where the mutual information forms the definition of a well-known internal drive known as empowerment. Using a variational lower bound on the mutual information, combined with convolutional networks for handling visual input streams, we develop a stochastic optimisation algorithm that allows for scalable information maximisation and empowerment-based reasoning directly from pixels to actions.

• The paper presents the SVIM algorithm, which efficiently computes mutual information using a variational lower bound to drive intrinsic motivation.
• The authors validate the method in dynamic mazelike environments, demonstrating scalable empowerment computation compared to traditional approaches.
• The study shows that integrating variational inference with deep learning enables agents to extract actionable insights from high-dimensional sensory data.

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

In the presented paper, Shakir Mohamed and Danilo J. Rezende introduce a novel approach to optimizing mutual information for applications in machine learning, particularly focusing on its utility in intrinsically motivated reinforcement learning (IMRL). The work addresses the computational and scalability limitations of existing mutual information estimation methods, such as the Blahut-Arimoto algorithm, by integrating techniques from variational inference and deep learning.

The primary contribution of the paper is the development of a stochastic variational information maximization (SVIM) algorithm. SVIM enables the efficient computation of mutual information by leveraging a variational lower bound. This approach diverges from traditional methods, providing a significant reduction in computational complexity and making it applicable across high-dimensional sensory input domains.

Key Contributions and Findings

1. Algorithm Development: The authors present a stochastic optimization algorithm that combines variational information optimization with deep learning. The algorithm effectively scales mutual information estimation for IMRL, bypassing the exponential complexity present in classical methods such as Blahut-Arimoto.
2. Empowerment Framework: Within the reinforcement learning context, the paper introduces the concept of "empowerment" — an intrinsic reward signal derived from mutual information. Empowerment quantifies the effect an agent's actions have on future states, guiding the agent towards states with maximal potential influence.
3. Scalability and Efficiency: The proposed methods are extensively evaluated in mazelike environments, including dynamic settings with flowing lava or predator scenarios. Results demonstrate that empowerment-based agents can learn meaningful behaviors by maximizing mutual information directly from sensory input like images, as opposed to fully resolving transition dynamics.
4. Numerical Results: The SVIM algorithm is validated against exact computations of empowerment in small, discrete environments. The findings confirm that the variational lower bound approach can reproduce the empowerment landscape accurately, leading to comparable agent behavior to exact methods while offering improved scalability.

The implications of these findings are manifold. Practically, the ability to compute empowerment from pixel data opens up new possibilities for autonomous agents to explore and learn effectively in complex, reward-sparse environments. Theoretically, the work hints at the potential of extending information-theoretic intrinsic motivation to more comprehensive decision-making frameworks in AI, such as robotics and adaptive systems, where direct human intervention is minimal or non-existent.

Future Directions

This paper sets a stage for several future explorations. One avenue is the refinement of intrinsic motivation measures and their integration with other advanced reinforcement learning models, potentially incorporating elements of hierarchical planning or multi-agent environments. Furthermore, understanding the behavior policy selection, particularly in aligning intrinsic motivations with long-term mastery of environments, remains an open area.

The presented work is a substantive step forward in applying information theory to AI, highlighting the potential of mutual information as a cornerstone in overcoming the intrinsic drive problem faced by autonomous agents. Overall, it offers not only methodological advancements but also encourages a deeper understanding of how intrinsic motivations can be quantified and utilized in creating more adaptive intelligent systems.