Active inference: demystified and compared (1909.10863v3)

Abstract: Active inference is a first principle account of how autonomous agents operate in dynamic, non-stationary environments. This problem is also considered in reinforcement learning (RL), but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in RL; 2) an explicit discrete-state comparison between active inference and RL on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of RL. We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration, and account for uncertainty about their environment in a Bayes-optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in RL is removed in active inference, where reward can simply be treated as another observation; even in the total absence of rewards, agent behaviors are learned through preference learning. We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based RL agents; by placing zero prior preferences over rewards and by learning the prior preferences over the observations corresponding to reward. We conclude by noting that this formalism can be applied to more complex settings if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside RL agents.

• The paper demonstrates that active inference optimizes agent behavior without explicit rewards through belief-based policies.
• It employs discrete-state tests using OpenAI gym benchmarks to rigorously compare active inference with reinforcement learning.
• Findings indicate that active inference enables rapid adaptation and intrinsic exploration, particularly in non-stationary settings.

Overview of Active Inference: Demystified and Compared

The paper "Active inference: demystified and compared" presents a meticulous comparison between the frameworks of active inference and reinforcement learning (RL), particularly within discrete-state environments. Active inference is advanced as a comprehensive principle accounting for the behavior of autonomous agents in dynamic and non-stationary environments. This work aims to clarify the discrete-state formulation of active inference and delineates its natural behaviors, which are typically engineered in RL paradigms. A comparative analysis was conducted using standard discrete-state environments, exemplified using the OpenAI gym benchmark.

Active inference builds upon the free energy principle and offers an integrative model for agent behavior, optimizing both action and perception under uncertainty. The paper highlights that active inference agents operate optimally in belief-based settings, emphasizing epistemic exploration and uncertainty management in a Bayesian optimal manner. A distinctive feature is the dispensation of explicit reward signals that are quintessential in RL, by treating rewards as mere observations and preferences. The investigation illustrates the potential of active inference agents to function in reward-absent environments by adopting preference learning principles.

Theoretical Foundations of Active Inference

Active inference invokes the free energy principle to define the interaction dynamics of agents with their environments, focusing on homeostasis via surprise minimization. It posits that agents perceive the world through outcomes rather than definitive state valuations. This process is facilitated by a generative model wherein beliefs about hidden states are inferred through observed outcomes, enabling agents to make informed decisions based on anticipated policy outcomes.

Distinctively, in contrast to RL's reward maximization objective, active inference seeks to minimize the expected free energy (EFE). This minimization accountably integrates both epistemic and extrinsic value, naturally fostering exploration-exploitation balance. The disciplinary framework leverages Bayesian formulations to refine an agent's behavioral models, where salient behavioral features are derived, such as intrinsic motivation for exploration and pragmatic exploitation.

Empirical Comparison: Active Inference vs. Reinforcement Learning

The empirical aspect of the paper compares active inference with RL through iterations on scenarios within the OpenAI gym environment "FrozenLake." The simulations showcase that active inference agents derive meaningful behaviors, learning complex agent-environment interaction patterns without needing explicit reward cues. The active inference agents demonstrated a robust capacity for online learning, adapting efficiently to the environment's stochastic dynamics compared to RL agents.

In stationary environments, active inference, and Bayesian RL agents achieved high average rewards in fewer episodes due to their belief-based policies. However, in non-stationary settings, active inference stood out with rapid adjustment to environmental changes owing to modular generative model updates, a feat challenging for classical RL agents suffering from reward-driven optimization inertia.

Practical Implications and Future Directions

This work has foundational implications for developing adaptive AI systems capable of functioning robustly in varying conditions. By modeling exploration as an intrinsic behavior rather than a function of extrinsic rewards, active inference may offer enhanced paradigms for developing autonomous systems, especially where dynamic adaptability is crucial. Future research could further explore hierarchical generative models within active inference frameworks and how they compete or complement state-of-the-art RL techniques in more complex applications like robotics or extensive gaming environments.

In summary, by providing a comprehensive exposition of active inference alongside a rigorous comparative analysis with RL, this paper illuminates the potential of belief-based frameworks in AI and paves the way for further exploration of agents that embody flexible, adaptive, and robust decision-making competencies.