Active Inference as a Model of Agency (2401.12917v1)

Abstract: Is there a canonical way to think of agency beyond reward maximisation? In this paper, we show that any type of behaviour complying with physically sound assumptions about how macroscopic biological agents interact with the world canonically integrates exploration and exploitation in the sense of minimising risk and ambiguity about states of the world. This description, known as active inference, refines the free energy principle, a popular descriptive framework for action and perception originating in neuroscience. Active inference provides a normative Bayesian framework to simulate and model agency that is widely used in behavioural neuroscience, reinforcement learning (RL) and robotics. The usefulness of active inference for RL is three-fold. \emph{a}) Active inference provides a principled solution to the exploration-exploitation dilemma that usefully simulates biological agency. \emph{b}) It provides an explainable recipe to simulate behaviour, whence behaviour follows as an explainable mixture of exploration and exploitation under a generative world model, and all differences in behaviour are explicit in differences in world model. \emph{c}) This framework is universal in the sense that it is theoretically possible to rewrite any RL algorithm conforming to the descriptive assumptions of active inference as an active inference algorithm. Thus, active inference can be used as a tool to uncover and compare the commitments and assumptions of more specific models of agency.

• The paper introduces active inference as a Bayesian approach to model agency by unifying exploration and exploitation under a free energy minimization framework.
• It demonstrates a shift from traditional reward-maximization paradigms to a principled formulation that mirrors biological decision-making.
• The framework offers an explainable alternative for agent-based modeling and encourages integrating hierarchical probabilistic models with deep neural architectures.

Overview of Active Inference and Agency

In the paper titled "Active Inference as a Model of Agency," a comprehensive framework for understanding and modeling agency is presented. Agency, within this context, refers to the ability of an entity—be it biological or artificial—to make decisions and take actions that influence the environment. The authors propose active inference as a normative Bayesian approach that naturally integrates both exploratory and exploitative behaviors by minimizing risk and ambiguity concerning external states.

Agency Beyond Reward Maximization

Active inference is highlighted as an advance over traditional reinforcement learning (RL) models, which primarily focus on reward maximization. The authors argue that active inference, by minimizing expected free energy, can address the exploration-exploitation dilemma without resorting to ad hoc methods for inducing exploratory behavior. This paradigm shift is significant as it aligns with the natural behavior observed in biological systems, providing a more realistic and principled methodology for simulating intelligent behavior while offering explainability through explicit generative world models.

Theoretical Foundations in Physics

The foundation of active inference lies in well-established physical principles, as the paper elucidates by discussing the behavior of macroscopic biological systems subject to classical mechanics, known as precise agents. A precise agent responds deterministically to environmental stimuli given the information available up until the current moment. The mathematical formalization of this interaction delineates the nature of decision-making and agency as optimization problems, governed by an expected free energy functional—a criterion that balances objective probabilities of states with subjective preferences or utilities.

Advantages and Applications

Active inference presents several advantages over traditional models of decision-making in living organisms and artificial systems. Firstly, it emphasizes the value of understanding preferences and predictions in a unified framework, leading to robust exploration-exploration solutions. Secondly, the universality of the active inference framework allows RL algorithms to be recast as active inference algorithms, fostering comparisons and advancing theory development in agent-based modeling. Finally, by explicitly encoding predictions and preferences, active inference lays down a blueprint for transparent and justifiable algorithmic decision-making, proving highly applicable to safety-critical domains.

Path Forward

The authors conclude by reflecting on the potential of active inference and pose several open questions important for future research. One area of intrigue is identifying the generative models humans use to represent their environments and the computational mechanisms through which these representations evolve from childhood to adulthood. The integration of hierarchical probabilistic models and deep neural architectures is suggested as a promising path forward, with the goal of understanding and eventually replicating the development of human intelligence in artificial agents.