pymdp: A Python library for active inference in discrete state spaces (2201.03904v2)

Abstract: Active inference is an account of cognition and behavior in complex systems which brings together action, perception, and learning under the theoretical mantle of Bayesian inference. Active inference has seen growing applications in academic research, especially in fields that seek to model human or animal behavior. While in recent years, some of the code arising from the active inference literature has been written in open source languages like Python and Julia, to-date, the most popular software for simulating active inference agents is the DEM toolbox of SPM, a MATLAB library originally developed for the statistical analysis and modelling of neuroimaging data. Increasing interest in active inference, manifested both in terms of sheer number as well as diversifying applications across scientific disciplines, has thus created a need for generic, widely-available, and user-friendly code for simulating active inference in open-source scientific computing languages like Python. The Python package we present here, pymdp (see https://github.com/infer-actively/pymdp), represents a significant step in this direction: namely, we provide the first open-source package for simulating active inference with partially-observable Markov Decision Processes or POMDPs. We review the package's structure and explain its advantages like modular design and customizability, while providing in-text code blocks along the way to demonstrate how it can be used to build and run active inference processes with ease. We developed pymdp to increase the accessibility and exposure of the active inference framework to researchers, engineers, and developers with diverse disciplinary backgrounds. In the spirit of open-source software, we also hope that it spurs new innovation, development, and collaboration in the growing active inference community.

• The paper presents pymdp, a Python library enabling simulation of active inference agents using discrete POMDP generative models.
• It details a modular architecture with variational Bayesian methods like fixed-point iteration and marginal message passing for state and policy inference.
• It demonstrates adaptive parameter learning via Hebbian-like updates, broadening access to active inference research across diverse fields.

Overview of pymdp: A Python Library for Active Inference

The paper presents pymdp, a Python-based library designed to facilitate the simulation of active inference agents operating within discrete state spaces. This package aims to enhance accessibility and broaden the adoption of the active inference framework beyond disciplines like cognitive neuroscience, encompassing fields such as reinforcement learning and real-world engineering applications.

Core Features and Structure

pymdp provides a suite of modular routines to simulate agents equipped with partially observable Markov Decision Process (POMDP) generative models. This model comprises stochastic variables—observations, hidden states, control states, and hyperparameters—underpinned by categorical and Dirichlet distributions.

The library's high-level API, encapsulated by the Agent class, abstracts the complexities of active inference processes including optimization routines for state estimation, action selection, and learning. pymdp's modular structure improves user customizability, enabling agents to be specified at various levels of abstraction. This modularity accommodates the integration of different inference and policy selection algorithms.

Inference and Control Mechanisms

Active inference, within this framework, is implemented as variational Bayesian inference. The paper details fixed-point iteration and marginal message passing methods for state inference. For policy inference, pymdp incorporates the computation of expected free energy (EFE), which combines epistemic value and utility derived from prior preferences to account for both exploratory and goal-directed behaviors.

Parameter Learning

The software also supports parameter learning by updating the Dirichlet priors over the generative model's categorical distributions. These updates occur via biologically plausible, Hebbian-like rules that iterate adjustments based on observations and inferred hidden states. Such capabilities assist in dynamically adapting the agent's model to align with environmental feedback.

Practical Implications and Future Directions

The introduction of pymdp is significant in its aim to democratize active inference research by offering an open-source and accessible alternative to traditional MATLAB-based approaches. The potential applications span across modeling human and animal behavior, control systems, and the enhancement of reinforcement learning paradigms. The library stands to encourage interdisciplinary innovations, allowing researchers from diverse fields to leverage active inference without the constraints of proprietary software.

Future enhancements may focus on expanding the library's applicability to continuous state spaces and integrating more sophisticated learning algorithms that further automate the adaptation to complex tasks. The developments in pymdp also suggest a promising avenue for advancing AI systems capable of more nuanced decision-making reflective of human-like cognitive processes. The open-source nature of the project encourages community involvement, promising continual improvement and sustained relevance in the evolving landscape of computational modeling.

In conclusion, pymdp represents a crucial step in the evolution of active inference tools, offering a comprehensive, adaptable, and user-friendly platform for researchers and developers to explore, simulate, and innovate across a wide spectrum of scientific and engineering domains.