Extended predictive coding framework as variational free-energy minimisation under exponential-family assumption

Abstract: The sensory cortices of the brain perform perceptual inference efficiently through their complex networks of neurons. One of the theoretical accounts of this process is the free-energy principle (FEP), which postulates that the brain performs variational Bayesian inference. Pioneering studies have shown that FEP can correspond to the predictive coding (PC) hypothesis under the Gaussian assumption and Laplace approximation. However, PC-based implementations of FEP within such a limited Gaussian regime have failed to capture several properties of biological neural networks, such as nonlinearity and heterogeneity of input--output properties within a network, and the biological implausibility of negative firing rates. This study shows that, when a broader class of probability distributions, namely the exponential family of distributions (EFD), is assumed for the variational posterior and prior, these missing characteristics are exhibited within the network, maintaining the FEP--PC correspondence up to the second cumulant of the posterior. We also show that the proposed model can be trained by biologically plausible local plasticity rules. Our results enrich the explanatory power of FEP regarding neural dynamics involved in perception as variational inference.

• The paper extends the predictive coding framework by deriving update rules for variational free-energy minimization under exponential family assumptions.
• It introduces both ordinary and natural gradient descent methods that yield local, biologically plausible synaptic plasticity rules.
• The work reconciles neural heterogeneities by enabling nonlinear activation functions and diverse F–I curves, addressing limitations in Gaussian models.

Extended Predictive Coding as Variational Free Energy Minimization in the Exponential-Family Regime

Introduction

This work presents a rigorous extension of the correspondence between the Free-Energy Principle (FEP) and Predictive Coding (PC) to the Exponential Family of Distributions (EFD). The FEP, foundational in theoretical neuroscience, postulates that perceptual inference in the brain operates via variational Bayesian inference, seeking to minimize Variational Free Energy (VFE). Previous implementations have relied heavily on the Gaussian assumption, resulting in linearities incompatible with experimentally observed neural heterogeneity and biophysical constraints (e.g., strictly non-negative firing rates). This paper generalizes the correspondence to the EFD, demonstrating that PCs can emerge naturally as the neural implementation of VFE minimization when both the prior and posterior are members of the EFD. Crucially, this extension maintains biological plausibility and admits biological features such as nonlinear activation functions and diverse F–I curves across neurons.

Theoretical Framework

FEP and Predictive Coding

FEP frames neural inference as minimization of VFE, an upper bound on the negative log marginal likelihood. The standard correspondence to PC—where the variational posterior and prior are Gaussian—maps representational neurons to belief encoders and error-coding neurons to the propagation of prediction errors. The gradient descent on VFE yields update rules for neural activities, interpretable as inference and learning mechanisms.

Exponential Family Assumptions

The primary advance of this study relaxes the Gaussian constraint by assuming prior and posterior from the EFD, a broad class that includes Bernoulli, Poisson, Binomial, Categorical, and Gaussian (as a special case). The EFD is characterized by natural parameters $\eta$, sufficient statistics $T(x)$, and log-partition functions $A(\eta)$, which are strictly convex and twice differentiable. The Legendre duality relates mean parameters to natural parameters, and the Fisher information matrix emerges as the Hessian of $A(\eta)$. This mathematical structure leads to both theoretical tractability and biological plausibility.

Hierarchical Factorization

Following the hierarchical PC literature, the model factorizes hierarchical generative models into Markov chains, with each layer's approximate posterior and prior independent and parameterized by EFD members.

Inferential Dynamics and Learning Rules

Derivation

A central contribution is the derivation of gradient descent dynamics on VFE under EFD assumptions:

• Update of posterior natural parameters $\eta_{q_l}$ involves three additive drives: drift toward the prior, prediction error feedback, and natural geometry of parameter space given by Fisher information.
• Both ordinary gradient descent (OGD) and natural gradient descent (NGD) forms are exhibited. NGD, in particular, cancels the Fisher information scaling, yielding computational and biological simplicity.

Figure 1: Schematic overview of the EFD–FEP model. Representational neurons encode internal states $\eta_{q_l}$ and sampled activities $\check x_l$, generating predictions, integrating errors, and implementing local plasticity.

Learning Mechanisms

The synaptic updates for prediction, lateral, and top-down weights are local and interpretable in terms of plausible plasticity:

• Prediction weights update via an error-modulated Hebbian rule.
• Prior-regulating weights (lateral and top-down) update following the difference between current and prior firing rates, echoing dendritic integration and plasticity.
• Both deterministic (mean-based) and stochastic (spike-based) variants accommodate biological noise.

Approximation and Validity

The formulation neglects third cumulant terms, focusing largely on the first two moments, justified when the posterior’s skewness is limited. Larger deviations may introduce approximation errors in strong non-Gaussian regimes.

Biological Implications and Mapping

Nonlinearity and Heterogeneity

A significant theoretical strength is the emergence of arbitrary, monotonically increasing and differentiable neuron activation functions—orthogonal to the homogeneity imposed by previous Gaussian models. This feature both resolves the negative firing rate issue and accounts for experimentally observed F–I heterogeneity.

Subtypes and Network Diversity

Two subtypes are defined:

• Type-A: Posterior factorizes across variables, yielding diagonal Fisher matrices and allowing one-to-one mapping between neurons and latent variables.
• Type-B: Non-factorizable posteriors (e.g., full Gaussian, categorical), whose implementation is less straightforward but potentially realized through mechanisms such as lateral inhibition or collective neural dynamics (e.g., softmax via winner-take-all).

Synaptic and Compartmental Correspondence

Local plasticity and integration, including BAC firing and dendritic plateau potentials, are mapped onto model parameters and update rules, particularly when representational neurons are interpreted as pyramidal cells.

Error-Coding Neuron Constraints

Error-coding neurons in the model are granted special properties (e.g., no intrinsic dynamics, ability to represent negative values) that are not fully realized in physiology—an acknowledged challenge. Remedies like splitting into excitatory/inhibitory populations and leveraging feedback alignment are discussed, though inherent limitations remain.

Theoretical and Practical Implications

• Unified Model: The EFD extension boosts the explanatory power of FEP–PC theory, capturing both neural and behavioral heterogeneity and supporting a richer class of perceptual computations.
• Information Geometry: The relation to natural gradients offers geometric insights into inference efficiency and adaptability, with OGD favoring stability and NGD maximizing convergence speed.
• Energetics: There is a direct analogy to thermodynamic systems, where the EFD plays the role of equilibrium distributions—opening avenues for metabolically grounded theories of neural computation.

Directions for Future Research

• Beyond Gaussian Likelihood: Further generalization to non-Gaussian EFD likelihoods is shown to break the simple PC-form of the dynamics, pointing to open questions about alternative biological approximations and surrogate objective functions.
• Nonlinear and Nonlocal Computations: Type-B models' neural implementation—particularly for correlated posteriors—warrants investigation into circuit-level or population-level computational motifs.
• Energetic Constraints and Non-equilibrium: Mapping nonstationary, non-equilibrium dynamics found in biological systems to principled statistical and physical frameworks remains a promising direction.

Conclusion

This study rigorously establishes that the predictive coding framework, as an implementation of the Free-Energy Principle, robustly generalizes when the variational posterior and prior are extended to the exponential family of distributions. The resulting inferential and learning dynamics not only accommodate observed biological nonlinearities and heterogeneity but also yield simple, local plasticity rules compatible with neurophysiological observations. These findings enrich both the theoretical foundations and empirical relevance of FEP-based models for neural inference, while delineating clear opportunities and challenges for further theoretical and experimental investigation.