Self-orthogonalizing attractor neural networks emerging from the free energy principle (2505.22749v1)

Abstract: Attractor dynamics are a haLLMark of many complex systems, including the brain. Understanding how such self-organizing dynamics emerge from first principles is crucial for advancing our understanding of neuronal computations and the design of artificial intelligence systems. Here we formalize how attractor networks emerge from the free energy principle applied to a universal partitioning of random dynamical systems. Our approach obviates the need for explicitly imposed learning and inference rules and identifies emergent, but efficient and biologically plausible inference and learning dynamics for such self-organizing systems. These result in a collective, multi-level Bayesian active inference process. Attractors on the free energy landscape encode prior beliefs; inference integrates sensory data into posterior beliefs; and learning fine-tunes couplings to minimize long-term surprise. Analytically and via simulations, we establish that the proposed networks favor approximately orthogonalized attractor representations, a consequence of simultaneously optimizing predictive accuracy and model complexity. These attractors efficiently span the input subspace, enhancing generalization and the mutual information between hidden causes and observable effects. Furthermore, while random data presentation leads to symmetric and sparse couplings, sequential data fosters asymmetric couplings and non-equilibrium steady-state dynamics, offering a natural extension to conventional Boltzmann Machines. Our findings offer a unifying theory of self-organizing attractor networks, providing novel insights for AI and neuroscience.

• The paper derives a framework where self-organizing attractor neural networks emerge through variational free energy minimization.
• It introduces local inference and Hebbian/anti-Hebbian learning rules that yield orthogonal attractor states and robust Bayesian sampling.
• The approach offers practical insights for neuromorphic computing, continuous learning, and enhanced pattern recognition.

This paper, "Self-orthogonalizing attractor neural networks emerging from the free energy principle" (2505.22749), presents a theoretical framework deriving a class of adaptive, self-organizing attractor neural networks directly from the Free Energy Principle (FEP). The core idea is that any system maintaining a statistical separation from its environment (a "particular partition") behaves as if it is minimizing variational free energy (VFE), which in turn drives emergent dynamics for inference and learning.

The authors show that by applying the FEP to a hierarchical partitioning of a dynamical system into "subparticles" (a "deep particular partition"), a specific type of recurrent neural network naturally emerges. This network structure functionally resembles continuous-state Boltzmann machines or stochastic Hopfield networks but allows for asymmetric couplings and continuous-valued states.

Key Concepts and Emergent Properties:

1. Particular Partitions and Deep Partitions: The framework starts with the concept of a particular partition, which divides a system's states into internal ($\mu$), external ($\eta$), and boundary states (sensory $s$, active $a$) based on sparse coupling. The crucial point is the conditional independence between internal and external states given the boundary states. A "deep particular partition" extends this recursively, treating internal states as collections of interacting subparticles, each forming its own particular partition.
2. Attractor Network Structure: By choosing a specific parametrization for the subparticle states (continuous Bernoulli distribution, $x \in [-1, 1]$) and couplings (deterministic scaling via weights $J_{ij}$), the joint probability distribution over the subparticle internal states ($\boldsymbol{\sigma}$) takes the form of a Boltzmann distribution: $$p(\mathbf{\sigma}) \propto \exp \Biggl\{ \sum_{i} b_i \sigma_i + \frac{1}{2} \sum_{ij} (J_{ij}+J_{ji})\sigma_i\sigma_j \Biggr\}$$. Here, $b_i$ is a bias term (potentially influenced by sensory input) and the coupling depends on the symmetric part of the weight matrix $\mathbf{J}$. Asymmetric couplings $(J_{ij} \neq J_{ji})$ introduce an antisymmetric component that drives non-equilibrium dynamics (solenoidal flows) but, under certain conditions (conservative particles), does not alter the stationary distribution's Boltzmann form.
3. Emergent Inference Dynamics: Minimizing the variational free energy (VFE) for a single subparticle $\sigma_i$, given the states of others $\sigma_{\backslash i}$, yields a stochastic update rule for the expected value of $\sigma_i$:

   $$\mathbb{E}_q[\sigma_i] = L \left( b_i + \sum_{j \ne i} J_{ij} \sigma_j \right)$$

   where $L(\cdot)$ is the Langevin function (the expected value of the continuous Bernoulli distribution). This rule resembles Gibbs sampling in a Boltzmann machine, performing local approximate Bayesian inference where the subparticle's belief (bias $b_i$) is updated by evidence from connected subparticles weighted by $J_{ij}$. The inherent stochasticity allows the network to sample from the posterior distribution over states.

4. Emergent Learning Dynamics: Minimizing VFE with respect to the coupling weights $J_{ij}$ results in a learning rule:

   $$\Delta J_{ij} \propto \sigma_i \sigma_j - L(b_i + \sum_{k\neq i} J_{ik}\,\sigma_k ) \sigma_j$$

   This is a form of Hebbian/anti-Hebbian learning. The Hebbian term ($\sigma_i \sigma_j$) reinforces co-activity, while the anti-Hebbian term ($L(\cdot) \sigma_j$) subtracts variance explained by the network's prediction. This implements a predictive coding-like update, driving weights towards configurations that minimize prediction error.

5. Self-Orthogonalizing Attractors: A key finding is that the learning rule naturally promotes approximately orthogonal attractor representations. By minimizing VFE, the network implicitly minimizes complexity, which penalizes redundant or highly correlated representations. The anti-Hebbian term ensures that weight updates primarily capture the component of new inputs orthogonal to previously learned patterns, leading to decorrelated attractor states analogous to online orthogonalization methods like Sanger's rule for PCA.
6. Macro-Scale Bayesian Inference: The collective stochastic dynamics of the network, driven by local VFE minimization, effectively perform Markov Chain Monte Carlo (MCMC) sampling from the posterior distribution over the network's global state $\boldsymbol{\sigma}$, given external inputs (encoded in biases $b_i$). The network's learned weights and baseline biases encode the prior distribution. The time-averaged state of the network approximates the expected value under the posterior. Orthogonal attractors facilitate more robust and less ambiguous inference.

Practical Implementation and Applications:

The paper provides Python code for a basic implementation of the derived network model, focusing on clarity rather than efficiency. This provides a starting point for developers interested in exploring this architecture.

The core implementation involves:

• Representing subparticles/neurons with continuous states $\sigma_i \in [-1, 1]$.
• Storing coupling weights in a matrix $\mathbf{J}$.
• Implementing the inference rule: Update $\sigma_i$ based on its bias $b_i$ and weighted sum of inputs $\sum_{j \ne i} J_{ij} \sigma_j$, potentially involving the Langevin function $L(\cdot)$ or its inverse to update the bias parameter $b_q$, and sampling from the continuous Bernoulli distribution $\mathcal{CB}(b_q)$. A simplified deterministic update using $L(\cdot)$ can be used to find fixed-point attractors.
• Implementing the learning rule: Update $J_{ij}$ based on the difference between observed correlation $\sigma_i \sigma_j$ and a predicted correlation $L(u_i) \sigma_j$, where $u_i$ is the net input to node $i$. This requires simultaneous inference and learning steps.
• Incorporating temperature (inverse precision $iT$) and learning rate ($\alpha$) parameters to control the dynamics and learning speed.

for each neuron i: # Calculate net input to neuron i net_input_i = bias_i + sum(J[j, i] * sigma[j] for j in all neurons except i) # Perform inference (update variational bias) # This is a simplified representation, actual derivation involves Langevin/coth variational_bias_i = bias_i + sum(J[j, i] * sigma[j] for j in all neurons except i) # From dF/db_q=0 # Update state (sampling from CB or deterministic update) # Use Langevin for expected value (deterministic) or sample from CB(variational_bias_i) (stochastic) sigma_predicted_i = langevin(variational_bias_i) # L(b_q) # Perform learning (update weights TO neuron i) for each neuron j: # Calculate predicted correlation (based on current state of j and PREDICTED state of i) predicted_correlation_ij = sigma_j * sigma_predicted_i # Simplified, see paper eq (16) # Calculate observed correlation observed_correlation_ij = sigma_j * sigma_i # current states # Update weight J_ji (influencing neuron i from neuron j) # Note the index order J_ji corresponds to action of i on j in definition, # but learning rule dF/dJ_ij relates to action of j on i through sensory input s_ij # Learning rule is dF/dJ_ij, so J_ij is updated based on sigma_i, sigma_j relation delta_J_ij = alpha * (sigma_i * sigma_j - sigma_predicted_i * sigma_j) # Simplified J[i, j] = J[i, j] + delta_J_ij # Learning rule updates J_ij (synapse from j to i) # Update neuron i state (e.g., sampling for stochastic inference) sigma[i] = sample_from_continuous_bernoulli(variational_bias_i) # Or set sigma[i] = sigma_predicted_i for deterministic

Real-World Applications:

• Neuromorphic Computing: The stochastic nature and local update rules are well-suited for energy-efficient hardware implementations.
• Continuous Learning and Lifelong Learning: The simultaneous learning and inference process, coupled with the resistance to catastrophic forgetting demonstrated in simulations, is highly relevant for AI systems that need to adapt and learn continuously in dynamic environments without forgetting previous knowledge.
• Pattern Recognition and Generalization: The ability to form approximately orthogonal attractors enhances the network's capacity and ability to generalize to novel inputs by combining existing representations.
• Sequence Learning: The framework naturally accommodates asymmetric couplings for learning and reproducing temporal sequences, relevant for tasks like time-series prediction or motor control.
• Brain Modeling: Provides a principled framework for understanding how attractor dynamics, memory, and learning could emerge from fundamental principles in the brain, supported by empirical findings like the orthogonality of large-scale brain attractors.
• Robotics: Enables adaptive control and perception in robots that need to self-organize and learn from interactions with their environment.

Implementation Considerations:

• Computational Cost: While local updates are efficient, simulating the network's stochastic dynamics over time to sample from the posterior can be computationally intensive, especially for large networks, similar to MCMC sampling in Boltzmann machines.
• Hyperparameter Tuning: The paper highlights the importance of parameters like inverse temperature (precision) and learning rate, which influence the balance between accuracy and complexity, and thereby retrieval vs. generalization performance. Finding optimal parameters for specific tasks requires tuning.
• Scalability: While theoretically scalable due to local updates, practical implementation on conventional hardware may face challenges with very large, dense networks. Exploring sparse connectivity or hierarchical structures might be necessary.
• Continuous States: Handling continuous states numerically requires careful discretization or approximation methods.
• Stability of Asymmetric Dynamics: While the paper argues that asymmetric couplings drive conservative solenoidal flows that don't alter the stationary distribution's form, implementing stable non-equilibrium dynamics in practice might require careful consideration of numerical integration and parameter choices.

In summary, this research offers a powerful theoretical foundation for self-organizing neural networks rooted in fundamental principles of self-organization, providing both insights into natural intelligence and a blueprint for developing more adaptive and biologically plausible artificial intelligence systems. The provided simulation code serves as a practical starting point for engineers and researchers to implement and experiment with this class of FEP-derived attractor networks.