Self-Orthogonalizing Attractor Networks

• Self-orthogonalizing attractor neural networks are recurrent systems that enforce mutual orthogonality among stored patterns to enhance memory capacity.
• They employ architectural constraints and regularized learning—such as orthogonality-constrained RNNs and iterative unlearning—to maintain stable, expressive dynamics.
• Balancing complexity and accuracy via free energy minimization, these systems support both fixed-point and sequence memory with analytically tractable properties.

Self-orthogonalizing attractor neural networks comprise a class of recurrent and associative memory systems in which the architecture and dynamics enforce, either exactly or approximately, the mutual orthogonality of stored attractors. This property is achieved via explicit constraints on the synaptic connectivity or emerges naturally from the optimization principles guiding learning and inference. Such networks maximize long-term memory capacity, enable robust generalization, and avoid pathological interference among stored patterns, all while maintaining analytically tractable dynamical regimes. The self-orthogonalization phenomenon is established in continuous-time RNNs with orthogonality-constrained recurrence (Ribeiro et al., 2019), in Hopfield and Boltzmann-type architectures via regularized learning and “dreaming” (iterative unlearning) (Agliari et al., 2023), and through principled approaches rooted in the free energy principle (Spisak et al., 28 May 2025).

1. Mathematical Frameworks and Orthogonality Criteria

Three primary frameworks realize self-orthogonalizing attractor networks:

• Orthogonality-Constrained RNNs: For discrete-time RNNs with hidden state $x_t\in\mathbb{R}^n$, the recurrence $x_{t+1}=f(x_t, u_t;\theta)$ admits attractor sets $A\subset\mathbb{R}^n$ that are invariant, possess attraction basins, and are minimal with these properties. The spectral radius of the hidden-to-hidden Jacobian $J_t=\partial x_t/\partial x_{t-1}$ determines gradient stability and the feasibility of complex attractor structures. Setting $W^\top W=I$ enforces $\rho(J_t)=1$, preventing both vanishing and exploding gradients and enabling maintenance of rich attractor sets (Ribeiro et al., 2019).
• Regularized Hopfield Networks (“Dreaming Kernels”): Given pattern vectors $\{\xi^\mu\}_{\mu=1}^P\subset\{\pm1\}^N$, the optimal synaptic matrix $W$ is obtained by minimizing a regularized quadratic loss, leading to closed-form solutions such as $W^{(D)}=(1/P)\xi[t_d/(I+t_d\,C)]\xi^\top$, with $C=(1/P)\xi^\top\xi$. As the regularization parameter (or “dream time” $t_d$) increases, $W^{(D)}$ becomes a projector that orthogonalizes all stored patterns (Agliari et al., 2023).
• Free Energy Principle-Based Attractor Nets: Here, learning arises from minimizing the variational free energy $$F(s,a,\mu)=\mathbb{E}_{q_\mu(\eta)}[\ln q_\mu(\eta)-\ln p(s,a,\eta)]$$ over internal states $\mu$, with inference and learning rules yielding attractor networks whose equilibrium points systematically minimize pattern overlap. The attractor set is characterized by near-orthogonality, especially when model complexity is balanced with predictive accuracy (Spisak et al., 28 May 2025).

Pairwise attractor orthogonality is quantitatively assessed by the normalized inner product (overlap) and the angle $$\theta_{\mu\nu}=\arccos\left(\frac{\sigma^{(\mu)}\cdot\sigma^{(\nu)}}{\|\sigma^{(\mu)}\|\|\sigma^{(\nu)}\|}\right)$$, with $\theta_{\mu\nu}\approx90^\circ$ signaling effective orthogonalization (Spisak et al., 28 May 2025).

2. Mechanisms for Achieving Self-Orthogonalization

The emergence of self-orthogonalizing attractor sets results from the interplay between architectural constraints and learning dynamics:

• Exact Spectral Norm Preservation: In RNNs, constraining the recurrent weight matrix to remain orthogonal or unitary ensures $\rho(J_t)=1$, thereby preventing gradient norm distortion across time steps. This regulation—achieved via Cayley-transform, Householder reflection, or Lie-group retraction—facilitates the coexistence of multiple distinct attractors by maintaining local dynamical sensitivity without loss of global stability (Ribeiro et al., 2019).
• Iterative Unlearning and Regularization: In Hopfield architectures, regularization or iterative “dreaming” progressively suppresses off-diagonal (cross-pattern) terms in $W$, such that in the large-$t_d$ (small-$\lambda$) limit, $W$ projects onto the attractor subspace and orthogonalizes all stored pattern vectors (Agliari et al., 2023). For finite $t_d$, small overlaps are strongly attenuated, providing effective but approximate orthogonality.
• Complexity–Accuracy Trade-Offs: Under the free energy principle, inference and learning adjust biases and couplings to fit data (accuracy) while penalizing departures from prior expectations (complexity). The fixed points of learning enforce residual covariance minimization, causing new attractors to be extracted orthogonally to those already learned. The resulting synaptic matrix $J$ approaches a diagonal form in the attractor basis (Spisak et al., 28 May 2025).

A further mechanism is retained in sequence learning under temporally structured input, where learning induces asymmetry in $J$ (i.e., $J\neq J^\top$), establishing both fixed-point and sequence (heteroclinic chain) attractors.

3. Dynamics, Stability, and Attractor Expressivity

Self-orthogonalizing attractor nets display several dynamical and functional regimes:

• Attractor Existence and Memory: Contractive networks ($L_f<1$) admit only a single fixed point and cannot support memory-rich attractor structures. Only by operating at or near the critical point ($L_f=1$) do networks permit expressive fixed points, limit cycles, and even chaotic attractors (Ribeiro et al., 2019).
• Stable Training Dynamics: Gradient norms remain constant during optimization due to exact spectral norm preservation, eliminating the need for ad-hoc gradient clipping or tuning (Ribeiro et al., 2019). The loss landscape is smoothed globally while allowing localized regions of high curvature necessary for memory storage.
• Generalization and Overfitting: Hopfield models with regularization reveal three operational regimes: (i) Failure (low $t_d$, high overlap, poor recall), (ii) Success (moderate $t_d$, high orthogonality, strong generalization), and (iii) Overfitting (large $t_d$, even intra-class minima become separated, recalling single examples only) (Agliari et al., 2023). Cross-validation or early stopping allows selection of the optimal regime.
• Non-equilibrium and Sequence Memory: For sequential inputs, antisymmetric matrix components drive solenoidal flows corresponding to spontaneous sequence recall (non-equilibrium steady states), with fixed-point attractors coexisting in the symmetric component (Spisak et al., 28 May 2025).

4. Learning Rules, Inference, and Algorithmic Prescriptions

Central prescriptions for building and training self-orthogonalizing attractor networks include:

• Orthogonal Recurrence and Norm-Preserving Activations: Maintain $W^\top W=I$ and choose activations with nearly unit Jacobian norm (e.g., leaky ReLU, modReLU) to avoid both information loss and instability (Ribeiro et al., 2019).
• Local Synaptic Update Rules: In free energy-based models, Hebbian/anti-Hebbian weight updates $$\Delta J_{ij} \propto \sigma_i \sigma_j - L(u_i) \sigma_j$$ induce orthogonalization by reinforcement of unexplained components of new patterns (Spisak et al., 28 May 2025).
• Regularization or Dreaming Kernels: Compute $W^{(D)}=(1/P)\xi[t_d/(I + t_d C)]\xi^\top$ or, in the infinite limit, $W^{(P)}=(1/P) \xi C^{-1}\xi^\top$ for exact pattern orthogonalization (Agliari et al., 2023).
• Design Recipes: Preprocess data to ensure zero mean, compute the Gram kernel, select an appropriate regularization parameter via validation, and realize pattern retrieval by thresholding $W^{(D)}\sigma$ (Agliari et al., 2023).
• Retraction Methods: Employ Cayley transform or similar operations to ensure that orthogonality of $W$ is preserved throughout training (Ribeiro et al., 2019).

5. Simulation Results and Empirical Regimes

Key empirical findings across frameworks demonstrate the impact and operational boundaries of self-orthogonalizing attractor networks:

Simulation	Principle	Measured Outcomes
Correlated Patterns	Free energy	Orthogonalization (angle $\approx 101^\circ$ for r=0.77 patterns), monotonic free-energy drop (Spisak et al., 28 May 2025)
Handwritten Digits	Free energy	Retrieval/generalization optimal at mean attractor angles near $90^\circ$
Sequence Learning	Free energy	Asymmetry in $J$ yields sequence recall and NESS dynamics
Overfitting/Generalization	Hopfield (regularized)	"Success" at intermediate dreaming time; projector kernel yields orthogonal recall; overfitting at large $t_d$ (Agliari et al., 2023)

Generalization is maximized when attractors are orthogonal and the complexity–accuracy trade-off is balanced (Spisak et al., 28 May 2025). Continuous replay, in free energy models, conferred resilience to catastrophic forgetting, with attractor orthogonality preserved over long epochs.

6. Equilibrium, Non-equilibrium, and Memory Structure

A central dichotomy emerges between equilibrium memory storage and non-equilibrium sequence memory:

• Equilibrium Regime: Random presentation of unstructured inputs leads to symmetric $J$ (Hopfield/Boltzmann machine behavior), detailed balance, and equilibrium Gibbs sampling (Spisak et al., 28 May 2025).
• Non-equilibrium Regime: Temporally ordered or structured inputs cause antisymmetrization of $J$, undermining detailed balance and generating sequence attractors corresponding to heteroclinic chains and persistent memory replay.

This duality exposes the capacity of self-orthogonalizing networks not only for static memory (fixed-point, orthogonal attractors) but also for dynamic, temporally ordered sequence memory and generation.

7. Design Principles and Practical Guidelines

Synthesis of the reviewed frameworks yields an implementation strategy as follows:

• Enforce global orthogonality of recurrent or synaptic matrices to preserve attraction basins and gradient stability (Ribeiro et al., 2019).
• Apply regularization or unlearning to the coupling kernel to suppress overlap among stored patterns and avert overfitting (Agliari et al., 2023).
• Balance model complexity with predictive accuracy using explicit free energy minimization, ensuring that attractors remain approximately orthogonal while providing robust generalization (Spisak et al., 28 May 2025).
• Structured input regimens enable the encoding of sequence memories via matrix asymmetry, exploiting non-equilibrium neural dynamics (Spisak et al., 28 May 2025).
• Employ smooth retraction methods (e.g., Cayley transform) during optimization to control higher-order derivatives and maintain $\beta$-smoothness of the cost landscape (Ribeiro et al., 2019).

Collectively, these principles establish self-orthogonalizing attractor neural networks as a maximally expressive, robust solution to memory storage and sequence learning, grounded in both dynamical systems and information-theoretic optimality.