Emergent Self-Attention from Astrocyte-Gated Associative Memory Dynamics

Abstract: We introduce a Hopfield-type associative memory in which effective connectivity is multiplicatively modulated by astrocytic gains evolving under an entropy-regularized replicator equation. The coupled neuron-astrocyte dynamics admit a Lyapunov function, ensuring global convergence. At fixed points, astrocytic gains implement a softmax-normalized allocation over pattern similarity scores, yielding a mechanistic realization of self-attention as emergent routing on the gain simplex. In regimes of high memory load and interference, the model significantly improves retrieval accuracy relative to classical Hopfield dynamics and recent neuron-astrocyte baselines. These results establish a dynamical systems framework linking glial modulation, competitive resource allocation, and attention-like computation.

• The paper introduces a principled framework where astrocyte-modulated gain dynamics yield a softmax self-attention mechanism over stored memory patterns.
• Simulations demonstrate that optimal astrocytic and neuronal timescales minimize retrieval error, outperforming classical Hopfield models.
• Analytical results confirm global convergence to fixed points, showing that competitive gating enhances interference suppression and memory accuracy.

Emergent Mechanistic Self-Attention via Astrocyte-Gated Associative Memory Dynamics

Introduction and Theoretical Framework

This work presents a rigorous dynamical framework in which attention-like computation emerges natively from the interplay between neuronal and astrocytic subsystems within an associative memory architecture. The classical Hopfield network, while a canonical substrate for attractor-based memory, is fundamentally limited by low storage capacity and catastrophic interference under dense loading. Extensions such as dense associative memories, and the class of "modern Hopfield networks," achieve improvements via non-quadratic energy landscapes, which in turn link to self-attention through fixed-point analysis. In contrast, this paper proposes that attention-weighted routing can instead emerge through astrocyte-modulated gain dynamics governed by an entropy-regularized replicator equation, yielding a mechanistic—rather than architecturally imposed—softmax allocation over stored memory patterns (2604.25481).

The astrocyte gain variables $p_\mu$ modulate the effective connection strengths for each memory "engram," constrained on the probability simplex, thus enforcing hard competitive allocation across patterns. Astrocyte dynamics evolve via a softmax-regularized replicator equation, with pattern scores derived from squared overlaps, and an inverse temperature parameter $T$ controlling sharpness of pattern selection. Neuronal dynamics follow a nonlinear rate-based Hopfield evolution, with the connectivity modulated multiplicatively by the pattern-wise gains. The joint neuron-astrocyte system admits a Lyapunov function, guaranteeing global convergence to fixed points, which correspond to softmax allocations over pattern similarity scores.

Emergent Softmax Routing and Analytical Results

The central analytic result is explicit characterization of equilibria: the neuron-astrocyte fixed points admit a coupled solution where the gain distribution on patterns is exactly a softmax over similarity scores:

$$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$

while the neuronal state satisfies $x^* = W(p^*)\phi(x^*)$ with $W(p) = \frac{K}{N}\sum_\mu p_\mu \xi_\mu\xi_\mu^T$. This mixture-of-experts structure (with adaptive soft assignment) yields convex combinations of stored memories, in direct analogy to self-attention readouts.

Two limiting cases formally recover classical Hopfield dynamics: either taking the gain timescale $\tau_p \to \infty$ (astrocytic adaptation frozen, $p_\mu=1/K$), or sending $T\to\infty$ (entropic regularization dominating, gain uniform by maximum entropy). Thus, the gain dynamics intervene only when astrocyte adaptation is both finite and competitive.

Dynamics: Parameter Dependencies and Convergence

Extensive simulation analyses probe the effects of astrocytic and neuronal timescales $(\tau_p, \tau_x)$ and the gain temperature $T$ on retrieval accuracy and routing selectivity. Retrieval is initialized from corrupted memory states, and both retrieval error $T$0 (mean soft Hamming distance to target) and gain perplexity $T$1 (exponential Shannon entropy) are tracked as summary statistics.

Astrocytic timescale $T$2 controls how rapidly gains can track pattern overlap dynamics. Fast astrocytic adaptation ($T$3) yields sharp routing and low retrieval error, while large $T$4 delays pattern selection and mimics Hopfield-like performance. The interaction between error, gain perplexity, and effective convergence time as a function of $T$5 is quantitatively illustrated:

Figure 1: $T$6 and $T$7 vs. astrocytic timescale $T$8 highlight regimes of optimal rapid gain allocation and slowing-induced convergence limitations.

Neuronal timescale $T$9 governs the relative speed of network relaxation. For $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$0, neurons commit immediately to (possibly spurious) attractors before astrocyte gains can select relevant patterns, resulting in high-confidence misclassification. Increasing $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$1 progressively allows astrocytes to reshape the attractor landscape, reducing retrieval error and increasing routing entropy until the process becomes too diffuse, at which point convergence slows and error plateaus.

Figure 2: Retrieval error and perplexity dependence on neuronal timescale $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$2, showing complex interplay with astrocyte dynamics and error/perplexity trade-off.

Temperature $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$3 in the replicator equation determines the selectivity–robustness tradeoff. As $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$4, routing approaches winner-take-all (minimal perplexity, maximal selectivity), with robust interference suppression and low error. Large $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$5 enforces near-uniform routing, degrades accuracy, and recovers classical Hopfield behavior.

Figure 3: Dependence of retrieval error and perplexity on gain temperature $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$6, controlling routing sharpness and selectivity.

Retrieval Performance Benchmarking

A comprehensive benchmarking versus classical Hopfield and the recent neuron-astrocyte associative memory framework of Kozachkov et al. is performed across memory loads $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$7 and query corruption levels $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$8. The critical result is that across the $$p_\mu^* = \operatorname{softmax}_\mu(f_\mu/T), \quad f_\mu = \frac{1}{2N}(\xi_\mu^T \phi(x))^2,$$9 grid, the proposed model achieves lower average Hamming retrieval error than both baselines, with the most substantial gains in the high-interference, high-corruption regime. The competitive allocation of modulator resources over patterns enables selective suppression of interference that would otherwise drive spurious attractors.

Figure 4: Error heatmap benchmarking showing substantial improvements in retrieval accuracy, particularly at large $x^* = W(p^*)\phi(x^*)$0 and high corruption $x^* = W(p^*)\phi(x^*)$1, relative to established models.

Mechanistic and Practical Implications

This framework identifies and mathematically formalizes a biologically plausible mechanism by which self-attention can emerge as a dynamical property in neural circuits—without explicit architectural encoding—via the interaction of pattern-wise gain modulation and competitive resource allocation. The coupling is inspired by astrocyte physiology (Ca$x^* = W(p^*)\phi(x^*)$2-mediated modulation, tripartite synapse), but $x^* = W(p^*)\phi(x^*)$3 serves as a coarse phenomenological proxy for modular astrocytic effects.

The model's parameter dependencies immediately suggest experimentally testable hypotheses: modulating astrocytic kinetics or background "modulatory tone" could adjust pattern selectivity and retrieval capacity. The critical ratio $x^* = W(p^*)\phi(x^*)$4 determines the competition between routing adaptation and network relaxation, reflecting a systems-level trade-off investable in both neurobiology and future AI models.

On the computational side, the results imply that resource-constrained competitive routing—implemented via differentiable replicator flow or similar simplex-constrained gain dynamics—can boost retrieval accuracy and interference robustness in associative memory architectures. This mechanism offers a clear design template for scalable mixture-of-experts or continual-learning architectures with explicit competitive gating, separate routing/state-update timescales, and entropic regularization of gating entropy.

Conclusion

The paper delivers a principled theoretical and empirical investigation of astrocyte-gated associative memory, demonstrating that softmax self-attention arises as an emergent property of competitive gain dynamics on the simplex. This formulation extends both the neuroscience and machine learning perspectives: on the former, by linking glial modulation to emergent attentional selection; on the latter, by providing a dynamical systems blueprint for attention and robust associative recall. The framework is directly extensible to differentiated, scalable ML architectures and motivates further exploration of competitive modulator-based mechanisms for adaptive memory allocation (2604.25481).