Self-Attention Dynamics: Models and Mechanisms

• Self-attention dynamics is the study of how evolving attention weights, token states, and interaction geometries convert similarity structures into dynamic evolution.
• It employs continuous and discrete formulations, revealing nonlocal transport phenomena, spectral selection, and clustering mechanisms in neural networks.
• Architectural models leveraging dynamic routing, energy-based, and probabilistic approaches demonstrate practical applications in NLP, vision, and biological inference.

Self-attention dynamics denotes the study of how attention weights, token states, and induced interaction geometries evolve under repeated attention updates, continuous-depth limits, or training. In the current literature, the term spans several distinct but related objects: interacting-particle flows on the unit sphere, mean-field transport equations, recurrent discrete-time attention maps, dynamic-routing sentence encoders, task-conditioned directed graphs over multivariate time series, and probabilistic generative models in which attention directly parameterizes transition or exponential-family conditionals (Kuehn et al., 28 Apr 2026, Burger et al., 6 Jan 2025, Yoon et al., 2018, Mahmood et al., 2021, Ildiz et al., 2024, Wibisono et al., 28 Jan 2025). Across these settings, the common question is how attention converts similarity structure into dynamical evolution—whether by clustering, spectral selection, synchronization, adaptive routing, or optimization-induced geometric bias.

1. Core formulations of self-attention as a dynamical system

A canonical continuous-time formulation models tokens $x_i(t)\in S^{d-1}$ on a product of spheres, with attention matrix $A(x)$ defined row-wise by

$$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$

and dynamics

$$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$

In this representation, time corresponds to continuous depth, the tangent projection models layer normalization, and the interaction graph is fully connected but state-dependent (Altafini, 14 Nov 2025). Closely related formulations impose the symmetry constraint $Q^\top K=V=V^\top$, yielding an idealized Transformer-type flow

$$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$

which makes the spectral structure of $V$ directly visible in the induced motion (Kuehn et al., 28 Apr 2026).

A second line of work treats self-attention dynamics as a mean-field transport problem. For layer-normalized attention on the sphere, the empirical measure of particles converges formally to a continuity equation

$$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$

with velocity

$$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$

or, in a related fixed-weight setting,

$$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$

These PDE limits make explicit that self-attention is a nonlocal transport mechanism whose velocity depends on the entire token distribution (Burger et al., 6 Jan 2025, Geshkovski et al., 2023).

Not all self-attention dynamics are continuous. In Dynamic Self-Attention, the dynamics are iterative and sentence-specific: projected word features $A(x)$0 are combined through routing logits $A(x)$1, weights $A(x)$2, aggregates $A(x)$3, and dynamic vectors $A(x)$4, with updates $A(x)$5. The resulting $A(x)$6 are dynamic weight vectors that vary across sentences and routing iterations rather than being fixed learned queries (Yoon et al., 2018). In CARSA, the temporal dimension is first compressed per variable through

$A(x)$7

after which self-attention acts across variables rather than across time: $A(x)$8 Here the induced attention weights define a subject-specific directed adjacency $A(x)$9 from component $$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$0 to component $$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$1 (Mahmood et al., 2021).

These formulations differ in state space, iteration variable, and semantics, but they share a common structure: attention weights are endogenous, state-dependent couplings, and the ensuing dynamics can be studied either at the level of individual tokens, variables, or distributions.

2. Geometric, variational, and energy-based structures

A major mathematical theme is that certain self-attention systems admit a gradient-flow interpretation. For multi-head self-attention with symmetric score matrices $$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$2 and value alignment $$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$3, the interaction energy

$$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$4

satisfies the exact Lyapunov identity

$$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$5

for both flat and spherical dynamics (Pendharkar, 5 May 2026). In the symmetric single-kernel setting, the same phenomenon appears as a weighted Riemannian gradient flow: with

$$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$6

the induced vector field is precisely the weighted gradient of $$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$7 under a state-dependent metric involving the partition functions $$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$8 (Kuehn et al., 28 Apr 2026).

At the mean-field level, layer-normalized self-attention with $$A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},$$9 yields the interaction energy

$$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$0

together with a Wasserstein-type geometry with nonlocal mobility

$$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$1

The resulting PDE is a metric gradient flow in $$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$2, and stationary measures are exactly those for which $$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$3 on the support (Burger et al., 6 Jan 2025). This places self-attention near the theory of aggregation equations, but with two nonstandard ingredients: confinement to the sphere and a mobility depending on the current measure.

A distinct but related construction expresses self-attention as the negative gradient of local energies rather than a single global energy. With vector spins $$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$4 and pairwise couplings $$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$5, define

$$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$6

Then

$$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$7

so the local attention update is exactly a negative local-energy gradient, with a pseudo-likelihood interpretation rather than a global Hopfield energy (D'Amico et al., 2024).

These variational viewpoints are powerful but conditional. Multi-head monotonicity can fail at the per-head level on the sphere because of radial shadow terms $$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$8, and approximate value alignment, orthogonality, or projection structure are treated as controlled idealizations rather than generic properties of trained Transformers (Pendharkar, 5 May 2026). The mean-field and local-energy theories similarly rely on symmetry or architectural tying assumptions that are mathematically advantageous but narrower than standard large-scale practice (Burger et al., 6 Jan 2025, D'Amico et al., 2024).

3. Spectral selection, clustering, multistability, and metastability

When self-attention is viewed as an interacting particle system, asymptotic behavior is organized by clustering and by the spectrum of the value operator. In the fixed-weight continuous-time model

$$\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.$$9

the rescaled dynamics $Q^\top K=V=V^\top$0 exhibits convergence to specific geometric limiting objects (Geshkovski et al., 2023). For $Q^\top K=V=V^\top$1 and $Q^\top K=V=V^\top$2, the convex hull of the rescaled particles is non-increasing and converges to a convex polytope, and each token converges either to $Q^\top K=V=V^\top$3 or to a point in a finite set $Q^\top K=V=V^\top$4. In one dimension, the self-attention matrix itself converges to a low-rank Boolean matrix, analytically confirming the emergence of leaders (Geshkovski et al., 2023).

The symmetric spectral theory sharpens this picture. In the eigenbasis of $Q^\top K=V=V^\top$5, coefficient dynamics take the replicator-type form

$Q^\top K=V=V^\top$6

with $Q^\top K=V=V^\top$7 the softmax kernel and $Q^\top K=V=V^\top$8 the modal average. This reveals a mode-selection mechanism: if one positive eigenvalue $Q^\top K=V=V^\top$9 strictly dominates all others in modulus and initial conditions lie in a forward-invariant cone, then all tokens converge to the dominant eigendirection; if $$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$0 is negative definite, the dynamics selects sign-split polarization toward the most negative eigendirection (Kuehn et al., 28 Apr 2026). The same spectral dependence appears in the broader single-head ODE on spheres, where equilibria fall into four classes—consensus, bipartite consensus, clustering, and polygonal—and multistability is typical rather than exceptional (Altafini, 14 Nov 2025).

Metastability introduces a slower timescale. For self-attention on the unit sphere with separated low-temperature initial clusters, particles remain trapped near a multi-cluster configuration for an exponentially long period, even though the global asymptotic state is full collapse to a single cluster (Geshkovski et al., 2024). The metastability theorem gives explicit times

$$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$1

such that particles remain within their spherical caps up to $$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$2 and intra-cap distances are exponentially small on $$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$3. Beyond this regime, after an appropriate time-rescaling, the energy exhibits a staircase profile with saddle-to-saddle-like behavior (Geshkovski et al., 2024).

Clustering is accompanied by entropy dynamics. In the scalar-head equiangular regime of multi-head attention, the attention entropy

$$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$4

obeys

$$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$5

and increases monotonically toward $$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$6 as clustering progresses (Pendharkar, 5 May 2026). A plausible implication is that “collapse” in self-attention need not mean low-entropy concentration of row distributions; in several symmetric regimes, geometric clustering of states coincides with equalization of scores and more uniform attention.

4. Training dynamics, optimization geometry, and objective-induced structure

Another use of the phrase “self-attention dynamics” concerns how training shapes the score geometry itself. In a single-layer softmax self-attention model trained for linear regression, the finite-data predictor

$$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$7

reduces in the infinite-data Gaussian limit to the bilinear map $$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$8. The population loss becomes

$$\dot{x}_i = P_{x_i}^\perp \left( \frac{1}{Z_{\beta,i}} \sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j \right), \qquad Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},$$9

which is a weighted matrix factorization problem. A balancing regularizer

$V$0

and a geometry-aware preconditioner in the $V$1 metric yield geometric convergence, with population excess loss

$V$2

under spectral initialization and sufficiently small step size (Goel et al., 2 Mar 2026). This establishes a rare global optimization result for softmax self-attention.

Training objective also leaves a structural signature on the bilinear form $V$3. The update

$V$4

decomposes into rank-1 token-pair outer products, making symmetry and directionality analytically tractable (Saponati et al., 15 Feb 2025). Bidirectional objectives induce approximate symmetry because pairwise contributions appear in both directions, whereas autoregressive objectives induce directionality and column dominance because a token is used disproportionately as context for future predictions. Empirically, encoder-only models such as BERT, RoBERTa, XLM-R, ALBERT, and ModernBERT exhibit high symmetry scores, while decoder-only models such as GPT, LLaMA2/3, Mistral, Mixtral, and Phi exhibit strong negative directionality scores and low symmetry (Saponati et al., 15 Feb 2025).

When energy structure is absent or relaxed, Jacobian analysis becomes central. For recurrent self-attention systems such as

$V$5

the Jacobian factorizes as

$V$6

and normalization layers suppress the spectral radius by removing radial components and shrinking complex modes (Tomihari et al., 26 May 2025). The normalization bound

$V$7

formalizes this effect. Empirically, learned $V$8 remains small, Lyapunov exponents concentrate near zero, and the best-performing models operate slightly on the chaotic side of criticality (Tomihari et al., 26 May 2025).

A complementary large-$V$9 theory derives dynamical mean-field equations for a simplified 1-bit self-attention network. The order parameters

$$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$0

and attention field $$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$1 satisfy closed equations obtained from a generating functional, and the resulting mean-field dynamics exhibits periodic, quasi-periodic, and chaotic regimes, with nonequilibrium phase transitions and chaotic bifurcations even for short context $$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$2 and few features $$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$3 (Poc-López et al., 2024). This moves the subject beyond asymptotic equilibrium analysis and shows that rich nonstationary behavior already appears in minimal self-attention systems.

5. Architectural realizations across domains

Several architectures build the dynamics directly into the attention mechanism rather than merely analyzing it post hoc. Kuramoto Attention places each hidden coordinate on the torus, $$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$4, and scores tokens by gated cosine similarity with rotary phase drift,

$$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$5

Because values are raw phase states, the tangent update is exactly

$$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$6

the classical Kuramoto coupling term (Nunley, 10 Jun 2026). The model therefore interprets attention as “selection then synchronization.” On enwiki8, its validation bits-per-character stays within $$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$7 BPC of a matched RoPE+SwiGLU transformer at one million parameters and is level on the median at five million parameters (Nunley, 10 Jun 2026).

In spiking Transformers, Neural Dynamics Self-Attention replaces explicit matrix storage by membrane dynamics. Standard Spiking Self-Attention computes

$$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$8

but still requires the $$\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,$$9 intermediate $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$0 (Zhang et al., 9 Mar 2026). LRF-Dyn adds localized receptive fields and rewrites attention as charge-fire-reset dynamics,

$$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$1

thereby eliminating explicit $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$2 storage. On Spikformer-8-512, LRF-Dyn reduces inference memory by $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$3 while improving accuracy by $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$4, and Table 1 lists storage complexity $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$5 rather than $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$6 (Zhang et al., 9 Mar 2026).

Task-specific attention dynamics also appear in more conventional domains. CARSA uses a single-layer bidirectional LSTM with hidden size $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$7 to encode per-component autoregression, then applies self-attention across $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$8 ICA components, treating the learned attention matrix as a subject-specific directed connectivity graph for schizophrenia classification (Mahmood et al., 2021). Dynamic Self-Attention adapts capsule-network routing to NLP: with $$v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)} {\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),$$9 routing iterations, it computes sentence-specific dynamic vectors $$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$0 and achieves $$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$1 on SNLI for the $$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$2D single-DSA model and $$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$3 for the $$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$4D multiple-DSA model, while the single model uses $$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$5M parameters (Yoon et al., 2018).

A different biological route derives self-attention as an emergent property of neuron–astrocyte dynamics. In astrocyte-gated associative memory, gains $$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$6 evolve on the simplex by the entropy-regularized replicator equation

$$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$7

with squared-overlap scores

$$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$8

At fixed points,

$$\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)} {\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.$$9

so astrocytic gains implement softmax-normalized routing over memory patterns (Vivet et al., 28 Apr 2026). In high-load, high-interference retrieval, this mechanism improves accuracy relative to classical Hopfield dynamics and recent neuron–astrocyte baselines (Vivet et al., 28 Apr 2026).

6. Probabilistic interpretations, interpretability, and limitations

A probabilistic reading identifies self-attention with conditional generative structure. For a one-layer model with tied embeddings and identity values, the attention output

$A(x)$00

is exactly equivalent to a context-conditioned Markov chain whose transition probabilities are reweighted by the empirical token frequencies of the prompt (Ildiz et al., 2024). The CCMC transition is

$A(x)$01

and this equivalence yields consistency and sample-complexity results under co-occurrence connectivity conditions (Ildiz et al., 2024). For a single autoregressive trajectory, however, the same mechanism produces a winner-takes-all effect: in the $A(x)$02 case with one weak token, the weak token’s empirical frequency decays polynomially, providing a mathematical explanation for repetitive generation (Ildiz et al., 2024).

Exponential Family Attention generalizes this probabilistic viewpoint beyond categorical next-token prediction. It models

$A(x)$03

with natural parameter $A(x)$04 obtained from the attended context

$A(x)$05

This allows the same self-attention dynamics to parameterize categorical, Bernoulli, Gaussian, or Poisson conditionals, and the theory establishes linear identifiability up to invertible transforms together with a $A(x)$06-type excess-risk bound (Wibisono et al., 28 Jan 2025).

Interpretability claims are domain-dependent and must be qualified. In CARSA, attention-derived adjacency matrices are interpretable as subject-specific effective connectivity and highlight predictive components, but they are explicitly not a proof of causality; hemodynamic lag, sampling rate, and noise remain confounds, and the learned graph is tuned for classification rather than mechanistic inference (Mahmood et al., 2021). In next-token prediction, Token Priority Graphs give a graph-theoretic account of why topic stability dominates: the priority order induced by self-attention is preserved under mixed-topic training, and a spontaneous topic change can occur only if lower-priority tokens outnumber all higher-priority tokens of the input topic; longer contexts and more ambiguous topics reduce the likelihood of such a change (Jia et al., 10 Jan 2025).

Theoretical limitations are equally explicit. Energy-based monotonicity results typically require symmetry, value alignment, projection structure, or orthogonality conditions, and approximate robustness is only partially understood (Pendharkar, 5 May 2026). The Jacobian-based perspective emphasizes that standard multi-head, discrete-time, normalized attention often does not admit a useful global energy function at all, motivating pseudo-energy diagnostics such as

$A(x)$07

for monitoring inference instead of asserting true Lyapunov behavior (Tomihari et al., 26 May 2025). A plausible synthesis is that “self-attention dynamics” names not one dynamical theory but a family of compatible lenses—variational, spectral, probabilistic, recurrent, and training-geometric—each exact only on its own regime of assumptions.