---
title: 'Self-Attention Dynamics: Models and Mechanisms'
url: https://www.emergentmind.com/topics/self-attention-dynamics
type: topic
---

# Self-Attention Dynamics: Models and Mechanisms

Self-attention dynamics denotes the study of how attention weights, token states, and induced interaction geometries evolve under repeated attention updates, continuous-depth limits, or training. In the current literature, the term spans several distinct but related objects: interacting-particle flows on the unit sphere, mean-field transport equations, recurrent discrete-time attention maps, dynamic-routing sentence encoders, task-conditioned directed graphs over multivariate time series, and probabilistic generative models in which attention directly parameterizes transition or exponential-family conditionals [2604.26085] [2501.03096] [1808.07383] [2111.01271] [2402.13512] [2501.16790]. Across these settings, the common question is how attention converts similarity structure into dynamical evolution—whether by clustering, spectral selection, synchronization, adaptive routing, or optimization-induced geometric bias.

## 1. Core formulations of self-attention as a dynamical system

A canonical continuous-time formulation models tokens \(x_i(t)\in S^{d-1}\) on a product of spheres, with attention matrix \(A(x)\) defined row-wise by
\[
A_{ij}(x)=\frac{\exp(\langle Qx_i,Kx_j\rangle)}{\sum_{\ell=1}^n \exp(\langle Qx_i,Kx_\ell\rangle)},
\]
and dynamics
\[
\dot{x}_i=(I-x_i x_i^\top)\sum_{j=1}^n A_{ij}(x)Vx_j.
\]
In this representation, time corresponds to continuous depth, the tangent projection models layer normalization, and the interaction graph is fully connected but state-dependent [2511.11553]. Closely related formulations impose the symmetry constraint \(Q^\top K=V=V^\top\), yielding an idealized Transformer-type flow
\[
\dot{x}_i
=
P_{x_i}^\perp
\left(
\frac{1}{Z_{\beta,i}}
\sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle}Vx_j
\right),
\qquad
Z_{\beta,i}=\sum_{k=1}^N e^{\beta\langle x_i,Vx_k\rangle},
\]
which makes the spectral structure of \(V\) directly visible in the induced motion [2604.26085].

A second line of work treats self-attention dynamics as a mean-field transport problem. For layer-normalized attention on the sphere, the empirical measure of particles converges formally to a continuity equation
\[
\partial_t \mu_t+\operatorname{div}_{\mathbb{S}^{d-1}}(\mu_t\,v_t)=0,
\]
with velocity
\[
v_t(x)=P_x^\perp\left(\frac{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}Vy\,d\mu_t(y)}
{\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}d\mu_t(y)}\right),
\]
or, in a related fixed-weight setting,
\[
\mathcal{X}[\mu](x)=\frac{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}Vy\,d\mu(y)}
{\int_{\mathbb{R}^d} e^{\langle Qx,Ky\rangle}d\mu(y)}.
\]
These PDE limits make explicit that self-attention is a nonlocal transport mechanism whose velocity depends on the entire token distribution [2501.03096] [2305.05465].

Not all self-attention dynamics are continuous. In Dynamic Self-Attention, the dynamics are iterative and sentence-specific: projected word features \(\hat{x}_{j|i}\) are combined through routing logits \(q_{ij}\), weights \(a_{ij}\), aggregates \(s_j=\sum_i a_{ij}\hat{x}_{j|i}\), and dynamic vectors \(z_j=\tanh(s_j)\), with updates \(q_{ij}\leftarrow q_{ij}+\hat{x}_{j|i}^\top z_j\). The resulting \(z_j\) are dynamic weight vectors that vary across sentences and routing iterations rather than being fixed learned queries [1808.07383]. In CARSA, the temporal dimension is first compressed per variable through
\[
c_t^i=f_\theta(x_t^i \mid f_\theta(x_j^i);j=1\to t-1),\qquad
y^i=\sum_{t=1}^n c_t^i,
\]
after which self-attention acts across variables rather than across time:
\[
z^i=f_\phi(y^i \mid y^j;j=1\to m).
\]
Here the induced attention weights define a subject-specific directed adjacency \(A_{ij}\) from component \(j\) to component \(i\) [2111.01271].

These formulations differ in state space, iteration variable, and semantics, but they share a common structure: attention weights are endogenous, state-dependent couplings, and the ensuing dynamics can be studied either at the level of individual tokens, variables, or distributions.

## 2. Geometric, variational, and energy-based structures

A major mathematical theme is that certain self-attention systems admit a gradient-flow interpretation. For multi-head self-attention with symmetric score matrices \(M_h=M_h^\top\) and value alignment \(W^{V,h}=M_h\), the interaction energy
\[
E^h(X)=\frac{1}{2\beta n^2}\sum_{i=1}^n\sum_{j=1}^n e^{\beta\langle x_i,M_hx_j\rangle},
\qquad
E_{\mathrm{multi}}=\sum_{h=1}^H E^h
\]
satisfies the exact Lyapunov identity
\[
\frac{dE_{\mathrm{multi}}}{dt}=\frac{1}{n}\sum_{i=1}^n \|\dot{x}_i\|^2\ge 0
\]
for both flat and spherical dynamics [2605.04279]. In the symmetric single-kernel setting, the same phenomenon appears as a weighted Riemannian gradient flow: with
\[
E_\beta(X)=\frac{1}{2\beta}\sum_{i=1}^N\sum_{j=1}^N e^{\beta\langle x_i,Vx_j\rangle},
\]
the induced vector field is precisely the weighted gradient of \(E_\beta\) under a state-dependent metric involving the partition functions \(Z_{\beta,i}\) [2604.26085].

At the mean-field level, layer-normalized self-attention with \(V=\pm D\) yields the interaction energy
\[
\mathcal{E}(\mu)=\pm \frac12\int_{\mathbb{S}^{d-1}}\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}\,d\mu(x)\,d\mu(y),
\]
together with a Wasserstein-type geometry with nonlocal mobility
\[
m_\mu(x)=\int_{\mathbb{S}^{d-1}} e^{\langle x,Dy\rangle}\,d\mu(y).
\]
The resulting PDE is a metric gradient flow in \((\mathcal{P}(S^{d-1}),W_{m,2})\), and stationary measures are exactly those for which \(\nabla_{\mathbb{S}^{d-1}}W[\mu](x)=0\) on the support [2501.03096]. This places self-attention near the theory of aggregation equations, but with two nonstandard ingredients: confinement to the sphere and a mobility depending on the current measure.

A distinct but related construction expresses self-attention as the negative gradient of local energies rather than a single global energy. With vector spins \(x_i\in S^d\) and pairwise couplings \(J_{ij}\), define
\[
e_i(x_i,\{x_j\}_{j\neq i})=-\log\sum_{j\neq i}\exp(x_i^\top J_{ij}x_j).
\]
Then
\[
-\nabla_{x_i}e_i=\sum_{j\neq i}\alpha_{i\leftarrow j}(x)J_{ij}x_j,
\qquad
\alpha_{i\leftarrow j}(x)=\frac{\exp(x_i^\top J_{ij}x_j)}
{\sum_{\ell\neq i}\exp(x_i^\top J_{i\ell}x_\ell)},
\]
so the local attention update is exactly a negative local-energy gradient, with a pseudo-likelihood interpretation rather than a global Hopfield energy [2409.16112].

These variational viewpoints are powerful but conditional. Multi-head monotonicity can fail at the per-head level on the sphere because of radial shadow terms \(a_i^{h'}=\langle f_i^{h'},x_i\rangle\), and approximate value alignment, orthogonality, or projection structure are treated as controlled idealizations rather than generic properties of trained Transformers [2605.04279]. The mean-field and local-energy theories similarly rely on symmetry or architectural tying assumptions that are mathematically advantageous but narrower than standard large-scale practice [2501.03096] [2409.16112].

## 3. Spectral selection, clustering, multistability, and metastability

When self-attention is viewed as an interacting particle system, asymptotic behavior is organized by clustering and by the spectrum of the value operator. In the fixed-weight continuous-time model
\[
\dot{x}_i(t)=\sum_{j=1}^n P_{ij}(t)Vx_j(t),
\qquad
P_{ij}(t)=\frac{\exp(\langle Qx_i(t),Kx_j(t)\rangle)}
{\sum_{\ell=1}^n \exp(\langle Qx_i(t),Kx_\ell(t)\rangle)},
\]
the rescaled dynamics \(z_i(t)=e^{-tV}x_i(t)\) exhibits convergence to specific geometric limiting objects [2305.05465]. For \(V=I_d\) and \(Q^\top K\succ 0\), the convex hull of the rescaled particles is non-increasing and converges to a convex polytope, and each token converges either to \(0\) or to a point in a finite set \(S\subset \partial\mathbb{K}\cup\{0\}\). In one dimension, the self-attention matrix itself converges to a low-rank Boolean matrix, analytically confirming the emergence of leaders [2305.05465].

The symmetric spectral theory sharpens this picture. In the eigenbasis of \(V\), coefficient dynamics take the replicator-type form
\[
\dot{c}_{i,k}=\lambda_k\sum_{j=1}^N K_{ij}(C)c_{j,k}-\phi_i(C)c_{i,k},
\]
with \(K_{ij}(C)\) the softmax kernel and \(\phi_i\) the modal average. This reveals a mode-selection mechanism: if one positive eigenvalue \(\lambda_1\) strictly dominates all others in modulus and initial conditions lie in a forward-invariant cone, then all tokens converge to the dominant eigendirection; if \(V\) is negative definite, the dynamics selects sign-split polarization toward the most negative eigendirection [2604.26085]. The same spectral dependence appears in the broader single-head ODE on spheres, where equilibria fall into four classes—consensus, bipartite consensus, clustering, and polygonal—and multistability is typical rather than exceptional [2511.11553].

Metastability introduces a slower timescale. For self-attention on the unit sphere with separated low-temperature initial clusters, particles remain trapped near a multi-cluster configuration for an exponentially long period, even though the global asymptotic state is full collapse to a single cluster [2410.06833]. The metastability theorem gives explicit times
\[
T_1 \le 2n\, e^{8\varepsilon\beta} + \frac{n\beta^2\lambda e}{\beta-1},
\qquad
T_2 \ge \frac{\varepsilon}{n}e^{(1-\alpha)\beta},
\]
such that particles remain within their spherical caps up to \(T_2\) and intra-cap distances are exponentially small on \([T_1,T_2]\). Beyond this regime, after an appropriate time-rescaling, the energy exhibits a staircase profile with saddle-to-saddle-like behavior [2410.06833].

Clustering is accompanied by entropy dynamics. In the scalar-head equiangular regime of multi-head attention, the attention entropy
\[
H_i^h=-\sum_{j=1}^n p_{ij}^h\log p_{ij}^h
\]
obeys
\[
\frac{dH_i^h}{dt}=-\beta^2\,\mathrm{Cov}_{p_{i\cdot}^h}(s_j,\dot{s}_j),
\]
and increases monotonically toward \(\log n\) as clustering progresses [2605.04279]. A plausible implication is that “collapse” in self-attention need not mean low-entropy concentration of row distributions; in several symmetric regimes, geometric clustering of states coincides with equalization of scores and more uniform attention.

## 4. Training dynamics, optimization geometry, and objective-induced structure

Another use of the phrase “self-attention dynamics” concerns how training shapes the score geometry itself. In a single-layer softmax self-attention model trained for linear regression, the finite-data predictor
\[
\hat{y}(x;\theta)=A\,\mu(x;B),\qquad
\mu(x;B)=\sum_{j=1}^n w_j(x;B)x_j
\]
reduces in the infinite-data Gaussian limit to the bilinear map \(A\Sigma B^\top x\). The population loss becomes
\[
L(\theta)=L^\star+\frac12\|A\Sigma B^\top \Sigma^{1/2}-M\Sigma^{1/2}\|_F^2,
\]
which is a weighted matrix factorization problem. A balancing regularizer
\[
R(\theta)=\frac18\|\Sigma^{1/2}(A^\top A-B^\top \Sigma B)\Sigma^{1/2}\|_F^2
\]
and a geometry-aware preconditioner in the \(P=\mathrm{diag}(I_p,\Sigma)\) metric yield geometric convergence, with population excess loss
\[
L(\theta_m)-L^\star \lesssim n^{-2}\log^6 n+\mu^m
\]
under spectral initialization and sufficiently small step size [2603.01514]. This establishes a rare global optimization result for softmax self-attention.

Training objective also leaves a structural signature on the bilinear form \(W_{qk}=W_QW_K^\top\). The update
\[
\frac{\partial L}{\partial W_{qk}^l}=\sum_i\sum_{j\in C_i}\beta_{ij}^l K_{ij}^{l-1},
\qquad
K_{ij}^{l-1}=x_i^{l-1}{x_j^{l-1}}^\top
\]
decomposes into rank-1 token-pair outer products, making symmetry and directionality analytically tractable [2502.10927]. Bidirectional objectives induce approximate symmetry because pairwise contributions appear in both directions, whereas autoregressive objectives induce directionality and column dominance because a token is used disproportionately as context for future predictions. Empirically, encoder-only models such as BERT, RoBERTa, XLM-R, ALBERT, and ModernBERT exhibit high symmetry scores, while decoder-only models such as GPT, LLaMA2/3, Mistral, Mixtral, and Phi exhibit strong negative directionality scores and low symmetry [2502.10927].

When energy structure is absent or relaxed, Jacobian analysis becomes central. For recurrent self-attention systems such as
\[
X^{(t+1)}=\mathrm{RMSNorm}(X^{(t)}+\eta(C+\mathrm{MSA}(X^{(t)}))),
\]
the Jacobian factorizes as
\[
J_f(X)=J_{\mathrm{RMSNorm}}(Y)\cdot (I_{SD}+\eta J_{\mathrm{MSA}}(X)),
\]
and normalization layers suppress the spectral radius by removing radial components and shrinking complex modes [2505.19458]. The normalization bound
\[
\left\|\frac{\partial \mathrm{RMSNorm}(X+\eta \Delta X)}{\partial X}\right\|_2
\le
\frac{\max_j |\gamma_j|}{R}\bigl(1+|\eta|\|J_{\mathrm{MSA}}(X)\|_2\bigr)
\]
formalizes this effect. Empirically, learned \(\gamma\) remains small, Lyapunov exponents concentrate near zero, and the best-performing models operate slightly on the chaotic side of criticality [2505.19458].

A complementary large-\(N\) theory derives dynamical mean-field equations for a simplified 1-bit self-attention network. The order parameters
\[
m^\alpha_{a,t}=\frac1N\sum_i W^\alpha_{i,a}\langle x_{i,t}\rangle
\]
and attention field \(\hat{A}_t\) satisfy closed equations obtained from a generating functional, and the resulting mean-field dynamics exhibits periodic, quasi-periodic, and chaotic regimes, with nonequilibrium phase transitions and chaotic bifurcations even for short context \(L=4\) and few features \(M\approx 3\) [2406.07247]. This moves the subject beyond asymptotic equilibrium analysis and shows that rich nonstationary behavior already appears in minimal self-attention systems.

## 5. Architectural realizations across domains

Several architectures build the dynamics directly into the attention mechanism rather than merely analyzing it post hoc. Kuramoto Attention places each hidden coordinate on the torus, \(\theta\in \mathbb{R}^k/(2\pi\mathbb{Z})^k\), and scores tokens by gated cosine similarity with rotary phase drift,
\[
s_{t,u}=\frac{\tau}{\sqrt{k}}\sum_j g^q_{t,j}g^k_{u,j}
\cos\bigl(\theta_{t,j}-\theta_{u,j}+\omega_j(t-u)\bigr),
\qquad
A_{t,\cdot}=\mathrm{softmax}_{u\le t}\,s_{t,\cdot}.
\]
Because values are raw phase states, the tangent update is exactly
\[
a_{t,j}=\sum_u A_{t,u}\sin(\theta_{u,j}-\theta_{t,j}),
\]
the classical Kuramoto coupling term [2606.11585]. The model therefore interprets attention as “selection then synchronization.” On enwiki8, its validation bits-per-character stays within \(0.02\) BPC of a matched RoPE+SwiGLU transformer at one million parameters and is level on the median at five million parameters [2606.11585].

In spiking Transformers, Neural Dynamics Self-Attention replaces explicit matrix storage by membrane dynamics. Standard Spiking Self-Attention computes
\[
\mathrm{Score}=s\cdot (QK^\top),\qquad
\mathrm{Attn}'=\mathrm{Score}\,V,\qquad
\mathrm{Attn}=\mathrm{SN}\{\mathrm{Attn}'\},
\]
but still requires the \(d\times d\) intermediate \(M[t]=K^\top V\) [2603.19290]. LRF-Dyn adds localized receptive fields and rewrites attention as charge-fire-reset dynamics,
\[
X_n[t]=A\odot X_{n-1}[t]+\mathrm{Token}_n[t],
\]
thereby eliminating explicit \(d^2\) storage. On Spikformer-8-512, LRF-Dyn reduces inference memory by \(49.4\%\) while improving accuracy by \(1.13\%\), and Table 1 lists storage complexity \(O(kd)\) rather than \(O(d^2)\) [2603.19290].

Task-specific attention dynamics also appear in more conventional domains. CARSA uses a single-layer bidirectional LSTM with hidden size \(64\) to encode per-component autoregression, then applies self-attention across \(m=100\) ICA components, treating the learned attention matrix as a subject-specific directed connectivity graph for schizophrenia classification [2111.01271]. Dynamic Self-Attention adapts capsule-network routing to NLP: with \(r=2\) routing iterations, it computes sentence-specific dynamic vectors \(z_j\) and achieves \(86.8\%\) on SNLI for the \(600\)D single-DSA model and \(87.4\%\) for the \(2400\)D multiple-DSA model, while the single model uses \(2.1\)M parameters [1808.07383].

A different biological route derives self-attention as an emergent property of neuron–astrocyte dynamics. In astrocyte-gated associative memory, gains \(p_\mu\) evolve on the simplex by the entropy-regularized replicator equation
\[
\tau_p \dot{p}=(\mathrm{diag}(p)-pp^\top)(f(x)-T\log p),
\]
with squared-overlap scores
\[
f_\mu(x)=\frac{1}{2N}\langle \xi_\mu,\phi(x)\rangle^2.
\]
At fixed points,
\[
p_\mu^\star=\operatorname{softmax}_\mu\!\left(\frac{f_\mu(x^\star)}{T}\right),
\]
so astrocytic gains implement softmax-normalized routing over memory patterns [2604.25481]. In high-load, high-interference retrieval, this mechanism improves accuracy relative to classical Hopfield dynamics and recent neuron–astrocyte baselines [2604.25481].

## 6. Probabilistic interpretations, interpretability, and limitations

A probabilistic reading identifies self-attention with conditional generative structure. For a one-layer model with tied embeddings and identity values, the attention output
\[
f_W(X)=X^\top s_X,\qquad s_X=S\{XW\},
\]
is exactly equivalent to a context-conditioned Markov chain whose transition probabilities are reweighted by the empirical token frequencies of the prompt [2402.13512]. The CCMC transition is
\[
P(y\mid X)=\pi_y^X=\frac{m_y\cdot \pi_{x_L,y}}{m^\top \pi_{x_L}},
\]
and this equivalence yields consistency and sample-complexity results under co-occurrence connectivity conditions [2402.13512]. For a single autoregressive trajectory, however, the same mechanism produces a winner-takes-all effect: in the \(K=2\) case with one weak token, the weak token’s empirical frequency decays polynomially, providing a mathematical explanation for repetitive generation [2402.13512].

Exponential Family Attention generalizes this probabilistic viewpoint beyond categorical next-token prediction. It models
\[
p(z_i\mid z_{-i})=h(z_i)\exp(\eta_i^\top T(z_i)-A(\eta_i)),
\]
with natural parameter \(\eta_i=\lambda_2(c_i)\) obtained from the attended context
\[
c_i=\sum_{j\in \mathrm{ctx}(i)} \alpha_{ij}V_j,\qquad
\alpha_{ij}=\mathrm{softmax}_j\!\left(\frac{Q_i^\top K_j}{\sqrt d}+m_{ij}\right).
\]
This allows the same self-attention dynamics to parameterize categorical, Bernoulli, Gaussian, or Poisson conditionals, and the theory establishes linear identifiability up to invertible transforms together with a \(1/\sqrt{F}\)-type excess-risk bound [2501.16790].

Interpretability claims are domain-dependent and must be qualified. In CARSA, attention-derived adjacency matrices are interpretable as subject-specific effective connectivity and highlight predictive components, but they are explicitly not a proof of causality; hemodynamic lag, sampling rate, and noise remain confounds, and the learned graph is tuned for classification rather than mechanistic inference [2111.01271]. In next-token prediction, Token Priority Graphs give a graph-theoretic account of why topic stability dominates: the priority order induced by self-attention is preserved under mixed-topic training, and a spontaneous topic change can occur only if lower-priority tokens outnumber all higher-priority tokens of the input topic; longer contexts and more ambiguous topics reduce the likelihood of such a change [2501.06382].

Theoretical limitations are equally explicit. Energy-based monotonicity results typically require symmetry, value alignment, projection structure, or orthogonality conditions, and approximate robustness is only partially understood [2605.04279]. The Jacobian-based perspective emphasizes that standard multi-head, discrete-time, normalized attention often does not admit a useful global energy function at all, motivating pseudo-energy diagnostics such as
\[
E_{\mathrm{pseudo}}(t)=-\operatorname{Tr}(X^{(t)\top}\Delta X^{(t)})
\]
for monitoring inference instead of asserting true Lyapunov behavior [2505.19458]. A plausible synthesis is that “self-attention dynamics” names not one dynamical theory but a family of compatible lenses—variational, spectral, probabilistic, recurrent, and training-geometric—each exact only on its own regime of assumptions.

Source: https://www.emergentmind.com/topics/self-attention-dynamics