Mean-Field Measure-Theoretic Transformer

• Mean-field measure-theoretic transformers are models that replace discrete token embeddings with probability measures, enabling analysis via nonlocal continuity equations.
• They employ gradient-flow structures and support-preserving maps to rigorously characterize self-attention dynamics, clustering, and expressivity in transformer architectures.
• The framework unifies concepts from optimal transport, aggregation PDEs, and measure-to-measure regression, providing insights into metastability and training-induced effects.

Searching arXiv for relevant papers on mean-field, measure-theoretic transformers and closely related formulations. Mean-field measure-theoretic transformers are mathematical idealizations of transformer architectures in which a finite collection of token embeddings is replaced by a probability measure, and the layerwise action of self-attention is represented either as a pushforward map on measures or as a transport equation on the space of probability measures. In this formulation, tokens are treated as particles, layer depth is treated as a time variable, and the empirical token cloud converges, in the large-token or large-depth limit, to a continuum law governed by a nonlocal continuity equation. Across recent work, this framework has been used to study self-attention dynamics on the unit sphere under layer normalization, characterize transformers as support-preserving maps between measures, establish universality results for measure-to-measure regression, and analyze clustering, metastability, low-temperature concentration, auxiliary-variable anti-collapse, and control-theoretic training effects (Burger et al., 6 Jan 2025, Chen et al., 20 Apr 2025, Furuya et al., 30 Sep 2025, Vandergrift et al., 27 May 2026).

1. Measure-theoretic formulation of transformer dynamics

A central starting point is to represent a prompt of $n$ tokens by points $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$, with the sphere arising from layer normalization or related normalization schemes. In the simplest self-attention model with no learned $Q,K,V$ and temperature parameter $\beta\ge 0$, the token dynamics are modeled by the interacting-particle ODE

$$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$

so that each token moves along the tangent plane of the sphere according to a weighted average of all other tokens (Chen et al., 20 Apr 2025). Closely related formulations retain learned matrices $Q,K,V$ and write the continuous-time flow as

$\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$

with the normalizing factor $Z_{\beta,i}(t)$ implementing row-wise softmax (Bruno et al., 29 Sep 2025).

The empirical measure

$\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$

encodes the token cloud as an element of $\mathcal P(\mathbb S^{d-1})$. In the mean-field limit $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$0, this empirical measure converges to a continuum measure $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$1 satisfying a nonlinear continuity equation. For the simplest isotropic model, one obtains

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$2

where $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$3 is the surface divergence on the sphere (Chen et al., 20 Apr 2025). In the more general $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$4 setting, the mean-field PDE takes Vlasov-type form,

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$5

again expressing self-attention as a nonlocal transport field on measures (Bruno et al., 29 Sep 2025).

This measure-theoretic perspective admits two complementary readings. First, it is a continuum approximation of a large token system, derived from empirical measures and propagation-of-chaos arguments. Second, it is an exact formulation of a class of “in-context maps” that act on measures by pushforward, so that a transformer layer becomes a map $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$6 for some context-dependent token map $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$7 (Furuya et al., 30 Sep 2025).

2. Gradient-flow structure and metric geometry

For symmetric interaction kernels, the mean-field transformer dynamics admit a Wasserstein-gradient-flow structure. In the layer-normalized spherical model with symmetric attention matrix $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$8, the interaction kernel is

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$9

and the associated energy functional is

$Q,K,V$0

or, in the isotropic temperature-parametrized case,

$Q,K,V$1

(Burger et al., 6 Jan 2025, Chen et al., 20 Apr 2025).

A distinctive feature of the layer-normalized setting is that the natural geometry is not always the standard $Q,K,V$2 geometry. One introduces a nonlocal mobility

$Q,K,V$3

and defines a “restricted-Wasserstein” distance $Q,K,V$4 through the Benamou–Brenier-type formula

$Q,K,V$5

where $Q,K,V$6 satisfies the continuity equation. Under this construction, $Q,K,V$7 is a complete, geodesic metric space whose topology coincides with narrow convergence (Burger et al., 6 Jan 2025).

Within this geometry, the gradient-flow law takes the form

$Q,K,V$8

and the continuity equation becomes a nonlocal nonlinear PDE on the sphere (Burger et al., 6 Jan 2025). In the isotropic model studied in the clustering work, the same dynamics can be viewed as the reversed Wasserstein-gradient flow of $Q,K,V$9, with energy dissipation identity

$\beta\ge 0$0

(Chen et al., 20 Apr 2025).

This gradient-flow interpretation links transformer dynamics to aggregation equations, synchronization models, and optimal transport. The analogy with the Kuramoto model is particularly explicit: for $\beta\ge 0$1 and $\beta\ge 0$2, the unnormalized self-attention dynamics reduce to the homogeneous Kuramoto model on $\beta\ge 0$3, with synchronization corresponding to collapse of the measure onto a Dirac mass (Rigollet, 1 Dec 2025).

3. Support-preserving maps and transformer expressivity on measures

A distinct strand of the theory studies transformers not only as dynamical systems but as a class of maps between measures. On a compact state space $\beta\ge 0$4, an in-context map is defined by a measurable function

$\beta\ge 0$5

with induced map

$\beta\ge 0$6

For an atomic input $\beta\ge 0$7, this means

$\beta\ge 0$8

so the measure transformation preserves atom multiplicities and moves each support point through a context-dependent map (Furuya et al., 30 Sep 2025).

The characterization theorem in "Transformers through the lens of support-preserving maps between measures" identifies exactly when a continuous map on measures admits such an in-context representation. The two conditions are support preservation and uniform continuity of the regular part of the derivative $\beta\ge 0$9 in a Wasserstein-plus-state metric (Furuya et al., 30 Sep 2025). This yields a formal answer to the question of what kind of measure maps transformers represent: they are precisely those continuous measure maps compatible with support-preserving pushforwards and sufficiently regular local derivatives.

Measure-theoretic self-attention fits this framework. A multi-head self-attention layer is written as

$$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$0

and the induced map $$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$1 is support-preserving. Under boundedness and Lipschitz assumptions on the weights, its regular derivative satisfies the hypotheses of the characterization theorem (Furuya et al., 30 Sep 2025).

This viewpoint also connects finite-depth transformers to transport PDEs. For a Vlasov-type equation

$$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$2

the terminal-time solution map $$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$3 is itself support-preserving and has a Lipschitz regular derivative, so it belongs to the transformer-representable class. Conversely, the infinite-depth residual limit of transformers converges to such a nonlocal transport flow, establishing a dictionary between residual layers and Vlasov dynamics (Furuya et al., 30 Sep 2025).

A related expressivity result appears in "Universal Approximation of Mean-Field Models via Transformers," where a finite transformer $$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$4 approximating an $$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$5-particle vector field induces an “expected transformer”

$$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$6

and the $$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$7 error between $$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$8 and the target mean-field vector field is bounded uniformly in $$\dot x_i(t) \;=\;P_{x_i(t)}\!\Bigl[\frac1n\sum_{j=1}^n x_j(t)\,e^{\beta\,\langle x_i(t),x_j(t)\rangle}\Bigr], \qquad P_x[y]=y-\langle x,y\rangle x,$$9 by a finite-particle approximation term plus a sampling term (Biswal et al., 2024). This suggests that finite transformers can approximate continuum mean-field vector fields whenever they approximate the corresponding finite-particle dynamics.

4. Measure-to-measure regression and dynamic transformer operators

The measure-theoretic transformer framework extends beyond forward-pass analysis to supervised learning problems in which both inputs and outputs are probability measures. In "Measure-to-measure Regression with Transformers," the task is to learn an operator

$Q,K,V$0

from empirical measure pairs $Q,K,V$1, with a parametric family $Q,K,V$2 satisfying $Q,K,V$3 (Vandergrift et al., 27 May 2026).

Two constructions are given. The static version uses a pushforward-by-transformer map,

$Q,K,V$4

which, for empirical $Q,K,V$5, becomes

$Q,K,V$6

Inside one transformer block,

$Q,K,V$7

$Q,K,V$8

which realizes a mean-field interaction because the attention weights depend on the entire empirical measure (Vandergrift et al., 27 May 2026).

The dynamic version interprets $Q,K,V$9 as the endpoint of a continuity equation,

$\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$0

with velocity field $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$1 parameterized by a time-conditioned transformer $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$2. Training uses Conditional Flow Matching with loss

$\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$3

for interpolations $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$4 (Vandergrift et al., 27 May 2026).

The paper also provides an informal mean-field analysis: as $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$5, empirical measures converge in Wasserstein metric to a population law; each attention block converges to a limiting integral operator; and continuity of $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$6 in $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$7 follows from uniform bounds on $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$8 (Vandergrift et al., 27 May 2026). This makes explicit that the measure-theoretic transformer is not merely an asymptotic description of standard attention, but a general nonlinear operator class on $\dot x_i(t)\;=\;P_{x_i(t)}\!\Bigl(\tfrac{1}{Z_{\beta,i}(t)}\sum_{j=1}^N e^{\beta\<Qx_i(t),Kx_j(t)\>}Vx_j(t)\Bigr),$9.

5. Clustering, synchronization, and metastability

One of the most studied consequences of the mean-field transformer formalism is token clustering. In "Quantitative Clustering in Mean-Field Transformer Models," for an initial measure $Z_{\beta,i}(t)$0 with density $Z_{\beta,i}(t)$1 and nonzero mean

$Z_{\beta,i}(t)$2

there exist constants $Z_{\beta,i}(t)$3 depending on $Z_{\beta,i}(t)$4 and $Z_{\beta,i}(t)$5 such that for all $Z_{\beta,i}(t)$6, the solution converges exponentially fast in $Z_{\beta,i}(t)$7-Wasserstein distance to a Dirac mass: $Z_{\beta,i}(t)$8 (Chen et al., 20 Apr 2025).

The proof combines several ingredients. First, stationary points satisfy $Z_{\beta,i}(t)$9, and the only local maxima of the energy are Dirac measures. Second, on a small spherical cap of opening $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$0, a Polyak–Łojasiewicz inequality holds: $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$1 with an explicit cap estimate stated as Theorem 2.3 (Chen et al., 20 Apr 2025). Third, one shows that after finite time $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$2 the flow concentrates into such a cap, after which the PL inequality yields exponential contraction (Chen et al., 20 Apr 2025).

A broader qualitative theory appears in "The Mean-Field Dynamics of Transformers." There, self-attention is treated as an interacting particle system on the sphere, with both normalized and unnormalized models. The paper states an almost-sure clustering theorem: for $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$3, $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$4, and almost every initial configuration, solutions of the self-attention flow converge to complete synchronization,

$\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$5

equivalently $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$6 (Rigollet, 1 Dec 2025). The argument uses analyticity of the gradient flow, Łojasiewicz convergence, identification of non-synchronized equilibria as strict saddles, and the center-stable manifold theorem (Rigollet, 1 Dec 2025).

At the same time, this collapse is not necessarily immediate. Several papers analyze metastable multi-cluster states. "Emergence of meta-stable clustering in mean-field transformer models" linearizes the mean-field PDE around uniform initialization and expands in spherical harmonics. The unstable mode is determined by the index

$\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$7

with $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$8 the Gegenbauer coefficients of the kernel (Bruno et al., 2024). In $\mu_t^N=\frac1N\sum_{i=1}^N\delta_{x_i(t)}$9, for $\mathcal P(\mathbb S^{d-1})$0, this becomes

$\mathcal P(\mathbb S^{d-1})$1

and the dominant mode specifies a metastable manifold carrying $\mathcal P(\mathbb S^{d-1})$2-fold structure (Bruno et al., 2024). The paper describes a regime in which the solution lingers near this manifold for long times before eventual collapse.

"A multiscale analysis of mean-field transformers in the moderate interaction regime" sharpens this by identifying three phases when $\mathcal P(\mathbb S^{d-1})$3 is large and $\mathcal P(\mathbb S^{d-1})$4 slowly: a fast alignment phase governed by a linear transport PDE, an intermediate heat phase after rescaling time by $\mathcal P(\mathbb S^{d-1})$5, and a slow pairing phase with exponentially rescaled time in which only the closest pair of clusters moves (Bruno et al., 29 Sep 2025). This multiscale decomposition suggests that clustering is not a single event but a hierarchy of concentration mechanisms.

6. Low-temperature concentration, normalization effects, and anti-collapse mechanisms

The mean-field literature also studies how architectural details modify clustering. In the low-temperature regime, "Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime" considers a single-head self-attention model with matrices $\mathcal P(\mathbb S^{d-1})$6, writing $\mathcal P(\mathbb S^{d-1})$7. The mean-field equation is

$\mathcal P(\mathbb S^{d-1})$8

with softmax-based velocity field, and its zero-temperature limit replaces the softmax by the hard maximizer

$\mathcal P(\mathbb S^{d-1})$9

Assuming $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$00 symmetric with dominant eigenspace $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$01, the projection

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$02

onto $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$03 defines the limiting stationary distribution $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$04 (Alcalde et al., 11 May 2026).

The main quantitative result is a Wasserstein bound: $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$05 equivalently with explicit constants as

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$06

(Alcalde et al., 11 May 2026). The paper interprets this as an initial collapse toward $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$07, followed by a metastable plateau of length $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$08, after which a different terminal phase may emerge (Alcalde et al., 11 May 2026).

Normalization changes contraction rates as well. In the equiangular reduction studied in "The Mean-Field Dynamics of Transformers," post-layer normalization yields

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$09

whereas pre-layer normalization yields

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$10

showing that pre-LN delays representational collapse from exponential to polynomial rate (Rigollet, 1 Dec 2025). The same work describes a long-context phase transition under scaling $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$11: below a critical threshold the attention behaves like averaging and collapses representations, while above it the diagonal term dominates and attention approaches the identity (Rigollet, 1 Dec 2025).

Several papers challenge the view that mode collapse is inevitable. "Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables" augments the state space from $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$12 to $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$13, where $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$14 carries auxiliary variables such as positions or prompt identifiers, and constrains admissible measures to have fixed marginal $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$15 on $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$16. The energy becomes

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$17

with kernels of the form

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$18

When $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$19 is nonatomic, global maximizers can be chosen as graph laws

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$20

so the conditional distribution in $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$21 is Dirac at each $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$22, but the marginal

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$23

need not collapse (Imaizumi et al., 28 May 2026).

Two explicit constructions are given. For rotary positional encoding,

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$24

the graph law $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$25 is a global maximizer, and its marginal is uniform on a non-degenerate orbit (Imaizumi et al., 28 May 2026). For prefix-token or gauge variables, any target law can be realized exactly as a marginal pushforward $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$26, yielding what the paper calls “gauge-level exact realization” (Imaizumi et al., 28 May 2026). This establishes an anti-collapse mechanism absent from the vanilla mean-field model.

7. Training, control, and broader theoretical extensions

The mean-field measure-theoretic transformer has also been extended to include training dynamics. "A First-Order Mean Field Control Analysis of Transformer Layers under Cross-Entropy Training" treats depth $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$27 as time and layer parameters as a control $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$28. The hidden state law $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$29 satisfies

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$30

where $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$31 is the vector field induced by one residual transformer block (Huan et al., 22 Jun 2026). The population objective is the cross-entropy risk

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$32

and the discrete residual recursion is interpreted as explicit Euler applied to the controlled ODE, with pathwise error $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$33 for step size $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$34 (Huan et al., 22 Jun 2026). The limiting optimal-control problem satisfies a Pontryagin condition with terminal adjoint

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$35

making the softmax residual appear explicitly in the adjoint equation (Huan et al., 22 Jun 2026).

A different training-aware formulation appears in "Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers." There, attention induces an entropy-regularized interaction energy

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$36

while a trainable FFN contributes a control drift $x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$37 inside the mean-field PDE (Isobe et al., 8 May 2026). The optimal-control problem

$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$38

exhibits a three-phase trained depth profile: early clustering, a long turnpike plateau near the static minimizers, and a terminal escape region close to the final layers (Isobe et al., 8 May 2026). This provides a training-induced correction to purely inference-time clustering theories.

Beyond forward inference and control, the framework has been adapted to other questions. "Perceptrons and localization of attention’s mean-field landscape" adds a two-layer perceptron potential to the attention energy and proves that stationary measures are generically atomic and localized on subsets of the sphere (Álvarez-López et al., 29 Jan 2026). "Dynamical Mean-Field Theory of Self-Attention Neural Networks" studies a different large-$x_1(t),\dots,x_n(t)\in \mathbb S^{d-1}$39 limit using path-integral methods for 1-bit tokens and weights, deriving closed dynamical mean-field equations for low-dimensional order parameters and identifying nonequilibrium phase transitions including periodic, quasi-periodic, and chaotic regimes (Poc-López et al., 2024). "Transformers as Measure-Theoretic Associative Memory" formulates long-context recall as a problem on mixture measures and proves a minimax-optimal statistical rate for a shallow measure-theoretic transformer composed with an MLP under an RKHS spectral assumption (Kawata et al., 2 Feb 2026).

A recurring misconception is that the measure-theoretic transformer is a single model class with a single asymptotic prediction of collapse. The literature instead shows several distinct regimes. In some symmetric and unregularized settings, Dirac clustering is the dominant attractor (Chen et al., 20 Apr 2025, Rigollet, 1 Dec 2025). In other settings, uniform laws, continuous stationary families, multi-cluster metastable states, or non-Dirac maximizing marginals arise due to the spectrum of the attention matrix, auxiliary variables, normalization, entropy, or training-induced drifts (Burger et al., 6 Jan 2025, Bruno et al., 2024, Imaizumi et al., 28 May 2026, Isobe et al., 8 May 2026). A plausible implication is that “mean-field measure-theoretic transformer” designates a general analytical framework rather than a unique canonical equation.

Within that framework, the unifying idea is that transformer architectures can be interpreted as nonlinear operators and transport flows on probability spaces. This has produced a shared vocabulary—empirical measures, support-preserving pushforwards, Wasserstein or restricted-Wasserstein geometry, nonlocal continuity equations, Lyapunov and PL inequalities, metastable manifolds, and control-theoretic adjoints—for connecting transformer theory to synchronization, aggregation PDEs, associative memory, and optimal transport (Burger et al., 6 Jan 2025, Furuya et al., 30 Sep 2025, Vandergrift et al., 27 May 2026).