Measure-Theoretic Self-Attention

• Measure-Theoretic Self-Attention is a framework that expresses attention mechanisms using probability measures, Markov kernels, and vector-valued operators.
• It reconstructs classical attention via Boltzmann–Gibbs reweighting, lookup kernels, mean-field theories, and finite-state operator analogues to capture token dynamics.
• The approach enables rigorous analysis through spectral, graph-diffusion, and PDE-based methods, offering insights into convergence properties and stability criteria.

Searching arXiv for papers on measure-theoretic and operator-theoretic formulations of self-attention. Measure-theoretic self-attention denotes a family of formulations in which attention is expressed in the language of probability measures, Markov kernels, vector-valued integral operators, empirical measures, or measure-valued gradient flows. In the strict sense, it includes exact reconstructions of classical attention via Boltzmann–Gibbs reweighting, lookup kernels, and moment projection, as well as mean-field theories in which the network state is a probability measure over attention-head parameters (Vuckovic et al., 2020, Huan et al., 9 Jun 2026). In a broader but weaker sense, it also includes discrete operator theories in which attention rows are finite-state kernels, geometric transport operators on token graphs, spectral theories of centered attention matrices, and graph-diffusion resolvents with Markovian interpretations (Lin et al., 12 Jul 2026, Hayase et al., 8 Oct 2025, Roffo et al., 26 Feb 2026). Taken together, these works suggest that “measure-theoretic self-attention” is not a single formalism but a cluster of related viewpoints ranging from exact measure theory to finite-dimensional probabilistic and operator-theoretic analogues.

1. Scope and main formulations

The literature separates naturally by the space on which the measure lives. In token-space formulations, a sequence is encoded by an empirical measure over token representations, and attention becomes a kernel acting on that measure. In graph/operator formulations, the relevant object is a nonnegative matrix acting on token-indexed quantities, often with a Markov or substochastic interpretation. In mean-field formulations, the measure lives not on tokens but on head-parameter space, so training becomes a PDE on probability measures. A further strand studies empirical spectral measures of attention matrices, which is measure-theoretic in the sense of random probability measures on the spectrum rather than on token space.

Formulation	State variable	Central object
Exact attention kernel	Empirical token measure	$$\mathbf A_\mu(x,dz)=\Pi[\Psi_{G(x,\cdot)}(\mu)L](dz)$$
Connection-walk attention	Token-position field	$(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$
Mean-field head training	Probability measure on head parameters	$z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$
Spectral/diffusion attention	Attention operator or attention matrix	$(I-\gamma A)^{-1}-I$, or empirical spectral measure $\nu_Y$

The strictest measure-theoretic constructions occur in the attention-kernel model and in mean-field training theory. The operator, spectral, and diffusion formulations are usually finite-dimensional and discrete. Several papers explicitly note that they are not full abstract measure theory even when the analogy is close (Lin et al., 12 Jul 2026, Roffo et al., 26 Feb 2026).

2. Exact token-space kernel theories

A mathematically equivalent measure-theoretic reconstruction of classical attention is given by “A Mathematical Theory of Attention” (Vuckovic et al., 2020). The ambient space is a representation space $E \subseteq \mathbb{R}^d$ with its Borel $\sigma$-algebra, tokens are identified with Dirac measures, and a collection of tokens $X=\{x_1,\dots,x_T\}$ is encoded by the empirical measure

$$m(X):=\frac{1}{|\mathcal T|}\sum_{t\in\mathcal T}\delta_{x_t}.$$

The similarity function is represented by a positive interaction potential

$G(x,y)=\exp(a(x,y)),$

and the softmax weighting step is written as a Boltzmann–Gibbs transformation

$(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$0

Keys are transported to values by a lookup kernel $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$1, and the weighted value distribution is collapsed back to a finite-dimensional representation by a moment projection $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$2. The resulting attention kernel is

$(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$3

For the standard deterministic finite setting, the paper proves exact equivalence with ordinary attention. It also interprets self-attention as the nonlinear map $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$4, so a stack of attention layers becomes a deterministic self-interacting particle system. Within the same framework, the softmatch distribution is characterized as the unique maximum-entropy solution under a moment constraint, and self-attention is shown to be Lipschitz-continuous under suitable assumptions in Wasserstein geometry (Vuckovic et al., 2020).

A complementary discrete operator formulation is developed in “From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers” (Lin et al., 12 Jul 2026). There, a token sequence is a vector field $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$5 over the token-position graph, and attention is expressed as a connection propagation operator

$(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$6

where $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$7 is a nonnegative walk matrix and $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$8 is an edge transport. On the finite discrete space $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$9 with discrete $z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$0-algebra, this is already the discrete analogue of the vector-valued kernel operator

$z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$1

The paper proves that single-head attention is exactly connection propagation with constant transport, and that multi-head attention is exactly a single edge-dependent connection walk whose effective transport is an attention-gated mixture of headwise transports. When the effective walk is reversible and the transports are metric-compatible and inverse-consistent, the generator reduces to a random-walk connection Laplacian with Dirichlet energy

$z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$2

This is a precise discrete kernel theory, but the paper is explicit that it is not yet a full abstract measurable-space treatment (Lin et al., 12 Jul 2026).

3. Probabilistic attention, particle dynamics, and conditional mixing

The token-space kernel view leads naturally to a probabilistic interpretation. In the exact measure-theoretic model, each attention update reweights an empirical measure by a Gibbs factor, pushes that weighted measure through a lookup kernel, and projects the resulting value distribution back to a representation. This makes self-attention a nonlinear Markov transport on empirical measures, and the network depth index plays the role of a discrete-time interacting-particle evolution (Vuckovic et al., 2020).

A statistically different but closely related reformulation appears in “Bidirectional Attention as a Mixture of Continuous Word Experts” (Wibisono et al., 2023). For single-layer, single-head bidirectional self-attention trained with the masked LLM objective, the paper proves exact equivalence to a continuous bag-of-words predictor with context-dependent mixture-of-experts weights:

$z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$3

Here the $z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$4-th expert is attached to token position $z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$5, and the attention weights $z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$6 are the softmax gating probabilities. In finite discrete terms, this supports the representation

$z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$7

where $z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$8. That expectation notation is an interpretive restatement rather than the paper’s own notation, but it is exact on the finite position space. The same paper states that multi-head attention corresponds to stacked MoEs and multiple layers to a mixture of MoEs (Wibisono et al., 2023).

A more explicitly stochastic account of attention rows is given in “Linear Log-Normal Attention with Unbiased Concentration” (Nahshan et al., 2023). The paper treats each row of the attention matrix as a probability distribution over token indices and studies the law of these normalized random weights. Under Gaussian assumptions on queries and keys, it argues that softmax attention entries are approximately log-normal, with row concentration quantified by entropy and spectral gap. It also clarifies a recurring terminological point: “unbiased concentration” does not mean unbiasedness in the Monte Carlo estimator sense; it refers to spectral-gap-based concentration analysis not confounded by systematic column bias (Nahshan et al., 2023).

4. Mean-field and Wasserstein theories of head training

The most explicit genuinely measure-theoretic treatment of self-attention training in the provided literature is “A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training” (Huan et al., 9 Jun 2026). The architecture is deliberately restricted to a single masked self-attention layer with no feed-forward block, no residual connection, no layer normalization, and a fixed vocabulary projection $z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$9. Each attention head is treated as a particle in parameter space, with head parameter

$(I-\gamma A)^{-1}-I$0

and the finite collection of heads is summarized by the empirical measure

$(I-\gamma A)^{-1}-I$1

The model output is rewritten exactly in terms of this empirical law:

$(I-\gamma A)^{-1}-I$2

and in the infinite-head limit the logits are

$(I-\gamma A)^{-1}-I$3

The cross-entropy risk becomes a functional on probability measures,

$(I-\gamma A)^{-1}-I$4

The first variation is

$(I-\gamma A)^{-1}-I$5

and its particle gradient generates a nonlinear Wasserstein gradient flow:

$(I-\gamma A)^{-1}-I$6

This PDE is the mean-field training dynamics. It is accompanied by a static finite-head approximation bound,

$(I-\gamma A)^{-1}-I$7

and by a quantitative finite-time propagation-of-chaos estimate comparing finite-head SGD with the limiting PDE. The paper then analyzes long-time behavior: energy dissipation, convergence to the stationary set under compactness, convergence to a single stationary measure under topological or Kurdyka–Łojasiewicz assumptions, explicit convergence rates under gradient-domination conditions, and local exponential stability under a Wasserstein strong-monotonicity condition. It also gives verifiable stability and instability criteria for Dirac stationary measures (Huan et al., 9 Jun 2026).

This framework is measure-theoretic in a strict sense: the state space is $(I-\gamma A)^{-1}-I$8 or $(I-\gamma A)^{-1}-I$9, the model is parameterized by a probability measure, and training is a transport equation on that measure space. The principal limitation is architectural scope. The paper is explicit that it is not a full transformer theory; multiple layers, residuals, layer normalization, feed-forward blocks, trainable output heads, and stronger noncompact effects are left outside the analysis (Huan et al., 9 Jun 2026).

5. Spectral, graph-diffusion, and resolvent viewpoints

Several recent works move beyond one-step softmax mixing by treating attention as an operator whose multi-step or large-system behavior is the object of analysis. “Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attention” reformulates an attention layer as graph diffusion on a content-adaptive token graph (Roffo et al., 26 Feb 2026). Starting from

$\nu_Y$0

it defines path weights over powers of the attention matrix and introduces the discounted Neumann series

$\nu_Y$1

under $\nu_Y$2. In the row-stochastic case, $\nu_Y$3 is a random-walk operator. In the Frobenius-normalized InfSA construction, $\nu_Y$4 is generally not row-stochastic, but after multiplying by $\nu_Y$5 the matrix $\nu_Y$6 is treated as substochastic, with missing row mass interpreted as absorption. The associated absorbing Markov chain has fundamental matrix

$\nu_Y$7

and $\nu_Y$8 equals the expected number of visits to token $\nu_Y$9 before absorption when starting at token $E \subseteq \mathbb{R}^d$0. The paper is explicit that this is not formal measure theory: there are no general measurable spaces, integral operators, or Radon–Nikodym constructions. The contribution is instead an operator-theoretic, graph-diffusion, spectral, and probabilistic reformulation in finite dimensions. Its linear-time variant, Linear-InfSA, approximates the principal eigenvector of the implicit operator and interprets that vector as a global token-importance profile (Roffo et al., 26 Feb 2026).

A different spectral direction is developed in “Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix” (Hayase et al., 8 Oct 2025). There the attention matrix $E \subseteq \mathbb{R}^d$1 is a random row-stochastic matrix, hence a random Markov kernel on the finite set $E \subseteq \mathbb{R}^d$2. The central measure-theoretic object is the empirical squared singular-value distribution

$E \subseteq \mathbb{R}^d$3

a random probability measure on $E \subseteq \mathbb{R}^d$4. After removing the Perron rank-one mode,

$E \subseteq \mathbb{R}^d$5

the paper proves a Gaussian equivalence principle for the bulk singular-value law:

$E \subseteq \mathbb{R}^d$6

almost surely in the proportional regime with fixed inverse temperature. The limiting law is deterministic and is not Marchenko–Pastur. This is not a token-space measure theory of attention, but it is measure-theoretic at the level of weak convergence and moment convergence of random empirical spectral measures (Hayase et al., 8 Oct 2025).

The universal-approximation results in “Universal Approximation with Softmax Attention” fit the same broader operator narrative (Hu et al., 22 Apr 2025). The proofs repeatedly use finite-temperature softmax columns as simplex-valued kernels that concentrate on selected anchors, so each output is a barycentric average over a discrete measure supported on anchor values. On compact domains, the paper proves that two-layer self-attention and one-layer self-attention followed by a softmax function are universal approximators for continuous sequence-to-sequence functions. A plausible interpretation is that softmax attention acts as an adaptive discrete quadrature operator whose expressive power comes from concentration on learned anchor measures rather than from feed-forward sublayers alone (Hu et al., 22 Apr 2025).

6. Stability, misconceptions, and open boundaries

A major structural issue for any operator or measure interpretation is stability. “The Lipschitz Constant of Self-Attention” proves that standard dot-product self-attention is not Lipschitz on the unbounded domain $E \subseteq \mathbb{R}^d$7 for any vector $E \subseteq \mathbb{R}^d$8-norm, whereas a distance-based $E \subseteq \mathbb{R}^d$9 self-attention becomes globally Lipschitz under tied query and key maps and explicit structural constraints (Kim et al., 2020). For the $\sigma$0 construction, the paper derives

$\sigma$1

up to the explicit norm-dependent constants given in the theorem. This matters for any attempt to view attention as a stable nonlinear operator on token clouds or empirical measures: the standard dot-product mechanism lacks global stability on unbounded domains, while the modified distance-based mechanism admits it (Kim et al., 2020).

A recurrent misconception in this area is to identify every probabilistic or operator reformulation with full measure theory. The recent connection-walk and InfSA papers come close in the finite discrete setting, but both are explicit about the boundary. The connection-walk theory gives an exact discrete operator/kernel formalism and even writes continuous-looking analogues such as

$\sigma$2

yet it does not prove a continuum limit on arbitrary measurable spaces (Lin et al., 12 Jul 2026). InfSA provides a discrete probabilistic kernel interpretation through an absorbing Markov chain, but it also states that there is no use of sigma-algebras, measurable spaces, signed measures, or integral operators on general measure spaces (Roffo et al., 26 Feb 2026).

A second misconception concerns the phrase “unbiased concentration.” In the log-normal attention paper, that phrase refers to a spectral-gap-based concentration diagnostic under absence of column bias; it is not a theorem of unbiased estimation for a linearized attention mechanism (Nahshan et al., 2023). The distinction matters because probabilistic language around attention often shifts between row-wise probability measures, random matrices, and estimator terminology.

These works suggest several unresolved directions. A fully general theory would replace finite token sets by arbitrary measurable or Polish spaces, replace finite sums by Bochner-type integrals, and study attention as a nonlinear operator on measures or vector-valued function spaces. A plausible implication is that the discrete kernel, diffusion, and mean-field theories already isolate the ingredients such a theory would need: a base-space kernel, a transport or value map, normalization by Gibbs or softmax reweighting, and a topology on measures strong enough to control composition across layers. What remains largely open is a rigorous synthesis of these ingredients beyond fixed-length finite-dimensional settings.