---
title: Measure-Theoretic Self-Attention
url: https://www.emergentmind.com/topics/measure-theoretic-self-attention
type: topic
---

# Measure-Theoretic Self-Attention

Searching arXiv for recent papers on measure-theoretic and operator-theoretic formulations of self-attention.
Measure-theoretic self-attention denotes a family of formulations in which attention is expressed in the language of probability measures, Markov kernels, vector-valued integral operators, empirical measures, or measure-valued gradient flows. In the strict sense, it includes exact reconstructions of classical attention via Boltzmann–Gibbs reweighting, lookup kernels, and moment projection, as well as mean-field theories in which the network state is a probability measure over attention-head parameters [2007.02876] [2606.10469]. In a broader but weaker sense, it also includes discrete operator theories in which attention rows are finite-state kernels, geometric transport operators on token graphs, spectral theories of centered attention matrices, and graph-diffusion resolvents with Markovian interpretations [2607.10677] [2510.06685] [2603.00175]. Taken together, these works suggest that “measure-theoretic self-attention” is not a single formalism but a cluster of related viewpoints ranging from exact measure theory to finite-dimensional probabilistic and operator-theoretic analogues.

## 1. Scope and main formulations

The literature separates naturally by the space on which the measure lives. In token-space formulations, a sequence is encoded by an empirical measure over token representations, and attention becomes a kernel acting on that measure. In graph/operator formulations, the relevant object is a nonnegative matrix acting on token-indexed quantities, often with a Markov or substochastic interpretation. In mean-field formulations, the measure lives not on tokens but on head-parameter space, so training becomes a PDE on probability measures. A further strand studies empirical spectral measures of attention matrices, which is measure-theoretic in the sense of random probability measures on the spectrum rather than on token space.

| Formulation | State variable | Central object |
|---|---|---|
| Exact attention kernel | Empirical token measure | $\mathbf A_\mu(x,dz)=\Pi[\Psi_{G(x,\cdot)}(\mu)L](dz)$ |
| Connection-walk attention | Token-position field | $(\mathcal T(A,O)X)_i=\sum_j A_{ij}X_jO_{ij}$ |
| Mean-field head training | Probability measure on head parameters | $z_l^\rho(X)=\int \psi_l(X;\omega)\rho(d\omega)$ |
| Spectral/diffusion attention | Attention operator or attention matrix | $(I-\gamma A)^{-1}-I$, or empirical spectral measure $\nu_Y$ |

The strictest measure-theoretic constructions occur in the attention-kernel model and in mean-field training theory. The operator, spectral, and diffusion formulations are usually finite-dimensional and discrete. Several papers explicitly note that they are not full abstract measure theory even when the analogy is close [2607.10677] [2603.00175].

## 2. Exact token-space kernel theories

A mathematically equivalent measure-theoretic reconstruction of classical attention is given by “A Mathematical Theory of Attention” [2007.02876]. The ambient space is a representation space $E \subseteq \mathbb{R}^d$ with its Borel $\sigma$-algebra, tokens are identified with Dirac measures, and a collection of tokens $X=\{x_1,\dots,x_T\}$ is encoded by the empirical measure
$$
m(X):=\frac{1}{|\mathcal T|}\sum_{t\in\mathcal T}\delta_{x_t}.
$$
The similarity function is represented by a positive interaction potential
$$
G(x,y)=\exp(a(x,y)),
$$
and the softmax weighting step is written as a Boltzmann–Gibbs transformation
$$
\Psi_g(\nu)(dx):=\frac{g(x)\nu(dx)}{\nu(g)}.
$$
Keys are transported to values by a lookup kernel $L$, and the weighted value distribution is collapsed back to a finite-dimensional representation by a moment projection $\Pi$. The resulting attention kernel is
$$
\mathbf A_\mu(x,dz):=\Pi[\Psi_{G(x,\cdot)}(\mu)L](dz).
$$
For the standard deterministic finite setting, the paper proves exact equivalence with ordinary attention. It also interprets self-attention as the nonlinear map $\mu\mapsto \mu\mathbf A_\mu$, so a stack of attention layers becomes a deterministic self-interacting particle system. Within the same framework, the softmatch distribution is characterized as the unique maximum-entropy solution under a moment constraint, and self-attention is shown to be Lipschitz-continuous under suitable assumptions in Wasserstein geometry [2007.02876].

A complementary discrete operator formulation is developed in “From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers” [2607.10677]. There, a token sequence is a vector field $X\in\mathbb R^{n\times d}$ over the token-position graph, and attention is expressed as a connection propagation operator
$$
(\mathcal{T}(A,O)X)_i = \sum_{j=1}^n A_{ij}\,X_j\,O_{ij},
$$
where $A$ is a nonnegative walk matrix and $O_{ij}\in \operatorname{End}(\mathbb{R}^d)$ is an edge transport. On the finite discrete space $V=\{1,\dots,n\}$ with discrete $\sigma$-algebra, this is already the discrete analogue of the vector-valued kernel operator
$$
(Tf)(x)=\int K(x,y)\,\Gamma_{x\leftarrow y}f(y)\,d\mu(y).
$$
The paper proves that single-head attention is exactly connection propagation with constant transport, and that multi-head attention is exactly a single edge-dependent connection walk whose effective transport is an attention-gated mixture of headwise transports. When the effective walk is reversible and the transports are metric-compatible and inverse-consistent, the generator reduces to a random-walk connection Laplacian with Dirichlet energy
$$
\mathcal{E}_{\pi}(X)=\frac12\sum_{i,j}\pi_i A_{\mathrm{eff}(i,j)\,\|X_i-X_jO^{\mathrm{eff}}_{ij}\|_2^2.
$$
This is a precise discrete kernel theory, but the paper is explicit that it is not yet a full abstract measurable-space treatment [2607.10677].

## 3. Probabilistic attention, particle dynamics, and conditional mixing

The token-space kernel view leads naturally to a probabilistic interpretation. In the exact measure-theoretic model, each attention update reweights an empirical measure by a Gibbs factor, pushes that weighted measure through a lookup kernel, and projects the resulting value distribution back to a representation. This makes self-attention a nonlinear Markov transport on empirical measures, and the network depth index plays the role of a discrete-time interacting-particle evolution [2007.02876].

A statistically different but closely related reformulation appears in “Bidirectional Attention as a Mixture of Continuous Word Experts” [2307.04057]. For single-layer, single-head bidirectional self-attention trained with the masked language model objective, the paper proves exact equivalence to a continuous bag-of-words predictor with context-dependent mixture-of-experts weights:
$$
F(\overline{X})=\sum_{j\in[S]}\pi_j(\overline{X})\,f_j(\overline{X}).
$$
Here the $j$-th expert is attached to token position $j$, and the attention weights $\pi_j(\overline X)$ are the softmax gating probabilities. In finite discrete terms, this supports the representation
$$
F(\overline X)=\int f(j;\overline X,m)\,d\mu_{\overline X,m}(j),
$$
where $\mu_{\overline X,m}=\sum_{j=1}^S \pi_j(\overline X)\delta_j$. That expectation notation is an interpretive restatement rather than the paper’s own notation, but it is exact on the finite position space. The same paper states that multi-head attention corresponds to stacked MoEs and multiple layers to a mixture of MoEs [2307.04057].

A more explicitly stochastic account of attention rows is given in “Linear Log-Normal Attention with Unbiased Concentration” [2311.13541]. The paper treats each row of the attention matrix as a probability distribution over token indices and studies the law of these normalized random weights. Under Gaussian assumptions on queries and keys, it argues that softmax attention entries are approximately log-normal, with row concentration quantified by entropy and spectral gap. It also clarifies a recurring terminological point: “unbiased concentration” does not mean unbiasedness in the Monte Carlo estimator sense; it refers to spectral-gap-based concentration analysis not confounded by systematic column bias [2311.13541].

## 4. Mean-field and Wasserstein theories of head training

The most explicit genuinely measure-theoretic treatment of self-attention training in the provided literature is “A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training” [2606.10469]. The architecture is deliberately restricted to a single masked self-attention layer with no feed-forward block, no residual connection, no layer normalization, and a fixed vocabulary projection $W_{\mathrm{out}}$. Each attention head is treated as a particle in parameter space, with head parameter
$$
\omega=(W_Q,W_K,W_V,W_O)\in \mathbb{R}^{d_\omega},
$$
and the finite collection of heads is summarized by the empirical measure
$$
\widehat\rho^N=\frac{1}{N}\sum_{i=1}^N\delta_{\omega_i}.
$$
The model output is rewritten exactly in terms of this empirical law:
$$
z_l^{\widehat\rho^N}(X)=\frac1N\sum_{i=1}^N \psi_l(X;\omega_i),
$$
and in the infinite-head limit the logits are
$$
z_l^\rho(X)=\int_{\mathbb{R}^{d_\omega}} \psi_l(X;\omega)\,\rho(d\omega).
$$
The cross-entropy risk becomes a functional on probability measures,
$$
E(\rho)=\mathbb E_X\left[\frac1L\sum_{l=1}^L \ell\bigl(Y_l(X),z_l^\rho(X)\bigr)\right].
$$

The first variation is
$$
\Phi(\omega;\rho)=\mathbb E_X\left[\frac1L\sum_{l=1}^L \left\langle \operatorname{softmax}(z_l^\rho(X))-Y_l(X),\,\psi_l(X;\omega)\right\rangle\right],
$$
and its particle gradient generates a nonlinear Wasserstein gradient flow:
$$
\partial_t\rho_t=2\zeta(t)\nabla_\omega\cdot\left(\rho_t\nabla_\omega\Phi(\omega;\rho_t)\right).
$$
This PDE is the mean-field training dynamics. It is accompanied by a static finite-head approximation bound,
$$
0 \le \inf_{\omega_1,\ldots,\omega_N\in K}E_N(\omega_1,\ldots,\omega_N)-\inf_{\rho\in P(K)}E(\rho)\le \frac{C}{\sqrt N},
$$
and by a quantitative finite-time propagation-of-chaos estimate comparing finite-head SGD with the limiting PDE. The paper then analyzes long-time behavior: energy dissipation, convergence to the stationary set under compactness, convergence to a single stationary measure under topological or Kurdyka–Łojasiewicz assumptions, explicit convergence rates under gradient-domination conditions, and local exponential stability under a Wasserstein strong-monotonicity condition. It also gives verifiable stability and instability criteria for Dirac stationary measures [2606.10469].

This framework is measure-theoretic in a strict sense: the state space is $P(\mathbb R^{d_\omega})$ or $P(K)$, the model is parameterized by a probability measure, and training is a transport equation on that measure space. The principal limitation is architectural scope. The paper is explicit that it is not a full transformer theory; multiple layers, residuals, layer normalization, feed-forward blocks, trainable output heads, and stronger noncompact effects are left outside the analysis [2606.10469].

## 5. Spectral, graph-diffusion, and resolvent viewpoints

Several recent works move beyond one-step softmax mixing by treating attention as an operator whose multi-step or large-system behavior is the object of analysis. “Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attention” reformulates an attention layer as graph diffusion on a content-adaptive token graph [2603.00175]. Starting from
$$
Y=AV,
$$
it defines path weights over powers of the attention matrix and introduces the discounted Neumann series
$$
\check C=\sum_{t=1}^{\infty}(\gamma A)^t=(I-\gamma A)^{-1}-I,
$$
under $\gamma<1/\rho(A)$. In the row-stochastic case, $A$ is a random-walk operator. In the Frobenius-normalized InfSA construction, $\hat A$ is generally not row-stochastic, but after multiplying by $\gamma$ the matrix $\mathbf M=\gamma\hat A$ is treated as substochastic, with missing row mass interpreted as absorption. The associated absorbing Markov chain has fundamental matrix
$$
N=(I-\mathbf M)^{-1}=(I-\gamma\hat A)^{-1},
$$
and $N_{ij}$ equals the expected number of visits to token $j$ before absorption when starting at token $i$. The paper is explicit that this is not formal measure theory: there are no general measurable spaces, integral operators, or Radon–Nikodym constructions. The contribution is instead an operator-theoretic, graph-diffusion, spectral, and probabilistic reformulation in finite dimensions. Its linear-time variant, Linear-InfSA, approximates the principal eigenvector of the implicit operator and interprets that vector as a global token-importance profile [2603.00175].

A different spectral direction is developed in “Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix” [2510.06685]. There the attention matrix $A$ is a random row-stochastic matrix, hence a random Markov kernel on the finite set $\{1,\dots,n\}$. The central measure-theoretic object is the empirical squared singular-value distribution
$$
\nu_Y=\frac1d\sum_{i=1}^d \delta_{s_i(Y)^2},
$$
a random probability measure on $\mathbb R_{\ge 0}$. After removing the Perron rank-one mode,
$$
A^\perp=A-u_nu_n^\top,
$$
the paper proves a Gaussian equivalence principle for the bulk singular-value law:
$$
\nu_{\sqrt n\,A^\perp}\xrightarrow{\text{moments}}\nu_\infty,
\qquad
\nu_{\sqrt n\,A}\xrightarrow{\text{weakly}}\nu_\infty,
$$
almost surely in the proportional regime with fixed inverse temperature. The limiting law is deterministic and is not Marchenko–Pastur. This is not a token-space measure theory of attention, but it is measure-theoretic at the level of weak convergence and moment convergence of random empirical spectral measures [2510.06685].

The universal-approximation results in “Universal Approximation with Softmax Attention” fit the same broader operator narrative [2504.15956]. The proofs repeatedly use finite-temperature softmax columns as simplex-valued kernels that concentrate on selected anchors, so each output is a barycentric average over a discrete measure supported on anchor values. On compact domains, the paper proves that two-layer self-attention and one-layer self-attention followed by a softmax function are universal approximators for continuous sequence-to-sequence functions. A plausible interpretation is that softmax attention acts as an adaptive discrete quadrature operator whose expressive power comes from concentration on learned anchor measures rather than from feed-forward sublayers alone [2504.15956].

## 6. Stability, misconceptions, and open boundaries

A major structural issue for any operator or measure interpretation is stability. “The Lipschitz Constant of Self-Attention” proves that standard dot-product self-attention is not Lipschitz on the unbounded domain $\mathbb R^{N\times D}$ for any vector $p$-norm, whereas a distance-based $L2$ self-attention becomes globally Lipschitz under tied query and key maps and explicit structural constraints [2006.04710]. For the $L2$ construction, the paper derives
$$
\lip_\infty(F)=O(\log N),
\qquad
\lip_2(F)=O(\sqrt N\log N),
$$
up to the explicit norm-dependent constants given in the theorem. This matters for any attempt to view attention as a stable nonlinear operator on token clouds or empirical measures: the standard dot-product mechanism lacks global stability on unbounded domains, while the modified distance-based mechanism admits it [2006.04710].

A recurrent misconception in this area is to identify every probabilistic or operator reformulation with full measure theory. The recent connection-walk and InfSA papers come close in the finite discrete setting, but both are explicit about the boundary. The connection-walk theory gives an exact discrete operator/kernel formalism and even writes continuous-looking analogues such as
$$
(\mathcal T_tu)(s)=\int a_t(s,s')\,O_t(s,s')\,u(t,s')\,ds',
$$
yet it does not prove a continuum limit on arbitrary measurable spaces [2607.10677]. InfSA provides a discrete probabilistic kernel interpretation through an absorbing Markov chain, but it also states that there is no use of sigma-algebras, measurable spaces, signed measures, or integral operators on general measure spaces [2603.00175].

A second misconception concerns the phrase “unbiased concentration.” In the log-normal attention paper, that phrase refers to a spectral-gap-based concentration diagnostic under absence of column bias; it is not a theorem of unbiased estimation for a linearized attention mechanism [2311.13541]. The distinction matters because probabilistic language around attention often shifts between row-wise probability measures, random matrices, and estimator terminology.

These works suggest several unresolved directions. A fully general theory would replace finite token sets by arbitrary measurable or Polish spaces, replace finite sums by Bochner-type integrals, and study attention as a nonlinear operator on measures or vector-valued function spaces. A plausible implication is that the discrete kernel, diffusion, and mean-field theories already isolate the ingredients such a theory would need: a base-space kernel, a transport or value map, normalization by Gibbs or softmax reweighting, and a topology on measures strong enough to control composition across layers. What remains largely open is a rigorous synthesis of these ingredients beyond fixed-length finite-dimensional settings.

Source: https://www.emergentmind.com/topics/measure-theoretic-self-attention