Universal Approximation for Transformers

• The universal approximation theorem for transformers demonstrates that these architectures can approximate broad classes of continuous functions under specified symmetry and topological assumptions.
• Methodologies leverage key mechanisms such as single-layer attention, invariant global summation, and softmax interpolation to achieve precise approximations.
• Extensions include efficient variants like Linformer and Performer, and in-context as well as operator-level formulations, enhancing their applicability in sequence and function space learning.

Searching arXiv for recent and foundational papers on transformer universal approximation. The universal approximation theorem for transformers denotes a family of results establishing that transformer architectures can approximate broad classes of target maps arbitrarily well, under specified assumptions on the domain, symmetry, topology, norm, and architectural components. In the fixed-length sequence setting, foundational theorems show that transformers approximate continuous sequence-to-sequence functions on compact domains, initially for permutation-equivariant maps and, with positional encodings, for arbitrary continuous maps (Yun et al., 2019). Subsequent work has refined the theorem along several axes: strengthening the approximation norm from $L_p$ to uniform sup-norm, reducing the number of required attention layers to one, extending universality to efficient linear-time variants such as Linformer and Performer, isolating softmax attention itself as a universal approximator without feed-forward networks, and reformulating the theory in measure-theoretic and operator-learning terms to obtain token-count-independent in-context guarantees (Alberti et al., 2023, Hu et al., 22 Apr 2025, Furuya et al., 2024). The resulting body of theory no longer concerns a single theorem, but a landscape of approximation results for different transformer families and function spaces.

1. Historical emergence and theorem classes

The first widely cited transformer universality result proved that fixed-width transformers are universal approximators of continuous permutation-equivariant sequence-to-sequence functions with compact support, and that trainable positional encodings remove the equivariance restriction, yielding universality for arbitrary continuous sequence-to-sequence maps on compact domains (Yun et al., 2019). In that formulation, the approximation metric is an $L_p$ distance on functions, with $1 \le p < \infty$, and the construction uses fixed architectural widths $h=2$, head size $m=1$, and feed-forward hidden size $r=4$ (Yun et al., 2019).

A later line of work sharpened both the function class and the architectural minimality. "Sumformer: Universal Approximation for Efficient Transformers" (Alberti et al., 2023) proves uniform approximation in sup-norm on compact domains for continuous permutation-equivariant sequence-to-sequence maps and shows that one attention layer is sufficient for universal approximation; all remaining layers may be token-wise feed-forward layers. The same argument yields the first universality guarantees for Linformer and Performer (Alberti et al., 2023).

Another strand asks whether feed-forward subnetworks are essential to transformer universality. "Universal Approximation with Softmax Attention" (Hu et al., 22 Apr 2025) answers negatively: two-layer attention-only architectures, or one multi-head attention layer followed by an output softmax, suffice for universal approximation of continuous sequence-to-sequence functions on compact domains. The key mechanism is an interpolation-based construction showing that softmax attention can approximate generalized ReLU, or truncated linear functions, to arbitrary precision (Hu et al., 22 Apr 2025).

A separate extension moves from fixed-length sequences to in-context maps over probability measures. "Transformers are Universal In-context Learners" (Furuya et al., 2024) proves that deep transformers acting on contexts represented as probability measures $\mu \in \mathcal{P}(\Omega)$ and queries $x \in \Omega$ approximate any continuous in-context mapping $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$ uniformly on compact token domains, with a fixed embedding dimension $d_{\mathrm{emb}} = d + 3d'$ and a fixed number of heads $L_p$0 independent of the number of tokens (Furuya et al., 2024).

Recent results generalize universality further. These include Lipschitz-constrained in-context transformers on measure spaces (Furuya et al., 17 Feb 2026), unified non-constructive universal approximation criteria for transformer-type architectures based on token distinguishability (Cheng et al., 30 Jun 2025), exact or near-exact simulation of maxout and ReLU networks by transformers (Gu et al., 3 Mar 2026), operator-learning theorems for transformers between function spaces (Furuya et al., 18 May 2026, Zappala et al., 2024), and universality for special regimes such as visual autoregressive transformers (Chen et al., 10 Feb 2025). This suggests that the phrase "universal approximation theorem for transformers" now refers to a growing family of structurally related but technically distinct results.

2. Core formulations: domains, symmetries, and approximation criteria

A central distinction in transformer universality is the target function class. In the sequence setting, one commonly fixes sequence length $L_p$1 and studies maps

$L_p$2

for compact $L_p$3 and suitable $L_p$4. The earliest transformer theorem focuses on permutation-equivariant functions satisfying

$L_p$5

for all permutation matrices $L_p$6 acting on token positions (Yun et al., 2019). This reflects the intrinsic symmetry of self-attention without positional information. When absolute positional encodings are added, universality extends to arbitrary continuous sequence-to-sequence maps on compact domains (Yun et al., 2019).

The Sumformer formulation makes the equivariant structure explicit. It studies the class $L_p$7 of continuous permutation-equivariant maps $L_p$8 and uses the equivalence between equivariant sequence-to-sequence maps and semi-invariant sequence-to-point maps:

$L_p$9

where $1 \le p < \infty$0 is invariant under permutations of all arguments except the distinguished first token (Alberti et al., 2023). This reduction is central because it turns a structured sequence problem into a single-token reconstruction problem conditioned on a permutation-invariant summary.

Approximation criteria also vary. The original theorem in (Yun et al., 2019) uses $1 \le p < \infty$1 approximation. By contrast, (Alberti et al., 2023) proves uniform approximation in supremum norm:

$1 \le p < \infty$2

This is a stronger conclusion on compact domains. The softmax-attention-only results in (Hu et al., 22 Apr 2025) provide both $1 \le p < \infty$3 and $1 \le p < \infty$4-norm guarantees, including an $1 \le p < \infty$5-norm theorem requiring interpolation precision $1 \le p < \infty$6.

In-context formulations replace a finite sequence by a context measure. In (Furuya et al., 2024), the object of approximation is

$1 \le p < \infty$7

continuous jointly in the weak$1 \le p < \infty$8 topology on $1 \le p < \infty$9—equivalently, in Wasserstein topology on compact $h=2$0—and Euclidean topology in the query variable. This reformulation yields universality that is uniform over all empirical measures

$h=2$1

hence independent of token count (Furuya et al., 2024).

A related but stricter target class appears in Lipschitz-constrained universality. There, the approximation space is

$h=2$2

consisting of functions that are $h=2$3-Lipschitz in the query and $h=2$4-Lipschitz in the context under $h=2$5 (Furuya et al., 17 Feb 2026). This is not unrestricted universality, but density within a Lipschitz-constrained function class.

3. Mechanisms of universality: contextual mappings, sums, and interpolation

The proofs reveal several distinct mechanisms by which transformers attain universal approximation.

The earliest constructive analysis isolates the role of self-attention as a contextual mapping. In (Yun et al., 2019), token-wise feed-forward layers first quantize inputs to a grid. Self-attention then computes a context-dependent identifier for each token using selective shifts and hardmax-style attention, ensuring that token representations are unique both within a sequence and across non-equivalent sequences. A subsequent token-wise feed-forward phase memorizes the desired output values associated with these contextual identifiers (Yun et al., 2019). This decomposition clarifies the functional split between attention and FFN: attention creates context-aware codes; FFNs perform value lookup.

The Sumformer line replaces this coding argument with an invariant aggregation principle. A Sumformer layer has the form

$h=2$6

Because $h=2$7 is permutation-invariant and $h=2$8 is applied token-wise, the architecture is permutation-equivariant (Alberti et al., 2023). Universality follows by showing that sufficiently rich $h=2$9 can encode all multisymmetric power sums needed to represent equivariant polynomials, and that $m=1$0 can reconstruct the output from the current token and the global summary. This reduces transformer universality to the ability of attention to compute the global sum $m=1$1 (Alberti et al., 2023).

The same paper gives two complementary routes. The continuous route uses Stone–Weierstrass approximation by equivariant polynomials and the theorem that multisymmetric power sums of total degree at most $m=1$2 generate the multisymmetric polynomial algebra (Alberti et al., 2023). The discontinuous route uses piecewise-constant approximation on a finite grid plus Deep Sets–style invariant coding, achieving universality with latent dimension $m=1$3 at the cost of many token-wise layers (Alberti et al., 2023). This yields an explicit width–depth trade-off: the continuous construction uses

$m=1$4

which scales like $m=1$5 for fixed $m=1$6, whereas the discontinuous construction collapses the invariant summary to dimension one but requires layer count growing roughly like $m=1$7 (Alberti et al., 2023).

The softmax-attention-only line identifies a different primitive: attention implements interpolation. In (Hu et al., 22 Apr 2025), a linear pre-map embeds anchor values into the attention mechanism, and the column-wise softmax with temperature $m=1$8 approximates an argmax over anchors. This lets a single attention layer approximate

$m=1$9

a generalized ReLU or truncated linear nonlinearity, with error decomposed into a finite-$r=4$0 softmax error and an interpolation error (Hu et al., 22 Apr 2025). Multi-head attention improves the interpolation error from $r=4$1 to $r=4$2 (Hu et al., 22 Apr 2025). Since generalized ReLU functions suffice to build piecewise-linear approximators, softmax attention alone becomes universal.

A plausible implication is that different transformer universal approximation proofs are not merely technical variants, but expose different internal computational primitives: contextual labeling (Yun et al., 2019), invariant global summation (Alberti et al., 2023), and anchor-based interpolation (Hu et al., 22 Apr 2025). These mechanisms support overlapping, but not identical, intuitions about what attention contributes.

4. Architectural minimality and efficient attention variants

A recurring question is how much architecture is truly needed. The answer has shifted substantially.

The original theorem uses deep compositions of transformer blocks with self-attention, feed-forward layers, and residual connections, though with fixed small widths (Yun et al., 2019). By contrast, (Alberti et al., 2023) proves that a single nonzero attention layer suffices for universal approximation of continuous permutation-equivariant sequence-to-sequence functions in sup-norm. The construction first computes per-token features and packs them into

$r=4$3

for each token. It then chooses $r=4$4 so that all query-key rows are constant, yielding uniform attention

$r=4$5

A suitable value projection extracts the shared sum $r=4$6 into designated channels, after which token-wise feed-forward layers implement $r=4$7 (Alberti et al., 2023). The attention layer serves only as a global averaging or summation device.

The same architecture extends to Linformer and Performer. For Linformer,

$r=4$8

and choosing $r=4$9 and $\mu \in \mathcal{P}(\Omega)$0 as scaled all-ones matrices yields uniform mixing, so the same global sum can be recovered. The resulting theorem states that for each $\mu \in \mathcal{P}(\Omega)$1 and $\mu \in \mathcal{P}(\Omega)$2, there exists a Linformer with projection dimension $\mu \in \mathcal{P}(\Omega)$3 such that the sup-norm error is below $\mu \in \mathcal{P}(\Omega)$4 (Alberti et al., 2023). Crucially, $\mu \in \mathcal{P}(\Omega)$5 does not depend on sequence length $\mu \in \mathcal{P}(\Omega)$6.

For Performer,

$\mu \in \mathcal{P}(\Omega)$7

and the proof sets $\mu \in \mathcal{P}(\Omega)$8 so every row is identical. Then the random-feature kernel matrix collapses to a constant multiple of the all-ones matrix, again yielding global mixing up to a rescaling that a following token-wise linear layer corrects (Alberti et al., 2023). This gives universality for Performer with any fixed number of random features in the constructed setting (Alberti et al., 2023).

Sparse transformers admit related results. "O(n) Connections are Expressive Enough" (Yun et al., 2020) proves that sparse attention patterns with self-loops, a Hamiltonian path, and finite-hop reachability preserve universal approximability in $\mu \in \mathcal{P}(\Omega)$9, even with only $x \in \Omega$0 attention connections per layer. This covers structured sparsity patterns such as Star-Transformer, Longformer-style window-plus-global patterns, and BigBird-like schemes under the stated connectivity conditions (Yun et al., 2020).

More recently, (Cheng et al., 30 Jun 2025) provides a unifying criterion. It identifies token distinguishability of the mixing family, combined with nonlinear affine-invariant token-wise feed-forward maps, as a sufficient condition for universal approximation of transformer-type architectures in $x \in \Omega$1. Under an analyticity assumption on the attention family, checking token distinguishability reduces to the two-sample case (Cheng et al., 30 Jun 2025). This subsumes softmax attention, kernel attention, sparse attention, Linformer, and SkyFormer under a single framework (Cheng et al., 30 Jun 2025).

5. Beyond fixed-length seq2seq: in-context, Lipschitz, and operator-level universality

Transformer universality is no longer confined to fixed-length sequence-to-sequence approximation.

In-context universality treats the prompt or context as a probability measure. "Transformers are Universal In-context Learners" (Furuya et al., 2024) proves that deep transformers approximate any continuous in-context mapping

$x \in \Omega$2

uniformly over all context measures and queries on compact $x \in \Omega$3. The architecture is written measure-theoretically:

$x \in \Omega$4

composed with token-wise MLPs and pushforwards of the context measure through intermediate layers (Furuya et al., 2024). A key property is uniformity over empirical measures $x \in \Omega$5, so one transformer works for arbitrary finite numbers of context tokens (Furuya et al., 2024).

This line has a notable implication for claims about in-context learning as uniquely transformer-specific. "Is In-Context Universality Enough? MLPs are Also Universal In-Context" (Kratsios et al., 5 Feb 2025) proves that ReLU MLPs, when applied to an appropriate matrix representation of finite permutation-invariant contexts and queries, are also universal in-context learners. The paper then shows how to transform such MLP approximators into transformers with exactly $x \in \Omega$6 attention heads per block (Kratsios et al., 5 Feb 2025). This suggests that in-context universality alone does not explain transformer performance; the likely distinguishing factors are inductive bias or optimization behavior rather than expressivity per se (Kratsios et al., 5 Feb 2025).

Lipschitz-constrained universality introduces stability into the theorem itself. In (Furuya et al., 17 Feb 2026), both attention and MLP blocks are interpreted as explicit Euler steps of negative gradient flows:

$x \in \Omega$7

$x \in \Omega$8

With step sizes chosen appropriately, each block is nonexpansive in the query variable, and the full architecture is dense in the class of scalar maps that are $x \in \Omega$9-Lipschitz in the query and $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$0-Lipschitz in the context under Wasserstein distance (Furuya et al., 17 Feb 2026). The theorem is token-count independent because it is stated over probability measures rather than token lists (Furuya et al., 17 Feb 2026).

Operator-learning results extend the approximation target from functions on finite-dimensional spaces to operators between function spaces. "Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity" (Shih et al., 2024) proves an operator-level UAT by combining encoders, a finite-dimensional transformer approximant, and decoders, yielding uniform approximation of continuous operators on compact subsets of Banach spaces (Shih et al., 2024). Related measure-theoretic operator-learning theorems use graph measures and function graph transformers to prove universality for graph-preserving operators between function spaces, including settings with negative-order Sobolev inputs and query points on different output domains (Furuya et al., 18 May 2026). A separate line proves universality of transformers for integral operators between Hölder spaces and, with Leray–Schauder modifications, for operators between arbitrary Banach spaces (Zappala et al., 2024).

This suggests that transformer universality has evolved from a theorem about sequence models into a general approximation theory for measure-dependent and operator-valued mappings.

6. Misconceptions, quantitative caveats, and open problems

A common misconception is that universal approximation implies practical efficiency. The theorems are overwhelmingly existential. They usually do not provide sharp bounds on width, depth, head count, or optimization complexity as functions of $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$1, and the constructive bounds can be very large. In (Alberti et al., 2023), the latent dimension in the continuous Sumformer proof scales as

$$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$2

which is polynomial in $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$3 for fixed $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$4 but exponential in $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$5 for fixed $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$6. In (Hu et al., 22 Apr 2025), $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$7-norm control requires interpolation precision $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$8. In (Yun et al., 2019), the depth needed to memorize general continuous functions via grid constructions grows on the order of $$\Lambda^\star : \mathcal{P}(\Omega) \times \Omega \to \mathbb{R}^{d'}$$9 in the equivariant setting (Yun et al., 2019, Alberti et al., 2023, Hu et al., 22 Apr 2025).

Another misconception is that universality automatically concerns arbitrary variable-length sequences. Most classical seq2seq theorems fix sequence length $d_{\mathrm{emb}} = d + 3d'$0 (Yun et al., 2019, Alberti et al., 2023, Hu et al., 22 Apr 2025). Measure-theoretic in-context theorems remove this dependence by operating on probability measures, but they target a different function class: continuous mappings of a context measure and a query (Furuya et al., 2024, Furuya et al., 17 Feb 2026).

It is also incorrect to equate universality with attention being indispensable. Attention-only universality shows FFNs are not strictly necessary (Hu et al., 22 Apr 2025), but in-context MLP universality shows transformers are not uniquely universal in that regime either (Kratsios et al., 5 Feb 2025). A plausible implication is that universal approximation results identify necessary representational sufficiency, not the source of empirical superiority.

Several open problems recur across the literature. Tight quantitative rates remain largely unresolved (Alberti et al., 2023, Furuya et al., 17 Feb 2026, Cheng et al., 30 Jun 2025). Robustness of universality proofs to approximate kernelization in practical Performer-style attention is open beyond the exact constructions used in existence proofs (Alberti et al., 2023). Extending measure-theoretic universality to masked or causal attention remains an explicit future direction in (Furuya et al., 2024). The unified framework in (Cheng et al., 30 Jun 2025) leaves open whether analyticity is merely sufficient or close to necessary for its token-distinguishability shortcut. Lipschitz-constrained results in (Furuya et al., 17 Feb 2026) currently focus on scalar outputs; extending the lattice-based argument to vector-valued outputs is nontrivial.

A further controversy concerns what exactly attention contributes. The contextual mapping view (Yun et al., 2019), the global-sum view (Alberti et al., 2023), and the interpolation view (Hu et al., 22 Apr 2025) are mathematically compatible, but emphasize different computational roles. Rather than a single canonical explanation, transformer universality appears to admit multiple equivalent constructive decompositions.

In aggregate, the universal approximation theorem for transformers now designates a mature theoretical domain. Its central conclusion is stable: transformers, and many efficient or constrained variants, are expressive enough to approximate large classes of continuous maps under suitable assumptions. What remains unsettled are the quantitative sharpness of these theorems, their relation to trainability and generalization, and the extent to which their mechanisms explain empirical transformer behavior rather than merely certify representational capacity.