On the Expressive Power of Transformers

Abstract: Multi-layer transformers form the critical component of essentially all LLMs in use today. Because of their ubiquity and computational capability, there is a rapidly growing body of work that aims to precisely calibrate the expressive power of transformers as language recognizers by comparing them against standard models of computation studied for decades by the theoretical computer science community. In this endeavor, circuit complexity has by and large emerged as the "correct" branch of computational complexity to analyze the expressive power of transformers; the reason is that parameterizing transformers by the various resources they use, such as attention and precision, leads to direct comparisons with different classes of circuits parameterized by resources such as type of gates, size, and depth. Here, we present an overview of selected results that delineate the expressive power of transformers using concepts and methods from circuit complexity.

• The paper develops circuit-complexity simulations showing that constant-depth transformers without chain-of-thought operate within AC or TC depending on attention type and numerical precision.
• The paper finds that logarithmic precision generally supports TC-level power, while constant precision collapses softmax and average-attention models to AC, with tight characterizations requiring polynomial embedding width.
• The paper shows that chain-of-thought increases expressivity substantially, enabling polynomial-time computation with polynomial-length generation and Turing completeness under unbounded generation, even for softmax attention at logarithmic precision.

This survey by Kolaitis and Sengupta synthesizes a body of work that calibrates the expressive power of transformers as language recognizers against classical circuit complexity classes. Its central thesis is that circuit complexity, rather than the Chomsky hierarchy, is the appropriate framework for this analysis: transformers perform parallel, fixed-depth computation over continuous vectors, which maps naturally onto bounded-depth circuits with restrictions on gates, fan-in, size, and uniformity.

Transformers as language recognizers

The survey formalizes the transformer as a family of length-preserving layers applied to an embedded input string of arbitrary length $n$ (the context length), mirroring how a finite automaton processes arbitrarily long strings. The input layer computes $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$; each hidden layer composes $H$ self-attention sublayers (scaled dot-product attention with weighting function $\mathcal{S}$) with a position-wise feed-forward sublayer using ReLU activations and residual connections. The output layer projects a designated vector to a scalar probability via a sigmoid, accepting when $p_\text{out} \geq 1/2$.

Three architectural choices dominate the expressivity landscape:

• Attention type: unique hard attention ($UHAT$), average hard attention ($AHAT$), and softmax attention ($SMAT$). Soft attention is strictly more expressive than hard attention in general, though most hard-attention variants can be simulated by soft attention (Yang et al., 2024).
• Masking: encoders attend bidirectionally; decoders apply future masking. Encoders are provably strictly more powerful than decoders without chain-of-thought for language recognition.
• Chain-of-thought (CoT): autoregressive generation of $f(n)$ intermediate tokens, which is known to yield strict increases in expressive power.

The resource parameters are depth $L$, heads $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$0, embedding dimension $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$1, precision $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$2, and CoT budget $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$3; the product $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$4 is termed the width. The survey emphasizes that precision is the most consequential parameter: arbitrary real precision is unrealistic, while $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$5 bits collapses transformers into $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$6 even with polynomial embedding dimension and logarithmic CoT [(2608.12671)'s cited Theorem 3.1 of LLHZ24], because constant precision prevents uniform attention over growing inputs. The standard assumption is therefore $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$7.

Why circuit complexity

The survey motivates the choice of circuit classes via the inclusion chain

$$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$8

where the first inclusion is strict because MAJORITY is not in $$\mathbf{x}^{(0)}_i = \mathsf{WE}(w_i) + \mathsf{PE}(i)$$9 [FSS], while whether $H$0 remains open. Descriptive complexity supplies logical counterparts: $H$1 corresponds to first-order logic with BIT, $H$2 adds majority quantifiers, and $H$3 corresponds to least fixed-point logic on ordered structures. These correspondences are exploited directly — e.g., Chiang, Cholak, and Pillay use a counting extension of first-order logic to place fixed-precision softmax encoders in $H$4.

Upper bounds without chain-of-thought

The core simulation technique constructs Boolean circuit gadgets for each component of a constant-depth transformer and stitches them across layers, yielding constant-depth circuits whose gate basis depends on attention type and precision:

Model	Precision	Upper bound
$H$5 encoder	arbitrary rational	$H$6
$H$7/$H$8 encoder	$H$9	$\mathcal{S}$0
$\mathcal{S}$1/$\mathcal{S}$2 encoder	$\mathcal{S}$3	$\mathcal{S}$4

The intuition is that $\mathcal{S}$5 requires only AND/OR/NOT gadgets, whereas computing averages of $\mathcal{S}$6 numbers at log precision requires threshold gates. Tightness varies: Barceló et al. show the $\mathcal{S}$7 upper bound is not tight — some $\mathcal{S}$8 languages exceed $\mathcal{S}$9 power, though $p_\text{out} \geq 1/2$0 captures first-order logic with unary numerical predicates, and $p_\text{out} \geq 1/2$1 captures this logic plus counting terms. By contrast, with $p_\text{out} \geq 1/2$2 embedding dimension, $p_\text{out} \geq 1/2$3- and $p_\text{out} \geq 1/2$4-precision transformers capture all of $p_\text{out} \geq 1/2$5 and $p_\text{out} \geq 1/2$6 respectively, making those bounds essentially exact. A separate logical characterization shows masked $p_\text{out} \geq 1/2$7 decoders without positional encodings recognize exactly the star-free languages, via the intermediate formalism B-RASP [YCA24].

Chain-of-thought breaks the $p_\text{out} \geq 1/2$8 barrier

CoT moves transformers beyond $p_\text{out} \geq 1/2$9 into classes believed strictly more powerful:

Model	CoT	Precision	Class
$UHAT$0 decoder	$UHAT$1	$UHAT$2	$UHAT$3
$UHAT$4 decoder	$UHAT$5	$UHAT$6	$UHAT$7
$UHAT$8 decoder	$UHAT$9	$AHAT$0	$AHAT$1
$AHAT$2 decoder	$AHAT$3	$AHAT$4	exactly $AHAT$5
$AHAT$6 decoder	unbounded	arbitrary	all r.e. languages
$AHAT$7 decoder	unbounded	$AHAT$8	all r.e. languages

The simulation arguments encode Turing machine computation histories in generated tokens: autoregression reconstructs the head position (using techniques such as layernorm hash from Merrill–Sabharwal), attention locates the most recent timestep at which the head occupied that position, and the tape symbol is read off. The $AHAT$9 and Turing-completeness results are equivalences rather than mere containments. Notably, the Turing-completeness result for softmax attention at log precision [jiang2026softmax] removes the need for arbitrary-precision arithmetic that earlier completeness proofs required. Amiri et al. complement these upper bounds with systematic lower bounds on the CoT needed for natural algorithmic problems.

Limitations and open questions

The survey is candid about several caveats. The equivalence results for $SMAT$0 and $SMAT$1 capture require $SMAT$2 embedding dimension, which is unrealistic in practice where $SMAT$3 is constant. The unbounded-CoT Turing-completeness claims require a model variant not fully formalized in the survey — one where acceptance is triggered by a distinguished token rather than a pre-specified $SMAT$4. Layer normalization is omitted from the formal treatment despite drastically affecting expressivity depending on its modeling. Encoder lower bounds would imply lower bounds against constant-depth symmetric circuits, for which existing techniques such as random restriction do not apply — so the strict separation between encoders and decoders rests on indirect arguments. Most fundamentally, the survey leaves open the gap between worst-case expressivity and the behavior of trained models: relating these complexity-theoretic characterizations to empirical transformer capabilities remains unresolved.

Conclusion

The survey distills a coherent picture: constant-depth transformers without CoT sit within $SMAT$5 or uniform $SMAT$6 depending on precision, with matching lower bounds under generous width assumptions; CoT of sufficient length lifts them through $SMAT$7 to exactly $SMAT$8, and unbounded CoT yields Turing completeness even for softmax attention at log precision. The sharp sensitivity of these classifications to precision, attention type, and masking underscores that "the transformer" is not a single computational model but a family whose power must be stated relative to explicit resource bounds.