Expressiveness of Transformers

• Expressiveness of transformers is the formal measure of the range of functions they can efficiently represent using multi-head self-attention and dynamic depth scaling.
• Log-depth scaling, typically Θ(log n), is critical for enabling transformers to solve problems like regular language recognition and graph connectivity through iterative binary-tree reductions and matrix squaring.
• Resource trade-offs among depth, width, and chain-of-thought steps offer practical guidelines for designing models that balance performance, parallelism, and robustness.

Transformers are a class of sequence models whose core architectural feature is multi-head self-attention, enabling massive parallelism and flexible global dependency modeling. Their expressive power—i.e., the formal class of functions and computational problems they can efficiently represent—has become a central object of study. Over the past several years, a rigorous theoretical and empirical understanding of the expressiveness of transformers, especially as a function of depth, width, and test-time augmentation (e.g., looping or chain-of-thought prompting), has emerged. In particular, the exact role of log-depth scaling as the minimal requirement for a range of canonical algorithmic, language recognition, and reasoning tasks has been sharply characterized.

1. Formal Models and Depth Parameterization

Transformer expressiveness is most precisely analyzed under the “universal transformer” framework, parameterized by the triple $(s, r, t)$, where $s$ initial layers and $t$ final layers are each applied once per input, and a block of $r$ layers is looped $d(n)$ times for input length $n$, yielding total depth $D(n) = s + r d(n) + t$ (Merrill et al., 5 Mar 2025). This parameterization captures both traditional fixed-depth transformers and the “looped” or dynamically deepened transformers often used for reasoning tasks. Key architectural variants—such as “averaging-hard” self-attention (the limit as softmax temperature $\to\infty$), pre-norm layering, and mixed masking—allow precise alignment with established circuit complexity classes.

Test-time looping, sometimes called “dynamic depth,” is especially crucial. Here, a small set of layers is shared and repeatedly applied, so that the effective model depth grows with $n$. In practice, $d(n) = \lceil \log_2 n \rceil$ is a minimal growth schedule able to cross key expressivity thresholds.

2. Theoretical Expressivity: Log-Depth as a Phase Transition

A series of recent theorems have established that depth scaling of $\Theta(\log n)$ is both necessary and sufficient for transformers to solve classes of problems previously shown to be outside the fixed-depth regime. The two canonical problems are regular language recognition and graph connectivity:

• Regular Language Recognition: Any regular language $L \subseteq \Sigma^*$ can be recognized by a looped transformer with $\Theta(\log n)$ depth. The construction proceeds via binary-tree reductions over the input, where, in each round, pairs of adjacent states are aggregated using monoid products and stored within the model’s residual stream. Fixed-depth transformers of size $D$ are strictly limited to context length $n \leq 2^{\Theta(D)}$ (Merrill et al., 5 Mar 2025, Liu et al., 2022).
• Graph Connectivity (Reachability): Given an $n \times n$ adjacency matrix and nodes $s, t$, a looped transformer of $\Theta(\log n)$ depth can determine whether $t$ is reachable from $s$. The computation emulates iterative matrix squaring within the attention mechanism, leveraging all-pairs aggregation in self-attention layers. The proof encodes the progressive computation of reachability predicates $B_\ell(i, j)$ up to path length $2^\ell$ with only $O(\log n)$ rounds (Merrill et al., 5 Mar 2025, Sanford et al., 2024).

Both cases show that while constant (fixed) depth transformers are confined to circuit class $\mathsf{TC}^0$—functions of poly-size, constant depth threshold circuits—log-depth looping enables the leap to the parallel class $\mathsf{TC}^1$ (and more generally, $\mathsf{NC}^1$ with further polylogarithmic depth), circumventing barriers imposed by circuit-complexity separations (Merrill et al., 25 May 2025).

3. Depth–Width–Chain-of-Thought Tradeoffs

A clear hierarchy of resource tradeoffs emerges in the transformer expressiveness landscape (Merrill et al., 5 Mar 2025, Yehudai et al., 3 Mar 2025, Yehudai et al., 3 Mar 2025):

Scaling Knob	Context Length to Hold Fixed
Depth $d$	$n \leq 2^{\Theta(d)}$
Width $w$ (model dim)	$n \leq \Theta(\log w)$
Chain-of-Thought steps	$n \leq$ CoT length

• Depth Scaling: To track state over $n$ tokens (for regular languages, reachability, or compositional reasoning questions), depth must scale as $O(\log n)$. Empirical results reproduce the theoretical slope, with 4–8 additional layers required for each doubling of $n$ (Merrill et al., 5 Mar 2025).
• Width Scaling: At fixed constant depth, the model width must grow exponentially in $n$ to maintain expressiveness on sequential reasoning or global aggregation tasks. Doubling the context requires multiplicative increases in model width (Merrill et al., 5 Mar 2025, Yehudai et al., 3 Mar 2025).
• Chain-of-Thought (CoT): Inference-time CoT steps need to scale at least superlogarithmically in the input for beyond $\mathsf{TC}^0$ problems. Even $O(\log n)$ CoT steps alone are insufficient for, e.g., reachability (Merrill et al., 5 Mar 2025, Yehudai et al., 3 Mar 2025).

This asymmetric scaling demonstrates why depth is the efficient knob: modest increases in depth achieve exponential increases in context length, unmatched by width or CoT expansion.

4. Extensions: Context-Free Languages, Parallel Circuits, and Padded Transformers

The boundary of transformer expressiveness lies at the intersection of model depth and test-time auxiliary mechanisms:

• Context-Free Languages (CFLs): Pad-and-loop transformer constructions with $O(\log n)$ depth and polynomial padding can recognize all CFLs, matching classical parallel recognition bounds. Subclasses such as unambiguous or linear CFLs admit more efficient (lower padding, sometimes $O(\log^2 n)$ depth) solutions. However, the padding requirements can become impractically large (e.g., $O(n^6)$ for general CFLs) (Jerad et al., 5 Jan 2026).
• Threshold Circuits—$\mathsf{TC}^d$ and $\mathsf{NC}$: Padded+looped transformers with $O(\log^d n)$ depth and polynomial padding recognize exactly those languages decidable by poly-size, depth-$O(\log^d n)$ threshold circuits ($\mathsf{TC}^d$). Letting $d$ grow leads to the entire class $\mathsf{NC}$—problems solvable in polylogarithmic parallel time. This hierarchy shows that parallel test-time inference (by padding and looping) is a strictly more parallelizable alternative to chain-of-thought, but cannot escape the NC boundary unless $\mathsf{NC} = \mathsf{P}$ (Merrill et al., 25 May 2025).
• Compositional Reasoning, Formula Evaluation, and Trees of Transformers: Balanced trees of problem instances (e.g., Boolean formula evaluation, compositional reasoning questions) require transformer depth matching the formula tree depth; for random trees, this is typically $\Theta(\log n)$. TreeCoder architectures employ $k$-ary trees of transformer blocks, achieving path lengths of $O(\log_k n)$ and demonstrating a favorable sparsity–capacity tradeoff in practice (Yehudai et al., 3 Mar 2025, D'Istria et al., 2024).

5. Empirical Validation and Practical Guidelines

Empirical studies across algorithmic, linguistic, and algebraic tasks corroborate the sharp depth–expressiveness phase transitions:

• State Tracking in $A_5$: Linear fits between log-depth and maximal context length validate that $O(\log n)$ depth is predictive of empirical performance for regular languages (Merrill et al., 5 Mar 2025).
• Automata Simulation: Trained transformers reproduce prefix-sum and shortcut solutions predicted by theory, with in-distribution accuracies exceeding 99% at depths matching the theoretical lower bounds. For group-theoretic automata (e.g., $A_5$, $S_5$), the theoretical and empirical log-depth match is exact (Liu et al., 2022).
• Graph Tasks: Practical runs confirm that for sublinear width, shallow transformers cannot solve connectivity, but adding logarithmic depth suffices, and that further width-depth tradeoff allows constant depth at linear width (Yehudai et al., 3 Mar 2025).
• In-Context Learning (Linear Dynamical Systems): Single-layer transformers plateau at nonzero error, while $\Theta(\log n)$ layers attain the minimax rate (Cole et al., 12 Feb 2025).

Practical guidelines for architectural design can be distilled:

• Set model depth $d\approx c \log_2 n$, constant $c \sim 5$–$8$, for sequence length $n$.
• Doubling context length incurs only a modest (constant) depth increase; width increases cost exponentially.
• For robust in-context generalization on diverse task distributions, prefer looped (weight-shared) transformer blocks over deep unshared ones to prevent fragility under distribution shift (Gatmiry et al., 2024).

6. Limitations, Robustness, and Architecture Variants

Critical limitations and subtleties concern model robustness, generalization, and tradeoffs between shortcut and “iterative” solutions:

• Fragility and OOD Generalization: Parallel shortcut solutions—enabled by shallow, log-depth constructions—are brittle and may fail under distribution shift, length extrapolation, or incomplete supervision. Augmenting training with iterative scratchpads or looped parameter sharing can restore robustness but sacrifices full parallelism (Liu et al., 2022, Gatmiry et al., 2024).
• Padding and Practicality: Implementing general context-free recognition with padded, looped transformers is theoretically possible but uses $O(n^6)$ padding, which is impractical for large $n$ (Jerad et al., 5 Jan 2026).
• Tree vs. Linear Transformers: Sparse-tree architectures (e.g., TreeCoders) obtain logarithmic path length in inference and achieve 64%–76% empirical win rates vs. comparably sized linear transformers on language modeling benchmarks. Selector module design, branching factor, and routing logic are critical efficiency determinants (D'Istria et al., 2024).

A plausible implication is that exploiting controlled depth is essential for parallel algorithmic reasoning in transformers, but must be balanced against considerations of robustness and hardware-parallel inference.

7. Connections to Circuit Complexity and Parallel Computation

Transformers instantiate a direct correspondence with massively parallel computation (MPC) and circuit complexity theory:

• Equivalence with MPC: An $L$-layer transformer can simulate $O(L)$-round MPC protocols (with all-to-all communication), and vice versa. This correspondence allows transferring known results and lower bounds from parallel algorithms and communication complexity directly to transformer expressiveness (Sanford et al., 2024).
• Circuit Classes: Fixed-depth corresponds with $\mathsf{TC}^0$; log-depth with $\mathsf{TC}^1$; polylogarithmic depth (with sufficient padding and looping) yields all of $\mathsf{NC}$. These equivalences are witnessed constructively and provably tight (Merrill et al., 25 May 2025).
• Limitations of Efficient Approximations: Sub-quadratic attention approximations, window masking, and low-rank kernels destroy global connectivity, necessitating — for certain tasks — depths that scale linearly with the context, in contrast to the log-depth optimal parallel transformers (Sanford et al., 2024).

This tight integration with classical computational complexity anchors transformer expressiveness within a robust theoretical framework, simultaneously illuminating their power and inherent limitations.