Computational Hardness of Transformers

• Computational Hardness of Transformers is a study of transformer models' limitations, characterizing their expressive bounds through circuit complexity and precision constraints.
• The analysis reveals trade-offs between parallelism, depth, and arithmetic precision, showing that constant-depth architectures align with low-complexity classes like TC⁰.
• It highlights challenges in symbolic reasoning and parameter recovery, demonstrating that fixed-resource transformers struggle with functions like parity and complex compositional tasks.

The computational hardness of transformers refers to both the theoretical and practical limitations in the computational power and complexity of transformer architectures, especially as formalized by circuit complexity, communication complexity, and approximation theory. The transformer’s architecture imposes stringent upper bounds on the classes of functions and formal languages it can efficiently compute, and also reveals trade-offs between parallelism, depth, width, and inference-time strategies.

1. Circuit Complexity and Expressive Power

The circuit-complexity analysis for transformers quantifies their expressive power via Boolean and threshold circuits. A saturated-attention transformer, where attention is realized by uniform averaging over maximal scores, can be exactly simulated by constant-depth threshold circuits (the complexity class $\mathsf{TC}^0$), provided internal computation is bounded to $O(\log n)$-bit precision per token. Specifically, Theorem 3.3 in ["Saturated Transformers are Constant-Depth Threshold Circuits" (Merrill et al., 2021)] establishes that any fixed-depth, fixed-width saturated transformer over floating-point valued activations only recognizes languages in $\mathsf{TC}^0$. This is achieved by formally demonstrating:

• Each layer’s activations (vectors in $\mathbb{F}^k$) remain $O(\log n)$ bits;
• Each primitive transformation (score computation, maximization, masking, addition, division by powers of two) is realizable within constant-depth threshold circuits.

Thus, the recognition capacity of such transformers is upper-bounded by highly parallel classes, precluding the exact computation of $\mathsf{NC}^1$-complete or harder languages for constant-depth (see (Strobl, 2023) for a uniform version).

Crucially, transformers with "hard attention" (strict one-hot selection) restrict further to $\mathsf{AC}^0$—they cannot even compute majority functions, as shown by prior work. In contrast, saturated attention allows transformers to count, thus matching the power of threshold logic ($\mathsf{TC}^0$), and allows computation of threshold-based functions such as majority, delineating a strict expressive gap.

2. Parallelism Constraints and Tradeoffs

Transformers are engineered for extreme parallelism, which is reflected in their simulatability by shallow ($O(1)$-depth) threshold circuits when arithmetic precision per token is $O(\log n)$ bits (Merrill et al., 2022). The salient result is that any transformer with $O(\log n)$-bit precision and fixed depth $d$ can be simulated by log-space-uniform, constant-depth $\mathsf{TC}^0$ circuits. This parallelism imposes a tradeoff:

• Any model whose computation can be performed in $O(1)$ sequential rounds over $n$ processors, with $O(\log n)$ bits per processor, is necessarily restricted to $\mathsf{TC}^0$;
• Increasing precision or sequential depth pushes the simulation to higher complexity classes (such as $\mathsf{NC}^1$), but at the cost of parallel efficiency.

This tradeoff—termed the "parallelism tradeoff"—underscores inherent limits of large-scale, parallel transformer-based architectures: maximizing parallelization directly constrains expressive power, notably rendering intractable any exact solution to $P$-complete or $NP$-complete problems in the log-precision, constant-depth regime.

3. Limitations in Reasoning and Compositional Power

For multi-step or compositional reasoning (e.g., Boolean formula evaluation, multi-stage decision problems formalized as Compositional Reasoning Questions—CRQs), transformer architectures face lower bounds sharply dictated by their depth and precision. As established in ["Compositional Reasoning with Transformers, RNNs, and Chain of Thought" (Yehudai et al., 3 Mar 2025)], general CRQs are $\mathsf{NC}^1$-complete; constant-depth transformers are strictly contained in $\mathsf{TC}^0$, which is conjecturally less powerful than $\mathsf{NC}^1$. This yields:

• No fixed-depth transformer of polynomial size and $O(\log n)$-bit precision can solve all CRQs.
• To solve CRQs on trees of depth $O(\log n)$ (e.g., balanced binary trees of $n$ nodes), the transformer’s depth must scale as $\Omega(\log n)$; otherwise, shallower transformers necessarily fail to realize general compositional reasoning.
• Trade-offs are possible: RNNs require logarithmic hidden-state dimension, and transformers with chain-of-thought tokens (sequential intermediate tokens) can achieve the task with $O(n)$ CoT tokens at the expense of serial computation.

This reveals that compositional tasks inherently demand resource growth (depth, width, or intermediate computation length) with input size; constant-resource regimes are provably insufficient under standard complexity-theory assumptions.

4. Barriers in Symbolic and Discrete Reasoning

Empirical and theoretical work shows that transformers are fundamentally limited in discrete and symbolic reasoning, especially for tasks involving highly sensitive functions (e.g., PARITY), discontinuous decision boundaries, and algorithmic compositions. As synthesized in the survey (Yuan et al., 19 Jan 2026):

• Circuit complexity theory establishes inexpressibility of parity, majority, and sorting in constant depth (with precise lower bounds: e.g., any constant-depth, polynomial-size transformer cannot realize parity),
• Approximation theory highlights that functions with sharp discontinuities (e.g., step or parity functions) cannot be well-approximated without superpolynomial parameter growth in fixed-depth architectures; error composes multiplicatively across steps, creating an exponential blowup in approximation error for deep compositions,
• Communication complexity demonstrates that information transmission along sequences (e.g., for pointer-chasing, long-range equality) requires either deep architectures or wide per-token representations. For n-bit equality at distance, $L \cdot d \geq \Omega(n)$ is necessary.

Concrete lower bounds for parity show that a $1$-layer, $1$-head transformer cannot compute $\mathrm{PARITY}_n$—the average sensitivity is $O(\sqrt{n})$, whereas parity’s sensitivity is $n$ (Kozachinskiy et al., 5 Feb 2026, Hahn et al., 2024). Construction of parity in constant depth is possible but needs at least $2$ layers, polynomially bounded positional encodings, and nontrivial architectural choices.

5. Intractability of Efficient Attention Computation

Transformers inherently require $O(n^2)$ time for evaluating full attention (i.e., evaluating all $n^2$ pairwise comparisons). Attempts to circumvent this via more efficient alternatives (e.g., linear attention, fast heuristics, state-space models) are fundamentally limited in their capacity to reproduce certain tasks—specifically, document similarity tasks and associated combinatorial search problems. If the Strong Exponential Time Hypothesis (SETH) holds, no $o(n^2)$-time algorithm (including any attention replacement or heuristic) can outperform the transformer’s $\Theta(n^2)$ lower bound on such problems (Alman et al., 2024). Thus, any subquadratic approach must necessarily sacrifice correctness or generality on these tasks.

Related computational lower bounds further show that for multi-head, multi-layer transformers, the cost of evaluating all attention heads cannot be improved beyond $L H N^{2 + o(1)}$ in the regime where embedding dimension $m = N^{o(1)}$, assuming SETH (Saha et al., 11 Mar 2026). In the $m = N$ case, the $N^\omega$ complexity (matrix multiplication exponent) is optimal.

6. Hardness in Parameter Recovery and Learning Dynamics

Even when the functionally optimal predictions are convex and efficiently computable in an expanded parameter space (e.g., via population loss minimization in single-layer linear self-attention models), recovering explicit transformer parameters that realize the optimum is NP-hard in the dimension (Ding et al., 21 Oct 2025). This is due to the necessity of solving overdetermined bilinear constraints; such hardness persists as chain length or system size grows. Consequently, representing Markovian or otherwise structured dynamical functions with shallow transformers is subject to intractability, exposing a separation between what is learnable and what is representable by the architecture.

7. Implications, Open Questions, and Remedies

These hardness results collectively demarcate sharp boundaries for transformer computation:

• Adding inference-time padding and dynamic looping (test-time depth increases) can augment expressive power up to classes $\mathsf{TC}^d$, matching polylog depth threshold circuits; with enough looping and padding, transformers can simulate all of $\mathsf{NC}$, but not $\mathsf{P}$ unless $\mathsf{NC} = \mathsf{P}$ (Merrill et al., 25 May 2025). However, this approach remains parallelizable, unlike chain-of-thought, which introduces inherently sequential computation, potentially going beyond $\mathsf{NC}$.
• Overcoming discrete reasoning barriers requires architectural innovations: chain-of-thought for increased sequential reasoning, increased depth/width for longer-range dependencies, or architectural modifications such as recurrence, external memory, or higher-precision arithmetic.
• Empirically, transformers exhibit a generalization bias toward low-sensitivity, low-degree functions, explained via the geometry of the loss landscape and the isolation of sensitive solutions (Hahn et al., 2024).

Open questions remain about the precise trade-offs between precision, depth, circuit class, and practical algorithmic tasks—including the search for explicit natural-language processing benchmarks lying outside $\mathsf{TC}^0$ or $\mathsf{NC}$. Also, the formalization of general attention variants and their corresponding circuit classes is a substantial avenue of current research.

Summary Table: Circuit Class Limits and Transformer Model Variants

Transformer Model	Attention Type	Circuit Class	Can Compute Majority?	Can Compute Parity?
Hard attention (selector)	Hard, one-hot	$\mathsf{AC}^0$	No	No
Saturated/average-hard attention (max tie avg)	Saturated, averaging	$\mathsf{TC}^0$	Yes	No (constant depth)
Softmax attention w/ log-precision	Softmax ($O(\log n)$)	uniform $\mathsf{TC}^0$	Yes	No (constant depth)
Padded+looped, log-depth	Hard/avg, polypadding	$\mathsf{TC}^d$	Yes	No if $d = O(1)$
Deep/chain-of-thought (arbitrary serial steps)	N/A	up to $\mathsf{P}$	Yes	Yes (if length scales w/ $n$)

Any language or decision problem outside $\mathsf{TC}^0$ (e.g., context-sensitive languages, Boolean formula evaluation, general CRQs, or $P$-complete problems) lies beyond the reach of constant-depth, log-precision, highly parallel transformers with standard architectural constraints. This comprehensively characterizes the computational hardness of transformers in terms of theoretical circuit complexity, communication bottlenecks, and practical algorithmic limits.