Barriers to Universal Reasoning With Transformers (And How to Overcome Them)

Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness. However, whether Transformers can learn to generalize to CoT traces longer than those seen during training is understudied. We use recent theoretical frameworks for Transformer length generalization and find that -- under standard positional encodings and a finite alphabet -- Transformers with CoT cannot solve problems beyond $TC^0$, i.e. the expressivity benefits do not hold under the stricter requirement of length-generalizable learnability. However, if we allow the vocabulary to grow with problem size, we attain a length-generalizable simulation of Turing machines where the CoT trace length is linear in the simulated runtime up to a constant. Our construction overcomes two core obstacles to reliable length generalization: repeated copying and last-occurrence retrieval. We assign each tape position a unique signpost token, and log only value changes to enable recovery of the current tape symbol through counts circumventing both barriers. Further, we empirically show that the use of such signpost tokens and value change encodings provide actionable guidance to improve length generalization on hard problems.

• The paper demonstrates that standard transformers with a fixed vocabulary are limited to TC⁰, restricting length generalization in complex reasoning tasks.
• The paper introduces the use of signpost tokens and value-change logs to expand the vocabulary, enabling Turing-complete computation and improved empirical performance.
• The paper validates its theoretical claims through robust experiments on both small-scale models and large pretrained LLMs, highlighting enhanced algorithmic and symbolic reasoning.

Barriers to Universal Reasoning with Transformers: A Formal and Empirical Investigation

Introduction

"Barriers to Universal Reasoning With Transformers (And How to Overcome Them)" (2604.25800) provides a rigorous theoretical and empirical analysis of the limits of length generalization and algorithmic reasoning in transformers, with a focus on chain-of-thought (CoT) reasoning protocols. The study extends previous expressivity results by showing that under standard assumptions—fixed finite alphabet and common positional encodings—transformers with CoT are fundamentally limited in their ability to generalize to longer reasoning chains, failing to transcend the class TC$^0$ of constant-depth threshold circuits. However, the authors propose an architectural and representational refinement: allowing the model's vocabulary to scale with problem size via signpost tokens and logging value changes, they demonstrate provable length generalization up to Turing completeness. They further establish that such construction has actionable empirical implications, substantially improving generalization in hard algorithmic tasks.

Theoretical Framework and Formal Results

The analysis is grounded in recent advances on C-RASP and C$^*$-RASP, formal frameworks for understanding the algorithmic capabilities of (softmax-based) transformers with and without positional encoding across varying alphabet sizes. The study distinguishes between two regimes.

• Fixed Finite Alphabet (Standard Setup): When transformer models are restricted to a finite output vocabulary (even with CoT), provable length-generalizable learning coincides with the class TC$^0$ and its relatives. This class notably includes functions computable by constant-depth, polynomial-size threshold circuits, but not more general classes such as P. The main theorem asserts that any function with a CoT solution in Pos (Positional C-RASP) is definable in TC$^0$. Tasks such as Boolean formula evaluation or permutation in $S_5$ are outside TC$^0$ and thus provably not learnable in a length-generalizable CoT-prompted transformer with fixed-size vocabulary.
• Growing Alphabet (Signposts and Value Logs): By augmenting the input and intermediate representations with a polynomially growing set of tokens (formalized via $\mathcal{C} \sim \mathbb{N}$), transformers become theoretically capable of simulating arbitrary Turing machines. In the C$^*$-RASP formalism, signpost tokens uniquely index positions in the computation, and value-change logging enables state reconstruction through frequency counts, circumventing the two key algorithmic bottlenecks: repeated copying and last-occurrence retrieval.

  Figure 1: Simulation of a Turing machine by using unique signpost tokens per cell, along with value-change logs, enabling constant overhead and promoting reliable trace reconstruction for each position.

The coalescence of these principles enables, for the first time, length-generalizable Turing-complete computation in transformer architectures under practical training constraints. Critically, the construction is optimal up to a constant factor: the length of the CoT trace is linear in the number of TM steps.

Empirical Validation: Structured Reasoning and Length Generalization

The paper provides comprehensive empirical validation by training GPT-2-style transformers from scratch (6 layers; 8 heads; 25M parameters) on algorithmic tasks with varying input/output formats:

• PARITY (TC$^0$): When formatted according to a value-change encoding, transformers generalize from training on sequences of length 50 to test sequences of length 100 with no degradation; naive CoT completely fails even with more training data.

  Figure 2: Autoregressive CoT trace for PARITY task; with value-change formatting, correct generalization is obtained via local position-based transitions.

• Boolean Evaluation and S5 Permutation ($\notin$ TC$^*$0): With CoT input augmentation using signpost tokens (natural numbers as explicit positional hints), marked improvement in generalization is observed. While Boolean Evaluation demonstrates 100% generalization at double the training sequence length, S5 shows partial improvement, highlighting remaining optimization barriers due to possible shortcut strategies.

  Figure 3: Empirical evaluation of length generalization from sequences of length 50 (training) to 100 (test), with signpost and value-change formattings.

Analysis of failure modes in more complex permutation tasks points to the need for both unique position identification (signposts) and value-change logging, especially when object repetition breaks uniqueness of retrieval.

Prompting and Pretrained Models

The authors extend the insights to prompting large pretrained open-source LLMs (Llama, Mistral, Qwen). Few-shot prompts are constructed with and without signposts and value-change logs:

• With signposts: Substantial improvements in generalization for both Boolean and S5 tasks; performance without signposts sharply deteriorates as sequence length increases, sometimes even dropping below random baselines.
• With value-change logs: On code-execution tasks, variable-value annotations dramatically improve output correctness, indicating universality of the approach.

  Figure 4: Impact of prompt design—signposts and value-change logging—in large LLMs: generalization improves with theoretically-informed formatting even in the absence of additional training.

Implications for Theory and Practice

This work both tightens the limits of transformer reasoning under established architectural constraints and provides a constructive remedy through input/CoT representation design. The key theoretical message is that expressivity (e.g., Turing completeness of transformers with CoT [perez2019turing]) does not translate to learnability or generalization under finite, fixed-vocabulary setups; the addition of CoT alone is insufficient for hard algorithmic or symbolic reasoning. The construction with signposts and value-change logs serves as a template for bridging the gap between in-principle expressivity and practical learnability, operationalizing length generalization guarantees for any computable function.

On the empirical front, the work provides actionable guidelines for problem representation—namely, explicit indexing and minimal state update traces—ensuring transformers and LLMs can generalize well outside the i.i.d. regime, including on symbolic and stepwise reasoning benchmarks.

Outlook and Future Directions

The broader implication is that the learnable regime of transformers can be systematically extended by careful alignment of input/CoT symbolic structure and inductive bias. Future directions include:

• Integrating signpost and value-change schemes into large-scale pretraining, potentially yielding more robust generalization in complex algorithmic, logical, or programmatic tasks.
• Automated schema induction for signpost/value-change generation in natural tasks, reducing reliance on hand-crafted representations.
• Theoretical characterization of the required size and granularity of the growing alphabet across task classes and model scales.
• Exploring impacts for interpretable mechanistic circuit analysis, as explicit signposts and value logs facilitate tracing of internal model state transitions.

Conclusion

This paper presents a mathematically precise account of the fundamental limits of transformer-based models for universal algorithmic reasoning under standard architectures. The results delineate the separation between expressivity and guaranteed learnability with length generalization and propose explicit, theoretically principled architectural and representational augmentations that restore universality. The utility of these augmentations is validated empirically, impacting both models trained from scratch and large pretrained LLMs. The findings suggest a concrete path forward for extending the generalization capabilities of neural architectures on symbolic and algorithmic tasks and provide a foundation for future empirical and theoretical work on the interplay between CoT, symbolic representations, and transformer reasoning (2604.25800).