From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP

Abstract: A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of LLMs. There is much work analyzing the expressivity of attention-based models. By proposing handcrafted weights or using computational complexity arguments, a large amount of past theoretical works have sought to characterize which tasks are and which are not in the hypothesis class of Transformer models. However, little work investigates the learnability of such solutions. In this work, we make progress towards this goal. Inspired by recent loss landscape analysis work, we propose preliminary sample complexity bounds for learning C-RASP constructions with Transformers.

• The paper presents a constructive sample complexity bound linking C-RASP program depth to transformer performance using narrow teacher models.
• It provides an explicit upper bound on sample complexity, demonstrating that wider student networks achieve PAC generalization with significantly fewer samples compared to classical approaches.
• The framework is validated through the Dyck-1 language recognition task, highlighting its practical implications for algorithmic learning in attention-based architectures.

Bridging Transformer Expressivity and Learnability: Sample Complexity via C-RASP

Motivation and Background

A comprehensive understanding of the theoretical capabilities of Transformer models is critical for advancing both their practical performance and reliability in LLM applications. Previous analyses have rigorously investigated the expressivity of attention-based architectures, characterizing the representable functions and situating them within known computational complexity classes [hahn2020theoretical, bhattamishra2024separations, rizvi2024simulating]. Programmatic characterizations via RASP and its extension C-RASP have formalized the languages and logics that Transformers can recognize, demonstrating equivalence with classes of automata and temporal logics [weiss2021thinking, yang2024counting, yang2025knee]. However, while these works identify what is possible for a Transformer to encode, they largely omit considerations of learnability—specifically, the conditions under which a network acquires such solutions from data.

This paper addresses this gap by translating expressivity arguments into quantifiable sample complexity bounds. Leveraging recent insights on the loss landscape [chiang2022loss, buzaglo2024uniform], the framework employs the concept of "narrow teachers"—models with minimal parameter footprint that are consistent with labels—and analyses how wider student networks emulate these teachers.

C-RASP Formalism and Transformer Implementation

C-RASP extends RASP by introducing counting operations, enabling compilation into future-masked, soft-attention Transformers without input length constraints [yang2024counting]. The two operation types—Boolean-valued and Count-valued—allow for rich algorithmic constructs, such as Dyck-1 language recognition. Acceptance is determined by the final Boolean value at the last sequence position.

Correspondence between C-RASP program depth and Transformer depth is formalized: a program of depth $n$ maps to an $n$-layer Transformer with constant width proportional to the number of program variables [yang2025knee]. This establishes a tight relationship between C-RASP program size and implementable Transformer configuration, enabling exact and efficient emulation of logical operations by attention and MLP sublayers.

Sample Complexity Analysis

Problem Setup

Let $N$ i.i.d. sequences from a distribution $\mathcal{D}$ and a teacher model $h^*$ be given. The empirical and true risks are defined over Transformer predictors. Both teacher and student models are assumed to be $Q$-quantized, restricting parameter precision.

Guess-and-Check Learning

The proposed framework adopts the Guess-and-Check (G&C) learning procedure: randomly sample network weights, check empirical error, and retain weights achieving error zero. The generalization guarantee is linked to the fraction of parameter space consistent with the teacher (i.e., teacher-equivalent models).

Main Theoretical Results

The central result provides an explicit upper bound on the sample complexity required to learn a C-RASP construction. Given a depth-$n$ C-RASP program with $m$ variables and a single-head teacher Transformer (width $2m + c$, absolute constant $c$), a wider student (width $n$0, depth $n$1) achieves PAC-generalization with probability $n$2 as long as

$n$3

where

$n$4

The bound is of order $n$5, which is notably tighter than classical Occam-type arguments ($n$6). The proof is constructive and proceeds by embedding the teacher within the student via weight assignments, zeroing-out non-teacher activations to maintain isolation and consistency.

Application: Dyck-1 Recognition

Applying the bound to the Dyck-1 C-RASP program (7 operations, 9 variables), a single-head student with $n$7 and width $n$8 achieves correct learning with $n$9 samples, substantiating the practical feasibility of such constructions.

Practical and Theoretical Implications

This framework provides a principled bridge between expressivity and learnability, setting a foundation for rigorous analysis of Transformer training dynamics in the context of algorithmic tasks and formal language recognition. Sample complexity estimates derived here enable more targeted design of training regimes and architectures for structured sequence tasks.

The results indicate that expressive constructions characterized by C-RASP are not only implementable in theory but can be learned efficiently under random parameter initialization, provided the existence of narrow teachers. This advances understanding of why constant-size formal languages (e.g., Dyck-1, $N$0) are readily acquired by Transformers, even absent explicit inductive biases.

Limitations and Future Directions

Several avenues for extension exist:

• The theoretical analysis currently lacks empirical validation, particularly regarding loss landscape effects on length generalization in Transformers and SSMs.
• The bounds on the probability of sampling a teacher-equivalent model are conservative; improved encoding schemes or broader posterior exploration could yield tighter sample complexity guarantees.
• The framework may be adapted to variable-precision models or attention variants, broadening applicability to emerging architectures.

Conclusion

By linking C-RASP expressivity results to explicit sample complexity bounds, this work advances the theoretical understanding of what Transformer models can efficiently learn from data. The results elucidate key factors governing learnability in attention architectures—program depth, width, and parameter quantization—opening pathways for more robust and predictable model design. Further empirical and theoretical exploration is warranted to refine these guarantees and deepen insights into Transformer training dynamics.

References

• "From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP" (2607.11760)