Algebraic Decomposition Theory for Transformer Length Generalization

Abstract: Transformer-based LLMs are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit length generalization. It is not even known which regular languages transformers length-generalize on -- and this is a foundational class of languages. Our contributions are to establish the first complete characterization of which regular languages transformers length-generalize on and provide a decision algorithm running in polynomial time in the size of the language's syntactic monoid. These results rely on an effective characterization of the regular languages in C-RASP, a recently-established formalism that expresses which languages transformers length-generalize on. This characterization is challenging because classical tools like Krohn-Rhodes decomposition theory for finite semigroups are insufficient for C-RASP. Firstly, the basic building blocks of Krohn-Rhodes theory -- flip-flop and simple groups -- are not expressible in C-RASP. Secondly, the basic building block of C-RASP (unbounded counting) is not expressible by the finite semigroups of Krohn-Rhodes theory. Thus, length generalization on regular languages is controlled by an algebraic property that is invisible to classical finite decomposition theory. We generalize classical decomposition theory from finite semigroups to the infinite additive group on the integers, allowing us to characterize C-RASP in terms of iterated wreath products of the integers and derive a provable polynomial-time decision algorithm for regular language membership. Experiments across a broad test suite of regular languages confirm that our theory captures transformers' length-generalization behavior more accurately than existing classifications.

• The paper characterizes regular languages recognized by C-RASP as those whose syntactic monoids divide iterated typed wreath products of the additive integers, replacing classical finite decomposition theory.
• The paper presents a polynomial-time membership algorithm based on derived categories, bounded Dyck monoids, and finitely generated integer modules, with an equivalent automaton refinement procedure.
• The paper validates the theory across 125 regular languages, showing that C-RASP languages retain near-perfect accuracy when tested at up to 10 times their training length, while others fail rapidly.

Motivation and problem statement

Whether transformer LLMs can generalize to inputs longer than those seen during training is a question that, for regular languages, has lacked a precise answer. The paper addresses this gap by asking: exactly which regular languages do transformers length-generalize on? Prior work established strong empirical evidence that length generalization tracks definability in C-RASP (Counting-RASP), a formalism capturing a subclass of transformer-expressible languages (2608.13433). However, no complete characterization of the regular languages in C-RASP existed. The paper's contributions are threefold: an effective algebraic characterization of the regular languages in C-RASP via iterated wreath products of the additive group of integers; a polynomial-time decision algorithm for membership given a language's syntactic monoid; and empirical validation on a suite of 125 regular languages.

The motivating observation is a minimal pair: the structurally similar languages $(ab+bbaa)^*$ and $(ab+aabb)^*$ diverge sharply in transformer length generalization, and no existing theory — expressivity characterizations based on $\mathsf{AC}^0$, star-free languages, or $\mathcal{R}$-trivial languages — explains this discrepancy.

Why classical decomposition theory fails

The natural tool for such questions is Krohn–Rhodes decomposition theory, which states that every finite monoid divides an iterated wreath product of the flip-flop monoid $U_2$ and simple groups dividing it. Two obstacles make this framework inapplicable to C-RASP. First, the building blocks of Krohn–Rhodes theory are not expressible in C-RASP: $U_2$ is not definable there, and C-RASP contains no non-trivial finite groups. Second, and conversely, the basic unit of C-RASP — unbounded counting over the integers — is not captured by any finite semigroup. Indeed, the untyped wreath product $\mathbb{Z} \wr \mathbb{Z}$ already recognizes every language, since its elements can encode arbitrary binary strings; unrestricted infinite wreath products are therefore trivially too powerful. Length generalization on regular languages is thus governed by an algebraic property invisible to classical finite decomposition theory.

Typed monoids and the wreath product characterization

To tame the infinity of integer-based factors, the authors adopt and adapt the typed monoid framework of Krebs et al. A typed monoid $(M, \mathbf{M}, \mathcal{M})$ consists of a finitely generated monoid $M$, a finite Boolean algebra $\mathbf{M}$ of subsets ("types"), and a finite set $(ab+aabb)^*$0 of "units"; recognition requires homomorphisms mapping generators into units and acceptance into types. For example, MAJORITY is recognized by the typed monoid $(ab+aabb)^*$1. The typed wreath product restricts the left factor's functions to be type-respecting — their values depend only on Boolean combinations of type conditions $(ab+aabb)^*$2 for finitely many constants $(ab+aabb)^*$3 — which restores decidability-relevant structure while retaining exactly the counting power of C-RASP.

The central characterization theorem states:

$(ab+aabb)^*$4

i.e., membership holds precisely when the syntactic monoid divides an iterated typed wreath product of $(ab+aabb)^*$5. The proof proceeds via a typed wreath product principle connecting program composition ($(ab+aabb)^*$6 substitutions of linear RASP programs) to wreath products of pseudovarieties, with depth-$(ab+aabb)^*$7 bounded Dyck languages corresponding to $(ab+aabb)^*$8-fold iterates of $(ab+aabb)^*$9.

The decision procedure

Deciding whether $\mathsf{AC}^0$0 divides such an iterated wreath product faces three challenges: defining "division" by a wreath factor, maintaining computability despite the infinitude of $\mathsf{AC}^0$1, and identifying termination. The first is handled classically via Tilson's derived category construction: for a relational morphism $\mathsf{AC}^0$2, the derived category $\mathsf{AC}^0$3 plays the role of "$\mathsf{AC}^0$4", and the Derived Category Theorem yields $\mathsf{AC}^0$5 whenever $\mathsf{AC}^0$6.

The procedure processes the $\mathsf{AC}^0$7-classes of $\mathsf{AC}^0$8 in order, maintaining relational morphisms into iterated wreath products of bounded Dyck monoids $\mathsf{AC}^0$9 (finite truncations of $\mathcal{R}$0). At each step it searches for nontrivial relational morphisms whose values are bounded on the current $\mathcal{R}$1-class; the set of such morphisms forms a finitely generated $\mathcal{R}$2-module, so each accepted step finds a linearly independent new morphism, guaranteeing termination. Completeness is argued by contradiction: if $\mathcal{R}$3 divided a minimal-depth wreath product of $\mathcal{R}$4's, the rightmost component either stays bounded (and would have been exploited by the algorithm) or runs to infinity (contributing no usable information), allowing its removal and contradicting minimality. An equivalent automaton-based formulation checks whether every strongly connected component of the minimal DFA is separable by totally balanced terms — terms whose values remain bounded across all cycles — via iterative refinement of state equivalences.

Two consequences follow. First, the regular languages in C-RASP are exactly those whose syntactic monoids lie in the pseudovariety generated by the bounded-depth Dyck monoids: $\mathcal{R}$5. Second, membership is decidable in time polynomial in the size of the syntactic monoid — a notable contrast to related problems in finite semigroup theory, some of which are undecidable.

Position within known hierarchies

The characterization induces a strict chain of pseudovarieties:

$\mathcal{R}$6

where $\mathcal{R}$7 denotes aperiodic monoids. The intermediate class $\mathcal{R}$8 is characterized by a profinite equation, $\mathcal{R}$9, and concretely consists of aperiodic monoids with at most one idempotent per $U_2$0-class; equivalently $U_2$1. This equation gives a simple necessary-but-not-sufficient criterion for C-RASP membership, checkable merely by counting idempotents rather than constructing relational morphisms. Notably, C-RASP is incomparable to $U_2$2 at the level of all languages yet strictly contained in it among regular languages (e.g., $U_2$3 separates them), lies strictly inside $U_2$4 (PARITY is in $U_2$5 but not C-RASP), and intersects every level of the dot-depth hierarchy without covering it. The witness separating $U_2$6 from C-RASP is precisely the syntactic monoid of $U_2$7, one member of the motivating minimal pair.

Experimental validation

The authors evaluate GPT-2 models on a state-prediction task (tracking DFA states over prefixes) across 125 systematically generated regular languages plus 50 deeper-nesting ones. Training uses lengths up to 50 (or 200 for the complex suite); evaluation extends to lengths up to 500. Two architectural controls align the setup with theory: positional embeddings are removed (NoPE), and separator tokens mediate access to the most recent symbol, preventing shortcuts based on local information. Hyperparameter grids are swept exhaustively, with multi-seed replication (up to five successful seeds per language).

The result is a clean separation: languages in C-RASP maintain near-perfect accuracy well beyond twice the training length (generalizing from length $U_2$8 to $U_2$9), while languages outside C-RASP collapse rapidly after the training range. This holds across seeds, across increased training data (100K examples), and across both language suites. The paper notes this paints a more optimistic picture than prior work reporting failure outside $U_2$0, attributing the discrepancy to the small number of previously testable languages in $U_2$1 — a set that could not even be computed before this work.

Limitations and open questions

Several caveats bound the scope of the results. The characterization applies to transformers without absolute positional encodings; extending it to architectures with periodic or local positional encodings would require additional techniques and would yield a different classification. The experiments concern the architecture itself rather than pretrained LLMs, whose behavior depends on prompt format and training data, though prior work suggests LLM capabilities may ultimately be bounded by C-RASP-like classes. The relationship between the empirical findings here and conflicting results in earlier studies (e.g., reported failures on languages outside $U_2$2 under different task formats and length ratios) remains to be fully reconciled. Finally, the extension of these techniques to other architectures with limited recurrence, such as state-space models capable of simulating flip-flops and counting, is posed as an open direction rather than resolved.

Conclusion

This work supplies the first complete, polynomial-time decidable algebraic characterization of the regular languages on which transformers length-generalize, replacing the inapplicable Krohn–Rhodes framework with a decomposition theory built on typed wreath products of the integers. The resulting hierarchy situates C-RASP precisely among classical pseudovarieties, and large-scale experiments confirm that C-RASP membership predicts length-generalization behavior more accurately than any prior classification.