C*-RASP: Transformer Generalization

• C*-RASP is a formal extension of C-RASP that incorporates an infinite token alphabet and a Match-predicate to handle variable object IDs, ensuring length and vocabulary generalization.
• It employs counting and matching operations to verify plan validity in delete-free or well-formed STRIPS domains, tracking action effects and proposition states.
• The framework provides formal guarantees for transformer generalization while guiding structural adjustments in planning domains to ensure successful plan verification.

C*-RASP is an extension of C-RASP designed to establish length generalization guarantees for transformers under the simultaneous growth in sequence length and vocabulary size. In the plan-verification setting, it operates over an input tape whose alphabet is the disjoint union

$\Omega \;=\;\Sigma\;\cup\;\mathcal C,$

where $\Sigma$ is a fixed finite alphabet and $\mathcal C\cong\mathbb N$ is an unbounded object-ID alphabet. Its distinctive addition is a Match-predicate that supports local identity tests over object-ID tokens without hard-coding particular IDs, which is central when training uses instances with up to $k$ objects but testing uses instances with $k'>k$ objects (Sarrof et al., 20 Mar 2026).

1. Formal system and operational vocabulary

Following Definition 3.2, a C*-RASP program is a finite list of operations

$P_1,\dots,P_\ell$

over a string $w_1\cdots w_n$, indexed by positions $i=1,\dots,n$. Each $P_k$ is either Boolean-valued, $B(i)\in\{\bot,\top\}$, or count-valued, $\Sigma$0. The last operation $\Sigma$1 must be Boolean, and the program accepts $\Sigma$2 exactly if

$\Sigma$3

A finite set $\Sigma$4 of local binary relations $\Sigma$5 is fixed in advance. These relations are used to restrict counting to a local neighborhood; the paper gives as an example “$\Sigma$6” for small constant $\Sigma$7 (Sarrof et al., 20 Mar 2026).

The Boolean-valued operations are: $\Sigma$8

The count-valued operations are: $\Sigma$9

The Match-predicate is a conjunction of equalities among object-ID tokens in local neighborhoods, for example

$\mathcal C\cong\mathbb N$0

where each $\mathcal C\cong\mathbb N$1 is a fixed constant and $\mathcal C\cong\mathbb N$2 is the token at position $\mathcal C\cong\mathbb N$3. This design lets the formalism express identity-sensitive local structure while remaining invariant to the particular numeric names assigned to objects (Sarrof et al., 20 Mar 2026).

2. Relation to C-RASP and neighboring expressivity results

Original C-RASP is defined “exactly as above” except that its alphabet $\mathcal C\cong\mathbb N$4 is fixed and finite, there is no separate $\mathcal C\cong\mathbb N$5, it includes a restricted set of local positional relations $\mathcal C\cong\mathbb N$6, and it does not have a Match operation because one can query “is token $\mathcal C\cong\mathbb N$7” with the usual $\mathcal C\cong\mathbb N$8. C*-RASP therefore adds exactly two ingredients: an infinite token alphabet $\mathcal C\cong\mathbb N$9, and a Match-predicate $k$0 enabling local identity tests on $k$1 without hard-coding any particular object ID (Sarrof et al., 20 Mar 2026).

The underlying C-RASP formalism is part of a broader line of results on transformer expressivity. C-RASP was introduced alongside the temporal counting logic $k$2[#], and the central equivalence theorem states that, over any alphabet $k$3, a language $k$4 is recognized by a C-RASP program iff it is defined by some sentence of $k$5[#]. The same work shows that every $k$6[#] formula can be compiled into a standard, future-masked softmax transformer encoder, and conversely every fixed-precision, future-masked softmax transformer encoder without positional encoding can be translated into an equivalent sentence of $k$7#.

Later work extends the C-RASP viewpoint to chain-of-thought generation. In particular, “Softmax Transformers are Turing-Complete” proves that length-generalizable softmax CoT transformers are Turing-complete via a CoT extension of C-RASP; it proves Turing-completeness for CoT C-RASP with causal masking over a unary alphabet, and full Turing-completeness for arbitrary languages when relative positional encoding is added (Jiang et al., 25 Nov 2025). This suggests that C*-RASP should be read not as a replacement for C-RASP, but as a targeted extension for the variable-vocabulary regime relevant to planning domains with growing object universes.

3. Plan verification and formal generalization guarantees

The plan-verification application is stated in terms of a planning domain $k$8 and the language

$k$9

The main theorem specialized to the variable-universe setting is:

$k'>k$0

This is the paper’s Theorem 3.4 (Sarrof et al., 20 Mar 2026).

The corresponding transformer consequence is stated as a corollary: if $k'>k$1 C*-RASP, then any decoder-only transformer with APE, trained by minimizing a mild regularizer on valid vs invalid examples up to plan length $k'>k$2 and object-ID range $k'>k$3, will—once $k'>k$4 exceeds some threshold—be correct on all plan lengths and object-ranges $k'>k$5. The paper attributes this to Theorem 3.6 (informal) together with the formal limit-transformer guarantees in $k'>k$6 (Sarrof et al., 20 Mar 2026).

The proof strategy for membership is constructive. To show $k'>k$7 C*-RASP for delete-free or well-formed domains, the program uses counting operations to track net adds and deletes of each ground proposition via the sets $k'>k$8, uses match-predicates to align action arguments with proposition arguments, checks each action’s preconditions, and checks goals at the end. The negative direction reduces non-membership to languages such as FlipFlop and PARITY, which are stated not to belong to C*-RASP (Sarrof et al., 20 Mar 2026).

4. Structural conditions on planning domains

The paper isolates four subclasses of planning domains: conditional-effects STRIPS, plain STRIPS, well-formed STRIPS, and delete-free STRIPS. In the fixed-universe case, Theorem 3.1 states that delete-free and well-formed plan-verification languages belong to C-RASP[Pos], whereas there are STRIPS languages (FlipFlop) and conditional-effect languages (PARITY) that are not in C-RASPPos.

The significance assigned to these subclasses is structural rather than merely statistical. The paper’s interpretation is that well-formedness means each proposition’s truth is determined by counting net toggles, while delete-free means each proposition is true iff it was initially true or ever added. By contrast, full STRIPS can require “find the last action that affected $k'>k$9,” which reduces to FlipFlop, and conditional effects can encode parity (Sarrof et al., 20 Mar 2026).

A common misconception is that plan length alone determines whether a transformer should generalize. The paper argues otherwise: “The structural form of the planning formalism (well-formed, delete-free) rather than mere plan length or number of objects determines whether transformers can length-generalize.” It further states that domains that are well-formed, including “70% of IPC 2023 Learning Track benchmarks,” fall in C*-RASP and so admit provable length generalization, whereas conditional effects and full STRIPS can encode FlipFlop or PARITY and are therefore not expected to length generalize (Sarrof et al., 20 Mar 2026).

This yields a precise boundary. C*-RASP is not presented as a characterization of arbitrary planning verification. Rather, it captures a large class of verifiable domains with simultaneous length and vocabulary growth, while also identifying structural barriers in the remaining cases.

5. Constructing a C*-RASP verifier

Appendix A.2 gives an explicit recipe for writing a verifier for a delete-free or well-formed domain. The input is delimited as

$P_1,\dots,P_\ell$0

Boolean predicates $P_1,\dots,P_\ell$1, $P_1,\dots,P_\ell$2, $P_1,\dots,P_\ell$3, and $P_1,\dots,P_\ell$4 are introduced by counting the number of @ separators up to position $P_1,\dots,P_\ell$5 (Sarrof et al., 20 Mar 2026).

For each ground action schema $P_1,\dots,P_\ell$6 and each grounding of its arguments from $P_1,\dots,P_\ell$7, one defines a Boolean $P_1,\dots,P_\ell$8 that is true exactly when position $P_1,\dots,P_\ell$9 ends a copy of $w_1\cdots w_n$0. Analogous predicates are defined for each ground proposition $w_1\cdots w_n$1 in $w_1\cdots w_n$2 or $w_1\cdots w_n$3. These are built with small-offset $w_1\cdots w_n$4 predicates testing “$w_1\cdots w_n$5, $w_1\cdots w_n$6, …, $w_1\cdots w_n$7, and $w_1\cdots w_n$8.”

For each ground proposition $w_1\cdots w_n$9, the paper defines

$i=1,\dots,n$0

and then the count streams

$i=1,\dots,n$1

The current truth of $i=1,\dots,n$2 is then computed differently in the two tractable cases. In the delete-free case,

$i=1,\dots,n$3

In the well-formed case,

$i=1,\dots,n$4

and $i=1,\dots,n$5 checks that this net count is exactly $i=1,\dots,n$6.

Each action occurrence $i=1,\dots,n$7 at position $i=1,\dots,n$8 is validated by checking all of its preconditions $i=1,\dots,n$9 through $P_k$0. The construction marks $P_k$1 if any precondition fails, counts the total number of invalid actions, and requires it to be zero at $P_k$2. Goal literals are handled by the same current-truth computation over the goal block. The final Boolean stream is

$P_k$3

and acceptance occurs exactly when $P_k$4 (Sarrof et al., 20 Mar 2026).

When $P_k$5 varies, direct symbol tests on object names are replaced by Match-based alignment. The variable-$P_k$6 case uses

$P_k$7

This is the mechanism that makes the verifier independent of which concrete object IDs appear in training versus test instances.

6. Empirical evidence and practical implications

The empirical study evaluates three domains in two variants each, yielding six datasets: Heavy Grippers (well-formed vs delete-free; variable universe), Colors (well-formed vs plain STRIPS; variable universe), and Lights Out (5×5) (well-formed vs conditional-effects; fixed universe). Data generation uses balanced valid and invalid plans of length 11–200, with training on 11–100; invalid plans are either incomplete or non-executable; and in variable-$P_k$8 domains the number of objects grows with plan length (Sarrof et al., 20 Mar 2026).

The model is a GPT-2–style decoder-only transformer with 8 layers, hidden size 768, 12 heads, APE, and pre-layernorm. Training uses block-packed inputs of up to 4096 tokens, batch size 8, 50 k steps, AdamW, warmup 10%, weight decay 0.1, clip 0.5, bfloat16, and a single A100/H100. Cross-instance attention is disabled for Colors and Grippers. Learning rate is selected per domain, with training on 11–100, validation on ID(11–100) and OOD(101–200), and testing likewise (Sarrof et al., 20 Mar 2026).

The reported outcomes are sharply aligned with the formal classification. Well-formed and delete-free variants achieve near-perfect ID and OOD accuracy, even at plan length 200 and larger object set. Plain STRIPS Colors dips sharply in OOD. Conditional-effects Lights Out stays at chance both ID and OOD, even in fixed universe. Well-formed Lights Out is again perfect, despite approximately 512 action schemas. The paper summarizes this as: C*-RASP membership implies provable length generalization; non-membership predicts empirical failure (Sarrof et al., 20 Mar 2026).

The practical recommendations are correspondingly structural. If a domain is naturally well-formed or delete-free, the paper states that one can safely train transformers on short plans and objects and expect generalization. Otherwise, it recommends reformulating the domain or compiling away conditional effects into well-formed ground actions. It also recommends enforcing translation invariance during training through random position off-sets and shuffling of object IDs, and using absolute positional embeddings when the program relies on local $P_k$9 positional tests; if it uses only Match with zero offsets, positional encodings can be omitted (Sarrof et al., 20 Mar 2026).