Role of periodic relations in length generalization

Investigate the role of periodic positional relations (e.g., modular relations of the form i ≡ k (mod m)) in length generalization within the proposed $$ framework and Symbolic Limit Transformer model, and determine how such relations should be incorporated or analyzed to characterize transformers’ generalization behavior.

Background

The paper extends the C-RASP framework to $$ to handle simultaneous growth in sequence length and vocabulary size, providing guarantees via Symbolic Limit Transformers. Unlike prior work by Huang et al. (2025), the authors omit periodic positional relations (modular relations) from their framework, arguing these are not necessary for their main planning results and noting technical complications with rank penalties used previously.

They point out that periodic relations were useful in earlier analyses to explain APE length generalization for certain formal languages and suggest that understanding their role in the present framework remains unresolved.