Length-generalization characterization for alternative architectures

Determine which regular languages architectures with limited recurrence, including state-space models, length-generalize on.

Background

The paper develops a complete characterization of regular languages on which the studied transformer architecture length-generalizes. It explicitly notes that this characterization does not extend automatically to other architectures. Although log-depth architectures are known to simulate arbitrary automata, the regular languages on which they length-generalize have not been characterized. The authors likewise leave unresolved the corresponding question for architectures with limited recurrence, such as state-space models, whose ability to simulate flip-flops and counting depends on the assumptions imposed.