Analyzing transformer length generalization in other practical settings

Analyze and characterize the length generalization behavior of transformer architectures beyond the q-sparse token selection task, identifying conditions and mechanisms that enable models trained on shorter sequences to extrapolate reliably to longer sequences in practical applications.

Background

The paper proves that the trained transformer with stochastic positional encoding exhibits strong out-of-distribution length generalization on the q-sparse token selection task, both theoretically and empirically.

The authors highlight the need to understand and analyze length generalization more broadly across practical settings, beyond the specific synthetic task studied here, to determine general principles and limitations for real-world applications.