Length Generalization in Neural Models

• Length Generalization is the ability of models to extrapolate performance to inputs longer than those seen during training, influenced by factors like positional encoding and data representation.
• Empirical studies show that strategies such as relative positional embeddings, random float sampling, and task hinting can significantly improve LG on arithmetic, language, and algorithmic tasks.
• Innovative architectures including looped transformers, hybrid recurrence-attention models, and Turing-style process supervision offer promising solutions to overcome LG challenges.

Searching arXiv for papers on length generalization in transformers and related sequence models. Length generalization (LG) is the ability of a model to maintain performance on inputs longer than those seen during training or pretraining. Across the recent literature, LG is treated as a specific form of out-of-distribution extrapolation in sequence length, and it has become a central diagnostic for transformers, recurrent models, and hybrid architectures on arithmetic, symbolic reasoning, formal languages, algorithmic tasks, language modeling, and speech enhancement (Shimizu et al., 15 Feb 2026). The contemporary research record shows that LG is neither a single failure mode nor a single capability: it depends on positional encoding, data representation, auxiliary supervision, recurrence structure, training dynamics, and, in several formal analyses, the computability or structural complexity of the target function (Zhou et al., 2024, Xiao et al., 2024, Chen et al., 3 Jun 2025, Yang et al., 13 Feb 2026).

1. Definitions and problem setting

In the cited work, LG is consistently defined as extrapolation from shorter training sequences to longer test sequences. One formulation describes it as the ability of LLMs to maintain performance on inputs longer than those seen during pretraining (Shimizu et al., 15 Feb 2026). Another defines it as the ability to extrapolate from shorter training sequences to longer test ones, emphasizing arithmetic addition as a test case (Zhou et al., 2024). A more general theoretical treatment frames non-asymptotic LG as learning a hypothesis that generalizes to longer inputs than those in the training set, together with a computable upper bound on the minimum input length needed to guarantee that generalization (Chen et al., 3 Jun 2025).

This shared definition masks substantial variation in evaluation regimes. Some studies train on sequence lengths up to a fixed threshold and test on moderately longer inputs, such as training on sequences up to 20 and testing up to 40 on Copy, Reverse, Addition, Sort, Summation, and SCAN (Shimizu et al., 15 Feb 2026). Others use much larger extrapolation ratios: standard transformers trained on sequences of up to 40 digits can, under the right conditions, exceed 98% exact match accuracy on 100-digit addition, corresponding to 2.5× extrapolation (Zhou et al., 2024). In recurrent-model analyses, LG is evaluated at far larger scales, such as generalization from 2k training contexts to 128k validation contexts after post-training interventions (Ruiz et al., 3 Jul 2025).

The literature also distinguishes between empirical LG and formal LG. Empirical LG concerns observed accuracy curves as a function of test length. Formal LG concerns conditions under which a target function is identifiable from bounded-length data, or whether a computable training-length bound exists at all (Chen et al., 3 Jun 2025, Yang et al., 13 Feb 2026). This distinction is central because several papers show that strong in-distribution performance, low training loss, or large model scale do not reliably predict OOD length performance (Zhou et al., 2024, Anil et al., 2022).

2. Positional structure as a primary determinant

A dominant theme in the literature is that positional encoding (PE) strongly shapes LG. Early arithmetic results show that relative position embeddings enable length generalization for addition: encoder-only transformers with relative position embeddings, trained on 5-digit numbers, achieve 100% at 6 digits, 99.9% at 10 digits, 97.2% at 15 digits, and 21.3% at 20 digits, while absolute position embeddings collapse to near-zero OOD accuracy (Jelassi et al., 2023). In speech enhancement, relative positional encoding methods are reported to exhibit better LG than absolute positional encodings, and absolute methods may degrade below no-position baselines on long utterances (Zhang et al., 7 Jun 2025).

A major 2026 contribution proposes Random Float Sampling (RFS), a PE indexing strategy in which training positions are obtained by sampling continuous values from $[0,1)$, sorting them to preserve order, and then scaling by a factor $L$ with default $L=1000$ (Shimizu et al., 15 Feb 2026). At inference, evenly spaced indices are used:

$$p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}$$

before scaling. The stated rationale is that continuous random indices densely expose the model to diverse positional values during training and thereby avoid out-of-distribution positional indices at longer test lengths (Shimizu et al., 15 Feb 2026). Because the method changes only index generation, it is plug-and-play with absolute sinusoidal encoding, RoPE, and ALiBi. Empirically, on the Copy task with sinusoidal PE, RFS maintains approximately 80% accuracy on twice-longer inputs, whereas NoPE drops to approximately 20% accuracy on 1.5× longer inputs; on seven commonsense benchmarks, average OOD accuracy reaches 35.78%, close to baseline ID accuracy of 35.84% and above baseline OOD accuracy of 34.47% (Shimizu et al., 15 Feb 2026).

Other PE-centric proposals pursue related goals through different mechanisms. The FIRE positional encoding, especially when combined with randomization, is reported as crucial for strong addition extrapolation in standard transformers (Zhou et al., 2024). In speech enhancement, LearnLin introduces one trainable scalar per attention head to scale the real relative position between time frames, with bias

$P^h_{i,j} = \beta^h \cdot |i-j|,$

and is described as comparable or superior to other state-of-the-art PE strategies for LG (Zhang et al., 7 Jun 2025). In theoretical work on PE limitations, Linear Representation Complexity (LRC) and Sequential Representation Complexity (SRC) are introduced to argue that PEs do not expand computational capabilities but instead structure learned computations across positions; the paper further proposes Scale Hint and a Learning-Based Position Embedding framework for cases where suitable positional relations must be learned or scaled flexibly (Chen et al., 5 Oct 2025).

A consistent empirical pattern is that PE choice alone is not uniformly sufficient. In addition, FIRE with randomization helps, but performance remains fragile across seeds and data order (Zhou et al., 2024). In speech enhancement, RPE is necessary but not identical across methods (Zhang et al., 7 Jun 2025). In PE theory, LG is conjectured to be possible if and only if SRC remains invariant across scales, which suggests that PE effectiveness is constrained by the underlying operator complexity of the task rather than by positional features alone (Chen et al., 5 Oct 2025).

3. Representation, hints, and training data design

A second major axis of LG research concerns how the task is represented to the model. In arithmetic addition, presenting digits in reversed format aligns the computation order with carry propagation and is described as critical for generalization; adding index hints further improves alignment between operand pairs and output positions (Zhou et al., 2024). The same paper reports that removing index hints or using a standard printed format drastically reduces both in-distribution and OOD generalization (Zhou et al., 2024).

Representation engineering is given a formal foundation in a theory of reasoning tasks modeled as DAGs. That work defines a maximal input element distance $R$ and the broader notion of $(n,r)$-consistency. If $R<\infty$, or more generally if a task representation is $(n,r)$-consistent with finite causal-step input space, then full LG is achievable in principle; if not, unbounded sequence growth introduces fundamentally unresolvable extraction problems (Xiao et al., 2024). Within that framework, 1-line addition is described as having $R=\infty$ and as not $L$0-consistent, while 2-line or 3-line representations are said to admit LG. Similarly, naive multiplication formats fail the criterion, but carefully designed multi-line layouts can satisfy it (Xiao et al., 2024). This theoretical account directly supports the empirical observation that task formatting can determine whether a vanilla transformer succeeds or fails.

Auxiliary supervision has also been used to steer models toward LG-compatible internal computations. Task hinting trains a model jointly on the main task and a related auxiliary task. On sorting, training only on lengths up to 20 and adding a successor hint improves test accuracy on length-100 inputs from less than 1% to more than 92% (Awasthi et al., 2023). The same study reports that not all auxiliary tasks help: successor hints are effective, whereas count or fill hints are marginal or negative (Awasthi et al., 2023). It additionally introduces tempered softmax,

$L$1

with $L$2 in the theoretical construction, and shows that this length-dependent parameterization can raise length-100 sorting accuracy from 0% to 45.2% even without hinting (Awasthi et al., 2023).

A separate multitask perspective studies transfer of LG across related tasks. Joint training with a longer, structurally related auxiliary task can enable a short-length main task to generalize robustly up to the auxiliary task’s length, whereas unrelated auxiliary tasks do not confer the same benefit (Cai et al., 10 Jun 2025). This transfer is evaluated in arithmetic, string manipulation, and maze navigation, and is associated with shared use of attention heads between tasks. The paper defines a normalized generalization gap over lengths,

$L$3

to quantify full versus partial transfer (Cai et al., 10 Jun 2025).

In arithmetic transformers, train set priming offers a particularly data-efficient intervention for hard tasks such as multiplication. Adding only 10 to 50 long priming examples from the target distribution to a short-length training set allows models trained on $L$4-digit multiplication to generalize to $L$5 examples; the number of priming examples needed is reported to scale logarithmically with training set size (Jelassi et al., 2023). This is presented as preferable to fine-tuning on long examples, which requires around 1,000 long-sequence examples and causes catastrophic forgetting (Jelassi et al., 2023).

4. Architectural strategies beyond standard transformers

Although much LG work targets standard transformers, several papers propose architectural modifications intended to match the sequential or algorithmic structure of the task more directly. Looped Transformers replace a fixed stack with recurrent application of a shared decoder block, optionally with input injection, and use an adaptive number of steps at inference (Fan et al., 2024). The motivation is that many algorithmic tasks require a number of computational steps that grows with input length; a fixed-depth transformer cannot increase its effective computation depth at test time, whereas a looped model can apply the same learned step repeatedly. On tasks such as Parity, Copy, Addition, Binary Sum, Multiplication, and Unique Set, looped transformers are reported to generalize perfectly or nearly so far beyond training lengths, while standard next-token prediction baselines, pause-token variants, and weight-tied baselines degrade rapidly (Fan et al., 2024).

Hybrid architectures combine recurrence and attention to exploit complementary strengths. In a composite task requiring both modulo-10 state tracking and associative recall, LSTMs generalize well on state tracking but struggle with recall, while transformers excel at recall but adopt shortcuts that fail on longer sequences (Phan et al., 30 Sep 2025). Several hybrid variants are explored, including attention-on-LSTM, LSTM-on-attention, branch-and-add, hybrid blocks, and sandwich architectures. A training schedule called Delayed Attention Training (DAT) initially deactivates attention and MLP modules so that the recurrent pathway learns the LG-critical state-tracking behavior, then later activates attention to pick up recall. On sequences trained up to length 25, DAT enables near-perfect accuracy above 90% on hybrid sequences three times longer than those used during training (Phan et al., 30 Sep 2025). The results are interpreted as evidence that naive hybridization invites shortcut learning unless the division of labor is enforced during optimization.

Research on recurrent models reaches a related conclusion through a different diagnosis. The unexplored states hypothesis posits that recurrent models fail to length generalize because training exposes them only to a limited subset of all attainable hidden states, particularly when initial state is fixed and context length is short (Ruiz et al., 3 Jul 2025). The paper introduces interventions such as Gaussian noise initialization, fitted noise initialization, state passing, and truncated backpropagation through time. State passing and TBTT are reported to enable LG from 2k to 128k with only 500 post-training steps, approximately 0.1% of the pre-training budget (Ruiz et al., 3 Jul 2025). In this setting, the obstacle is not fixed transformer depth but state-distribution mismatch over long rollouts.

Log-depth recurrence offers yet another route. MLP-LDRU, a Log-Depth Recurrent Unit using a learned approximately associative operator under balanced reduction, is evaluated on 21 regular-language tasks and achieves 100% OOD accuracy on 18 tasks and at least 99.9% on the remaining 3 when increasing max training length (Pert et al., 25 May 2026). The design is motivated by the observation that regular-language computation can be expressed as reduction over an associative monoid, making $L$6 parallel depth a natural inductive bias. This work suggests that part of the LG problem is architectural mismatch between the computational symmetry of the target problem and the model’s update rule.

5. Turing-style process supervision and universality claims

A substantial line of work argues that robust LG requires models to imitate the execution trace of an algorithm rather than infer outputs from surface regularities. Turing Programs decompose an algorithmic task into steps mimicking the computation of a Turing Machine, with each step formed by copying text from context with small modifications (Hou et al., 2024). Using this format together with Hard-ALiBi positional encoding, transformers are reported to achieve 98% on addition from 50 to 100 digits, 97% on both $L$7 and $L$8 multiplication from 50 to 100 digits, and 95% on in-context SGD from 50 to 80 examples (Hou et al., 2024). The same work reports strong LG on random Turing Programs and provides a theoretical construction showing that a RASP program can simulate an arbitrary Turing machine under specified $L$9-gram conditions (Hou et al., 2024).

TAIL, or Turing Machine Imitation Learning, extends this perspective to LLMs by synthesizing chain-of-thought traces that imitate Turing Machine execution (Hua et al., 17 Jul 2025). Its three core modules are linear transition, atomic operation, and memory fetcher. Linear transition enforces exhaustive step order, atomic operation decomposes reasoning into smallest task-relevant transitions aligned with

$L=1000$0

and memory fetcher makes operands explicit to mitigate long-range access difficulties (Hua et al., 17 Jul 2025). On a synthetic dataset spanning 18 tasks over 8 classes of algorithms, Qwen2.5-7B fine-tuned only on short inputs is evaluated on sequences up to 10× longer; the paper reports that TAIL significantly improves LG, surpassing Index Hint, Reversed Format, and DeepSeek-R1, and that removing any of the three modules causes a significant collapse in LG (Hua et al., 17 Jul 2025). Attention visualizations are said to reveal read-and-write behaviors consistent with Turing Machine properties when the memory fetcher is present (Hua et al., 17 Jul 2025).

These Turing-inspired approaches are presented as more universal than task-specific tricks such as reversed arithmetic formats or index hints. A plausible implication is that the literature is separating into two broad families: methods that improve extrapolation for specific structural bottlenecks, and methods that attempt to induce a reusable algorithmic execution prior across heterogeneous tasks. The latter claim universality by appealing to Turing-computable reasoning rather than to a particular arithmetic or symbolic benchmark (Hou et al., 2024, Hua et al., 17 Jul 2025).

6. Theoretical frameworks, guarantees, and impossibility results

Formal work on LG has become increasingly precise. One strand studies non-asymptotic LG, defined through a computable upper bound on the minimum input length required for guaranteed generalization as a function of ground-truth complexity (Chen et al., 3 Jun 2025). The Minimum-Complexity Interpolator is shown to achieve optimal length complexity, and non-asymptotic LG is proved equivalent to decidability of the language-equivalence problem for the function class (Chen et al., 3 Jun 2025). This yields a sharp contrast: Deterministic Finite Automata admit explicit bounds, with length complexity $L=1000$1 for $L=1000$2-state DFAs, while Context-Free Grammars do not admit computable upper bounds (Chen et al., 3 Jun 2025). For transformer-related C-RASP classes, the same paper gives bounds of $L=1000$3 for 1-layer C-RASP with precision $L=1000$4, and $L=1000$5 for 2-layer C-RASP with precision $L=1000$6 and $L=1000$7 heads (Chen et al., 3 Jun 2025).

A companion theory develops quantitative bounds for transformers in several regimes. For one-layer finite-precision transformers under worst-case $L=1000$8 error, the required training length is bounded by

$L=1000$9

where $$p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}$$0 is precision, $$p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}$$1 an attention-logit margin, $$p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}$$2 parameter norm, $$p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}$$3 periodicity, $$p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}$$4 locality, and $$p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}$$5 vocabulary size (Izzo et al., 30 Oct 2025). For two-layer infinite-precision transformers, a different bound is given in terms of explicit complexity terms and positional margins (Izzo et al., 30 Oct 2025). The paper’s central intuition is simulation: if long-sequence behavior can be simulated by behavior on sufficiently rich shorter sequences, then agreement on short sequences implies agreement on all lengths (Izzo et al., 30 Oct 2025).

A different theoretical framework analyzes causal transformers with learnable absolute positional encodings under an idealized norm-regularized inference scheme. It introduces the notions of Periodic and Local limit transformers and connects LG to expressibility by periodic/local C-RASP programs (2410.02140). Under this account, LG succeeds when the target function can be implemented with local or periodic positional relations and fails for tasks requiring richer global communication, such as arbitrary copying and addition (2410.02140). The same work presents empirical validation across algorithmic and formal-language tasks, arguing that the theory predicts success and failure patterns for APE and NoPE transformers (2410.02140).

The most restrictive results are impossibility theorems. For CRASP, a class closely linked to transformers, one 2026 paper proves the non-existence of computable length generalization bounds already with two layers, and hence for transformers in general (Yang et al., 13 Feb 2026). The negative result is tied to undecidability via reductions to Diophantine equations. In contrast, the positive fragment of CRASP, shown equivalent to fixed-precision transformers, admits computable but exponential bounds, and those bounds are proved optimal (Yang et al., 13 Feb 2026). Taken together with non-asymptotic LG results, these papers show that the question “how much short-length data is enough?” has radically different answers depending on the expressivity and precision assumptions of the model class (Chen et al., 3 Jun 2025, Yang et al., 13 Feb 2026).

7. Empirical regularities, fragility, and open directions

Across empirical studies, several regularities recur. First, LG is often fragile. In standard transformers solving addition, success depends delicately on positional encoding, data format, random initialization, and data order, with large variance across random seeds even when in-distribution accuracy is perfect (Zhou et al., 2024). In LLMs, naive fine-tuning on parity and boolean variable assignment produces sharp OOD failure independent of model scale, and scratchpad fine-tuning alone does not resolve the problem (Anil et al., 2022). In hybrid architectures, attention modules can destroy the recurrent pathway’s otherwise strong state-tracking LG unless their training is delayed (Phan et al., 30 Sep 2025).

Second, successful LG is frequently linked to reducing shortcut learning. TAIL’s linear transitions, atomic steps, and memory fetcher are explicitly justified as mechanisms to prevent shortcut solutions (Hua et al., 17 Jul 2025). DAT removes the shortcut path during the recurrent-only phase (Phan et al., 30 Sep 2025). Task hinting works when the auxiliary task aligns with the model’s internal solution mechanism, not when it merely adds extra supervision (Awasthi et al., 2023). In-context scratchpad prompting helps LLMs when it elicits a pattern-copying structure that scales with problem length, whereas parameter learning tends to prefer parallel shortcuts over sequential algorithms (Anil et al., 2022).

Third, the field increasingly treats LG as a systems property rather than an isolated benchmark score. In language modeling, RFS improves zero-shot commonsense reasoning under OOD context length without architectural change (Shimizu et al., 15 Feb 2026). In speech enhancement, relative positional structure affects performance on runtime utterances substantially longer than training utterances (Zhang et al., 7 Jun 2025). In recurrent models, minimal post-training interventions can unlock orders-of-magnitude longer context processing (Ruiz et al., 3 Jul 2025). These results suggest that LG is relevant not only for algorithmic toy tasks but also for real deployment scenarios involving long documents, long speech segments, or small-context pretraining budgets.

The literature remains divided on universality. Some papers present task-specific recipes with dramatic improvements on particular domains, such as sorting hints or arithmetic formatting (Awasthi et al., 2023, Zhou et al., 2024). Others seek broader principles: Turing-style execution traces (Hou et al., 2024, Hua et al., 17 Jul 2025), task-association transfer (Cai et al., 10 Jun 2025), local/periodic expressibility (2410.02140), simulation-based bounds (Izzo et al., 30 Oct 2025), or complexity-theoretic decidability criteria (Chen et al., 3 Jun 2025, Yang et al., 13 Feb 2026). This suggests that “length generalization” is best understood not as a single capacity but as a family of extrapolation phenomena whose success depends on whether model, representation, supervision, and inference geometry align with the computational structure required at larger scales.