---
title: Length Generalization in Neural Models
url: https://www.emergentmind.com/topics/length-generalization-lg
type: topic
---

# Length Generalization in Neural Models

Searching arXiv for recent papers on length generalization in transformers and related sequence models.
Length generalization (LG) is the ability of a model to maintain performance on inputs longer than those seen during training or pretraining. Across the recent literature, LG is treated as a specific form of out-of-distribution extrapolation in sequence length, and it has become a central diagnostic for transformers, recurrent models, and hybrid architectures on arithmetic, symbolic reasoning, formal languages, algorithmic tasks, language modeling, and speech enhancement [2602.14050]. The contemporary research record shows that LG is neither a single failure mode nor a single capability: it depends on positional encoding, data representation, auxiliary supervision, recurrence structure, training dynamics, and, in several formal analyses, the computability or structural complexity of the target function [2402.09371][2404.00560][2506.03085][2603.02238].

## 1. Definitions and problem setting

In the cited work, LG is consistently defined as extrapolation from shorter training sequences to longer test sequences. One formulation describes it as the ability of language models to maintain performance on inputs longer than those seen during pretraining [2602.14050]. Another defines it as the ability to extrapolate from shorter training sequences to longer test ones, emphasizing arithmetic addition as a test case [2402.09371]. A more general theoretical treatment frames non-asymptotic LG as learning a hypothesis that generalizes to longer inputs than those in the training set, together with a computable upper bound on the minimum input length needed to guarantee that generalization [2506.03085].

This shared definition masks substantial variation in evaluation regimes. Some studies train on sequence lengths up to a fixed threshold and test on moderately longer inputs, such as training on sequences up to 20 and testing up to 40 on Copy, Reverse, Addition, Sort, Summation, and SCAN [2602.14050]. Others use much larger extrapolation ratios: standard transformers trained on sequences of up to 40 digits can, under the right conditions, exceed 98% exact match accuracy on 100-digit addition, corresponding to 2.5× extrapolation [2402.09371]. In recurrent-model analyses, LG is evaluated at far larger scales, such as generalization from 2k training contexts to 128k validation contexts after post-training interventions [2507.02782].

The literature also distinguishes between empirical LG and formal LG. Empirical LG concerns observed accuracy curves as a function of test length. Formal LG concerns conditions under which a target function is identifiable from bounded-length data, or whether a computable training-length bound exists at all [2506.03085][2603.02238]. This distinction is central because several papers show that strong in-distribution performance, low training loss, or large model scale do not reliably predict OOD length performance [2402.09371][2207.04901].

## 2. Positional structure as a primary determinant

A dominant theme in the literature is that positional encoding (PE) strongly shapes LG. Early arithmetic results show that relative position embeddings enable length generalization for addition: encoder-only transformers with relative position embeddings, trained on 5-digit numbers, achieve 100% at 6 digits, 99.9% at 10 digits, 97.2% at 15 digits, and 21.3% at 20 digits, while absolute position embeddings collapse to near-zero OOD accuracy [2306.15400]. In speech enhancement, relative positional encoding methods are reported to exhibit better LG than absolute positional encodings, and absolute methods may degrade below no-position baselines on long utterances [2506.06697].

A major 2026 contribution proposes Random Float Sampling (RFS), a PE indexing strategy in which training positions are obtained by sampling continuous values from $[0,1)$, sorting them to preserve order, and then scaling by a factor $L$ with default $L=1000$ [2602.14050]. At inference, evenly spaced indices are used:
$$
p_i = \frac{2i - 1}{2 \max(n_{\text{tr}}, n_{\text{in}})}
$$
before scaling. The stated rationale is that continuous random indices densely expose the model to diverse positional values during training and thereby avoid out-of-distribution positional indices at longer test lengths [2602.14050]. Because the method changes only index generation, it is plug-and-play with absolute sinusoidal encoding, RoPE, and ALiBi. Empirically, on the Copy task with sinusoidal PE, RFS maintains approximately 80% accuracy on twice-longer inputs, whereas NoPE drops to approximately 20% accuracy on 1.5× longer inputs; on seven commonsense benchmarks, average OOD accuracy reaches 35.78%, close to baseline ID accuracy of 35.84% and above baseline OOD accuracy of 34.47% [2602.14050].

Other PE-centric proposals pursue related goals through different mechanisms. The FIRE positional encoding, especially when combined with randomization, is reported as crucial for strong addition extrapolation in standard transformers [2402.09371]. In speech enhancement, LearnLin introduces one trainable scalar per attention head to scale the real relative position between time frames, with bias
$$
P^h_{i,j} = \beta^h \cdot |i-j|,
$$
and is described as comparable or superior to other state-of-the-art PE strategies for LG [2506.06697]. In theoretical work on PE limitations, Linear Representation Complexity (LRC) and Sequential Representation Complexity (SRC) are introduced to argue that PEs do not expand computational capabilities but instead structure learned computations across positions; the paper further proposes Scale Hint and a Learning-Based Position Embedding framework for cases where suitable positional relations must be learned or scaled flexibly [2510.04130].

A consistent empirical pattern is that PE choice alone is not uniformly sufficient. In addition, FIRE with randomization helps, but performance remains fragile across seeds and data order [2402.09371]. In speech enhancement, RPE is necessary but not identical across methods [2506.06697]. In PE theory, LG is conjectured to be possible if and only if SRC remains invariant across scales, which suggests that PE effectiveness is constrained by the underlying operator complexity of the task rather than by positional features alone [2510.04130].

## 3. Representation, hints, and training data design

A second major axis of LG research concerns how the task is represented to the model. In arithmetic addition, presenting digits in reversed format aligns the computation order with carry propagation and is described as critical for generalization; adding index hints further improves alignment between operand pairs and output positions [2402.09371]. The same paper reports that removing index hints or using a standard printed format drastically reduces both in-distribution and OOD generalization [2402.09371].

Representation engineering is given a formal foundation in a theory of reasoning tasks modeled as DAGs. That work defines a maximal input element distance $R$ and the broader notion of $(n,r)$-consistency. If $R<\infty$, or more generally if a task representation is $(n,r)$-consistent with finite causal-step input space, then full LG is achievable in principle; if not, unbounded sequence growth introduces fundamentally unresolvable extraction problems [2404.00560]. Within that framework, 1-line addition is described as having $R=\infty$ and as not $(n,r)$-consistent, while 2-line or 3-line representations are said to admit LG. Similarly, naive multiplication formats fail the criterion, but carefully designed multi-line layouts can satisfy it [2404.00560]. This theoretical account directly supports the empirical observation that task formatting can determine whether a vanilla transformer succeeds or fails.

Auxiliary supervision has also been used to steer models toward LG-compatible internal computations. Task hinting trains a model jointly on the main task and a related auxiliary task. On sorting, training only on lengths up to 20 and adding a successor hint improves test accuracy on length-100 inputs from less than 1% to more than 92% [2310.00726]. The same study reports that not all auxiliary tasks help: successor hints are effective, whereas count or fill hints are marginal or negative [2310.00726]. It additionally introduces tempered softmax,
$$
\operatorname{softmax}_\tau(x)_i = \frac{e^{\tau x_i}}{\sum_j e^{\tau x_j}},
$$
with $\tau = \beta \ln n$ in the theoretical construction, and shows that this length-dependent parameterization can raise length-100 sorting accuracy from 0% to 45.2% even without hinting [2310.00726].

A separate multitask perspective studies transfer of LG across related tasks. Joint training with a longer, structurally related auxiliary task can enable a short-length main task to generalize robustly up to the auxiliary task’s length, whereas unrelated auxiliary tasks do not confer the same benefit [2506.09251]. This transfer is evaluated in arithmetic, string manipulation, and maze navigation, and is associated with shared use of attention heads between tasks. The paper defines a normalized generalization gap over lengths,
$$
\text{GenGap} = \frac{1}{L} \sum_{\ell \in \mathcal{L}} \frac{|A_\text{main}(\ell) - A_\text{aux}(\ell)|}{\max(A_\text{aux}(\ell), \epsilon)},
$$
to quantify full versus partial transfer [2506.09251].

In arithmetic transformers, train set priming offers a particularly data-efficient intervention for hard tasks such as multiplication. Adding only 10 to 50 long priming examples from the target distribution to a short-length training set allows models trained on $5\times3$-digit multiplication to generalize to $35\times3$ examples; the number of priming examples needed is reported to scale logarithmically with training set size [2306.15400]. This is presented as preferable to fine-tuning on long examples, which requires around 1,000 long-sequence examples and causes catastrophic forgetting [2306.15400].

## 4. Architectural strategies beyond standard transformers

Although much LG work targets standard transformers, several papers propose architectural modifications intended to match the sequential or algorithmic structure of the task more directly. Looped Transformers replace a fixed stack with recurrent application of a shared decoder block, optionally with input injection, and use an adaptive number of steps at inference [2409.15647]. The motivation is that many algorithmic tasks require a number of computational steps that grows with input length; a fixed-depth transformer cannot increase its effective computation depth at test time, whereas a looped model can apply the same learned step repeatedly. On tasks such as Parity, Copy, Addition, Binary Sum, Multiplication, and Unique Set, looped transformers are reported to generalize perfectly or nearly so far beyond training lengths, while standard next-token prediction baselines, pause-token variants, and weight-tied baselines degrade rapidly [2409.15647].

Hybrid architectures combine recurrence and attention to exploit complementary strengths. In a composite task requiring both modulo-10 state tracking and associative recall, LSTMs generalize well on state tracking but struggle with recall, while transformers excel at recall but adopt shortcuts that fail on longer sequences [2510.00258]. Several hybrid variants are explored, including attention-on-LSTM, LSTM-on-attention, branch-and-add, hybrid blocks, and sandwich architectures. A training schedule called Delayed Attention Training (DAT) initially deactivates attention and MLP modules so that the recurrent pathway learns the LG-critical state-tracking behavior, then later activates attention to pick up recall. On sequences trained up to length 25, DAT enables near-perfect accuracy above 90% on hybrid sequences three times longer than those used during training [2510.00258]. The results are interpreted as evidence that naive hybridization invites shortcut learning unless the division of labor is enforced during optimization.

Research on recurrent models reaches a related conclusion through a different diagnosis. The unexplored states hypothesis posits that recurrent models fail to length generalize because training exposes them only to a limited subset of all attainable hidden states, particularly when initial state is fixed and context length is short [2507.02782]. The paper introduces interventions such as Gaussian noise initialization, fitted noise initialization, state passing, and truncated backpropagation through time. State passing and TBTT are reported to enable LG from 2k to 128k with only 500 post-training steps, approximately 0.1% of the pre-training budget [2507.02782]. In this setting, the obstacle is not fixed transformer depth but state-distribution mismatch over long rollouts.

Log-depth recurrence offers yet another route. MLP-LDRU, a Log-Depth Recurrent Unit using a learned approximately associative operator under balanced reduction, is evaluated on 21 regular-language tasks and achieves 100% OOD accuracy on 18 tasks and at least 99.9% on the remaining 3 when increasing max training length [2605.26035]. The design is motivated by the observation that regular-language computation can be expressed as reduction over an associative monoid, making $O(\log n)$ parallel depth a natural inductive bias. This work suggests that part of the LG problem is architectural mismatch between the computational symmetry of the target problem and the model’s update rule.

## 5. Turing-style process supervision and universality claims

A substantial line of work argues that robust LG requires models to imitate the execution trace of an algorithm rather than infer outputs from surface regularities. Turing Programs decompose an algorithmic task into steps mimicking the computation of a Turing Machine, with each step formed by copying text from context with small modifications [2407.03310]. Using this format together with Hard-ALiBi positional encoding, transformers are reported to achieve 98% on addition from 50 to 100 digits, 97% on both $n\times1$ and $n\times3$ multiplication from 50 to 100 digits, and 95% on in-context SGD from 50 to 80 examples [2407.03310]. The same work reports strong LG on random Turing Programs and provides a theoretical construction showing that a RASP program can simulate an arbitrary Turing machine under specified $n$-gram conditions [2407.03310].

TAIL, or Turing Machine Imitation Learning, extends this perspective to large language models by synthesizing chain-of-thought traces that imitate Turing Machine execution [2507.13332]. Its three core modules are linear transition, atomic operation, and memory fetcher. Linear transition enforces exhaustive step order, atomic operation decomposes reasoning into smallest task-relevant transitions aligned with
$$
\delta(q_s, a) = (q_{s+1}, b, D),
$$
and memory fetcher makes operands explicit to mitigate long-range access difficulties [2507.13332]. On a synthetic dataset spanning 18 tasks over 8 classes of algorithms, Qwen2.5-7B fine-tuned only on short inputs is evaluated on sequences up to 10× longer; the paper reports that TAIL significantly improves LG, surpassing Index Hint, Reversed Format, and DeepSeek-R1, and that removing any of the three modules causes a significant collapse in LG [2507.13332]. Attention visualizations are said to reveal read-and-write behaviors consistent with Turing Machine properties when the memory fetcher is present [2507.13332].

These Turing-inspired approaches are presented as more universal than task-specific tricks such as reversed arithmetic formats or index hints. A plausible implication is that the literature is separating into two broad families: methods that improve extrapolation for specific structural bottlenecks, and methods that attempt to induce a reusable algorithmic execution prior across heterogeneous tasks. The latter claim universality by appealing to Turing-computable reasoning rather than to a particular arithmetic or symbolic benchmark [2407.03310][2507.13332].

## 6. Theoretical frameworks, guarantees, and impossibility results

Formal work on LG has become increasingly precise. One strand studies non-asymptotic LG, defined through a computable upper bound on the minimum input length required for guaranteed generalization as a function of ground-truth complexity [2506.03085]. The Minimum-Complexity Interpolator is shown to achieve optimal length complexity, and non-asymptotic LG is proved equivalent to decidability of the language-equivalence problem for the function class [2506.03085]. This yields a sharp contrast: Deterministic Finite Automata admit explicit bounds, with length complexity $2n-2$ for $n$-state DFAs, while Context-Free Grammars do not admit computable upper bounds [2506.03085]. For transformer-related C-RASP classes, the same paper gives bounds of $O(T^2)$ for 1-layer C-RASP with precision $T$, and $O(T^{O(K)})$ for 2-layer C-RASP with precision $T$ and $K$ heads [2506.03085].

A companion theory develops quantitative bounds for transformers in several regimes. For one-layer finite-precision transformers under worst-case $\ell_\infty$ error, the required training length is bounded by
$$
N = O\left(\max\left\{ 2^{p/\gamma}, ~~\frac{L^2 \Delta^7 |\Sigma|^6 \tau^2}{\varepsilon^2} \right\}\right),
$$
where $p$ is precision, $\gamma$ an attention-logit margin, $L$ parameter norm, $\Delta$ periodicity, $\tau$ locality, and $|\Sigma|$ vocabulary size [2510.27015]. For two-layer infinite-precision transformers, a different bound is given in terms of explicit complexity terms and positional margins [2510.27015]. The paper’s central intuition is simulation: if long-sequence behavior can be simulated by behavior on sufficiently rich shorter sequences, then agreement on short sequences implies agreement on all lengths [2510.27015].

A different theoretical framework analyzes causal transformers with learnable absolute positional encodings under an idealized norm-regularized inference scheme. It introduces the notions of Periodic and Local limit transformers and connects LG to expressibility by periodic/local C-RASP programs [2410.02140]. Under this account, LG succeeds when the target function can be implemented with local or periodic positional relations and fails for tasks requiring richer global communication, such as arbitrary copying and addition [2410.02140]. The same work presents empirical validation across algorithmic and formal-language tasks, arguing that the theory predicts success and failure patterns for APE and NoPE transformers [2410.02140].

The most restrictive results are impossibility theorems. For CRASP, a class closely linked to transformers, one 2026 paper proves the non-existence of computable length generalization bounds already with two layers, and hence for transformers in general [2603.02238]. The negative result is tied to undecidability via reductions to Diophantine equations. In contrast, the positive fragment of CRASP, shown equivalent to fixed-precision transformers, admits computable but exponential bounds, and those bounds are proved optimal [2603.02238]. Taken together with non-asymptotic LG results, these papers show that the question “how much short-length data is enough?” has radically different answers depending on the expressivity and precision assumptions of the model class [2506.03085][2603.02238].

## 7. Empirical regularities, fragility, and open directions

Across empirical studies, several regularities recur. First, LG is often fragile. In standard transformers solving addition, success depends delicately on positional encoding, data format, random initialization, and data order, with large variance across random seeds even when in-distribution accuracy is perfect [2402.09371]. In large language models, naive fine-tuning on parity and boolean variable assignment produces sharp OOD failure independent of model scale, and scratchpad fine-tuning alone does not resolve the problem [2207.04901]. In hybrid architectures, attention modules can destroy the recurrent pathway’s otherwise strong state-tracking LG unless their training is delayed [2510.00258].

Second, successful LG is frequently linked to reducing shortcut learning. TAIL’s linear transitions, atomic steps, and memory fetcher are explicitly justified as mechanisms to prevent shortcut solutions [2507.13332]. DAT removes the shortcut path during the recurrent-only phase [2510.00258]. Task hinting works when the auxiliary task aligns with the model’s internal solution mechanism, not when it merely adds extra supervision [2310.00726]. In-context scratchpad prompting helps large language models when it elicits a pattern-copying structure that scales with problem length, whereas parameter learning tends to prefer parallel shortcuts over sequential algorithms [2207.04901].

Third, the field increasingly treats LG as a systems property rather than an isolated benchmark score. In language modeling, RFS improves zero-shot commonsense reasoning under OOD context length without architectural change [2602.14050]. In speech enhancement, relative positional structure affects performance on runtime utterances substantially longer than training utterances [2506.06697]. In recurrent models, minimal post-training interventions can unlock orders-of-magnitude longer context processing [2507.02782]. These results suggest that LG is relevant not only for algorithmic toy tasks but also for real deployment scenarios involving long documents, long speech segments, or small-context pretraining budgets.

The literature remains divided on universality. Some papers present task-specific recipes with dramatic improvements on particular domains, such as sorting hints or arithmetic formatting [2310.00726][2402.09371]. Others seek broader principles: Turing-style execution traces [2407.03310][2507.13332], task-association transfer [2506.09251], local/periodic expressibility [2410.02140], simulation-based bounds [2510.27015], or complexity-theoretic decidability criteria [2506.03085][2603.02238]. This suggests that “length generalization” is best understood not as a single capacity but as a family of extrapolation phenomena whose success depends on whether model, representation, supervision, and inference geometry align with the computational structure required at larger scales.

Source: https://www.emergentmind.com/topics/length-generalization-lg