GrowLength Pretraining

• GrowLength is a pretraining method that gradually increases the training sequence length, reducing compute costs and memory usage.
• The method employs a staged curriculum with intermediate lengths, ensuring smooth transitions and adherence to standard autoregressive training objectives.
• Empirical results show faster convergence, lower loss, and better long-context perplexity compared to fixed-length training baselines across different model sizes.

GrowLength is a pretraining method for LLMs that accelerates training by progressively growing the training sequence length instead of fixing a long context window from the start. In the reported formulation, pretraining begins with short sequences such as 128 tokens and later expands to longer lengths such as 1024 or 4096, while preserving the standard causal language-model objective and standard optimizer practice. The method is presented as a simple curriculum over sequence length: it exploits the lower runtime and memory cost of short-context training in early phases, then uses a final long-context phase so that the model learns to utilize the target window length. Reported experiments on GPT-like RoPE-based models show faster convergence, lower loss at equal wall-clock time, and improved long-context perplexity relative to fixed-length baselines (Jin et al., 2023).

1. Computational setting and motivation

The motivating bottleneck is the quadratic cost of self-attention with respect to sequence length. For a Transformer with hidden size $d$ and sequence length $T$, standard attention costs roughly $\mathcal{O}(T^2 d)$ in both compute and memory because an $T \times T$ attention matrix is formed per head per layer. As $T$ grows from 128 to 4096 or 16384 tokens, this scaling becomes a primary constraint in LLM pretraining (Jin et al., 2023).

The paper studies three directly measurable effects of sequence length: per-step runtime, memory usage, and tokens processed per step or per unit time. On a single A100-80G, with 16,384 tokens per batch, the reported runtime is 0.18 s at length 128, 0.22 s at 1024, and 0.60 s at 16384. Under the same setting, memory usage is 9.49 GB at 128, 13.45 GB at 1024, and 41.63 GB at 16384. When the GPU memory budget is instead pushed to the full 80G, the number of tokens per step is 97,010 at length 128, 69,228 at 1024, and 28,248 at 16384. The stated takeaway is that shorter sequences are much faster per step, use far less memory, and permit many more tokens per batch under a fixed memory budget (Jin et al., 2023).

This computational asymmetry motivates a mismatch claim about conventional pretraining. Typical pretraining uses a fixed context length such as 512, 1024, or 2048 from step 0, even though early training may not yet need long-range context. At the same time, many downstream tasks require long context at inference, and prior context-window extension work indicates that models trained at shorter lengths can later be adapted to longer ones with limited additional training. GrowLength transfers that intuition from fine-tuning into pretraining itself (Jin et al., 2023).

2. Method: progressive growth of training length

GrowLength is a stage-wise curriculum on sequence length. The reported core procedure is: begin pretraining with short sequences, gradually increase the training length to the target long context, and keep the total number of training tokens fixed or comparable to fixed-length baselines. The main schedule is discrete rather than continuous, using a list such as

$\{128, 256, 512, 1024, \dots, L_{\max}\}.$

Each phase uses a fixed sequence length, and training moves to the next phase after a predefined number of steps or tokens (Jin et al., 2023).

The implementation described in the paper is deliberately minimal. One prepares multiple data loaders specialized to different sequence lengths, for example 128_loader, 256_loader, 512_loader, and 1024_loader, and then trains sequentially over this list. The loss remains the standard autoregressive cross-entropy,

$\mathcal{L} = \text{CE}(\text{LLM}(x), y),$

and the training loop otherwise retains standard optimizer updates. The tokenizer, causal masking, and learning-rate schedule are unchanged. The reported implementation adopts RoPE and direct positional extrapolation when lengths increase (Jin et al., 2023).

Two batching regimes are distinguished. In an equal-tokens-per-batch regime, total tokens per batch are fixed, such as 16,384 tokens, so the number of sequences per batch is $16{,}384/T$. This isolates the effect of sequence length on runtime and memory. In a fixed-memory regime, each length uses as many sequences as fit in GPU memory; this yields the largest throughput gains because short lengths admit many more tokens per step. The paper’s central comparison keeps the same total tokens and same model across baselines, and argues that GrowLength finishes faster because many tokens are processed during cheaper short-context phases (Jin et al., 2023).

3. Computational rationale and theoretical framing

The paper gives a simple FLOPs-based account of why progressive length growth reduces cost. If a model is trained for $N$ total tokens entirely at a fixed target length $T_{\max}$, the attention-related cost is described as roughly proportional to

$T$0

Under GrowLength, training is split across lengths $T$1 with token shares $T$2, where $T$3, giving

$T$4

Because the average $T$5 is smaller than $T$6, the paper states that $T$7 at a fixed token budget (Jin et al., 2023).

The performance argument has two parts. First, RoPE-based models are said to have good positional extrapolation properties, so increasing length stage-wise does not induce catastrophic loss spikes when the transition is moderate. Second, shorter-length early training allows more distinct tokens to be seen per unit time; the interpretation given is that early phases emphasize local context and basic language modeling, while later long-context phases teach the model to use long-range patterns. The reported empirical consequence is faster convergence to a given loss level and, in several settings, better final loss than fixed short-length or fixed long-length training under the same time budget (Jin et al., 2023).

The method does not inherently reduce the total number of pretraining tokens. Its main mechanism is redistribution: more tokens are consumed at short lengths early, and fewer at the expensive long length. A separate variant denoted “-2” trains more total tokens, which the paper notes also improves perplexity, but this is not presented as the central point of the method (Jin et al., 2023).

4. Experimental design and model configurations

The default experimental model is a 160M-parameter LLM, and additional experiments use 70M, 160M, and 410M models. The architecture is the same configuration as Pythia, with only the model size varied. The paper follows Pythia’s general pretraining approach and standard tokenizer practice. A key comparison uses 0.36B total tokens (Jin et al., 2023).

Three core training conditions are contrasted. LLM$T$8 uses a fixed context length of 128 tokens. LLM$T$9 uses a fixed context length of 1024 tokens. GrowLength starts at 128 and grows to 1024 with the main schedule 128 $\mathcal{O}(T^2 d)$0 256 $\mathcal{O}(T^2 d)$1 512 $\mathcal{O}(T^2 d)$2 1024. All three consume 0.36B tokens in the central experiment. Additional ablations test reduced schedules such as 128 $\mathcal{O}(T^2 d)$3 512 $\mathcal{O}(T^2 d)$4 1024 and a direct 128 $\mathcal{O}(T^2 d)$5 1024 jump (Jin et al., 2023).

The complexity study is explicitly run on a single NVIDIA A100-80G GPU. For the full pretraining experiments, the paper states that GPU setups are consistent with training models of this scale, though cluster specifications are not detailed exhaustively. Long-context evaluation is performed on a long text dataset used by prior work, with perplexity measured as context length increases (Jin et al., 2023).

5. Reported empirical behavior

At equal total token budget, GrowLength is reported to give both speed and optimization advantages. In the wall-clock comparison among LLM$\mathcal{O}(T^2 d)$6, LLM$\mathcal{O}(T^2 d)$7, and GrowLength on 0.36B tokens, LLM$\mathcal{O}(T^2 d)$8 takes substantially longer than LLM$\mathcal{O}(T^2 d)$9, while GrowLength processes the same total tokens as LLM$T \times T$0 in significantly less time because early phases are cheaper. The reported loss curve shows that GrowLength achieves lower loss than LLM$T \times T$1 within that time and also lower loss than LLM$T \times T$2 because it ultimately trains at longer lengths (Jin et al., 2023).

The same-time comparison strengthens that claim. With training curves aligned by wall-clock time, GrowLength reaches a lower loss than both fixed-length baselines by the end of the time budget. The paper explains this asymmetrically: relative to LLM$T \times T$3, GrowLength has processed more tokens; relative to LLM$T \times T$4, it has been exposed to longer sequences and can leverage context better (Jin et al., 2023).

The scaling study across 70M, 160M, and 410M models reports the same qualitative result for every size pair: GrowLength achieves lower final loss than the no-GrowLength baseline under the same time budget. The scaling-law ordering is preserved, but the paper highlights a notable comparison in which a 70M model with GrowLength approaches or slightly outperforms a 160M model with standard training in final loss. The paper interprets this as evidence that the method is useful under compute constraints (Jin et al., 2023).

Long-context evaluation uses perplexity as the window grows. In that setting, GrowLength-1, which uses the same number of tokens as LLM$T \times T$5 and LLM$T \times T$6, outperforms both fixed-length baselines across all tested context lengths. LLM$T \times T$7 deteriorates heavily at large context because it never trained on long sequences. GrowLength-2, which uses more tokens, is reported as the strongest and most stable variant. The paper’s conclusion is that GrowLength improves long-context behavior even though much of training occurs at short lengths (Jin et al., 2023).

6. Ablations, constraints, and relation to long-context training

The main ablation concerns transition schedules. When intermediate lengths are included, the paper reports that GrowLength is “not sensitive to the ratio of different window size,” and that models trained with or without the 256-token stage can reach almost the same final loss with smooth loss transitions. By contrast, the direct 128 $T \times T$8 1024 schedule produces “dramatic loss rising and a drop in performance” at the transition. The practical recommendation is therefore to avoid large jumps and include at least one intermediate stage such as 512 (Jin et al., 2023).

Several limitations are stated explicitly. The method relies heavily on RoPE and its positional extrapolation behavior; for absolute positional embeddings or other schemes without good extrapolation, transfer across lengths may be less clean. The ratios of training time across lengths are heuristic rather than optimized. Very long contexts beyond those evaluated are not fully characterized, and the paper identifies optimal scheduling, interaction with larger-scale models, and combination with other curricula as open questions (Jin et al., 2023).

The implementation burden is presented as unusually low. GrowLength requires no special engineering effort, modifies only sequence sampling and maximum sequence length over time, and is described as orthogonal to FlashAttention, quantization, pruning, and parallelism. The broader long-context literature later places GrowLength in the category of data strategies for Long Context LLMs, specifically as a curriculum that starts from short contexts and gradually increases the maximum sequence length during training (Liu et al., 20 Mar 2025). Within that wider framing, GrowLength occupies a distinctive position: it is neither a new attention kernel nor a new positional encoding scheme, but a pretraining schedule that uses the compute asymmetry of sequence length as a primary optimization lever.