Optimal Sequence Length for BabyLM

• The paper demonstrates that optimal sequence length in BabyLM is task- and architecture-dependent, balancing training speed with memory requirements.
• It systematically evaluates varying lengths on a 100M-word corpus, comparing transformer and state-space models to determine performance trade-offs.
• The study guides practical choices by recommending shorter sequences for local syntactic tasks and longer contexts for capturing global dependencies.

The optimal sequence length for BabyLM refers to the choice of input context window size that maximizes pretraining and downstream evaluation efficiency for LLMs trained under the small-data BabyLM regime (typically ≤100M words) given a fixed compute or token budget. Empirical studies reveal that the "best" sequence length is not universal: it varies by model architecture, evaluation task, and practical constraints such as available memory. Extensive recent work has quantified how sequence length affects model performance, training speed, and memory requirements, providing precise guidelines for both fixed and curriculum-based approaches.

1. Experimental Regimes and Architectural Comparisons

Key findings derive from systematic ablations using the 100M-word BabyLM corpus with compute budgets fixed to ≈10B token-updates. Studies directly compare distinct model families—canonical transformer architectures (e.g., OPT-125M) and state-space models (e.g., Mamba-125M)—across powers-of-two sequence lengths from 64 to 8192 tokens (Salhan et al., 22 Oct 2025). Official BabyLM evaluation and challenge baselines primarily defaulted to 128-token contexts, inherited by both leaderboard and baseline models (Warstadt et al., 10 Apr 2025).

The principal architectural distinction rests in scaling behavior: OPT (transformers) incurs an O(L²) computational and memory cost for a sequence of length L (due to self-attention), while Mamba achieves linear-time recurrence without such penalties, potentially offering advantages at larger L.

2. Systematic Effects of Sequence Length

Empirical benchmarks reveal that increasing L does not uniformly improve model performance. Instead, sequence-length effects are strictly task- and architecture-dependent. The principal outcomes are summarized below.

Task/Eval Suite	OPT L*	OPT L_best	Mamba L*	Mamba L_best	Typical Gain Pattern
BLiMP	256	64	512	2048	Peaks at low L, declines at high L
BLiMP Supplement	256	64	64	64	Small L optimal, stable
Entity Tracking	4096	8192	1024	128	OPT improves with L, Mamba less so
Wug (Morph. Analogy)	4096	4096	128	128	OPT benefits from long L
EWoK (Novel Words)	4096	2048	1024	512	Both architectures favor longer L
Reading (Psychometric)	8192	8192	512	64	OPT: monotonically improves, Mamba peaks at tiny L

L*: minimum L achieving ≥98% of peak task accuracy, minimizing wall-clock. L_best: maximum observed accuracy, unconstrained by efficiency (Salhan et al., 22 Oct 2025).

Shorter context (L in 32–256 range) yields higher updates-per-token, faster convergence, and improved grammatical sensitivity (BLiMP, (Super)GLUE), especially in limited data scenarios (Warstadt et al., 10 Apr 2025, Edman et al., 2023). For tasks requiring global context, such as coreference or morphological analogical reasoning, longer contexts (L ≥ 1024) yield substantially better performance, especially in transformers.

3. Computational Efficiency and Trade-Offs

Memory and computational cost scale approximately as $O(L^2)$ for transformers:

• Memory: $M(L) = \kappa L^2$
• Steps per token: $U(L) \propto 1/L$ (Warstadt et al., 10 Apr 2025)

Shorter L allows more optimizer steps for a fixed corpus ("gradient diversity"), accelerating convergence and stabilizing learning. However, this comes at the cost of truncated dependency range. For BabyLM-scale models, memory constraints often dictate the practical upper bound on L.

Empirical data (OPT-125M baseline, Strict-Small): | Context Length | BLIMP Acc. | GLUE Macro | Aggregate | Epochs to 90% Conv. | Throughput | Peak Memory | |----------------|------------|------------|-----------|---------------------|------------|-------------| | 32 | 0.77 | 0.73 | 0.62 | 5 | 4× | 0.35× | | 128 | 0.75 | 0.70 | 0.60 | 10 | 1× | 1× | | 256 | – | – | – | 8 | 0.5× | 4× | | 512 | – | – | – | 6 | 0.25× | 16× | (Warstadt et al., 10 Apr 2025)

Doubling L approximately halves training throughput and quadruples peak memory usage. Break-even for most aggregate metrics occurs at L ≈ 64–128.

4. Task-Specific, Default, and Curriculum Recommendations

Authoritative recommendations integrate per-task optima, architecture, and compute constraints:

• Syntax (BLiMP, BLiMP Supplement):
  • OPT: L = 256–512; Mamba: L = 64–256.
• Morphological Productivity (Wug):
  • OPT: L = 4096; Mamba: L = 128 (absolute accuracy lower than OPT).
• Entity Tracking/EWoK:
  • OPT: L = 2048–4096; Mamba: L = 512–1024.
• Reading/Psychometrics:
  • OPT: L = 8192; Mamba: L = 64–512.

If a single universal L must be chosen:

• OPT: L=2048 (≈60% of max cost, robust across tasks)
• Mamba: L=512 (≈35% of max cost, near-optimal for most) (Salhan et al., 22 Oct 2025)

Variable-length curricula (VSL, e.g., "Grow-P₂" schedule) starting from L=64 and growing to L=1024 in 8 cycles further accelerate learning and enable efficient coverage of both local and global patterns at BabyLM scale, with negligible overhead and no loss in regular-task accuracy (Pouransari et al., 2024).

5. Curriculum and Two-Phase Training Regimes

Purely fixed-length training is often suboptimal. Two-phase schedules—beginning with extensive training at short context (e.g., L=32 tokens) followed by fine-tuning at longer context (e.g., L=128 or 256)—consistently improve aggregate scores, grammatical acceptability, and bias metrics (MSGS) (Edman et al., 2023). For example, BLiMP accuracy rises from 75% (128 tokens) and 78.6% (32 tokens) to 80% for a 32→128 two-phase schedule, and (Super)GLUE gains 4–5 points over the best single-context run.

Variable Sequence Length (VSL) curricula (dataset decomposition) generalize this insight: by decomposing the dataset into power-of-two length "buckets," and scheduling bucket sampling in cycles from short-to-long, overall training efficiency is improved and wall-clock time reduced, especially for BabyLM-sized models (Pouransari et al., 2024).

6. Implications and Outstanding Issues

The consensus is that "longer is not always better" for BabyLM. Instead, selecting sequence length according to model architecture and evaluation task yields significant time and memory savings (up to 65% pretraining time reduction) with negligible performance drop (Salhan et al., 22 Oct 2025). Crucially, scaling up L above 128–256 tokens yields sharply diminishing returns for local syntactic phenomena and general NLU, with considerable memory cost. Extending L beyond 1024 is justifiable only for benchmark tasks requiring true long-range dependencies or scientific evaluation of global coherence.

A plausible implication is that short-to-long curricula will likely remain best-practice for BabyLM-style regimes, but for contemporary large-scale settings with ample compute, a mixture of variable-length contexts or long fixed L (2048–8192) may be preferable, especially for tasks involving coreference, discourse, or morphological analogies.

7. Summary Table: Sequence Length Recommendations in BabyLM

Setting	Dataset/Model Size	Recommended L	Core Justification
BabyLM (Standard)	100M words, 125M	128 (default)	Compute/memory efficient, >0.98 best aggregate (GLUE/BLiMP)
Grammaticality	10M–100M, 100M	32–64	Peaks in BLiMP, fast convergence
NLU/GLUE	100M, 125M	64–128	Moderate coverage, stable throughput
Analogical/Global	100M, 125M	1024–4096	Needed for Wug/EWoK/Entity Tracking (OPT)
Curriculum (VSL)	10⁸–10⁹, any	64→1024	"Grow-P₂" schedule, optimal task coverage, efficient

(Edman et al., 2023, Warstadt et al., 10 Apr 2025, Salhan et al., 22 Oct 2025, Pouransari et al., 2024)

These findings indicate that BabyLM sequence length remains an optimization problem, conditional on the interplay of model class, evaluation task, and system constraints. Fixed 128 or two-phase 32→128 schedules are optimal for grammatical generalization and general NLU, while longer fixed L or curriculum-based VSL settings are required for robust generalization to document-level and analogical tasks.