BabyLM Corpus: A Developmental Language Dataset

• BabyLM Corpus is a multi-source, 100M-word dataset that replicates children's linguistic exposure using curated data from sources like CHILDES and BNC.
• It preserves natural speech characteristics by minimal preprocessing, retaining disfluencies and discourse boundaries to enhance realistic language modeling.
• Empirical findings show that high data quality and syntactic filtering significantly boost performance in sample-efficient and cognitively plausible language models.

The BabyLM corpus is a developmentally motivated, multi-source text dataset of approximately 100 million words, curated to serve as a realistic proxy for the type and quantity of linguistic input available to children by early adolescence. The corpus was introduced and refined through a series of community-wide shared tasks—most notably the BabyLM Challenge—that benchmark sample-efficient pretraining and cognitive plausibility of neural LLMs under strict data constraints. The collection is designed to facilitate research into both practical low-resource NLP and theoretical models of human language acquisition (Warstadt et al., 2023, Warstadt et al., 10 Apr 2025, Hu et al., 6 Dec 2024, Steuer et al., 2023, Güven et al., 11 Nov 2025).

1. Corpus Composition and Developmental Criteria

The BabyLM corpus is a composite of multiple developmentally plausible domains:

Subcorpus	Domain	Token Count (M)
CHILDES	Child-directed speech	Up to 29
British National Corpus (BNC)	Spoken/dialogue	~8–9
OpenSubtitles	Movie subtitles	20–31
Children’s Fiction (Gutenberg, CBT, etc.)	Written fiction (child-appropriate)	3–31
Simple/Standard Wikipedia	Encyclopedia (simplified/written)	14–17
QED Subtitles / Switchboard	Educational/video, phone dialogue	1–10

Strict variants aggregate to 98–110 million words, with spoken data comprising >50% of tokens in the top-level “strict” track (Warstadt et al., 2023, Warstadt et al., 10 Apr 2025, Hu et al., 6 Dec 2024, Güven et al., 11 Nov 2025, Steuer et al., 2023). The selection process enforces developmental plausibility by ensuring that dataset composition and quantity match estimates of the linguistic input received by children (≈24–84 million words by age 12, capped at 100 million). Age-inappropriate, highly technical, or artificially generated text (e.g., web scrapes) is expressly excluded. Proportional subsampling is used for the "Strict-Small" (10M) track—a stratified subset maintaining domain shares (Warstadt et al., 10 Apr 2025).

Corpus curation involves minimal preprocessing to preserve disfluencies, sentence fragments, discourse boundaries, and spoken properties characteristic of child language input. Duplicate removal applies only in known noisy sources (e.g., subtitles, Wikipedia). Annotation metadata, such as speaker labels, are removed for most tasks except where their presence is relevant (Güven et al., 11 Nov 2025).

2. Structural Properties and Statistical Measures

Corpus structure retains the statistical and syntactic diversity present in naturalistic language exposure. Average sequence lengths vary by domain (spoken: 7–12 tokens, Wikipedia: 15–20), and overall type–token ratio is approximately

$$\mathrm{TTR} = \frac{V}{N} \approx 5 \times 10^{-4}$$

for vocabularies of 50k and 100M tokens (Warstadt et al., 10 Apr 2025, Steuer et al., 2023). The word frequency spectrum exhibits an approximate Zipfian distribution $f(r) \propto r^{-s}$, with $s \approx 1.0–1.1$ (Warstadt et al., 10 Apr 2025).

Syntactic complexity is a central concern. Explicit syntactic categorization of corpus sentences—especially in CHILDES—has been systematized via Tregex patterns over constituency parses, yielding 13 fine-grained categories grouped into macro-classes (Simple, Interrogative, Complex) (Güven et al., 11 Nov 2025). The proportion of "syntactically categorizable" sentences in CHILDES exceeds 70%. However, contrary to developmental linguistics expectations, no significant monotonic increase in syntactic complexity by child age group is observable in CHILDES; interrogative constructions peak at 36–48 months but complex structure growth is non-monotonic.

3. Tracks, Splits, and Preprocessing Regimes

The challenge splits the BabyLM corpus into tracks that control available data budget and domain modalities:

• Strict-Small: 10M words from all sources, stratified subsample.
• Strict: Full 100M words, with fixed, balanced distribution.
• Loose: 100M words as cap, modality-agnostic (may incorporate text, audio, vision, code).

All tracks are split into training, development, and test sets (approximately 10:1:1 within each subcorpus). Subset creation prioritizes preservation of discourse features—no lowercasing, no punctuation normalization, newlines preserved, and minimal intervention on original structure (Warstadt et al., 2023, Warstadt et al., 10 Apr 2025).

Tokenization is model-dependent; most participants use subword-based algorithms (BPE, WordPiece, SentencePiece) with vocabulary sizes of 30–50k, built directly from the BabyLM corpus (Warstadt et al., 10 Apr 2025, Steuer et al., 2023, Hu et al., 6 Dec 2024).

4. Model Training Setups and Empirical Findings

BabyLM is explicitly agnostic to architecture and objective, enabling evaluation of diverse modeling paradigms:

• Autoregressive (CLM):

$$\mathcal{L}_{AR} = -\sum_{t=1}^{T} \log p(x_t \mid x_{<t})$$

• Masked language modeling (MLM):

$$\mathcal{L}_{MLM} = -\sum_{i\in M} \log p(x_i \mid x_{\setminus i})$$

• Hybrid Objectives: Notably, mixing schedules (e.g., 1:7 CLM:MLM) shown to outperform pure objectives (Hu et al., 6 Dec 2024).
• Architectures: OPT-style decoder-only Transformers, LTG-BERT, hybrid causal-masked models, RNNs, and others (Warstadt et al., 10 Apr 2025, Hu et al., 6 Dec 2024, Steuer et al., 2023).

Empirical observations include:

• Corpus Curation Effect: Systems using syntactically filtered or custom-crafted corpora consistently outperform those based on noisy or heterogeneous data. Data quality was the strongest positive predictor of downstream score ($p<0.05$) (Hu et al., 6 Dec 2024, Güven et al., 11 Nov 2025).
• Model Capacity: On BLiMP, GLUE, and MSGS, increased model size correlates positively with challenge task performance (Spearman ρ up to 0.88 on BLiMP), but negatively with psycholinguistic fit to human reading time ($\rho\approx -0.51$); moderate models (e.g., 2 layers × 192 hidden units) yield best processing-effort alignment (Steuer et al., 2023).
• Syntactic Filtering: Filtering for syntactically welldefined sentences (71% coverage in CHILDES) enables 40% reduction in data (77M tokens) with matched or higher task performance compared to the full 131M-token corpus (Güven et al., 11 Nov 2025).
• Curriculum Learning: Developmentally motivated curriculum (e.g., Simple → Interrogative → Complex) confers only modest, task-specific gains over random order; primary benefit derives from data quality/subset selection, not controlled exposure schedule (Güven et al., 11 Nov 2025, Warstadt et al., 10 Apr 2025).

5. Evaluation Methodology and Benchmarking

All submissions to the BabyLM Challenge pipeline are evaluated on a unified battery:

Benchmark	Evaluation Type	Metric(s)
BLiMP	Zero-shot grammaticality	Accuracy on minimal pairs
BLiMP Supplement	Syntactic contrasts	Accuracy
(Super)GLUE	Fine-tuned NLU	Accuracy, Cross-entropy, $F_1$
MSGS/COMPS	Generalization, semantics	Accuracy
EWoK	World knowledge probing	Macro-average, accuracy
WUG_ADJ/PAST	Generalization to nonce	Accuracy, human-alignment
READING	Psycholinguistic alignment	Correlation with human RT/eye-tracking
Vision (DevBench, VQA, Winoground)	Multimodal	Accuracy

Pseudolikelihood ($\mathrm{PLL}$), cross-entropy loss, and perplexity ($\mathrm{PPL}$) are used as density estimation metrics (Warstadt et al., 2023, Warstadt et al., 10 Apr 2025). Leaderboards are maintained on platforms such as Dynabench.

In challenge rounds, top-performing entries (e.g., GPT-BERT, LTG-BERT) achieved 86.1% BLiMP accuracy (vs. 69.2% baseline), 81.5% on (Super)GLUE, and 58.4% on EWoK, approaching human and trillion-word pretraining baselines (Hu et al., 6 Dec 2024). No vision–language system yet matched text-only performance under the same data constraint.

6. Syntactic Annotation, Curriculum Learning, and Data Efficiency

Detailed syntactic analysis revealed:

• 13 Tregex-defined constructions, grouped as Simple, Interrogative, Complex (Güven et al., 11 Nov 2025).
• Coverage rates for syntactically categorizable material in CHILDES by age are consistently high (70–92%), with minimal variation.
• Inclusion of only syntactically categorized sentences delivers the largest gains in data efficiency and quality for pretraining, exceeding gains from curriculum ordering per se.
• Curriculum strategies (Simple → Interrogative → Complex, gradual probabilistic S→C, Learn–Focus–Review) yield limited additional performance; data filtration is more effective (Güven et al., 11 Nov 2025).
• Sub-corpus distributions offer diagnostic value for aligning model training with expected strengths/limitations on syntax-sensitive benchmarks.

7. Impact, Recommendations, and Future Directions

The BabyLM corpus and associated shared tasks have established a new paradigm for cognitively plausible, sample-efficient language modeling research (Warstadt et al., 2023, Warstadt et al., 10 Apr 2025, Hu et al., 6 Dec 2024). Key recommendations include:

• Corpus Curation: Strong preference for developmentally realistic, syntactically filtered data to maximize the relevance and efficiency of pretraining.
• Objective Mixing: Hybrid modeling objectives yield notable gains.
• Compute: Within strict data caps, compute budget (FLOPs) remains a significant determinant of performance ($\beta_{\log\mathrm{FLOPs}} \approx 2.7,\,p<0.01$) (Hu et al., 6 Dec 2024).
• Modality: Pure text models outperform multimodal systems under data constraints; image–text alignment under strict caps remains an open research problem.
• Developmental Fidelity: Increasing focus on spoken interactions, child speech, prosody, language diversity, and spontaneous longitudinal data is recommended.

The BabyLM corpus and challenge datasets, together with open-source categorization tools and standardized evaluation, provide a benchmark infrastructure for future work on low-resource, cognitively motivated language modeling (Güven et al., 11 Nov 2025, Hu et al., 6 Dec 2024, Warstadt et al., 10 Apr 2025).