- The paper introduces an 88B-token corpus restricted to U.S. kindergarten–Grade 5 material and a 5B-parameter model, creating a controlled sandbox for studying knowledge acquisition.
- The precision-first filtering pipeline reduced out-of-scope retention to 0% on a CCSS-aligned benchmark while retaining about 35% of in-scope passages, with independent checks finding only 0.05% genuinely out-of-scope passages in WeeBit data.
- The model’s performance declined sharply beyond its exposure boundary, and scaling, SFT plus GRPO, and in-context learning mainly improved in-scope ability without reliably recovering advanced mathematical, scientific, or linguistic capabilities.
Motivation and approach
Studying how LLMs acquire knowledge is confounded by opaque web-scale pretraining corpora: when prior exposure is unknown, it is difficult to determine whether in-context learning (ICL) or post-training produces genuine capability growth or merely elicits latent knowledge. This paper takes a complementary route to benchmark-side novelty curation: instead of defining novelty at evaluation time, it constrains the training distribution itself. The authors release two artifacts: an 88B-token pretraining corpus filtered from FineWeb-Edu to contain only U.S. kindergarten-through-Grade-5 ("inexposure") material, and a 5B-parameter decoder trained from scratch on that corpus under a Qwen3-dense architecture. Together they form a sandbox in which the knowledge boundary is specified before training and can be interpreted against curriculum standards such as the Common Core State Standards (CCSS) and NGSS.
The design contrasts with BabyLM, which restricts corpus quantity rather than conceptual scope, and with concurrent work imposing a temporal cutoff via pre-1931 text, which constrains availability but not conceptual complexity. The developmental cutoff here yields an interpretable conceptual boundary at LLM scale.
Constructing the developmentally restricted corpus
The filtering pipeline is multi-stage and explicitly precision-first. An age-of-acquisition (AoA) rule-based pre-filter discards documents where more than 5% of words exceed a target age of 12; missing AoA values (10% of FineWeb-Edu vocabulary) are imputed from Zipf word frequencies, which offered the best coverage–alignment trade-off among tested frequency sources. LLM-as-a-judge annotation with Gemini Flash, with prompts optimized via DSPy/GEPA and OpenEvolve, provides labels for training a FastText classifier followed by a roughly 50× more expensive ModernBERT classifier on the ~266M samples whose grade band matches the AoA assignment. A regex-based symbolic filter removes notation-heavy content (equations, exponents, roots, calculus operators), removing only ~0.1% of retained documents, indicating it addresses a narrow residual tail rather than serving as a general mathematics filter. A final contrastive frequency-ratio sampling stage preferentially excludes documents containing terms disproportionately associated with out-of-scope material.
Validation supports the precision-first design. On CommonCoreText, a held-out CCSS-aligned ground-truth benchmark, the pipeline reduces out-of-scope retention to 0% while preserving only ~35% of in-scope passages; the authors concede this 65% rejection rate is deliberate, since recovering more in-scope material from noisy web data would require relaxing the filter and weakening confidence in the boundary. Removed in-scope passages are systematically longer and lexically more complex than retained ones. Independent checks corroborate specificity: applied without retuning to WeeBit, the filter retains only 2.48% of 6,000 out-of-scope passages, of which manual inspection attributes genuinely out-of-scope concepts to just three passages (0.05%), and a corpus-wide scan against 126 Grade-6+ technical n-grams finds matches in only 0.09% of retained passages.
A notable cost figure motivates the classifier-based design: full LLM-judge annotation of FineWeb-Edu would cost approximately USD 46M at Gemini Flash pricing, versus USD 45.87M computed directly—hence the multi-stage distillation into cheap classifiers.
Validating the model's exposure boundary
The 5B model tracks its controlled exposure across language, factual knowledge, and mathematical reasoning. On CLEAR readability data binned by Bradley–Terry easiness, its bits-per-byte rises steadily with text hardness while matched-size controls trained on unfiltered FineWeb-Edu (and Gemma 2B) remain flat—an effect attributable to pretraining data given the shared recipe. On CoMTA tutoring dialogues restricted to student turns, familiarity matches baselines on elementary math but diverges on algebra, trigonometry, and calculus. On Jeopardy science questions split by NGSS strand, accuracy collapses beyond the exposure horizon while the control performs comparably across bins. On MathCAMPS, aligned to individual CCSS standards, performance declines disproportionately at higher grades; notably, even at pass@1024 on Grade 8 problems the model solves fewer than half as many questions as the control, and pass@k curves plateau well before k = 1024, which the authors read as a capability ceiling rather than a sampling-budget artifact.
Two caveats qualify these results. First, CCSS grade labels are imperfect difficulty proxies for models: adjacent standards may differ mainly in expected fluency, and the standard 6.EE.B.7 was excluded because its generated problems are structurally indistinguishable from Grade-3 word problems. Second, capability acquisition does not follow human curricular ordering—the model performs better on multi-digit-divisor division than single-digit-divisor division—so the model should not be interpreted as a digital twelve-year-old. A tokenizer ablation with single-digit number tokenization raises out-of-scope MathCAMPS pass@1 for both models but leaves the qualitative gap intact, ruling out number segmentation as the primary driver.
Scaling, post-training, and ICL do not extend the boundary
Three interventions probe whether capabilities can be pushed past the boundary, and all three yield the same pattern: amplification within exposure, little or none beyond it.
Scaling. Training at 0.6B, 1.3B, and 5B parameters improves MathCAMPS performance within exposure and partially at the Grades 6–7 boundary, but has essentially no effect on Grade 8, where the base model remains near floor at all sizes. Interestingly, the smallest restricted-exposure model outperforms the control within exposure, suggesting that removing advanced material lets limited capacity produce more focused reasoning traces—a specialization advantage that disappears at larger scale.
Post-training. A two-stage SFT + GRPO pipeline lifts both models above their base performance within exposure, but out-of-scope gains are modest for the restricted model and considerably larger for the control, widening the gap. Crucially, post-training the restricted model on out-of-scope data produces no measurable difference from post-training it on in-scope data within the tested budgets, indicating that bridging the gap requires more fundamental changes than GRPO provides.
In-context learning. With hand-authored natural-prose chain-of-thought exemplars, few-shot prompting yields only modest in-scope gains (34.0% → 36.8%) and no out-of-scope gain (6.0% → 5.9%). Format matters sharply: compact algebraic shots degrade performance to 17.0%/1.6% and answer-only shots to 9.2%/0.2%, despite the model visibly steering its output style toward the exemplars. Procedural explanations have no effect at all—responses do not change with the explanation provided. The authors frame this not as a negative result but as evidence that the pretraining filter, rather than any tested intervention, sets the effective capability ceiling, consistent with prior findings that RL largely re-elicits pretraining behaviors.
Limitations and open questions
The paper concedes several constraints. The 5B scale means emergent behaviors such as ICL may be less pronounced than at frontier scales, so the null ICL result may not transfer. The corpus sacrifices recall for precision, so it is not an exhaustive sample of elementary-school material, and the developmental framing is a tool for controlling exposure, not a claim of human-like development. Open questions left by the experiments include whether RL with verifiers, self-play, or search can drive genuine discovery beyond the boundary in this controlled setting; whether algorithmic extrapolation (e.g., small-operand arithmetic to large numbers) reflects acquired procedures or interpolation; and how sample efficiency, retention, and interference behave when new concepts such as negative numbers are introduced post hoc.
Conclusion
This work contributes a pedagogically grounded, precisely bounded 88B-token corpus and a 5B model trained on it, validated through language-modeling, factual, and mathematical evaluations showing sharp degradation exactly beyond the specified exposure. Across scaling, SFT+GRPO post-training, and ICL, none of the tested interventions recovers out-of-scope capability, reinforcing the view that the pretraining distribution dominates downstream behavior. The released sandbox enables cleanly attributed studies of knowledge injection, continual learning, interpretability, calibration, and RL-driven extrapolation that are difficult to conduct on models with opaque training data.
