Dolmino Mix 1124: Mid-Training Curriculum

• Dolmino Mix 1124 is a mid-training curriculum designed to patch mathematical, factual, and instruction-following weaknesses in OLMo 2 pretraining.
• It employs a fixed mix of high-quality sources—including Wikipedia, FLAN, and synthetic math data—to reinforce academic and logical reasoning.
• The curriculum uses a two-phase training schedule with model soup checkpoint averaging, delivering significant performance gains on benchmarks like GSM8K and MMLU.

Dolmino Mix 1124 is the mid-training dataset curriculum for the OLMo 2 open LLMs, engineered to be introduced in the final 5–10% of pretraining compute. Its primary goal is to efficiently patch deficiencies—particularly in mathematical reasoning and factual knowledge—that persist after the main, web-centric phase of pretraining. The mix and its associated training schedule exemplify a late-stage curriculum learning approach, leveraging high-quality, carefully constructed data to drive performance improvements across a range of downstream benchmarks (OLMo et al., 2024).

1. Definition and Objectives

Dolmino Mix 1124, finalized in November 2024 (“Late-stage Data Mixture 11/24”), defines Phase II of OLMo 2 pretraining. It consists of a compact, high-quality mixture of data sources designed to optimize the model for:

• Patching factual, encyclopedic, and academic knowledge with up-sampled Wikipedia/Wikibooks, peer-reviewed academic sources (peS2o, arXiv), and curated reference materials.
• Improving instruction-following behaviors using clean, decontaminated instruction sets (FLAN) and real Q&A data (Stack Exchange).
• Boosting mathematical ability via synthetic and curated corpora targeting arithmetic, symbolic reasoning, and multistep logic (e.g., TuluMath, DolminoSynthMath, TinyGSM-MIND, GSM8K train, Metamath filtered, MathCoder2).

By dedicating a small fraction of compute to this specialized mixture after broad language modeling on web data, Dolmino Mix 1124 seeks to move the model to better minima in parameter space with respect to complex downstream tasks (OLMo et al., 2024).

2. Data Composition and Source Rationale

Dolmino Mix 1124 is composed of six fixed data sources, with proportions held constant for each mid-training run. The specific token allocation across common training runs is shown in the following table:

Source	50B run	100B run	300B run
Filtered DCLM-Baseline	47.2%	50.2%	51.9%
Decontaminated FLAN (Dolma 1.7)	16.6%	16.7%	11.3%
Stack Exchange Q&A (2024-09-30 dump)	2.45%	2.47%	1.68%
peS2o (academic)	5.85%	9.52%	19.4%
Wikipedia/Wikibooks	7.11%	3.57%	4.86%
Dolmino Math (synthetic & filtered)	20.8%	17.5%	10.8%

The rationale for the constituent sources is as follows:

• Filtered DCLM: Up-sampled to ~50% for linguistic diversity and data quality (top-7% fastText score, FineWeb ≥ 2).
• FLAN: Provides clean instruction tuning examples (~16%), aligning the model with supervised fine-tuning (SFT) formats.
• Stack Exchange Q&A: Supplies real-world Q&A pairs (~2%) under permissive CC licenses.
• peS2o and Wikipedia/Wikibooks: Academic and encyclopedic material (3–19%) boosts factual coverage.
• Dolmino Math: Synthetic and heavily filtered, directly boosts GSM8K, arithmetic, proofs, and multi-step reasoning performance (11–21%).

No dynamic annealing or reweighting of source proportions occurs within Phase II. Each sampler run uses these fixed percentages for all tokens (OLMo et al., 2024).

3. Training Schedule and Curriculum Integration

The OLMo 2 curriculum under Dolmino Mix 1124 adheres to a two-phase, token-based learning rate schedule:

• Phase I (“Pretraining”): Consumes ≈90% of FLOPs on OLMo 2 Mix 1124 (main web-plus-academic corpus). Learning rate follows:
  • Warmup: $$\eta(t)=\frac{t}{t_{\mathrm{warmup}}}\eta_{\mathrm{peak}}$$, $t_{\mathrm{warmup}} = 2{,}000$ steps.
  • Cosine decay: $$\eta(t) = \eta_\mathrm{peak} \cdot \frac{1}{2}\left[1+\cos\left(\pi\frac{t-t_\mathrm{warmup}}{T_{\cos}-t_\mathrm{warmup}}\right)\right]$$, with $T_{\cos} = 4\times10^{12}$ tokens for 7B, $5\times10^{12}$ for 13B.
• Phase II (“Mid-training”): Conducted for ≈5–10% of total FLOPs using only Dolmino Mix 1124. Learning rate is linearly decayed: $$\eta(t) = \eta_\mathrm{peak}\left(1-\frac{t-T_\mathrm{cos}}{T_\mathrm{mid}}\right)$$.
  • For 7B: $T_{\mathrm{mid}} = 3 \times 50 \times 10^{9}$ tokens (three independent runs).
  • For 13B: $$T_{\mathrm{mid}} = 3 \times 100 \times 10^{9} + 1 \times 300 \times 10^{9}$$ tokens (four runs).

Final Phase II checkpoints are combined via arithmetic mean (“model soup”) across independent seed orders (OLMo et al., 2024).

4. Hyperparameters and Implementation

Dolmino Mix 1124 mid-training retains most optimizer and architecture settings from Phase I, with the following phase-specific parameters:

• Token budgets: 7B: $3\times50$B; 13B: $3\times100$B + $t_{\mathrm{warmup}} = 2{,}000$0B tokens.
• Optimizer: AdamW $t_{\mathrm{warmup}} = 2{,}000$1.
• Peak learning rate: $t_{\mathrm{warmup}} = 2{,}000$2 for 7B; $t_{\mathrm{warmup}} = 2{,}000$3 for 13B; linearly decayed to zero.
• Batch size/seq length: 7B: BS $t_{\mathrm{warmup}} = 2{,}000$4, L $t_{\mathrm{warmup}} = 2{,}000$5; 13B: BS $t_{\mathrm{warmup}} = 2{,}000$6, L $t_{\mathrm{warmup}} = 2{,}000$7.
• Weight decay: 0.1 (embeddings exempted); Dropout: none.
• Model averaging: Arithmetic mean of last three (7B) or four (13B) checkpoints maximizes dev performance.

Implementation involves simple two-stage training with sequential data loader and learning rate schedule transitions:

$t_{\mathrm{warmup}} = 2{,}000$8

The data sampler uses fixed source weights:

$t_{\mathrm{warmup}} = 2{,}000$9

Final models aggregate checkpoints as follows:

$$\eta(t) = \eta_\mathrm{peak} \cdot \frac{1}{2}\left[1+\cos\left(\pi\frac{t-t_\mathrm{warmup}}{T_{\cos}-t_\mathrm{warmup}}\right)\right]$$0 (OLMo et al., 2024)

5. Empirical Performance and Ablation Insights

The introduction of Dolmino Mix 1124 leads to substantial improvements across both development and held-out benchmarks. Phase II mid-training delivers gains in OLMES, MMLU, GS8MK, and other tasks, as summarized below:

Model	Avg OLMES	MMLU	ARC_C	HSwag	WinoG	NQ	DROP	GSM8K
7B pretrain	50.6	59.8	72.6	81.3	75.8	29.0	40.7	24.1
7B midtrain	61.2 (+10.6)	63.7 (+3.9)	79.8 (+7.2)	83.8 (+2.5)	77.2 (+1.4)	36.9 (+7.9)	60.8 (+20.1)	67.5 (+43.4)
13B pretrain	56.5	63.4	80.2	84.8	79.4	34.6	49.6	37.3
13B midtrain	66.8 (+10.3)	67.5 (+4.1)	83.5 (+3.3)	86.4 (+1.6)	81.5 (+2.1)	46.7 (+12.1)	70.7 (+21.1)	75.1 (+37.8)

Ablation (“microanneal”) studies confirm that even low percentages (10%) of math-specialist data can double GSM8K performance, with further gains from increasing the math data proportion. Model soups from independent runs consistently yield best dev results (OLMo et al., 2024).

6. Design Motivations and Training Insights

Key insights underlying the efficacy of Dolmino Mix 1124 include:

• Specialization late in pretraining: Deferring domain-specific content until most broad-language capabilities are acquired focuses optimization on resolving targeted weaknesses, notably mathematics and instruction-following.
• Microannealing: Short-duration (1–2B token) experiments efficiently determine the utility of individual data sources before executing full-scale runs.
• Model soup checkpoint averaging: Reduces variance and consistently locates better-performing optima by averaging weights from independent runs.
• Synthetic-rewritten math data: Translating code-like stepwise solutions (e.g., TinyGSM) into natural-language chain-of-thought via datasets like MIND makes mathematical reasoning accessible to predominantly text-based LLMs.
• Data-filtering synergy: High-quality web subsets and carefully selected instruction sources stabilize training losses, while math data addresses arithmetic and chain-of-thought deficiencies.

These design choices are rooted in empirical study and inform the fixed ~21% proportion of math material in Dolmino Mix 1124 (OLMo et al., 2024).

7. Significance in Open Foundation Model Training

Dolmino Mix 1124 exemplifies the impact of late-phase, curriculum-based data design in high-parameter LLMs. By introducing a fixed, high-quality data mixture in the closing stages of pretraining, substantial improvements are attained on reasoning, math, and instruction-following benchmarks, all with modest additional compute overhead (5–10%). Its transparent data curation, benchmarking, and model release distinguish OLMo 2 in the open LLM landscape and provide a template for future mid-training curricula (OLMo et al., 2024).