---
title: 'Dolmino Mix 1124: Mid-Training Curriculum'
url: https://www.emergentmind.com/topics/dolmino-mix-1124-data-curriculum
type: topic
---

# Dolmino Mix 1124: Mid-Training Curriculum

Dolmino Mix 1124 is the mid-training dataset curriculum for the OLMo 2 open language models, engineered to be introduced in the final 5–10% of pretraining compute. Its primary goal is to efficiently patch deficiencies—particularly in mathematical reasoning and factual knowledge—that persist after the main, web-centric phase of pretraining. The mix and its associated training schedule exemplify a late-stage curriculum learning approach, leveraging high-quality, carefully constructed data to drive performance improvements across a range of downstream benchmarks [2501.00656].

## 1. Definition and Objectives

Dolmino Mix 1124, finalized in November 2024 (“Late-stage Data Mixture 11/24”), defines Phase II of OLMo 2 pretraining. It consists of a compact, high-quality mixture of data sources designed to optimize the model for:

- Patching factual, encyclopedic, and academic knowledge with up-sampled Wikipedia/Wikibooks, peer-reviewed academic sources (peS2o, arXiv), and curated reference materials.
- Improving instruction-following behaviors using clean, decontaminated instruction sets (FLAN) and real Q&A data (Stack Exchange).
- Boosting mathematical ability via synthetic and curated corpora targeting arithmetic, symbolic reasoning, and multistep logic (e.g., TuluMath, DolminoSynthMath, TinyGSM-MIND, GSM8K train, Metamath filtered, MathCoder2).

By dedicating a small fraction of compute to this specialized mixture after broad language modeling on web data, Dolmino Mix 1124 seeks to move the model to better minima in parameter space with respect to complex downstream tasks [2501.00656].

## 2. Data Composition and Source Rationale

Dolmino Mix 1124 is composed of six fixed data sources, with proportions held constant for each mid-training run. The specific token allocation across common training runs is shown in the following table:

| Source                               | 50B run | 100B run | 300B run |
|---------------------------------------|---------|----------|----------|
| Filtered DCLM-Baseline                | 47.2%   | 50.2%    | 51.9%    |
| Decontaminated FLAN (Dolma 1.7)       | 16.6%   | 16.7%    | 11.3%    |
| Stack Exchange Q&A (2024-09-30 dump)  | 2.45%   | 2.47%    | 1.68%    |
| peS2o (academic)                      | 5.85%   | 9.52%    | 19.4%    |
| Wikipedia/Wikibooks                   | 7.11%   | 3.57%    | 4.86%    |
| Dolmino Math (synthetic & filtered)   | 20.8%   | 17.5%    | 10.8%    |

The rationale for the constituent sources is as follows:

- **Filtered DCLM**: Up-sampled to ~50% for linguistic diversity and data quality (top-7% fastText score, FineWeb ≥ 2).
- **FLAN**: Provides clean instruction tuning examples (~16%), aligning the model with supervised fine-tuning (SFT) formats.
- **Stack Exchange Q&A**: Supplies real-world Q&A pairs (~2%) under permissive CC licenses.
- **peS2o and Wikipedia/Wikibooks**: Academic and encyclopedic material (3–19%) boosts factual coverage.
- **Dolmino Math**: Synthetic and heavily filtered, directly boosts GSM8K, arithmetic, proofs, and multi-step reasoning performance (11–21%).

No dynamic annealing or reweighting of source proportions occurs within Phase II. Each sampler run uses these fixed percentages for all tokens [2501.00656].

## 3. Training Schedule and Curriculum Integration

The OLMo 2 curriculum under Dolmino Mix 1124 adheres to a two-phase, token-based learning rate schedule:

- **Phase I (“Pretraining”)**: Consumes ≈90% of FLOPs on OLMo 2 Mix 1124 (main web-plus-academic corpus). Learning rate follows:
  - Warmup: $\eta(t)=\frac{t}{t_{\mathrm{warmup}}}\eta_{\mathrm{peak}}$, $t_{\mathrm{warmup}} = 2{,}000$ steps.
  - Cosine decay: $\eta(t) = \eta_\mathrm{peak} \cdot \frac{1}{2}\left[1+\cos\left(\pi\frac{t-t_\mathrm{warmup}}{T_{\cos}-t_\mathrm{warmup}}\right)\right]$, with $T_{\cos} = 4\times10^{12}$ tokens for 7B, $5\times10^{12}$ for 13B.

- **Phase II (“Mid-training”)**: Conducted for ≈5–10% of total FLOPs using only Dolmino Mix 1124. Learning rate is linearly decayed: $\eta(t) = \eta_\mathrm{peak}\left(1-\frac{t-T_\mathrm{cos}}{T_\mathrm{mid}}\right)$.
  - For 7B: $T_{\mathrm{mid}} = 3 \times 50 \times 10^{9}$ tokens (three independent runs).
  - For 13B: $T_{\mathrm{mid}} = 3 \times 100 \times 10^{9} + 1 \times 300 \times 10^{9}$ tokens (four runs).

Final Phase II checkpoints are combined via arithmetic mean (“model soup”) across independent seed orders [2501.00656].

## 4. Hyperparameters and Implementation

Dolmino Mix 1124 mid-training retains most optimizer and architecture settings from Phase I, with the following phase-specific parameters:

- **Token budgets**: 7B: $3\times50$B; 13B: $3\times100$B + $1\times300$B tokens.
- **Optimizer**: AdamW $(\beta_1=0.9,\, \beta_2=0.95,\, \epsilon=10^{-8})$.
- **Peak learning rate**: $3 \times 10^{-4}$ for 7B; $9 \times 10^{-4}$ for 13B; linearly decayed to zero.
- **Batch size/seq length**: 7B: BS $=1{,}024$, L $=4{,}096$; 13B: BS $=2{,}048$, L $=4{,}096$.
- **Weight decay**: 0.1 (embeddings exempted); **Dropout**: none.
- **Model averaging**: Arithmetic mean of last three (7B) or four (13B) checkpoints maximizes dev performance.

Implementation involves simple two-stage training with sequential data loader and learning rate schedule transitions:

```python
# Pseudo‐code for integrating Dolmino Mix 1124
trainer = Trainer(
  model=model,
  optimizer=AdamW(...),
  lr_schedule=CompositeSchedule([
    CosineDecayWarmup(peak=η_peak, warmup_steps=2e3, decay_tokens=T_cos),
    LinearDecay(start_lr=η_peak, end_lr=0, decay_tokens=T_mid)
  ]),
)
# Stage I data:
trainer.set_data_loader(olmo_mix_1124_loader)
trainer.train_until(token_count=T_cos)
# Stage II data:
trainer.set_data_loader(dolmino_mix_1124_loader)
trainer.train_for(tokens=T_mid)
```

The data sampler uses fixed source weights:

```python
class DolminoSampler(torch.utils.data.Dataset):
  def __init__(self, sources, weights):
    # sources: dict(name→Dataset), weights summing to 1.0
    self.mix = torch.utils.data.WeightedRandomSampler(weights, total_samples)
  def __getitem__(self, idx):
    src_idx = self.mix[idx]
    return self.sources[src_idx][ random.randint(0, len(self.sources[src_idx])-1) ]
```

Final models aggregate checkpoints as follows:

```python
def soup_checkpoints(fnames):
  states = [torch.load(f) for f in fnames]
  avg_state = {}
  for k in states[0]:
    avg_state[k] = torch.mean(torch.stack([s[k] for s in states]), dim=0)
  return avg_state
```
[2501.00656]

## 5. Empirical Performance and Ablation Insights

The introduction of Dolmino Mix 1124 leads to substantial improvements across both development and held-out benchmarks. Phase II mid-training delivers gains in OLMES, MMLU, GS8MK, and other tasks, as summarized below:

| Model      | Avg OLMES | MMLU | ARC_C | HSwag | WinoG | NQ   | DROP | GSM8K |
|------------|-----------|------|-------|-------|-------|------|------|-------|
| 7B pretrain | 50.6      | 59.8 | 72.6  | 81.3  | 75.8  | 29.0 | 40.7 | 24.1  |
| 7B midtrain | 61.2 (+10.6)| 63.7 (+3.9)| 79.8 (+7.2)|83.8 (+2.5)|77.2 (+1.4)|36.9 (+7.9)|60.8 (+20.1)|67.5 (+43.4)|
| 13B pretrain| 56.5      | 63.4 | 80.2  | 84.8  | 79.4  | 34.6 | 49.6 | 37.3  |
| 13B midtrain| 66.8 (+10.3)| 67.5 (+4.1)| 83.5 (+3.3)|86.4 (+1.6)|81.5 (+2.1)|46.7 (+12.1)|70.7 (+21.1)|75.1 (+37.8)|

Ablation (“microanneal”) studies confirm that even low percentages (10%) of math-specialist data can double GSM8K performance, with further gains from increasing the math data proportion. Model soups from independent runs consistently yield best dev results [2501.00656].

## 6. Design Motivations and Training Insights

Key insights underlying the efficacy of Dolmino Mix 1124 include:

- **Specialization late in pretraining**: Deferring domain-specific content until most broad-language capabilities are acquired focuses optimization on resolving targeted weaknesses, notably mathematics and instruction-following.
- **Microannealing**: Short-duration (1–2B token) experiments efficiently determine the utility of individual data sources before executing full-scale runs.
- **Model soup checkpoint averaging**: Reduces variance and consistently locates better-performing optima by averaging weights from independent runs.
- **Synthetic-rewritten math data**: Translating code-like stepwise solutions (e.g., TinyGSM) into natural-language chain-of-thought via datasets like MIND makes mathematical reasoning accessible to predominantly text-based language models.
- **Data-filtering synergy**: High-quality web subsets and carefully selected instruction sources stabilize training losses, while math data addresses arithmetic and chain-of-thought deficiencies.

These design choices are rooted in empirical study and inform the fixed ~21% proportion of math material in Dolmino Mix 1124 [2501.00656].

## 7. Significance in Open Foundation Model Training

Dolmino Mix 1124 exemplifies the impact of late-phase, curriculum-based data design in high-parameter language models. By introducing a fixed, high-quality data mixture in the closing stages of pretraining, substantial improvements are attained on reasoning, math, and instruction-following benchmarks, all with modest additional compute overhead (5–10%). Its transparent data curation, benchmarking, and model release distinguish OLMo 2 in the open LLM landscape and provide a template for future mid-training curricula [2501.00656].

Source: https://www.emergentmind.com/topics/dolmino-mix-1124-data-curriculum