---
title: 'OLMo 2: AllenAI''s Open Language Models'
url: https://www.emergentmind.com/topics/olmo-2
type: topic
---

# OLMo 2: AllenAI's Open Language Models

to=arxiv_search.search  玩彩神争霸json
{"query":"OLMo 2 2501.00656 OLMo 2 Furious AllenAI", "max_results": 5}
to=arxiv_search.search  天天中彩票会json
{"query":"OLMo 2 arXiv Allen Institute for AI", "max_results": 10}
to=arxiv_search.search  大发游戏json
{"query":"2501.00656", "max_results": 3}
OLMo 2 is the second generation of Allen Institute for AI’s fully open language models: a family of dense autoregressive transformers released with open weights, pretraining data mixtures, training code and recipes, training logs, and thousands of intermediate checkpoints. The initial OLMo 2 release centered on 7B and 13B base and instruction-tuned models, while later work also evaluated additional family members such as OLMo‑2‑32B in safety-critical settings [2501.00656][2510.27521]. Within the open-model ecosystem, OLMo 2 is notable not only for benchmark competitiveness but for exposing the full development stack needed for reproducible training, post-training, mechanistic analysis, and safety evaluation [2501.00656].

## 1. Family scope and release model

OLMo 2 was introduced as a successor to earlier OLMo releases, with revisions to architecture, optimization, pretraining data, and post-training. The core release comprises dense decoder-only transformers at 7B and 13B scale, each with a 4096-token context length, and corresponding instruction-tuned variants branded OLMo 2‑Instruct [2501.00656]. The project’s defining claim of “full openness” is literal: weights, data manifests, code, recipes, logs, and intermediate checkpoints are all public, which has made OLMo 2 a recurring reference model in later work on decoding, safety, memorization, and interpretability [2501.00656].

The initial model sizes are summarized below.

| Model | Layers | Hidden size | Attention heads | Total training tokens |
|---|---:|---:|---:|---:|
| OLMo 2 7B | 32 | 4096 | 32 | 4.05T |
| OLMo 2 13B | 40 | 5120 | 40 | 5.6T |

A later study of suicide-risk communication describes **OLMo‑2‑32B** as a 32‑billion‑parameter, general-purpose model released in April 2025 as part of the OLMo 2 family, emphasizing again that the OLMo 2 line is best understood as a family rather than a single checkpoint [2510.27521].

## 2. Architecture and optimization changes

OLMo 2 retains the dense decoder-only transformer template but changes several stability-critical components relative to earlier OLMo models. It uses no biases, SwiGLU feed-forward blocks with hidden dimension approximately $\frac{8}{3}d$ rounded to a multiple of 128, rotary positional embeddings, RMSNorm, and a “post-norm” block layout [2501.00656]. The block update is given as
$$
\begin{aligned}
\mathbf{h} &= \mathbf{x} + \mathrm{RMSNorm}(\mathrm{Attention}(\mathbf{x})) \\
\mathbf{h}_{\text{out}} &= \mathbf{h} + \mathrm{RMSNorm}(\mathrm{MLP}(\mathbf{x}))
\end{aligned}
$$
with QK-norm applied before dot-product attention to control attention-logit growth [2501.00656].

The training recipe also introduces a softmax stabilizer:
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \lambda_z \log^2 Z, \quad \lambda_z = 10^{-5},
$$
where $Z$ is the softmax denominator. OLMo 2 further adopts a larger RoPE base, $\theta = 5 \times 10^5$, excludes embeddings from AdamW weight decay, uses global-norm clipping at 1.0, and reduces AdamW $\epsilon$ to $10^{-8}$ [2501.00656]. Initialization is simplified to
$$
\theta \sim \mathcal{N}(0, 0.02^2),
$$
replacing the scaled initialization used in OLMo‑0424; the stated motivation is better activation and gradient behavior across depth [2501.00656].

The project formalizes training stability with two diagnostics. The first is a layerwise growth exponent,
$$
\lambda = \frac{1}{n_{\text{layers}}} \log \left( \frac{\|\mathbf{v}'\|_2}{\|\mathbf{v}\|_2} \right),
$$
used for both activations and gradients. The second is a “spike score,” defined as the percentage of values at least seven standard deviations away from a rolling mean over the previous 1000 steps. Both diagnostics improved relative to earlier OLMo recipes after the architectural and data-cleaning changes, especially once repeated n-gram artifacts in the data were filtered or masked during loss computation [2501.00656].

A smaller but consequential change is the tokenizer. OLMo 2 replaces the earlier modified GPT‑NeoX‑20B tokenizer with the cl100k tokenizer and adds several special PII-mask tokens for compatibility with Dolma-derived sources. In a 1B, 100B-token experiment reported in the OLMo 2 paper, this tokenizer slightly improved downstream accuracy relative to the old tokenizer [2501.00656].

## 3. Pretraining data, curriculum, and Dolmino

OLMo 2 pretraining is explicitly two-stage. The long first stage uses **OLMo 2 Mix 1124**, a roughly 3.90T-token mixture dominated by DCLM Baseline 1.0 web data, which accounts for about 95% of the mix, with additional code, scholarly, mathematical, and encyclopedic sources including filtered StarCoder repositories, peS2o, arXiv, OpenWebMath, Algebraic Stack, Wikipedia, and Wikibooks [2501.00656]. The 7B model trains to 4.05T total tokens and the 13B model to 5.6T, with the latter spending 5T tokens in the first stage [2501.00656].

The distinctive contribution is the late-stage curriculum **Dolmino Mix 1124**, used during annealing. Dolmino combines a high-quality filtered web-and-reference mixture with a specialized math-heavy component. The high-quality side includes filtered DCLM web, decontaminated FLAN instruction data, peS2o, Wikipedia/Wikibooks, and StackExchange question–answer pairs. The math side includes TuluMath, Dolmino SynthMath, TinyGSM‑MIND, MathCoder2‑Synthetic, ProofPile-derived sources, and GSM8K training data [2501.00656]. The paper’s microannealing experiments argue that modest injections of domain-specific data can materially improve math capability without hurting broad performance, and that representation format matters: TinyGSM‑Inline reduced GSM* to 25.0, while TinyGSM‑MIND reached 65.5 and duplicated TinyGSM‑MIND reached 70.0, with MMLU also improved [2501.00656].

The empirical effect of this curriculum is large. After Dolmino mid-training, the 7B model’s average development score rises from 50.6 to 61.2 and GSM8K from 24.1 to 67.5; the 13B model’s average development score rises from 56.5 to 66.8 and GSM8K from 37.3 to 75.1 [2501.00656]. Gains also appear on MMLU, ARC‑C, Natural Questions, and DROP. The final base models are checkpoint soups over multiple annealing runs on the same Dolmino mixtures, a choice reported to consistently improve or match the best single run [2501.00656].

A common misconception is that OLMo 2’s gains come solely from scale. The release paper instead presents OLMo 2 as a recipe change: stability-oriented architecture, altered optimizer settings, curated late-stage data, and model souping all contribute materially, and the late-stage curriculum is treated as a central capability driver rather than a minor refinement [2501.00656].

## 4. Post-training and OLMo 2‑Instruct

OLMo 2‑Instruct follows a Tülu‑3-style post-training stack: supervised finetuning, preference optimization, and reinforcement learning with verifiable rewards [2501.00656]. The SFT mixture contains 939,104 prompts drawn from high-quality instruction and chat corpora, including permissively licensed replacements for non-permissive Tülu‑3 components. An ablation reported in the OLMo 2 paper found that removing multilingual subsets slightly reduced average performance, so multilingual sources such as Aya and multilingual WildChat were retained despite the family’s primary English orientation [2501.00656].

Preference tuning uses DPO over prompt–chosen–rejected triples assembled from a large prompt pool and responses generated by a permissively licensed model pool plus on-policy OLMo 2‑SFT generations. GPT‑4o acts as an LM judge over helpfulness, truthfulness, honesty, and instruction following, after which Argilla-style binarization yields chosen and rejected responses for DPO training [2501.00656]. The final RLVR stage targets tasks with algorithmically checkable rewards, particularly math reasoning on GSM8K and MATH and instruction-following tasks with explicit constraints [2501.00656].

Later work relies heavily on the openness of this staged post-training pipeline. Because public checkpoints exist for Base $\rightarrow$ SFT $\rightarrow$ DPO $\rightarrow$ RLVR, OLMo and OLMo 2 became unusually useful for causal analyses of post-training effects. One such analysis found that, in OLMo and OLMo 2 7B checkpoints, supervised finetuning reduces output diversity, DPO causes the sharpest additional reduction, and RLVR has comparatively little further effect; the same paper proposes conformative decoding, which mixes the instruct model with its base model at inference time to recover diversity while maintaining or slightly improving quality [2507.20956].

## 5. Performance, openness in practice, and diversity-oriented follow-up work

The OLMo 2 paper positions the base models on the Pareto frontier of performance to compute, often matching or outperforming open-weight-only models such as Llama 3.1 and Qwen 2.5 while using fewer FLOPs and fully transparent data and recipes [2501.00656]. The instruction-tuned models are described as competitive with or surpassing open-weight-only models of comparable size, including Qwen 2.5, Llama 3.1, and Gemma 2 [2501.00656]. A separate clinical-evaluation paper also characterizes OLMo‑2‑32B as competitive with GPT‑4o mini on academic benchmarks, which partly explains why OLMo 2 family members have become recurring “systems under test” in downstream research [2510.27521].

A second line of follow-up work studies response diversity in OLMo 2 specifically. The paper “Diverse, not Short” trains OLMo‑2‑7B and OLMo‑2‑13B with only 3,000 preference pairs under a length-controlled self-learning framework, showing that common diversity metrics and reward models are length-biased and that length control is necessary to improve diversity without simply collapsing outputs to shorter responses [2505.16245]. For OLMo‑2‑7B on the creative writing task, Diverse‑NS raises DSI from 0.8499 to 0.8563 and ArmoRM from 0.1435 to 0.1464; for OLMo‑2‑13B, trained only on 7B-generated preference pairs, DSI rises from 0.8557 to 0.8614 and ArmoRM from 0.1571 to 0.1590 [2505.16245]. The same study argues that OLMo‑2‑7B can act as an effective “diversity teacher” for OLMo‑2‑13B, which is unusual because it shows smaller-model-generated preference data improving a larger model on creativity-oriented tasks [2505.16245].

These results complicate a simplistic view of OLMo 2 post-training. The public checkpoint chain shows that post-training can substantially reshape behavioral properties without changing architecture; DPO in particular can trade diversity for preference alignment, and dedicated corrective procedures can re-expand lexical and semantic variety [2507.20956][2505.16245].

## 6. Safety, robustness, and mechanistic analyses

Because OLMo 2 is unusually transparent, later work has used it to study failure modes that are difficult to examine in closed systems. One example is syntactic-domain shortcut learning. A 2025 study fine-tunes OLMo‑2 base and instruct models at 1B, 7B, and 13B scale on synthetic data with spurious syntax–domain correlations and reports large in-domain/cross-domain gaps; it further shows that OLMo‑2‑7B‑Instruct can exhibit refusal bypasses when harmful requests are embedded in syntactic templates associated with benign domains [2509.21155]. The broader implication is not that OLMo 2 is uniquely vulnerable, but that its openness makes these shortcut behaviors empirically tractable.

Safety-critical conversational behavior has also been examined directly. An independent clinical evaluation of OLMo‑2‑32B on suicide-risk conversations found that the model is generally empathetic and often encourages help-seeking, but becomes less likely to invite continued dialogue as more suicidal-thought-and-behavior risk factors accumulate across turns, a pattern the authors interpret as a potential hazard because it may feel dismissive or abandoning in high-risk interactions [2510.27521]. That paper leverages OLMo‑2‑32B precisely because the model’s weights, data, and tools are open enough to support reproducible safety analysis.

On the optimization side, sharpness-aware mid-training has been tested directly on OLMo‑2‑1B. Starting from the official 4T-token OLMo‑2‑1B checkpoint, a 50B-token SAM mid-training phase on Dolmino reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization, despite slightly weaker base-model performance before modification [2605.02105]. This suggests that, for OLMo‑2‑style systems, the geometry of the pretrained solution can matter as much as raw base-model strength when subsequent SFT, RL, or quantization is expected.

Mechanistic work has likewise used OLMo 2 as a primary testbed. A study of FFN channel importance finds that, in OLMo‑2‑7B, the top 1% of channels per layer account for a median 71.7% of LP mass, with a median max/mean LP ratio of 3690; across pretraining checkpoints, the median top‑1% LP mass rises from 18.6% to 71.7%, showing that a small loss-critical core emerges over training [2604.23475]. Another study of long-context retrieval heads reports 87 detected retrieval heads in OLMo‑2‑7B and shows that masking them collapses needle-in-a-haystack recall from 1.00 to 0.00, while masking matched random heads has no effect; the same paper further shows that low-frequency RoPE dimensions, rather than simple query-norm “utility,” are the causal positional substrate of these retrieval heads [2606.21249].

Taken together, these later studies make OLMo 2 significant beyond its release benchmarks. It functions simultaneously as a competitive open model family and as a research substrate for analyzing post-training dynamics, mechanistic structure, memorization, robustness to modification, and safety behavior under reproducible conditions [2501.00656]. A plausible implication is that OLMo 2’s most durable contribution may be methodological: it provides a rare case where frontier-adjacent language models can be studied not merely as opaque endpoints but as fully inspectable training trajectories.

Source: https://www.emergentmind.com/topics/olmo-2