OLMo 2: AllenAI's Open Language Models

• OLMo 2 is a family of fully open, dense autoregressive transformer models released by AllenAI with public weights, data, code, and training logs.
• The architecture employs innovations like SwiGLU blocks, rotary embeddings, and rigorous diagnostics to enhance training stability and performance.
• Detailed experiments and post-training analyses demonstrate that OLMo 2 achieves competitive benchmarks and facilitates reproducible safety and mechanistic studies.

to=arxiv_search.search 玩彩神争霸json {"query":"OLMo 2 (OLMo et al., 2024) OLMo 2 Furious AllenAI", "max_results": 5} to=arxiv_search.search 天天中彩票会json {"query":"OLMo 2 arXiv Allen Institute for AI", "max_results": 10} to=arxiv_search.search 大发游戏json {"query":"(OLMo et al., 2024)", "max_results": 3} OLMo 2 is the second generation of Allen Institute for AI’s fully open LLMs: a family of dense autoregressive transformers released with open weights, pretraining data mixtures, training code and recipes, training logs, and thousands of intermediate checkpoints. The initial OLMo 2 release centered on 7B and 13B base and instruction-tuned models, while later work also evaluated additional family members such as OLMo‑2‑32B in safety-critical settings (OLMo et al., 2024, Judd et al., 31 Oct 2025). Within the open-model ecosystem, OLMo 2 is notable not only for benchmark competitiveness but for exposing the full development stack needed for reproducible training, post-training, mechanistic analysis, and safety evaluation (OLMo et al., 2024).

1. Family scope and release model

OLMo 2 was introduced as a successor to earlier OLMo releases, with revisions to architecture, optimization, pretraining data, and post-training. The core release comprises dense decoder-only transformers at 7B and 13B scale, each with a 4096-token context length, and corresponding instruction-tuned variants branded OLMo 2‑Instruct (OLMo et al., 2024). The project’s defining claim of “full openness” is literal: weights, data manifests, code, recipes, logs, and intermediate checkpoints are all public, which has made OLMo 2 a recurring reference model in later work on decoding, safety, memorization, and interpretability (OLMo et al., 2024).

The initial model sizes are summarized below.

Model	Layers	Hidden size	Attention heads	Total training tokens
OLMo 2 7B	32	4096	32	4.05T
OLMo 2 13B	40	5120	40	5.6T

A later study of suicide-risk communication describes OLMo‑2‑32B as a 32‑billion‑parameter, general-purpose model released in April 2025 as part of the OLMo 2 family, emphasizing again that the OLMo 2 line is best understood as a family rather than a single checkpoint (Judd et al., 31 Oct 2025).

2. Architecture and optimization changes

OLMo 2 retains the dense decoder-only transformer template but changes several stability-critical components relative to earlier OLMo models. It uses no biases, SwiGLU feed-forward blocks with hidden dimension approximately $\frac{8}{3}d$ rounded to a multiple of 128, rotary positional embeddings, RMSNorm, and a “post-norm” block layout (OLMo et al., 2024). The block update is given as

$$\begin{aligned} \mathbf{h} &= \mathbf{x} + \mathrm{RMSNorm}(\mathrm{Attention}(\mathbf{x})) \ \mathbf{h}_{\text{out}} &= \mathbf{h} + \mathrm{RMSNorm}(\mathrm{MLP}(\mathbf{x})) \end{aligned}$$

with QK-norm applied before dot-product attention to control attention-logit growth (OLMo et al., 2024).

The training recipe also introduces a softmax stabilizer:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \lambda_z \log^2 Z, \quad \lambda_z = 10^{-5},$$

where $Z$ is the softmax denominator. OLMo 2 further adopts a larger RoPE base, $\theta = 5 \times 10^5$, excludes embeddings from AdamW weight decay, uses global-norm clipping at 1.0, and reduces AdamW $\epsilon$ to $10^{-8}$ (OLMo et al., 2024). Initialization is simplified to

$\theta \sim \mathcal{N}(0, 0.02^2),$

replacing the scaled initialization used in OLMo‑0424; the stated motivation is better activation and gradient behavior across depth (OLMo et al., 2024).

The project formalizes training stability with two diagnostics. The first is a layerwise growth exponent,

$$\lambda = \frac{1}{n_{\text{layers}}} \log \left( \frac{\|\mathbf{v}'\|_2}{\|\mathbf{v}\|_2} \right),$$

used for both activations and gradients. The second is a “spike score,” defined as the percentage of values at least seven standard deviations away from a rolling mean over the previous 1000 steps. Both diagnostics improved relative to earlier OLMo recipes after the architectural and data-cleaning changes, especially once repeated n-gram artifacts in the data were filtered or masked during loss computation (OLMo et al., 2024).

A smaller but consequential change is the tokenizer. OLMo 2 replaces the earlier modified GPT‑NeoX‑20B tokenizer with the cl100k tokenizer and adds several special PII-mask tokens for compatibility with Dolma-derived sources. In a 1B, 100B-token experiment reported in the OLMo 2 paper, this tokenizer slightly improved downstream accuracy relative to the old tokenizer (OLMo et al., 2024).

3. Pretraining data, curriculum, and Dolmino

OLMo 2 pretraining is explicitly two-stage. The long first stage uses OLMo 2 Mix 1124, a roughly 3.90T-token mixture dominated by DCLM Baseline 1.0 web data, which accounts for about 95% of the mix, with additional code, scholarly, mathematical, and encyclopedic sources including filtered StarCoder repositories, peS2o, arXiv, OpenWebMath, Algebraic Stack, Wikipedia, and Wikibooks (OLMo et al., 2024). The 7B model trains to 4.05T total tokens and the 13B model to 5.6T, with the latter spending 5T tokens in the first stage (OLMo et al., 2024).

The distinctive contribution is the late-stage curriculum Dolmino Mix 1124, used during annealing. Dolmino combines a high-quality filtered web-and-reference mixture with a specialized math-heavy component. The high-quality side includes filtered DCLM web, decontaminated FLAN instruction data, peS2o, Wikipedia/Wikibooks, and StackExchange question–answer pairs. The math side includes TuluMath, Dolmino SynthMath, TinyGSM‑MIND, MathCoder2‑Synthetic, ProofPile-derived sources, and GSM8K training data (OLMo et al., 2024). The paper’s microannealing experiments argue that modest injections of domain-specific data can materially improve math capability without hurting broad performance, and that representation format matters: TinyGSM‑Inline reduced GSM* to 25.0, while TinyGSM‑MIND reached 65.5 and duplicated TinyGSM‑MIND reached 70.0, with MMLU also improved (OLMo et al., 2024).

The empirical effect of this curriculum is large. After Dolmino mid-training, the 7B model’s average development score rises from 50.6 to 61.2 and GSM8K from 24.1 to 67.5; the 13B model’s average development score rises from 56.5 to 66.8 and GSM8K from 37.3 to 75.1 (OLMo et al., 2024). Gains also appear on MMLU, ARC‑C, Natural Questions, and DROP. The final base models are checkpoint soups over multiple annealing runs on the same Dolmino mixtures, a choice reported to consistently improve or match the best single run (OLMo et al., 2024).

A common misconception is that OLMo 2’s gains come solely from scale. The release paper instead presents OLMo 2 as a recipe change: stability-oriented architecture, altered optimizer settings, curated late-stage data, and model souping all contribute materially, and the late-stage curriculum is treated as a central capability driver rather than a minor refinement (OLMo et al., 2024).

4. Post-training and OLMo 2‑Instruct

OLMo 2‑Instruct follows a Tülu‑3-style post-training stack: supervised finetuning, preference optimization, and reinforcement learning with verifiable rewards (OLMo et al., 2024). The SFT mixture contains 939,104 prompts drawn from high-quality instruction and chat corpora, including permissively licensed replacements for non-permissive Tülu‑3 components. An ablation reported in the OLMo 2 paper found that removing multilingual subsets slightly reduced average performance, so multilingual sources such as Aya and multilingual WildChat were retained despite the family’s primary English orientation (OLMo et al., 2024).

Preference tuning uses DPO over prompt–chosen–rejected triples assembled from a large prompt pool and responses generated by a permissively licensed model pool plus on-policy OLMo 2‑SFT generations. GPT‑4o acts as an LM judge over helpfulness, truthfulness, honesty, and instruction following, after which Argilla-style binarization yields chosen and rejected responses for DPO training (OLMo et al., 2024). The final RLVR stage targets tasks with algorithmically checkable rewards, particularly math reasoning on GSM8K and MATH and instruction-following tasks with explicit constraints (OLMo et al., 2024).

Later work relies heavily on the openness of this staged post-training pipeline. Because public checkpoints exist for Base $\rightarrow$ SFT $$\begin{aligned} \mathbf{h} &= \mathbf{x} + \mathrm{RMSNorm}(\mathrm{Attention}(\mathbf{x})) \ \mathbf{h}_{\text{out}} &= \mathbf{h} + \mathrm{RMSNorm}(\mathrm{MLP}(\mathbf{x})) \end{aligned}$$0 DPO $$\begin{aligned} \mathbf{h} &= \mathbf{x} + \mathrm{RMSNorm}(\mathrm{Attention}(\mathbf{x})) \ \mathbf{h}_{\text{out}} &= \mathbf{h} + \mathrm{RMSNorm}(\mathrm{MLP}(\mathbf{x})) \end{aligned}$$1 RLVR, OLMo and OLMo 2 became unusually useful for causal analyses of post-training effects. One such analysis found that, in OLMo and OLMo 2 7B checkpoints, supervised finetuning reduces output diversity, DPO causes the sharpest additional reduction, and RLVR has comparatively little further effect; the same paper proposes conformative decoding, which mixes the instruct model with its base model at inference time to recover diversity while maintaining or slightly improving quality (Peeperkorn et al., 28 Jul 2025).

5. Performance, openness in practice, and diversity-oriented follow-up work

The OLMo 2 paper positions the base models on the Pareto frontier of performance to compute, often matching or outperforming open-weight-only models such as Llama 3.1 and Qwen 2.5 while using fewer FLOPs and fully transparent data and recipes (OLMo et al., 2024). The instruction-tuned models are described as competitive with or surpassing open-weight-only models of comparable size, including Qwen 2.5, Llama 3.1, and Gemma 2 (OLMo et al., 2024). A separate clinical-evaluation paper also characterizes OLMo‑2‑32B as competitive with GPT‑4o mini on academic benchmarks, which partly explains why OLMo 2 family members have become recurring “systems under test” in downstream research (Judd et al., 31 Oct 2025).

A second line of follow-up work studies response diversity in OLMo 2 specifically. The paper “Diverse, not Short” trains OLMo‑2‑7B and OLMo‑2‑13B with only 3,000 preference pairs under a length-controlled self-learning framework, showing that common diversity metrics and reward models are length-biased and that length control is necessary to improve diversity without simply collapsing outputs to shorter responses (Deshpande et al., 22 May 2025). For OLMo‑2‑7B on the creative writing task, Diverse‑NS raises DSI from 0.8499 to 0.8563 and ArmoRM from 0.1435 to 0.1464; for OLMo‑2‑13B, trained only on 7B-generated preference pairs, DSI rises from 0.8557 to 0.8614 and ArmoRM from 0.1571 to 0.1590 (Deshpande et al., 22 May 2025). The same study argues that OLMo‑2‑7B can act as an effective “diversity teacher” for OLMo‑2‑13B, which is unusual because it shows smaller-model-generated preference data improving a larger model on creativity-oriented tasks (Deshpande et al., 22 May 2025).

These results complicate a simplistic view of OLMo 2 post-training. The public checkpoint chain shows that post-training can substantially reshape behavioral properties without changing architecture; DPO in particular can trade diversity for preference alignment, and dedicated corrective procedures can re-expand lexical and semantic variety (Peeperkorn et al., 28 Jul 2025, Deshpande et al., 22 May 2025).

6. Safety, robustness, and mechanistic analyses

Because OLMo 2 is unusually transparent, later work has used it to study failure modes that are difficult to examine in closed systems. One example is syntactic-domain shortcut learning. A 2025 study fine-tunes OLMo‑2 base and instruct models at 1B, 7B, and 13B scale on synthetic data with spurious syntax–domain correlations and reports large in-domain/cross-domain gaps; it further shows that OLMo‑2‑7B‑Instruct can exhibit refusal bypasses when harmful requests are embedded in syntactic templates associated with benign domains (Shaib et al., 25 Sep 2025). The broader implication is not that OLMo 2 is uniquely vulnerable, but that its openness makes these shortcut behaviors empirically tractable.

Safety-critical conversational behavior has also been examined directly. An independent clinical evaluation of OLMo‑2‑32B on suicide-risk conversations found that the model is generally empathetic and often encourages help-seeking, but becomes less likely to invite continued dialogue as more suicidal-thought-and-behavior risk factors accumulate across turns, a pattern the authors interpret as a potential hazard because it may feel dismissive or abandoning in high-risk interactions (Judd et al., 31 Oct 2025). That paper leverages OLMo‑2‑32B precisely because the model’s weights, data, and tools are open enough to support reproducible safety analysis.

On the optimization side, sharpness-aware mid-training has been tested directly on OLMo‑2‑1B. Starting from the official 4T-token OLMo‑2‑1B checkpoint, a 50B-token SAM mid-training phase on Dolmino reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization, despite slightly weaker base-model performance before modification (Watts et al., 4 May 2026). This suggests that, for OLMo‑2‑style systems, the geometry of the pretrained solution can matter as much as raw base-model strength when subsequent SFT, RL, or quantization is expected.

Mechanistic work has likewise used OLMo 2 as a primary testbed. A study of FFN channel importance finds that, in OLMo‑2‑7B, the top 1% of channels per layer account for a median 71.7% of LP mass, with a median max/mean LP ratio of 3690; across pretraining checkpoints, the median top‑1% LP mass rises from 18.6% to 71.7%, showing that a small loss-critical core emerges over training (Cherilyn et al., 26 Apr 2026). Another study of long-context retrieval heads reports 87 detected retrieval heads in OLMo‑2‑7B and shows that masking them collapses needle-in-a-haystack recall from 1.00 to 0.00, while masking matched random heads has no effect; the same paper further shows that low-frequency RoPE dimensions, rather than simple query-norm “utility,” are the causal positional substrate of these retrieval heads (Bayram, 19 Jun 2026).

Taken together, these later studies make OLMo 2 significant beyond its release benchmarks. It functions simultaneously as a competitive open model family and as a research substrate for analyzing post-training dynamics, mechanistic structure, memorization, robustness to modification, and safety behavior under reproducible conditions (OLMo et al., 2024). A plausible implication is that OLMo 2’s most durable contribution may be methodological: it provides a rare case where frontier-adjacent LLMs can be studied not merely as opaque endpoints but as fully inspectable training trajectories.