OLMo 2 Language Models

• OLMo 2 language models are second-generation open LLMs that introduce refined transformer architectures and a two-stage pretraining curriculum using diverse, massive token mixtures.
• They employ advanced techniques like RMSNorm, QK-Norm, rotary positional embeddings, and RLVR to enhance training stability and instruction-tuning effectiveness.
• OLMo 2 models excel in compute-efficiency and transparency, setting new benchmarks while enabling research in positional encoding, interpretability, and bias mitigation.

OLMo 2 LLMs constitute the second generation of open LLMs released by the Allen Institute for AI, building on the architectural foundation and open-source philosophy of the original OLMo models. OLMo 2 introduces refinements in model architecture, a two-stage pretraining curriculum with specialized data mixtures, novel instruction-tuning methodology incorporating reinforcement learning with verifiable rewards (RLVR), and a comprehensive release of all training data, code, and checkpoints. The models are positioned on the Pareto frontier for compute-efficiency and transparency in the open-weight LLM ecosystem, competing with contemporaneous models such as Llama 3.1, Qwen 2.5, and Gemma 2 (OLMo et al., 2024). OLMo 2 is also a prominent backbone in recent research addressing position encoding and interpretability.

1. Model Architecture and Variants

OLMo 2 comprises decoder-only, autoregressive transformer models trained with a causal (left-to-right) next-token prediction objective. Two principal model sizes are released—7 billion (7B) and 13 billion (13B) parameters. Key architectural updates include:

• RMSNorm over LayerNorm: RMSNorm is applied after each attention and MLP sub-block, replacing previous non-parametric pre-norm, to improve training stability.
• QK-Norm: Before computing attention, both Query (Q) and Key (K) projections are normalized with RMSNorm.
• Rotary Positional Embeddings (RoPE): θ is set to $5 \times 10^5$, substantially higher than prior OLMo versions.
• Z-Loss Regularization: Applied to the log-partition function in the head, with regularization weight $10^{-5}$.
• Bias-Free and Weight Decay Adjustments: No bias terms; embedding weights are excluded from weight decay during training.

The block-level computation for each layer is given by:

$$\text{Self-Attention: } q = \mathrm{RMSNorm}(W_q x), \quad k = \mathrm{RMSNorm}(W_k x), \quad A(x) = \mathrm{softmax}\!\left( \frac{q k^\top}{\sqrt{d}} \right)v, \quad x' = x + \mathrm{RMSNorm}(A(x))$$

$$\text{MLP: } \mathrm{MLP}(x') = W_2 (\mathrm{SwiGLU}(W_1 x')), \quad x'' = x' + \mathrm{RMSNorm}(\mathrm{MLP}(x'))$$

Both 7B and 13B models use a context window of 4,096 tokens, batch sizes of 1,024 (7B) or 2,048 (13B), and SwiGLU activations in MLP layers (OLMo et al., 2024).

2. Pretraining Curriculum and Data Mixtures

OLMo 2 introduces a two-stage pretraining curriculum comprised of:

1. Pretraining on "OLMo 2 Mix 1124": A large web-derived corpus (3.9–5.0 trillion tokens), encompassing web pages (DCLM-Baseline), code (StarCoder), academic content, arXiv, mathematical datasets, and Wiki/Wikibooks. Weight initialization uses $\mathcal{N}(0,0.02^2)$ for improved stability.
2. Mid-Training (Annealing Stage) on "Dolmino Mix 1124": A high-quality, targeted mixture (~50–300B tokens) to patch domain gaps in math and text. Components include filtered web, decontaminated FLAN instruction, StackExchange Q&A, high-quality/synthetic math, academic, and wiki sources.

Curriculum proceeds with a warmup and cosine decay learning rate schedule, truncating pretraining at 4T tokens for 7B before annealing, with larger token budgets for 13B. The "model soup" method (averaging independent runs with different data permutations) is used to further improve robustness (OLMo et al., 2024).

3. Post-Training: Instruction-Tuning and RLVR

The OLMo 2 "Instruct" variants undergo a post-training pipeline inspired by the "T recipe" (originally from Tülu 3):

• Supervised Finetuning (SFT): Instruction tasks are curated from permissively-licensed datasets, including synthetic math/code and PersonaHub contributions. For 7B, SFT is run for 2 epochs at learning rate $1 \times 10^{-5}$.
• Direct Preference Optimization (DPO): Both on-policy data (samples from SFT models) and off-policy data (outputs of 20 open instruct models) are scored by GPT-4o for helpfulness or truthfulness and binarized. DPO is optimized with learning rate $\sim 6 \times 10^{-7}$.
• Reinforcement Learning with Verifiable Rewards (RLVR): Proximal Policy Optimization (PPO) is used with rewards supervised on math (GSM8K, MATH) or truth-constraint signals. The reward is $r=1$ for correct, parseable numerical answers. KL penalty to the SFT policy is set at $\beta \approx 0.05$–0.1.

Total RLVR episodes are ~100k (7B) and ~200k (13B), with fine control via Generalized Advantage Estimation (GAE) and regularization (OLMo et al., 2024).

4. Performance Analysis and Comparisons

OLMo 2 models match or surpass leading open-weight models of similar size on compute-constrained and task-specific benchmarks, occupying the Pareto frontier in FLOPs-to-performance scaling. Empirical results from (OLMo et al., 2024) show:

Model	Avg Score	FLOPs ×$10^{23}$	MMLU	GSM8K
Llama 3.1 8B	59.7	7.2	56.5	56.5
Qwen 2.5 7B	67.2	8.2	74.4	81.5
Gemma 2 9B	66.3	4.4	70.6	70.1
OLMo 2 7B	61.2	1.8	63.7	67.5
OLMo 2 13B	66.8	4.6	67.5	75.1

Late-stage Dolmino Mix mid-training yields substantial performance improvements: for 7B, pretrain average = 50.6, post-anneal = 61.2; for 13B, pretrain = 56.5, post-anneal = 66.8.

Instruction-tuned OLMo 2 models are competitive with Qwen 2.5, Gemma 2, and Llama 3.1 on instruct-style tasks (e.g., OLMo 2 13B-Instruct avg = 62.4 vs Qwen 2.5 14B = 65.2) (OLMo et al., 2024).

5. Internal Representations and Linguistic Probing

Probing experiments on OLMo 2 (particularly the 7B variant) reveal a marked separation between the encoding of lexical identity and inflectional morphology (Li et al., 2 Jun 2025). Layer-wise analyses find:

• Lexical Identity: Decodability (linear/MLP classifiers) is highest in early layers (linear acc. ≈ 0.90 at layer 1), decreasing monotonically toward final layers (≈ 0.75 at layer 24), with MLP achieving marginally higher accuracy due to growing nonlinearity. The linear separability gap rises from ~0.02 to ~0.15 across layers. Selectivity is near zero across layers—reflecting memorization rather than abstraction.
• Inflectional Morphology: Accuracies remain high (0.93–0.97) and nearly flat across all layers for both linear and MLP classifiers, indicating inflectional features (tense, number, etc.) are linearly separable throughout. Selectivity rises to 0.4–0.6, showing that this information is captured via generalizable abstraction.
• Intrinsic Dimensionality: Variance is maximally compressed in mid-layers, with principal components for 50–70% variance dropping from 400+ (input/final) to ≈ 40–300 (middle layers).

These patterns persist across instruction-tuned and base models, suggesting a fundamental organizational principle that distinguishes memorized lexical identity from distributed, abstract inflectional features (Li et al., 2 Jun 2025).

6. Spurious Correlations and Syntactic Bias

OLMo 2 models exhibit significant sensitivity to syntactic–domain correlations, particularly when such patterns are present in pretraining or synthetic fine-tuning data (Shaib et al., 25 Sep 2025). Using controlled datasets constructed from Wikipedia-derived facts (TREx), researchers manipulated the association between syntactic templates (sequences of POS tags) and knowledge domains.

The evaluation framework defines "in-domain" and "cross-domain" settings:

• In-domain: Template-entity pairs correctly answered under an exact syntactic template.
• Cross-domain: All other template-entity combinations.

Key findings:

Model	In-Domain Acc. (μ)	Cross-Domain Acc. (μ)	Δ (=in−cross)
1B	0.92 ± 0.02	0.41 ± 0.01	0.51 ± 0.03
7B	0.91 ± 0.02	0.50 ± 0.01	0.41 ± 0.04
13B	0.93 ± 0.01	0.39 ± 0.03	0.54 ± 0.02

The mean performance drop is $10^{-5}$0. Cross-domain accuracies can collapse to nearly random chance, even as in-domain remains high, indicating a strong reliance on superficial syntactic cues.

A case study on OLMo-2-7B-Instruct reveals that appending a "cross-domain" chain-of-thought template to harmful prompts bypasses refusal mechanisms: refusal rates drop from 40% (baseline) to 2.5% when prepended with a specific cross-domain syntactic prefix.

These observations underscore the need for explicit testing for syntactic–domain confounds and ensuring syntactic diversity within each domain during training (Shaib et al., 25 Sep 2025).

7. Positional Encoding Innovations: RePo on OLMo-2

The OLMo-2-1B model provides a backbone for research into alternative positional encoding schemes, including the RePo ("context re-positioning") mechanism (Li et al., 16 Dec 2025). Standard models employ RoPE with fixed linear token indices; RePo replaces these with continuous, head-wise, data-driven positions $10^{-5}$1 computed via a differentiable module $10^{-5}$2:

$10^{-5}$3

These positions are injected into the rotary encoding; self-attention is adjusted via $10^{-5}$4. RePo is applied to layers 5–24, with position representation dimension $10^{-5}$5 and independent $10^{-5}$6 per head.

Empirical results from continual pretraining (50B tokens) show:

• Noisy-context robustness: Average recall/EM scores on RULER improve from 44.64 (RoPE) to 55.68 (+11 points).
• Structured-data reasoning: End-to-end EM increases from 25.93 to 27.87.
• Long-context generalization (up to 16k tokens): Gains of 7.24 points on LongBench aggregate F1/Rouge-L; substantial benefits on "needle-in-a-haystack" tasks.
• Short-context performance: Remains unchanged versus the vanilla backbone.

Attention-mass analysis demonstrates RePo allocates ≈15% more attention to distant, salient tokens, counteracting the locality bias of default encodings. The learned $10^{-5}$7 sequence exhibits non-monotonic, chunked, or collapsed patterns depending on local context, suggesting increased representational flexibility (Li et al., 16 Dec 2025).

---

OLMo 2 thus represents a critical axis in open LLM research, providing both a performant, transparent foundation and a flexible platform for probing, safety, and interpretability innovations.