OLMo 2-Instruct Models: Architecture & Tuning

• OLMo 2-Instruct models are fully open large-scale language models built on a dense autoregressive Transformer architecture optimized for advanced instruction following.
• They incorporate key architectural innovations like bias-free design, SwiGLU activations, RoPE, and QK-norm normalization to ensure training stability and improved long-context handling.
• A rigorous multi-phase training pipeline—including SFT, DPO, and RLVR—coupled with a two-stage pretraining curriculum, enhances both downstream performance and output diversity.

OLMo 2-Instruct models are a family of fully open, large-scale LLMs built upon the OLMo 2 dense autoregressive Transformer architecture, optimized for both upstream efficiency and downstream performance through a rigorous multi-phase pretraining and post-training curriculum. These models implement architectural enhancements, novel data mixtures, and advanced post-training methodologies—including Direct Preference Optimization (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR)—all focused on maximizing instruction-following ability while maintaining flexibility for diverse use cases. The OLMo 2-Instruct release is notable for its comprehensive transparency, with all models (7B and 13B parameters), training data, code, and recipes made openly available, positioning it as a central resource in the open LLM ecosystem (OLMo et al., 2024).

1. Architectural Innovations and Training Stability

OLMo 2-Instruct inherits a standard decoder-only Transformer but introduces substantial refinements over prior OLMo iterations. Noteworthy features include:

• Bias Free Design and SwiGLU Activations: All linear layers omit bias terms. The feedforward hidden dimension is set to approximately $\frac{8}{3}d$ (rounded to a multiple of 128), coupled with the SwiGLU activation to enhance expressivity.
• Rotary Positional Embeddings: Rotational positional embeddings (RoPE) employ a frequency range parameter $\vartheta$ increased from $10^3$ to $5 \times 10^5$, improving the handling of longer contexts.
• RMSNorm Normalization: Layer normalization is replaced with RMSNorm (no bias), applied to output residuals after each attention and MLP block, rather than to the inputs.
• QK-norm for Attention Stability: Pre-attention queries ($Q$) and keys ($K$) are separately normalized with RMSNorm prior to dot-product computation, mitigating early training instability.
• Z-loss Regularization: A supplementary term, $L = L_{\mathrm{CE}} + \lambda(\log Z)^2$ ($\lambda=10^{-5}$, $Z$ the softmax partition function), regularizes logits and stabilizes optimization.

Model scales:

• 7B: 32 layers, $d_{\text{model}}=4096$, 32 heads ($\vartheta$0 params)
• 13B: 40 layers, $\vartheta$1, 40 heads ($\vartheta$2 params) The pretraining FLOP investment is approximately $\vartheta$3 for 7B (4T tokens) and $\vartheta$4 for 13B (5T tokens) (OLMo et al., 2024).

2. Pretraining and Data Curriculum

OLMo 2-Instruct employs a two-stage curriculum pretraining strategy explicitly designed to shape generalization and capability:

• Stage 1 (OLMo 2 Mix 1124): ($\vartheta$590–95 % FLOPs, $\vartheta$63.9T tokens). Domain composition includes baseline web corpora (95.2%), permissively licensed code (2.1%), academic publications (1.5%), and Wikipedia (0.1%).
• Stage 2 (Dolmino Mix 1124): A late-phase, high-quality/interleaved curriculum (5–10% FLOPs) up-samples instruction and math-reasoning sources. For the 7B model’s 50B-token pass: 47.2% high-quality web, 16.6% decontaminated FLAN, 2.45% StackExchange, 5.85% academic, 7.11% Wikipedia, 20.8% synthetic/math. Larger 13B model passes (100B, 300B) repeat FLAN accordingly.

Model checkpoints from different random seeds are often averaged (“model soups”), consistently outperforming individual runs. This staged, annealed training uniquely enhances downstream performance, especially on instruction-following and math reasoning (OLMo et al., 2024).

3. Instruction-Tuning Pipeline: SFT, DPO, RLVR

Instruction tuning of OLMo 2-Instruct follows a Tülu 3–inspired, fully open, three-phase protocol:

• 3A. Supervised Finetuning (SFT): Conducted on a 940K-prompt mix (“tulu-3-sft-olmo-2” plus PersonaHub, WildChat, and curated sets), strictly filtered for permissive licensing. Hyperparameters: AdamW, batch size ≈ 256, LR $\vartheta$7, 2 epochs.
• 3B. Direct Preference Optimization (DPO): Preference data combines on-policy (SFT-sampled) and off-policy (20 open LLMs) prompts, labeled by GPT-4o. The objective directly optimizes the pairwise preference across response pairs:

$\vartheta$8

DPO is critical and, as shown, is the chief driver of diversity loss (Peeperkorn et al., 28 Jul 2025).

• 3C. Reinforcement Learning with Verifiable Rewards (RLVR): Final PPO-based RL step uses automatic, verifiable task-specific reward functions (e.g. GSM8K/MATH exact match) to directly optimize behaviors relevant to specific benchmarks. The baseline is a reward model trained on preference data; the KL-penalty term keeps the policy close to the DPO reference.

RLVR is staged (mixed data, then GSM8K, then MATH) and requires no human labelers, aligning policy behaviors with verifiable objectives (OLMo et al., 2024).

4. Output Diversity, Conformative Decoding, and the Diversity Gap

Instruction tuning is shown to compress model output diversity, with the largest losses occurring during DPO. For OLMo 2 7B, metrics such as lexical Vendi Score (VS) and semantic truncated entropy (TE) show the following losses (Peeperkorn et al., 28 Jul 2025):

Stage	VS (lexical)	TE (semantic)
Base	≈ 160	≈ 8.0
after SFT	–10–15%	–
after DPO	–15–20%	–
after RLVR	<5% change	–

DPO alone induces roughly two-thirds of the shift from base to instruct model diversity. Precision is largely preserved, while recall (coverage of reference responses) and the Mauve score (quality-diversity frontier) drop.

Conformative decoding interpolates next-token logits between instruct ($\vartheta$9) and base ($10^3$0) models:

$10^3$1

For OLMo 2, setting $10^3$2 increases VS and TE by 5–10%, recall by 5–8 ppt, and Mauve by 0.03–0.05, without compromising precision. This recovers a substantial fraction of diversity loss post-DPO without sacrificing completion quality (Peeperkorn et al., 28 Jul 2025).

5. In-Context Learning–Instruction Following Trade-Offs via Partial Adaptation

OLMo 2-Instruct displays a marked trade-off between in-context few-shot learning (ICL) and instruction following, which can be tuned continuously via the Partial Adaptation (PAd) method (İrsoy et al., 15 Apr 2025). Let $10^3$3 and $10^3$4 be the base and instruct parameter tensors, and define $10^3$5. PAd constructs:

$10^3$6

Sweeping $10^3$7 interpolates between base (0) and full instruct (1):

Model	Base/Inst ICL	Best ICL (%)	$10^3$8	$10^3$9WinRate (%)
OLMo 2 7B 1124 Instruct	55.7 / 55.4	57.9	0.5	–9.95
OLMo 2 13B 1124 Instruct	60.2 / 61.1	61.5	0.75	–4.43

For OLMo 2-7B, $5 \times 10^5$0 yields a $5 \times 10^5$12.5 pt ICL boost but a $5 \times 10^5$210 pt AlpacaEval win-rate drop; for 13B, $5 \times 10^5$3 produces a 1.4 pt ICL gain at a 4 pt win-rate loss. Larger models tolerate lower $5 \times 10^5$4 (e.g., $5 \times 10^5$5–$5 \times 10^5$6) with minimal wr loss yet gain $5 \times 10^5$7–$5 \times 10^5$8 pts ICL.

Deployment is guided by:

• For critical ICL, select $5 \times 10^5$9–$Q$0
• For maximal instruction following, fix $Q$1 or $Q$2
• To preserve a maximum win-rate drop $Q$3, select maximal $Q$4 that maintains $Q$5

Partial Adaptation provides a zero-cost, post-hoc control for end-users to adjust the trade-off between structured ICL performance and conversational utility (İrsoy et al., 15 Apr 2025).

6. Syntactic-Template Reliance, Domain Shortcuts, and Safety

OLMo 2-Instruct is empirically shown to develop spurious correlations between prompt syntactic templates (frequent PoS-tag sequences) and domain targets, especially in entity knowledge tasks (Shaib et al., 25 Sep 2025). This manifests as high in-domain accuracy (Exact, Synonym, Antonym perturbations $Q$6–$Q$7) but severe cross-domain degradation (mean $Q$8 for instruct variants 1B–13B, Exact/Synonym/Antonym).

For OLMo 2-7B Instruct:

Setting	In-Domain Acc	Cross-Domain Acc	ΔAcc
Exact	0.92	0.51	–0.41
Synonym	0.91	0.43	–0.48
Antonym	0.90	0.49	–0.41
Disfluent	0.22	0.27	+0.05
Paraphrase	0.75	0.63	–0.12

Such shortcuts can bypass alignment: a cross-domain syntactic template (e.g. chain-of-thought) appended to an unsafe query can reduce OLMo-2-7B’s refusal rate from 40% to 2.5% in 1,000 requests.

Mitigation strategies:

• Explicitly test for syntax-domain reliance using benchmarked $Q$9 metrics.
• Ensure intra-domain syntactic diversity via synthetic paraphrases and template interleaving.
• Incorporate invariant risk minimization–style objectives during tuning.

These findings indicate that even 13B-scale OLMo 2-Instruct models are vulnerable to such domain shortcuts, impacting both factual generalization and safety (Shaib et al., 25 Sep 2025).

7. Release, Reproducibility, and Ecosystem Position

OLMo 2-Instruct is fully open: models (both base and Instruct, 7B and 13B), curated pretraining and post-training datasets, training and evaluation code, and logs are released under Apache 2.0 (where feasible). Downloadable checkpoints and artifacts are available via HuggingFace and the AllenAI Playground (OLMo et al., 2024). The models achieve state-of-the-art results on OLMES, GSM8K, MMLU, and BigBenchHard given their scale and pretraining FLOPs, and are competitive with open-weight only counterparts (Qwen 2.5, Llama 3.1, Gemma 2).

The transparent open-source release cycle and availability of fine-grained artifacts—including intermediate post-training checkpoints and model soups—make OLMo 2-Instruct a standard reference for research on instruction tuning, curriculum learning, and open dataset/model governance.

---

References:

• (OLMo et al., 2024) 2 OLMo 2 Furious
• (İrsoy et al., 15 Apr 2025) Improving Instruct Models for Free: A Study on Partial Adaptation
• (Peeperkorn et al., 28 Jul 2025) Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned LLMs
• (Shaib et al., 25 Sep 2025) Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in LLMs