--- title: 'OLMo 2-Instruct Models: Architecture & Tuning' url: https://www.emergentmind.com/topics/olmo-2-instruct-models type: topic --- # OLMo 2-Instruct Models: Architecture & Tuning OLMo 2-Instruct models are a family of fully open, large-scale language models built upon the OLMo 2 dense autoregressive Transformer architecture, optimized for both upstream efficiency and downstream performance through a rigorous multi-phase pretraining and post-training curriculum. These models implement architectural enhancements, novel data mixtures, and advanced post-training methodologies—including Direct Preference Optimization (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR)—all focused on maximizing instruction-following ability while maintaining flexibility for diverse use cases. The OLMo 2-Instruct release is notable for its comprehensive transparency, with all models (7B and 13B parameters), training data, code, and recipes made openly available, positioning it as a central resource in the open LLM ecosystem [2501.00656]. ## 1. Architectural Innovations and Training Stability OLMo 2-Instruct inherits a standard decoder-only Transformer but introduces substantial refinements over prior OLMo iterations. Noteworthy features include: - **Bias Free Design and SwiGLU Activations:** All linear layers omit bias terms. The feedforward hidden dimension is set to approximately $\frac{8}{3}d$ (rounded to a multiple of 128), coupled with the SwiGLU activation to enhance expressivity. - **Rotary Positional Embeddings:** Rotational positional embeddings (RoPE) employ a frequency range parameter $\vartheta$ increased from $10^3$ to $5 \times 10^5$, improving the handling of longer contexts. - **RMSNorm Normalization:** Layer normalization is replaced with RMSNorm (no bias), applied to output residuals after each attention and MLP block, rather than to the inputs. - **QK-norm for Attention Stability:** Pre-attention queries ($Q$) and keys ($K$) are separately normalized with RMSNorm prior to dot-product computation, mitigating early training instability. - **Z-loss Regularization:** A supplementary term, $L = L_{\mathrm{CE}} + \lambda(\log Z)^2$ ($\lambda=10^{-5}$, $Z$ the softmax partition function), regularizes logits and stabilizes optimization. Model scales: - 7B: 32 layers, $d_{\text{model}}=4096$, 32 heads ($\approx 7 \times 10^9$ params) - 13B: 40 layers, $d_{\text{model}}=5120$, 40 heads ($\approx 1.3 \times 10^{10}$ params) The pretraining FLOP investment is approximately $1.8 \times 10^{23}$ for 7B (4T tokens) and $4.6 \times 10^{23}$ for 13B (5T tokens) [2501.00656]. ## 2. Pretraining and Data Curriculum OLMo 2-Instruct employs a two-stage curriculum pretraining strategy explicitly designed to shape generalization and capability: - **Stage 1 (OLMo 2 Mix 1124):** ($\approx$90–95 % FLOPs, $\sim$3.9T tokens). Domain composition includes baseline web corpora (95.2%), permissively licensed code (2.1%), academic publications (1.5%), and Wikipedia (0.1%). - **Stage 2 (Dolmino Mix 1124):** A late-phase, high-quality/interleaved curriculum (5–10% FLOPs) up-samples instruction and math-reasoning sources. For the 7B model’s 50B-token pass: 47.2% high-quality web, 16.6% decontaminated FLAN, 2.45% StackExchange, 5.85% academic, 7.11% Wikipedia, 20.8% synthetic/math. Larger 13B model passes (100B, 300B) repeat FLAN accordingly. Model checkpoints from different random seeds are often averaged (“model soups”), consistently outperforming individual runs. This staged, annealed training uniquely enhances downstream performance, especially on instruction-following and math reasoning [2501.00656]. ## 3. Instruction-Tuning Pipeline: SFT, DPO, RLVR Instruction tuning of OLMo 2-Instruct follows a Tülu 3–inspired, fully open, three-phase protocol: - **3A. Supervised Finetuning (SFT):** Conducted on a 940K-prompt mix (“tulu-3-sft-olmo-2” plus PersonaHub, WildChat, and curated sets), strictly filtered for permissive licensing. Hyperparameters: AdamW, batch size ≈ 256, LR $1 \times 10^{-5}$, 2 epochs. - **3B. Direct Preference Optimization (DPO):** Preference data combines on-policy (SFT-sampled) and off-policy (20 open LLMs) prompts, labeled by GPT-4o. The objective directly optimizes the pairwise preference across response pairs: $$ \min_\theta \sum_{\langle \pi_A,\pi_B \rangle} -\log \sigma\big( \log \pi_\theta(A|x) - \log \pi_\theta(B|x) \big) $$ DPO is critical and, as shown, is the chief driver of diversity loss [2507.20956]. - **3C. Reinforcement Learning with Verifiable Rewards (RLVR):** Final PPO-based RL step uses automatic, verifiable task-specific reward functions (e.g. GSM8K/MATH exact match) to directly optimize behaviors relevant to specific benchmarks. The baseline is a reward model trained on preference data; the KL-penalty term keeps the policy close to the DPO reference. RLVR is staged (mixed data, then GSM8K, then MATH) and requires no human labelers, aligning policy behaviors with verifiable objectives [2501.00656]. ## 4. Output Diversity, Conformative Decoding, and the Diversity Gap Instruction tuning is shown to compress model output diversity, with the largest losses occurring during DPO. For OLMo 2 7B, metrics such as lexical Vendi Score (VS) and semantic truncated entropy (TE) show the following losses [2507.20956]: | Stage | VS (lexical) | TE (semantic) | |-----------|--------------|--------------| | Base | ≈ 160 | ≈ 8.0 | | after SFT | –10–15% | – | | after DPO | –15–20% | – | | after RLVR| <5% change | – | DPO alone induces roughly two-thirds of the shift from base to instruct model diversity. Precision is largely preserved, while recall (coverage of reference responses) and the Mauve score (quality-diversity frontier) drop. **Conformative decoding** interpolates next-token logits between instruct ($\theta$) and base ($\phi$) models: $$ \text{ConformativeLogits}(x_t|x_{