---
title: 'Mellum 2: 12B MoE Code & Reasoning Model'
url: https://www.emergentmind.com/topics/mellum-2
type: topic
---

# Mellum 2: 12B MoE Code & Reasoning Model

Mellum 2 is an open-weight, 12-billion parameter Mixture-of-Experts (MoE) language model, architected for high-performance software engineering tasks including code generation, editing, debugging, multi-step reasoning, agentic tool use, and conversational programming. As the successor to the completion-focused 4B dense Mellum model, Mellum 2 integrates sparse MoE computation, advanced attention mechanisms, long-context handling, and multi-objective training strategies to balance compute efficiency and frontier performance across code and reasoning benchmarks. Released under the Apache 2.0 license alongside its base, instruct, and thinking checkpoints, Mellum 2 represents a systematically ablated design for latency-constrained, GPU-serving environments [2605.31268].

## 1. Model Architecture

Mellum 2 employs a decoder-only Transformer architecture with substantial innovations in parameter sparsity and attention.

- **Parameterization and Sparsity**: The model comprises $P_{\rm total}\approx12\times10^9$ parameters, implemented as an MoE with $E=64$ experts per layer and $A=8$ active experts per token via top-8 gating. Thus, the effective parameters per token are $P_{\rm active} \approx 2.5\times10^9$, satisfying
  $$
  P_{\rm active} = A \times P_{\rm expert} + P_{\rm non\text{-}MoE}
  $$
  with $P_{\rm expert}\approx P_{\rm total}/E$. Intermediate expert size is 896, and no always-on expert is used.

- **Attention Mechanisms**:
  - *Grouped-Query Attention (GQA)*: Queries partitioned into 32 groups, each mapped to 4 shared key/value projections, yielding reduced KV-cache memory traffic and ∼20% sustained throughput improvement versus 8-head attention under high concurrency, without loss of softmax fidelity.
  - *Sliding-Window Attention (SWA)*: In each block of four layers, three utilize local windows ($W=1024$ tokens), cutting self-attention FLOPs by ≈75% to $O(Wd)$, while one retains full $O(Ld)$ global context.
  - *Multi-Token Prediction (MTP) Head*: An auxiliary Transformer layer predicts $t+1$ tokens with a loss weight $\alpha=0.1$, removed at inference but leveraged for speculative decoding acceleration, and shown to boost HumanEval pass@1 (+10.4pp) and MMLU (+3.6pp) in ablation with a 7% training overhead.

- **Hyperparameter Summary**:

| Component         | Value                              | Notes                              |
|-------------------|------------------------------------|------------------------------------|
| Backbone          | 28 layers, hidden size 2304        |                                    |
| Attention         | 32 query heads, 4 KV heads, dim 128| RoPE $\theta=500,000$              |
| Sliding Window    | 1024 tokens, 3/4 layers            |                                    |
| MoE               | 64 experts, top-8 routing          | Expert size 896                    |
| MTP               | 1 layer, $\alpha=0.1$              |                                    |
| Vocab/Context     | 98,304 tokens, 8,192 max base      |                                    |

## 2. Pre-training Pipeline

The pre-training regime utilizes a three-phase curriculum across ≈10.65 trillion tokens, shifting data composition toward greater code and mathematical content.

- **Data Curriculum** (see table):

| Phase                  | Tokens (T) | % Run  | Web % | Code % | Math % |
|------------------------|-----------:|-------:|------:|-------:|-------:|
| Foundation Building    |    6.18    |   58   |   70  |   23   |     6  |
| Quality Uplift         |    2.79    |   26   |   44  |   42   |    14  |
| Capability Sharpening  |    1.69    |   16   |   23  |   59   |    18  |
| **Total**              |  10.65     |  100   |       |        |        |

  - The code ratio increases from 23% to 59% over phases.
  - Fill-in-the-Middle (FIM) objective: 50% of samples in Phases 1/3, 10% in Phase 2, with a code emphasis in Phase 3.

- **Optimization Regimen**:
  - *Optimizer*: Muon (Moonlight configuration; per-layer orthogonal updates + AdamW on embeddings/output).
  - *Precision*: BF16 base, FP8 tensorwise, FP32 gradient reduction.
  - *Learning Rate*: Warmup-Hold-Decay ($\eta_{\rm peak}=3\times10^{-4}$, 2,000 steps warmup, ≈49,306 steps linear decay). Batch ramps 2,048→4,096 sequences ($\approx$33.6M tokens/step), gradient clip 1.0, weight decay 0.1.

## 3. Context Extension and Post-training

Mellum 2 is extended for ultra-long contexts and post-trained with both supervision and RL.

- **128K Context via Layer-selective YaRN**:
  - YaRN frequency remapping is selectively applied to global/full-attention layers only, with local layers retaining original RoPE. This achieves 128K–131K token contexts with RULER scores at 64K: 0.64 (layer-selective YaRN), outperforming 0.52 (uniform θ-bump) and 0.33 (baseline).

- **Supervised Fine-Tuning (SFT)**:
  - Both "Instruct" and "Thinking" variants are SFT at full 131,072 context length for 3 epochs. The "Instruct" model produces direct answers, discarding reasoning, with loss on all assistant outputs. The "Thinking" variant emits chain-of-thought ("<think>…</think>") before answers, applies loss only to the final turn plus reasoning, and filters conversations accordingly.
  - Shared hyperparameters: peak LR $3\times10^{-5}$, cosine decay to $3\times10^{-6}$, MoE auxiliary loss $10^{-4}$, batch size 64 packs ($\approx$8.4M tokens/step), BF16+FP8 precision.

- **Reinforcement Learning with Verifiable Rewards (RLVR)**:
  - Policy: GRPO (with leave-one-out baseline), token-level PPO clipping $[1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}}]$ with asymmetric clip ($\epsilon_{\text{low}}=0.2$, $\epsilon_{\text{high}}=0.28$), IcePop truncation for train/infer balance.
  - Two separate RLVR phases for Instruction and Thinking, each over ≈260K prompts covering code, math, tool use, instruction, reasoning, knowledge. Output length budgets: ∼16,384 (Instruct), ∼40,960 (Thinking). Reward shaping uses DAPO length penalty and unwanted reasoning concision penalty.

## 4. Performance Across Benchmarks and Efficiency

Mellum 2 is evaluated in comparison to open-weight baselines in the 4B–14B range, emphasizing compute efficiency and benchmark competitiveness.

- **Benchmark Results** (select highlights):

| Benchmark          | Mellum 2 Base | Qwen2.5-7B | OLMo-3-7B | Qwen3-4B |
|--------------------|--------------:|-----------:|----------:|---------:|
| HumanEval pass@1   | 41.5%         | 55.5%      | 45.1%     | 57.3%    |
| MMLU               | 70.9%         | 71.8%      | 62.1%     | 71.1%    |
| GSM8K              | 81.7%         | 81.9%      | 73.5%     | 82.0%    |
| BBH                | 74.9%         | 69.0%      | 63.6%     | 71.3%    |
| MBPP               | 62.4%         | 63.6%      | 50.6%     | 67.0%    |

  - After RLVR, "Instruct" and "Thinking" checkpoints provide task-specialized output, with notable results on EvalPlus (78.4%), LiveCodeBench (69.9%), MMLU-Redux (78.1%), and JetBrains pairwise win rate (68.1%) for Instruct.

- **Inference Efficiency**:
  - On a single NVIDIA H100 (80GB) leveraging vLLM and FP8 quantization at ISL=2304, OSL=256:
    - Single-request latency: 192 tokens/s (Mellum 2), matching Qwen2.5-7B (193 tokens/s).
    - Concurrency throughput: Mellum 2 achieves 5,179 tokens/s (≈21% increase over Qwen2.5-7B at 4,286 tokens/s).
  - Thus, Mellum 2 achieves accuracy of a 7B–12B model at the compute profile of a 2.5B dense model.

## 5. Design Rationale and Trade-offs

All architecture and training choices underwent empirical ablation with inference efficiency on commodity GPUs as a constraint.

- **Density vs. Sparsity**: 5.5B dense configurations (with latent attention) matched Qwen2.5-7B for speed but not quality. 12B MoE with 8/64 routing preserved 2.5B active FLOPs latency.
- **Attention**: 4 KV heads chosen to balance quality and serve-at-scale throughput; more heads (e.g., 8) degraded throughput, fewer hurt quality.
- **SWA Pattern**: The 3:1 local vs. full attention (window=1024) reduces steady-state self-attention compute by ∼50% while maintaining global context retention.
- **MTP Head**: +7% training time delivered substantial (+3–10pp) gains on key benchmarks, no negative effect on next-token loss convergence.
- **MoE Routing Stability**: A global-batch auxiliary loss ($10^{-3}$) ensured expert balance; dropless routing mitigated capacity drops from 15% overhead to negligible by training end.
- **Data Stability**: Fewer than 82 unique tokens were filtered; data-shard hash sorting validated with no adverse effects; Megatron cluster size changes only transiently affected balancing-loss magnitude.

*This systematic approach establishes Mellum 2 as a robust model for real-time, large-scale code assistance and program reasoning on commodity hardware* [2605.31268].

Source: https://www.emergentmind.com/topics/mellum-2