Mellum 2: 12B MoE Code & Reasoning Model

• Mellum 2 is a 12B parameter MoE language model featuring 64 experts and top-8 gating, designed for high-performance code generation, debugging, and reasoning.
• It incorporates advanced techniques like Grouped-Query and Sliding-Window Attention to optimize memory use and speed while retaining global context.
• The model is pretrained on 10.65 trillion tokens with a curriculum focused on code and math, achieving competitive benchmark scores on tasks such as HumanEval and MMLU.

Mellum 2 is an open-weight, 12-billion parameter Mixture-of-Experts (MoE) LLM, architected for high-performance software engineering tasks including code generation, editing, debugging, multi-step reasoning, agentic tool use, and conversational programming. As the successor to the completion-focused 4B dense Mellum model, Mellum 2 integrates sparse MoE computation, advanced attention mechanisms, long-context handling, and multi-objective training strategies to balance compute efficiency and frontier performance across code and reasoning benchmarks. Released under the Apache 2.0 license alongside its base, instruct, and thinking checkpoints, Mellum 2 represents a systematically ablated design for latency-constrained, GPU-serving environments (Kojic et al., 29 May 2026).

1. Model Architecture

Mellum 2 employs a decoder-only Transformer architecture with substantial innovations in parameter sparsity and attention.

• Parameterization and Sparsity: The model comprises $P_{\rm total}\approx12\times10^9$ parameters, implemented as an MoE with $E=64$ experts per layer and $A=8$ active experts per token via top-8 gating. Thus, the effective parameters per token are $P_{\rm active} \approx 2.5\times10^9$, satisfying

$$P_{\rm active} = A \times P_{\rm expert} + P_{\rm non\text{-}MoE}$$

with $P_{\rm expert}\approx P_{\rm total}/E$. Intermediate expert size is 896, and no always-on expert is used.

• Attention Mechanisms:
  • Grouped-Query Attention (GQA): Queries partitioned into 32 groups, each mapped to 4 shared key/value projections, yielding reduced KV-cache memory traffic and ∼20% sustained throughput improvement versus 8-head attention under high concurrency, without loss of softmax fidelity.
  • Sliding-Window Attention (SWA): In each block of four layers, three utilize local windows ($W=1024$ tokens), cutting self-attention FLOPs by ≈75% to $O(Wd)$, while one retains full $O(Ld)$ global context.
  • Multi-Token Prediction (MTP) Head: An auxiliary Transformer layer predicts $t+1$ tokens with a loss weight $E=64$0, removed at inference but leveraged for speculative decoding acceleration, and shown to boost HumanEval pass@1 (+10.4pp) and MMLU (+3.6pp) in ablation with a 7% training overhead.
• Hyperparameter Summary:

Component	Value	Notes
Backbone	28 layers, hidden size 2304
Attention	32 query heads, 4 KV heads, dim 128	RoPE $E=64$1
Sliding Window	1024 tokens, 3/4 layers
MoE	64 experts, top-8 routing	Expert size 896
MTP	1 layer, $E=64$2
Vocab/Context	98,304 tokens, 8,192 max base

2. Pre-training Pipeline

The pre-training regime utilizes a three-phase curriculum across ≈10.65 trillion tokens, shifting data composition toward greater code and mathematical content.

• Data Curriculum (see table):

Phase	Tokens (T)	% Run	Web %	Code %	Math %
Foundation Building	6.18	58	70	23	6
Quality Uplift	2.79	26	44	42	14
Capability Sharpening	1.69	16	23	59	18
Total	10.65	100

• The code ratio increases from 23% to 59% over phases.
• Fill-in-the-Middle (FIM) objective: 50% of samples in Phases 1/3, 10% in Phase 2, with a code emphasis in Phase 3.
  • Optimization Regimen:
• Optimizer: Muon (Moonlight configuration; per-layer orthogonal updates + AdamW on embeddings/output).
• Precision: BF16 base, FP8 tensorwise, FP32 gradient reduction.
• Learning Rate: Warmup-Hold-Decay ($E=64$3, 2,000 steps warmup, ≈49,306 steps linear decay). Batch ramps 2,048→4,096 sequences ($E=64$433.6M tokens/step), gradient clip 1.0, weight decay 0.1.

3. Context Extension and Post-training

Mellum 2 is extended for ultra-long contexts and post-trained with both supervision and RL.

• 128K Context via Layer-selective YaRN:
  • YaRN frequency remapping is selectively applied to global/full-attention layers only, with local layers retaining original RoPE. This achieves 128K–131K token contexts with RULER scores at 64K: 0.64 (layer-selective YaRN), outperforming 0.52 (uniform θ-bump) and 0.33 (baseline).
• Supervised Fine-Tuning (SFT):
  • Both "Instruct" and "Thinking" variants are SFT at full 131,072 context length for 3 epochs. The "Instruct" model produces direct answers, discarding reasoning, with loss on all assistant outputs. The "Thinking" variant emits chain-of-thought ("> …") before answers, applies loss only to the final turn plus reasoning, and filters conversations accordingly.
  • Shared hyperparameters: peak LR $E=64$5, cosine decay to $E=64$6, MoE auxiliary loss $E=64$7, batch size 64 packs ($E=64$88.4M tokens/step), BF16+FP8 precision.
• Reinforcement Learning with Verifiable Rewards (RLVR):
  • Policy: GRPO (with leave-one-out baseline), token-level PPO clipping $E=64$9 with asymmetric clip ($A=8$0, $A=8$1), IcePop truncation for train/infer balance.
  • Two separate RLVR phases for Instruction and Thinking, each over ≈260K prompts covering code, math, tool use, instruction, reasoning, knowledge. Output length budgets: ∼16,384 (Instruct), ∼40,960 (Thinking). Reward shaping uses DAPO length penalty and unwanted reasoning concision penalty.

4. Performance Across Benchmarks and Efficiency

Mellum 2 is evaluated in comparison to open-weight baselines in the 4B–14B range, emphasizing compute efficiency and benchmark competitiveness.

• Benchmark Results (select highlights):

Benchmark	Mellum 2 Base	Qwen2.5-7B	OLMo-3-7B	Qwen3-4B
HumanEval pass@1	41.5%	55.5%	45.1%	57.3%
MMLU	70.9%	71.8%	62.1%	71.1%
GSM8K	81.7%	81.9%	73.5%	82.0%
BBH	74.9%	69.0%	63.6%	71.3%
MBPP	62.4%	63.6%	50.6%	67.0%

• After RLVR, "Instruct" and "Thinking" checkpoints provide task-specialized output, with notable results on EvalPlus (78.4%), LiveCodeBench (69.9%), MMLU-Redux (78.1%), and JetBrains pairwise win rate (68.1%) for Instruct.
  • Inference Efficiency:
• On a single NVIDIA H100 (80GB) leveraging vLLM and FP8 quantization at ISL=2304, OSL=256:
• Single-request latency: 192 tokens/s (Mellum 2), matching Qwen2.5-7B (193 tokens/s).
• Concurrency throughput: Mellum 2 achieves 5,179 tokens/s (≈21% increase over Qwen2.5-7B at 4,286 tokens/s).
• Thus, Mellum 2 achieves accuracy of a 7B–12B model at the compute profile of a 2.5B dense model.

5. Design Rationale and Trade-offs

All architecture and training choices underwent empirical ablation with inference efficiency on commodity GPUs as a constraint.

• Density vs. Sparsity: 5.5B dense configurations (with latent attention) matched Qwen2.5-7B for speed but not quality. 12B MoE with 8/64 routing preserved 2.5B active FLOPs latency.
• Attention: 4 KV heads chosen to balance quality and serve-at-scale throughput; more heads (e.g., 8) degraded throughput, fewer hurt quality.
• SWA Pattern: The 3:1 local vs. full attention (window=1024) reduces steady-state self-attention compute by ∼50% while maintaining global context retention.
• MTP Head: +7% training time delivered substantial (+3–10pp) gains on key benchmarks, no negative effect on next-token loss convergence.
• MoE Routing Stability: A global-batch auxiliary loss ($A=8$2) ensured expert balance; dropless routing mitigated capacity drops from 15% overhead to negligible by training end.
• Data Stability: Fewer than 82 unique tokens were filtered; data-shard hash sorting validated with no adverse effects; Megatron cluster size changes only transiently affected balancing-loss magnitude.

This systematic approach establishes Mellum 2 as a robust model for real-time, large-scale code assistance and program reasoning on commodity hardware (Kojic et al., 29 May 2026).