Mellum2 Technical Report

Abstract: We present Mellum 2, an open-weight 12B-parameter Mixture-of-Experts (MoE) LLM with 2.5B active parameters per token. Mellum 2 is a general-purpose LLM specialized in software engineering, spanning code generation and editing, debugging, multi-step reasoning, tool use and function calling, agentic coding, and conversational programming assistance, and it is the successor to the completion-focused 4B dense Mellum model. The architecture builds on the Mixture-of-Experts (64 experts, 8 active) and combines Grouped-Query Attention with 4 KV heads, Sliding Window Attention on three of every four layers, and a single Multi-Token Prediction head that doubles as both an auxiliary pre-training objective and a built-in draft model for speculative decoding; each choice was validated by ablation with inference efficiency on commodity GPUs as a design constraint. Pre-training spans approximately 10.6 trillion tokens through a three-phase curriculum that progressively shifts the mixture from diverse web data toward curated code and mathematical content, optimized with Muon under FP8 hybrid precision and a Warmup-Hold-Decay schedule with linear decay to zero. The pre-trained base is extended to a 128K context window via a layer-selective YaRN and then post-trained in two stages (supervised fine-tuning followed by RLVR), yielding two released variants: an Instruct model that answers directly and a Thinking model that emits an explicit reasoning trace before its final answer. Across code generation, math and reasoning, tool use, knowledge, and safety benchmarks, Mellum 2 is competitive with open-weight baselines in the 4B-14B range while running at the per-token compute of a 2.5B dense model. We release the base, instruct, and thinking checkpoints, together with this report on the architecture decisions, data pipeline, and training recipe behind them, under the Apache 2.0 license.

• The paper introduces Mellum 2, a 12B-parameter MoE LLM that optimizes software tasks with only 2.5B active parameters, balancing performance and efficiency.
• It details innovative design choices such as Grouped-Query Attention with 4 KV heads, sliding window attention, and a multi-token prediction head to improve throughput and quality.
• Extensive ablation studies and empirical results demonstrate competitive coding benchmarks and long-context handling, underscoring its practical deployment on commodity hardware.

Mellum 2: An Open-Weight 12B MoE LLM for Software Engineering

Model Motivation and Positioning

Mellum 2 is introduced as a 12B-parameter MoE LLM targeting software engineering tasks, with only 2.5B active parameters per token. It extends beyond completion to code generation, editing, debugging, multi-step reasoning, tool use, function calling, agentic workflows, and conversational programming. The model is positioned to fill the gap between small dense models (4–14B, efficient but plateau on complex reasoning/coding) and very large MoEs (frontier quality, high deployment cost), providing sufficient parameter scale but sparse per-token compute for practical deployment on modern commodity hardware.

Architecture Overview and Design Tradeoffs

Mellum 2’s architecture is a Qwen3-MoE variant with 64 experts (8 active per token), using Grouped-Query Attention (GQA) with 4 KV heads, a 3:1 pattern of Sliding Window Attention (SWA, window size 1024), and a Multi-Token Prediction (MTP) head serving both as auxiliary during training and a speculative decoding draft. Each architectural choice was supported by ablation studies under strict latency/throughput constraints.

Key architectural highlights:

• Sparse MoE Backbone: 12B total parameters, 2.5B active per token (8/64 experts). Ablations show that reducing active experts below 8 gains latency at significant cost to quality; 8 yields the optimal balance.
• GQA with 4 KV Heads: Minimizes throughput bottlenecks from key-value cache blowup in high-concurrency settings without major losses in performance. Increasing to 8 heads degrades throughput; reducing to 2 heads reduces quality.
• Sliding Window Attention: SWA is applied to 3/4 attention layers, which reduces latency for long input sequences while global attention layers preserve long-range reasoning ability. Window size 1024 maximizes benchmark quality.
• Multi-Token Prediction Head: Training with next-token plus MTP (α=0.1, 1 step ahead) yields improvements on HumanEval (+10.4), GSM8K (+3.0), MMLU (+3.6), and others, with negligible negative side-effects and 7% increase in training time.

Figure 1: Throughput mode evaluation. Mellum 2 surpasses Qwen2.5-7B and Qwen3-8B in throughput while matching single-request latency.

Pre-Training Strategy and Data Curation

The model is pre-trained on 10.65T tokens, organized into a curriculum shifting from web data toward curated code/math. The data mixture is dynamically managed across three phases:

• Phase 1 (Foundation, 6.2T, 58%): Emphasizes web (70%), with 23% code, 6% math.
• Phase 2 (Quality Uplift, 2.8T, 26%): Upsamples code (42%) and curated instruction/QA datasets.
• Phase 3 (Capability Sharpening, 1.7T, 16%): Code is 59%; heavily upsampled synthetic and high-quality data, minimal web.

A fill-in-the-middle (FIM) objective is intermittently deployed, crucial for code completion applications. Aggressive multi-stage quality filtering, classifier-driven web quality stratification, and deduplication ensure high calibration and minimize memorization of degenerate or low-utility samples.

Robust ablation-driven development also extends to data repetition: high-quality and synthetic datasets are repeated up to 4×, informed by observed expert specialization benefits in MoE training.

Training Infrastructure and Optimization

Training leverages state-of-the-art approaches:

• Muon Optimizer (Distributed, Moonlight variant, momentum 0.95, Newton–Schulz 5 iter, extra scale 0.2): Outperforms AdamW in loss on both dense and MoE without instabilities at large scale.
• BF16+FP8 Hybrid Precision: Maximizes throughput and resource utilization on H200 clusters.
• Dropless MoE Routing: Avoids expert capacity limits and token dropping, resulting in a stable global load-balancing loss as router adaptation occurs, observed empirically in training.
• Offline-then-streamed Data Pipeline: Ensures high reproducibility and modular dataset management across distributed training.

  Figure 2: Latency comparison for MoE models with sliding window attention, highlighting competitive throughput even at increased context lengths.

Long-Context Extension

Mellum 2 supports context lengths up to 128K via selective YaRN positional scaling applied only to global-attention layers. Ablations demonstrate that:

• Layer-selective YaRN outperforms uniform scaling or naive RoPE base increases at all tested context lengths.
• Training beyond 30B tokens for long-context adaptation exhibits diminishing returns in retrieval-oriented benchmarks; further training primarily equilibrates MoE routing—for more efficient expert utilization at extreme context lengths.

  Figure 3: RULER long-context retrieval scores under various scaling recipes, showcasing the superior context adaptation of the layer-selective method.

  

  Figure 4: RULER score convergence during the long-context extension phase, indicating rapid plateauing post-initial adaptation.

  

  Figure 5: Global MoE load-balancing loss declining during long-context adaptation, evidencing router dynamics stabilization.

Post-Training: SFT and RL

The pre-trained checkpoint is post-trained via supervised fine-tuning (SFT) and RL with verifiable rewards (RLVR):

• SFT produces two variants: ‘Instruct’ (direct answers, no explicit reasoning) and ‘Thinking’ (chain-of-thought emitted before final answer). These differ in target/loss masking and treatment of reasoning annotations.
• SFT data hazards: Self-identification oversampling addresses initial hallucination of upstream model origins.
• RL employs an asynchronous GRPO variant: With token-level updates, leave-one-out baselines, IcePop importance calibration to address non-determinism from MoE router stochasticity, and reward shaping (overlong and concision penalties). Tasks include code, math, tool use, instruction, and reasoning, all verifiable. The RL cluster splits training/inference roles for optimal throughput.

  Figure 6: RL training and validation accuracy for the Instruct RL run, demonstrating stable performance improvements across capability domains.

Numerical Results: Quality and Efficiency

Pre-training: Mellum 2 matches or outperforms 7B dense LLMs in math/reasoning (e.g., 81.7 GSM8K, 70.9 MMLU, 74.9 BBH, 62.4 MBPP, 45.4 CRUXEval-I) despite being 2.5B active parameters per token.

Post-training (SFT/RL):

• On function-level coding (EvalPlus): 78.4 SFT+RL, beating Seed-Coder-8B, Qwen3.5-9B.
• On competitive programming (LiveCodeBench): Thinking variant achieves 75.1, leading all baselines.
• On tool use (BFCL v4): RL thinking variant leads at 45.6.
• Math: RL thinking variant achieves 87.0 on GSM-Plus and 58.4 AIME.
• Safety: SFT model is safest in its class (8.4% HarmBench harmful rate).
• JetBrains-internal pairwise tests: RL thinking leads at 69.5% win rate over Qwen2.5-7B.

Efficiency: Mellum 2 matches Qwen2.5-7B in single-token latency (192–193 t/s) but is 21% faster under high concurrency, and 79% faster than Qwen3-8B.

Figure 7: Output tokens/s comparison on H100 hardware reveals Mellum 2’s throughput advantage relative to dense baselines, owing to architectural sparsity and optimized attention/routing.

Implications and Future Work

Mellum 2 demonstrates that, at modestly small scale (12B total, 2.5B active), judicious MoE allocation plus inference-conscious design decisions can yield models highly competitive with much larger dense LLMs on core software and reasoning tasks—while remaining feasible for commodity deployment (single H100).

Theoretical implications include:

• Confirmation that long-context transfer in MoE architectures benefits from targeted context extension only where necessary (selective YaRN).
• Evidence that strong multi-task, multi-domain performance is achievable with limited per-token compute, contingent on careful data curation, curriculum, and router dynamics.

Practically:

• Mellum 2 is poised for direct deployment in developer IDEs/applications.
• Release of base, instruct, and thinking checkpoints under a permissive license—plus an open recipe for reproducible, inference-aware small MoEs—enables further community experimentation.

Next steps include extending RL to real-world repository-scale SWE tasks, increased RL environment heterogeneity, and revisiting long-context mid-curriculum design for enhanced coverage. Kernel, framework, and hybrid architecture advances are also expected to further shrink efficiency gaps, potentially enabling even larger-scale MoE deployments within constrained inference budgets.

Conclusion

Mellum 2 provides a rigorously-optimized, open-weight MoE LLM for software engineering, achieving high-quality, cost-efficient, and extensible performance. The research systematically ablates design axes under strict deployment constraints, empirically demonstrating that modern MoE stacks—when carefully matched with data, training infrastructure, and post-training pipelines—can deliver practical, high-quality assistants with frontier-level reasoning and coding skills at commodity scale.