DeepSeekCoderV2: Open-Source MoE Code Model

• The paper introduces a state-of-the-art Mixture-of-Experts architecture using top-2 expert routing, optimizing parameter efficiency and predictive accuracy on multiple coding benchmarks.
• DeepSeekCoderV2 is a transformer model designed for code intelligence with extensive language support, context windows up to 128K tokens, and both supervised and reinforcement alignment protocols.
• Fine-tuning via QLoRA enables domain-specific semantic fault localization, significantly reducing debugging time and achieving statistically significant performance gains.

DeepSeekCoderV2 is a family of open-source, Mixture-of-Experts (MoE) transformer LLMs designed for code intelligence, code generation, mathematical reasoning, and repository-scale understanding. As a continued pretraining and expansion of the DeepSeek series, DeepSeekCoderV2 achieves performance competitive with or exceeding leading proprietary models (e.g., GPT-4-Turbo, Claude 3 Opus, Gemini 1.5 Pro) on a spectrum of coding and general AI benchmarks. The architecture supports extensive language coverage across 338 programming languages, with industry-scale context windows up to 128,000 tokens, and advanced alignment through supervised and reinforcement learning. Fine-tuning protocols, including QLoRA, enable robust adaptation for domain-specific semantic reasoning and software engineering workflows.

1. Model Architecture and Scaling

DeepSeekCoderV2 utilizes a transformer-based backbone with interleaved sparse expert MLP layers forming a Mixture-of-Experts (MoE) framework. The design implements a token-level expert routing mechanism: each token is dynamically routed via a gating network $g(x) = \text{softmax}(W_g x + b_g)$ to its top-2 experts within each MoE layer, activating only a subset of parameters per inference step. Two principal configurations are described:

Variant	Total Params	Active Params	Transformer Blocks	Hidden Dim	Attention Heads	MoE Experts/layer
Lite	16B	2.4B	40	5,120	40	8
Standard	236B	21B	72	12,288	96	16

Relative to DeepSeekCoder-33B, the V2 Standard model doubles the depth and width, expands the expert count per layer from 8 to 16, and stabilizes training with conventional LayerNorm in place of previous exponential variants. Only the top-2 experts per token are active, optimizing parameter efficiency and memory usage, while maintaining state-of-the-art predictive accuracy (DeepSeek-AI et al., 17 Jun 2024).

2. Pretraining Corpus and Objectives

The pretraining corpus comprises 10.2 trillion tokens:

• 4.2T from DeepSeek-V2 (natural language, code, math),
• 6.0T newly collected for DeepSeekCoderV2:
  • 60% source code (1.17T tokens) from GitHub, CommonCrawl (338 languages),
  • 10% math text (221B tokens),
  • 30% additional natural language.

Language and code content are filtered and deduplicated through fastText-based classification and multi-round web crawling. Pretraining objectives include standard next-token prediction (NTP) for all model variants, and a fill-in-the-middle (FIM) permutation span modeling objective at a 50% rate for DeepSeekCoderV2-Lite. Positional interpolation via YARN enables robust context window expansion to 128K tokens, with staged sequence length extension confirmed on “Needle in a Haystack” benchmarks (DeepSeek-AI et al., 17 Jun 2024).

3. Supervised Fine-tuning and Alignment

Alignment is achieved through a two-stage supervised and reinforcement learning pipeline:

• Supervised fine-tuning (SFT) utilizes 300M tokens of instructions (20K code, 30K math, and sampled general data) with a learning rate of $5 \times 10^{-6}$, cosine scheduling, and large batch sizes.
• Reinforcement learning via Group Relative Policy Optimization (GRPO) leverages a reward model trained on compiler-verified correctness and human preferences, optimizing on ~40K prompts without an explicit critic.

This protocol aims to align code completion, reasoning, and language usage with practical user expectations and correctness incentives (DeepSeek-AI et al., 17 Jun 2024).

4. QLoRA Fine-Tuning for Semantic Behavior Localization

For domain-specific semantic tasks, such as commit-level fine-grained behavior localization in software repositories, DeepSeekCoderV2 has been fine-tuned using QLoRA (Quantized Low-Rank Adapter LoRA) (Wang et al., 24 Nov 2025). The procedure introduces low-rank matrices $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times d}$ to frozen base weights $W_0 \in \mathbb{R}^{d \times d}$, yielding effective weights $W = W_0 + BA$. Only $A$ and $B$ are updated during fine-tuning. Training minimizes cross-entropy loss over the “bisect_mark” output token (good/bad), with no further regularization or consensus loss.

Weakly supervised annotation over a corpus of 1,000 curated code-diff pairs is employed. Auto-labeling by the base LLM is filtered by softmax confidence (≥0.8) and self-consistency among few-shot samples, with manual audit (“correct-and-commit”) reducing error rates by approximately 35%. No fixed train/validation/test split is reported for this corpus. This process enables robust semantic fault localization under noisy, non-monotonic, or flaky testing conditions.

5. Prompt Engineering and Temporal Reasoning

Prompting mechanisms are structured around commit-diff presentation and rich chain-of-thought (CoT) reasoning (Wang et al., 24 Nov 2025). The model receives per-step diffs annotated by line type (addition, deletion, relocation) and is required to populate a fixed JSON schema including:

• target_behavior (free text),
• has_compile_error (bool),
• behavior_change (categorical),
• sem_edits (categorized changes with semantic flags and likelihoods),
• counterfactual_fix (free text),
• reasoning_chain (list of reasoning steps),
• reflection (free text),
• bisect_mark (scored output: good | bad).

Only the bisect_mark is directly optimized, but intermediate fields—especially the reasoning_chain—steer latent reasoning activations. Self-consistency filtering during fine-tuning further refines label quality, but no additional regularization or consensus loss is applied beyond initial filtering.

6. Evaluation Results and Empirical Performance

Benchmarking on code intelligence and reasoning tasks demonstrates the following:

• HumanEval: 90.2% (DeepSeekCoderV2-Instruct, 236B)
• MBPP: 76.2%
• RepoBench (Python/Java): up to 43.3% (Lite), 46.1% (CodeStral 22B baseline)
• Defects4J/SWE-Bench/Aider: 73.7% (Aider), competitive with GPT-4-Turbo on other code repair tasks

For semantic fault localization, QLoRA-tuned DeepSeekCoderV2 achieves an absolute success rate gain of +6.4 points (from 74.2% to 80.6%) on 32 git-bisect runs, with paired Wilcoxon signed-rank test indicating statistical significance ($p < 0.01$). Mean debugging time is reduced by 45.6% in a developer paper, and up to 2× reduction in average bisect sequence time is observed. No explicit ablation studies on prompt structure or self-consistency filtering are provided (Wang et al., 24 Nov 2025).

7. Language Coverage, Context Handling, and Use Cases

DeepSeekCoderV2 supports 338 programming languages (expanded from 86), facilitated through comprehensive code crawling and classification. The context window is extended from 16K to 128K tokens using YARN positional interpolation. These capabilities enable project-scale code completion, documentation understanding, and cross-file reasoning.

Practical applications encompass repository-scale autocompletion (RepoBench), competitive programming assistance (LiveCodeBench, USACO), automated bug patch generation, and interactive chat agents specializing in code and mathematics. The “active parameter” efficiency of 2.4B (Lite) or 21B (Std) matches or exceeds much larger dense baselines, optimizing deployment costs and resource utilization (DeepSeek-AI et al., 17 Jun 2024).

---

DeepSeekCoderV2 establishes a new open-source benchmark for large-scale code-aware language modeling, demonstrating extensibility in both alignment-focused instruction tuning and domain-specific behavior localization (DeepSeek-AI et al., 17 Jun 2024, Wang et al., 24 Nov 2025).