DeepSeek-R1-Distill-Qwen Model

• The paper demonstrates effective knowledge transfer by distilling reasoning skills from a 671B-parameter DeepSeek-R1 teacher model into smaller Qwen models ranging from 1.5B to 32B parameters.
• It applies a multi-loss strategy combining cross-entropy, KL-divergence, and auxiliary losses, enhanced with skill-aware data selection and trajectory-aware token filtering for robust reasoning performance.
• Reinforcement learning and tool-augmented approaches further boost benchmark accuracy on tasks like mathematical problem solving, achieving significant industrial scalability and efficiency improvements.

DeepSeek-R1-Distill-Qwen models comprise a series of efficient, reasoning-specialized LLMs produced by distilling DeepSeek-R1, a powerful RL-trained teacher, into the Qwen family of decoder-only Transformers. These models were developed to transfer complex reasoning abilities—such as those required for mathematical problem solving, code synthesis, and logic tasks—from the very large and computationally expensive DeepSeek-R1 model (up to 671B parameters) into smaller, deployable students spanning 1.5B to 32B parameters. The resulting models, by combining supervised fine-tuning, targeted data selection, trajectory-aware objectives, and, in some cases, reinforcement learning, deliver substantial reasoning performance gains while offering practical improvements in efficiency, deployment, and industrial scalability.

1. Architectural Foundation and Distillation Protocols

The DeepSeek-R1-Distill-Qwen series encompasses student models of 1.5B, 3B, 7B, 8B, 14B, and 32B parameters, all based on Qwen2.5 family architectures. The canonical backbone is a decoder-only Transformer (e.g., Qwen2.5-32B: 40 layers, hidden size 8192, 32768-dim feedforward, 64 attention heads). Models are initialized from Qwen2.5-Instruct checkpoints and adapted for Chain-of-Thought (CoT) generation by extending the output head to handle reasoning tokens, with minor architectural modifications such as a verifier head (in slow-thinking variants, ~0.1% overhead) (Cai et al., 3 Nov 2025).

Distillation is performed using DeepSeek-R1 as the teacher model. Knowledge transfer uses combinations of three types of loss:

• Next-token cross-entropy ($L_{CE}$): Standard supervised imitation of teacher-generated traces.
• KL-divergence regularization ($L_{KL}$): Alignment of student output logits towards teacher soft targets with temperature scaling (e.g., $\tau=2.0$).
• Auxiliary losses ($L_{aux}$): Ranking or contrastive losses on CoT variants, e.g., to encourage preference for medium-difficulty chains.

Losses are aggregated via $$L_{total} = \alpha_1 L_{CE} + \alpha_2 L_{KL} + \alpha_3 L_{aux}$$, with typical coefficients $\alpha_1=1.0$, $\alpha_2=0.5$, $\alpha_3=0.1$. Training employs AdamW optimizer, batch sizes up to 256 (via data-parallelism), cosine decay learning rates, and 3–5 epochs of data (Cai et al., 3 Nov 2025).

For smaller models or in simpler SFT regimes, only cross-entropy on teacher outputs is used without soft targets; e.g., DeepSeek-R1-Distill-Qwen-32B in (Zhao et al., 25 Mar 2025) is trained exclusively via supervised fine-tuning on 1.4 million teacher-distilled reasoning exemplars.

2. Data Selection, Preprocessing, and Skill-Aware Distillation

While initial distillation efforts relied on large-scale, teacher-generated reasoning chains via rejection sampling, recent work demonstrates the effectiveness of skill-centric, data-efficient protocols. In the skill-aware approach (Zhang et al., 15 Jan 2026), a hierarchical skill taxonomy organizes mathematical problems by root, intermediate, and leaf skills. Each problem in a 100K DeepSeek-R1–produced QA corpus is attributed one or more leaf-level skills via prompting a LLM.

Student models (e.g., Qwen3-8B) are profiled to obtain per-skill accuracy $\mathrm{acc}_k$. Sampling probability for new SFT data is set proportional to the (clipped) inverse per-skill accuracy,

$$w_k = \mathrm{clip}\!\bigl(\mathrm{acc}_k^{-1},\,0,\,w_{\max}\bigr), \qquad P(k) = \frac{w_k}{\sum_j w_j}$$

enriching the fine-tuning batch with cases corresponding to the model's weakest competencies.

Skill-aware fine-tuning is achieved by prepending each example with its associated skill chain, explicitly signaling the decomposition of problem structure and enabling reasoning trace alignment:

$L_{KL}$0

where $L_{KL}$1 is the input, $L_{KL}$2 contains the ordered skill-chain tokens, and $L_{KL}$3 is the teacher-provided output.

Empirically, with only 1,000 strongly-selected data points, Qwen3-8B recoups and even surpasses baseline performance, achieving +1.4pp accuracy over random SFT and avoiding the degradation observed when fine-tuning on all 100K teacher traces (Zhang et al., 15 Jan 2026).

3. Reinforcement Learning, Reward Distillation, and Tool Augmentation

The DeepSeek-R1-Distill-Qwen family incorporates reinforcement learning (RL) in several capacities. Vanilla RL (proximal policy optimization, PPO) optimizes expected reward for correct and formatted answers, with dynamic KL penalties to maintain output diversity (Chen et al., 6 Mar 2025). For distilled reward models, an additional regression head is trained on teacher-computed Reasoning Verbosity (RV) and Cognitive Difficulty (CD), guiding subsequent RL fine-tuning with shaped rewards:

$L_{KL}$4

where $L_{KL}$5 and $L_{KL}$6 penalize deviation from optimal reasoning trajectories (Cai et al., 3 Nov 2025).

Tool manipulation extends the reasoning pipeline: models are fine-tuned with traces containing interleaved code cells, and a runtime controller executes code blocks (marked by special tokens), looping results back into the prompt. Notably, code-enabled reasoning raises AIME 2024 accuracy from 60% (standard DeepSeek-R1-Distill-Qwen-32B) to 86.67% (tool-augmented student) (Chen et al., 6 Mar 2025).

4. Training-Trajectory-Aware and Negative-Signal-Inclusive Objectives

Recent research underscores critical pitfalls in naïve continual distillation. Masking "Imitation-Anchor Tokens" (i.e., surface-pattern tokens that quickly reach high confidence during training) from the loss and focusing updates on "yet-to-learn tokens" substantially improves reasoning transfer. In T3S (Training-Trajectory-Aware Token Selection), tokens are partitioned by per-token confidence shift in the training trajectory, and the AR loss is reconstructed to ignore anchor tokens:

$L_{KL}$7

allowing focus on latent, high-difficulty reasoning (Shen et al., 15 Jan 2026). With only 200–1,000 examples, T3S-distilled Qwen3-8B surpasses DeepSeek-R1 on AIME24/AIME25 (80.63/73.96% accuracy; teacher: 79.8/70.00%).

Exploiting negative reasoning traces, as in REDI (Reinforcement Distillation), further advances offline reasoning transfer. REDI jointly maximizes correct-trace likelihood and minimizes incorrect-trace likelihood (without a reference model), enabling effective learning from both accepted and rejected teacher outputs:

$L_{KL}$8

On MATH-500, Qwen-REDI-1.5B (trained on 131K open examples) matches or exceeds DeepSeek-R1-Distill-Qwen-1.5B (trained on 800K proprietary positives) (Xu et al., 30 May 2025).

5. Evaluation, Scaling, Benchmarks, and Application Efficacy

The DeepSeek-R1-Distill-Qwen suite exhibits systematic gains in reasoning-specific and general benchmarks across model sizes.

Mathematical/logic tasks: Performance scales with parameter size and focused training. For example, DistilQwen2.5-7B-R1 (RL and slow-thinking) achieves 43.3% on AIME2024 vs. 10.0% for Qwen2.5-7B-Inst; the corresponding 32B model attains 70.0% (AIME2024), 93.8% (MATH-500), outpacing Llama-70B on several tasks (Cai et al., 3 Nov 2025, Zhao et al., 16 Feb 2025).

General benchmarks: On the A-Eval-2.0 suite (text understanding, information extraction, text generation, logical reasoning, task planning), DeepSeek-R1-Distill-Qwen-32B achieves B/A+ tier scores—significant reasoning gains, with some regressions on certain non-reasoning subtasks (Zhao et al., 16 Feb 2025).

Data scaling and diversity: Outperformance is linked to both dataset size and diversity. Distillation on 1.4M high-quality, reasoning-verified traces via simple SFT yields an average of 71.6% across four high-difficulty tasks; further improvements arise from careful category stratification and skill-targeted selection (Zhao et al., 25 Mar 2025). Conversely, naïve large-scale SFT can degrade specialized reasoning performance, justifying data efficiency protocols.

RL and reward model integration: RL-optimized students regularly improve on top of distillation alone. For example, Qwen2.5-7B, after reward-model-guided GRPO, reaches 81.4% (MATH-500), up from 73.6% (Cai et al., 3 Nov 2025).

Multilingual and inference scaling: Distilled models such as DeepSeek-R1-Distill-Qwen-7B manifest "test-time scaling"—reasoning performance rises with CoT generation length. However, scaling is erratic for low-resource languages, causing mid-generation language switches; tailored prefix-tuning (MITT) using high-resource initial thoughts eliminates such inconsistencies and boosts accuracy in both high- and low-resource settings (Bajpai et al., 21 May 2025).

6. Security, Adversarial Robustness, and Deployment

DeepSeek-R1-Distill-Qwen models are susceptible to excessive-reasoning attacks. Short, optimized suffixes appended to inputs trigger order-of-magnitude increases in reasoning trace length (from 169 to 1,531 tokens per query on Qwen), inflating inference latency and energy consumption (11.9s → 42.4s, 2,535J → 8,188J on GSM8K) without sacrificing accuracy (Si et al., 17 Jun 2025). These attacks utilize a composite loss:

• Priority Cross-Entropy (PCE) for key-token overemphasis,
• Excessive Reasoning (ER) for encouraging repeated branching,
• Delayed Termination (DT) for deferral of EOS/EOT.

Defense recommendations include early exit detectors, suffix-pattern filters, budgeted decoding, and loss-trajectory monitoring.

Industrial-scale deployment is realized through Alibaba Cloud PAI, supporting batch/streaming inference, prompt chaining, auto-scaling, and integration with MLOps suites, while providing RESTful APIs and elastic GPU orchestration (Cai et al., 3 Nov 2025).

7. Insights, Limitations, and Future Directions

Skill-aware and trajectory-aware techniques have shown that, contrary to earlier intuition, most distillation benefit arises from addressing skill-specific weaknesses, deferred or bottlenecked tokens, and negative feedback, rather than from sheer data scale (Zhang et al., 15 Jan 2026, Shen et al., 15 Jan 2026, Xu et al., 30 May 2025).

Nonetheless, current DeepSeek-R1-Distill-Qwen implementations exhibit pitfalls:

• Performance regressions on non-reasoning benchmarks and simple text tasks after reasoning-centric distillation (Zhao et al., 16 Feb 2025).
• Vulnerability to overthinking and adversarial inference time inflation (Si et al., 17 Jun 2025).
• Opaque or proprietary training details (e.g., batch sizes, LR schedules, 4-bit quantization protocols) in certain model reports (Zhao et al., 16 Feb 2025, Zhao et al., 25 Mar 2025).

Proposed future work spans formal theorem-proving for output verification, full training workflow publication, true soft-label (KL) distillation, and robust defenses against computational denial-of-service attacks.

The DeepSeek-R1-Distill-Qwen family thus represents a technically sophisticated pipeline for compressing and deploying high-performing reasoning models, integrating advances in data efficiency, optimization, security, and multilingual reasoning, with verifiable impact on both research benchmarks and large-scale industrial deployment.