DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Abstract: General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by LLMs and chain-of-thought prompting, have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models' capabilities are still insufficient for more complex problems. Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models.

• The paper's main contribution is the introduction of a reinforcement learning framework that directly improves LLM reasoning without relying on supervised fine-tuning.
• It details a multi-stage training process using Group Relative Policy Optimization, demonstrating enhanced reasoning accuracy and extended response times.
• Evaluations show that distilled models derived from DeepSeek-R1 deliver efficiency and performance comparable to larger, established reasoning models.

DeepSeek-R1: Harnessing Reinforcement Learning to Enhance LLM Reasoning Capabilities

Introduction

LLMs have dramatically improved in their ability to process and generate human-like language, approaching features of AGI. A key challenge remains in enhancing their reasoning capabilities, particularly in test-time scenarios where model scalability is an issue. Previous attempts using approaches like supervised fine-tuning (SFT) and search algorithms failed to match the reasoning prowess of models such as OpenAI's o1 series. The paper "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" introduces DeepSeek-R1, a generation of reasoning models that leverage reinforcement learning (RL) for training without the prior requirement of supervised fine-tuning.

Reinforcement Learning Model

DeepSeek-R1-Zero, the base model, is developed using pure RL techniques, bypassing the typical SFT steps. This model inherently acquires reasoning abilities via an elaborate reinforcement learning process. The focus on RL facilitates the development of sophisticated reasoning behaviors, albeit with certain challenges such as readability and language consistency. These limitations prompted further refinement, resulting in the creation of DeepSeek-R1.

Benchmark performance of DeepSeek-R1 is illustrated below.

Figure 1: Benchmark performance of DeepSeek-R1.

Implementation Approach

DeepSeek-R1-Zero was subjected to a reinforcement learning framework using Group Relative Policy Optimization (GRPO), which omits the need for a critic model, instead estimating baselines from group scores and optimizing policy models effectively. This strategy markedly enhanced reasoning performance.

To mitigate limitations seen in DeepSeek-R1-Zero, the researchers introduced a multi-stage training process in DeepSeek-R1. Cold-start data was used initially, followed by reasoning-oriented RL using specific methodologies to align model output with human preferences.

Throughout the training process, DeepSeek-R1-Zero exhibited a trajectory of improvement in accuracy as shown below.

Figure 2: AIME accuracy of DeepSeek-R1-Zero during training. For each question, 16 responses are sampled and the average accuracy is calculated.

Moreover, DeepSeek-R1-Zero's RL training process also demonstrated advancements in the model's inherent capabilities to allocate enhanced computation time for reasoning tasks, depicted in the subsequent figure.

Figure 3: The average response length of DeepSeek-R1-Zero on the training set during the RL process. DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time.

Evaluation and Distillation

Performance evaluation on educational and factual benchmarks shows DeepSeek-R1 achieves parity with OpenAI's models in various reasoning tasks. This solidifies the notion that enhancing LLMs with RL frameworks could bridge existing performance gaps. The open-sourcing of DeepSeek-R1 and its distilled variants generates new opportunities for utilizing smaller yet powerful models in practice.

Comparative Analysis

The paper offers insights into the effectiveness of distilling reasoning patterns into smaller models. Through experiments with several dense models like Qwen and Llama, it showcased that distilled models not only outperform non-reasoning focused models but also are more efficient when subjected to extensive RL.

Future Work and Conclusion

The study suggests future efforts could focus on broader capability expansion of models like DeepSeek-R1, addressing language consistency issues, and exploring optimization in software engineering tasks. The emphasis remains on leveraging reinforcement learning to enhance reasoning capabilities in LLMs, which is a promising avenue for subsequent research in AI. This research underlines the potential of RL in creating more robust, reasoning-capable LLMs, suggesting an exciting direction for further advancements in AI technologies.