DeepScaleR-1.5B Preview: Efficient Reasoning Model

• DeepScaleR-1.5B-Preview is a 1.5-billion parameter reasoning model employing chain-of-thought strategies to solve mathematical and logical challenges.
• It uses History-Aware Policy Optimization (HAPO) with a cosine-scaled length reward combined with a correctness reward to encourage concise yet accurate responses.
• Empirical evaluations show a 29–34% reduction in response length with only a modest 3–7% accuracy trade-off on math benchmarks, enhancing overall efficiency.

DeepScaleR-1.5B-Preview is a 1.5-billion parameter reasoning LLM trained using reinforcement learning with an emphasis on both response correctness and reasoning conciseness. Developed in the context of ongoing research into more efficient and controllable LLMs, it incorporates advanced reward shaping mechanisms rooted in history-aware signal design to optimize for both accuracy and brevity, with demonstrated improvements in resource usage and competitive performance across math reasoning benchmarks.

1. Model Design and Training Objectives

DeepScaleR-1.5B-Preview adopts a distilled 1.5B-parameter architecture tailored for chain-of-thought (CoT) reasoning, designed to solve mathematical and logical problems through explicit stepwise deliberation. The defining feature of DeepScaleR-1.5B-Preview’s training is the application of History-Aware Policy Optimization (HAPO), a reinforcement learning (RL) framework that tracks, for each task instance, the smallest response length at which the correct answer has previously been produced (Huang et al., 16 May 2025). The model’s reward function is therefore a composite of two terms:

• Correctness reward: rewards a response if its extracted answer matches the ground truth.
• Length reward: leverages the historical shortest correct solution, incentivizing correct responses that are more concise than any previously observed answer.

The length reward is formally defined using a cosine-based scaling:

$$f(\lvert y_i^{(j)} \rvert, h_i) = \cos\left(\min\left(\frac{\pi}{2} \cdot \frac{\lvert y_i^{(j)} \rvert}{h_i}, \pi\right)\right)$$

where $|y_i^{(j)}|$ is the output token count for candidate response $j$ on input $x_i$, and $h_i$ is the minimum token count of any prior correct response for $x_i$. The overall reward $r_i^{(j)}$ for sample $y_i^{(j)}$ combines correctness with the weighted length reward.

This approach ensures the model progressively learns to solve increasingly challenging problems using ever-shorter reasoning chains, promoting both efficiency and correctness.

2. Role of History-Aware Policy Optimization (HAPO)

HAPO introduces a query-specific historical state for each example in the training data, capturing the minimum-length successful solution seen so far (Huang et al., 16 May 2025). During training, the reward assigned to a new candidate response reflects not only its correctness but also its efficiency relative to this historical benchmark.

A key aspect of HAPO’s length reward formulation is its graduated penalty structure for incorrect or longer responses. If the current response is correct and shorter than $h_i$, it receives a high reward; if it is longer, the reward diminishes but remains positive. Incorrect responses are not excessively penalized for brevity, thereby supporting exploration of concise reasoning, which helps avoid converging prematurely to verbose but correct solutions.

The model’s reward function, parameterized by a weight $w$, enables explicit trade-offs between brevity and accuracy as determined by the practitioner.

3. Training Protocol and Resource Considerations

The training procedure for DeepScaleR-1.5B-Preview involves sampling diverse responses for each prompt and updating the history state after each batch (Huang et al., 16 May 2025). The historical minimum length is iteratively lowered as more concise correct solutions are discovered. The reward signal is normalized within each batch, and hyperparameters such as the clipping threshold $c$ and weight $w$ are carefully tuned to maintain stability and balance between accuracy and length optimization.

This methodology is designed to operate efficiently over large datasets and is robust to batch-level reward scaling. In distributed setups, the maintenance and synchronization of historical states at scale are necessary practical considerations.

4. Reasoning Performance and Efficiency

Experimental results on prominent math benchmarks (e.g., GSM8K, MATH500, AIME2024) indicate that DeepScaleR-1.5B-Preview, when trained with HAPO, achieves substantial reductions in average response length—ranging from approximately 29% to 34%—with only modest declines in problem-solving accuracy (roughly 3–7%) (Huang et al., 16 May 2025).

Analyses of training dynamics reveal an ongoing mutual reinforcement: as the average response length decreases due to the model's pursuit of more concise solutions, the minimum historical length $h_i$ is also lowered, further incentivizing succinctness without sacrificing accuracy. This suggests the model is capable of “pruning” its own reasoning chains to remove unnecessary steps.

A plausible implication is that, compared to previous methods relying solely on fixed length constraints or batch-only query-level optimization, HAPO’s mechanism of leveraging longitudinal history enables more persistent and generalizable improvements in generating concise reasoning.

5. Comparative Analysis with Contemporary Methods

Traditional techniques for controlling reasoning efficiency include imposing universal token budgets (i.e., hard length caps) or optimizing length only with respect to the best answer within the current batch. In contrast, DeepScaleR-1.5B-Preview’s HAPO framework enables it to iteratively surpass all previously observed solutions in terms of brevity, effectively creating a curriculum that tightens over time (Huang et al., 16 May 2025).

Empirical comparisons show that while universal budget constraints often degrade accuracy if brevity is prioritized, and batch-level optimization yields only incremental gains, HAPO achieves a superior balance—maintaining higher accuracy at much lower token usage.

6. Challenges and Optimization Strategies

Key challenges in training DeepScaleR-1.5B-Preview with HAPO include maintaining stability in reward normalization (the batchwise re-scaling can counteract length incentives) and tuning the weight $w$ of the length reward to avoid excessive brevity at the expense of correctness. The selection of the clipping threshold $c$ in the reward function also plays a critical role in ensuring correct answers are always prioritized over incorrect yet very short answers.

The need for exploration is addressed by not harshly penalizing short but incorrect attempts, allowing the model room to experiment and ultimately converge on optimal concise forms.

7. Implications and Applications

The central insight of DeepScaleR-1.5B-Preview’s training regime is that reinforcement learning with a history-aware dual objective can systematically produce LLMs that reason both accurately and efficiently. This is particularly valuable in computational environments constrained by token budgets or serving cost-sensitive applications that require consistently concise outputs (e.g., mobile deployment, batch processing).

Furthermore, the ability to induce concise chain-of-thought reasoning may facilitate better model interpretability and ease of downstream verification, as extraneous or redundant reasoning steps are minimized without significant loss of task accuracy.

The framework established by DeepScaleR-1.5B-Preview and HAPO is directly extendable to future reasoning models, particularly those aimed at high-stakes or cost-constrained settings, and can inform ongoing research into curriculum RL strategies and advanced reward shaping for LLM optimization.