Efficient Reasoning via Reward Model (2511.09158v1)

Abstract: Reinforcement learning with verifiable rewards (RLVR) has been shown to enhance the reasoning capabilities of LLMs, enabling the development of large reasoning models (LRMs). However, LRMs such as DeepSeek-R1 and OpenAI o1 often generate verbose responses containing redundant or irrelevant reasoning step-a phenomenon known as overthinking-which substantially increases computational costs. Prior efforts to mitigate this issue commonly incorporate length penalties into the reward function, but we find they frequently suffer from two critical issues: length collapse and training collapse, resulting in sub-optimal performance. To address them, we propose a pipeline for training a Conciseness Reward Model (CRM) that scores the conciseness of reasoning path. Additionally, we introduce a novel reward formulation named Conciseness Reward Function (CRF) with explicit dependency between the outcome reward and conciseness score, thereby fostering both more effective and more efficient reasoning. From a theoretical standpoint, we demonstrate the superiority of the new reward from the perspective of variance reduction and improved convergence properties. Besides, on the practical side, extensive experiments on five mathematical benchmark datasets demonstrate the method's effectiveness and token efficiency, which achieves an 8.1% accuracy improvement and a 19.9% reduction in response token length on Qwen2.5-7B. Furthermore, the method generalizes well to other LLMs including Llama and Mistral. The implementation code and datasets are publicly available for reproduction: https://anonymous.4open.science/r/CRM.

• The paper proposes a CRM+CRF framework that rewards concise reasoning only if the answer is correct.
• It employs an LLM-augmented conciseness model with annealing and difficulty penalties to stabilize training and prevent reward hacking.
• Empirical results show up to 8.1% accuracy improvement and 19.9% token reduction across various backbones on mathematical benchmarks.

Efficient Reasoning via Reward Model: Technical Summary and Analysis

Introduction and Problem Statement

Recent developments in Reinforcement Learning with Verifiable Rewards (RLVR) have strengthened the mathematical reasoning ability of LLMs, producing Large Reasoning Models (LRMs) such as DeepSeek-R1 and OpenAI o1. However, these RLVR-trained models demonstrate a persistent overthinking issue: they tend to generate unnecessarily verbose reasoning paths with repetitive or irrelevant steps, incurring substantial computational overhead due to increased output length. Attempts to address this with length penalties in the reward function typically lead either to length collapse—where the model is incentivized to produce trivial, non-informative outputs—and training collapse, where both logical quality and conciseness degrade.

This paper introduces a two-component framework designed to mitigate these collapse failures: (1) a Conciseness Reward Model (CRM) to quantify the quality of reasoning conciseness, and (2) a Conciseness Reward Function (CRF) that fuses the correctness and conciseness reward signals with explicit dependency, ensuring conciseness is only rewarded when the reasoning is correct. The overall objective is to optimize for models that maintain high accuracy while minimizing token usage in reasoning paths, thus achieving more efficient reasoning without sacrificing correctness or logical rigor.

Framework Overview

The proposed approach augments RLVR with a reward model pipeline for conciseness and integrates this via a tailored reward structure. The system comprises two main components:

1. Conciseness Reward Model (CRM): Trained to measure the conciseness of a solution, accounting for redundancy, irrelevance, and token efficiency.
2. Conciseness Reward Function (CRF): Introduces explicit dependency—only correct reasoning paths receive additional reward based on conciseness—and includes an annealing mechanism for training stability and a difficulty-adaptive penalty.

   Figure 1: An overview of the proposed framework taking GRPO as an example.

Conciseness Reward Model Construction

Training a reward model to reflect human preferences typically involves RLHF, but the resource-intensive nature and domain-expert inconsistency make this infeasible at large scale. This work leverages LLM-based data augmentation for scalable labeling:

• Data Source: The DeepMath-103K dataset is selected for diversity and difficulty coverage.
• Augmentation: For each question, two distinct reasoning paths are generated: one concise (higher quality), one redundant (lower quality).
• Scoring: Qwen2.5-72B-Instruct assigns a conciseness score between 0.1 and 1, considering factors such as repetition avoidance, step relevance, and token efficiency.
• Filtering: Only pairs differentiable by conciseness score are retained, yielding a 65,878-sample dataset.
• Supervised Fine-Tuning: The CRM is trained using Qwen2.5-3B-Instruct on this data.

Reward Function Design (CRF)

A naive weighted sum of correctness and conciseness is vulnerable: models optimize for short outputs at the expense of reasoning quality (length collapse/training collapse). The authors instead propose:

$$\hat{R}_i = R^o_i \cdot \left[ 1 + \alpha \cdot c_i \cdot (s + d_q) \right]$$

• $R^o_i$ is a binary correctness reward.
• $c_i$ is the conciseness score from CRM.
• $\alpha$ scales the influence of conciseness.
• $s$ is an annealing coefficient $\exp(-\frac{step}{T})$ to control exploration-exploitation trade-off during training.
• $d_q$ is a difficulty coefficient (higher penalty for easier questions).

This explicit dependency means conciseness only contributes for correct answers, addressing reward hacking.

Theoretical Analysis

Two propositions underpin the CRF’s optimization properties:

1. Variance Reduction: Assuming positive correlation between outcome reward and conciseness, the variance of the policy gradient estimator is strictly reduced compared to the standard objective. This addresses instability caused by binary, sparse reward signals:

$$\text{Var}[\nabla_\theta J(\theta)] \leq (1-\eta)\,\text{Var}[\nabla_\theta J_0(\theta)]$$

2. Improved Convergence Constants: Thanks to reduced gradient estimation variance, the stochastic optimization with suitably chosen learning rates achieves $O(1/\sqrt{T})$ convergence with improved constants, enhancing RL stability and sample efficiency.

Empirical Results and Ablation

The framework is evaluated on five mathematical benchmarks—MATH-500, AIME2025, OlympiadBench, AMC-23, and GPQA—across Qwen2.5-7B, Llama3.1-8B, and Mistral-7B-v0.1 backbones. Key metrics are Pass@$k$ (accuracy) and average output token count (efficiency).

Figure 2: Training curves of outcome reward and average number of tokens of the generated reasoning path on (a) Qwen2.5-7B, (b) Llama3.1-8B, and (c) Mistral-7B-v0.1

Results

• On Qwen2.5-7B, CRF achieves 8.1% accuracy improvement and 19.9% token reduction versus the original GRPO with binary reward.
• Competing length-penalized reward schemes (Cosine, Kimi 1.5) substantially reduce token count but incur significant accuracy loss—as high as 14.2% (Cosine) and 8.7% (Kimi) on average.
• ARM, which utilizes "short CoT mode," does improve efficiency but at the cost of severely increased output verbosity (up to 75.1% more tokens).

In ablation, explicit dependency between correctness and conciseness was critical; weighted-sum variants suffered a 9.8% reduction in accuracy. Both annealing and difficulty-adaptive penalties further contributed to stable and efficient optimization.

Generalization

CRF is compatible with other RL algorithms (DAPO, etc.) and multiple LLM backbones, with consistent gains in accuracy-efficiency trade-off. The method is robust to the cold-start problem observed in Llama3.1-8B and Mistral-7B, where direct RL on base models fails absent initial SFT.

Case Study

Qualitative analysis confirms that CRF-trained models avoid rote answer regurgitation (seen in Cosine reward) and redundant reasoning (seen in standard GRPO), achieving short, logically structured solutions matching expected mathematical rigor.

Practical Implications

The proposed CRM+CRF framework offers several practical advances:

• Enables deployment of LLMs for mathematical and deductive tasks with strict inference compute budgets.
• Reduced model output length mitigates API latency and memory overhead in real-world applications.
• Compatible with plug-and-play reward structures for different domains beyond mathematics, where concise reasoning is necessary (e.g., scientific QA, legal reasoning).

Theoretically, the approach stands as a robust RLVR paradigm, with improved sample efficiency and convergence characteristics, while sidestepping common reward hacking and collapse pitfalls.

Future Directions

Potential avenues for extension include:

• Scaling CRM to multi-domain reasoning beyond mathematics.
• Integrating human-in-the-loop feedback for outlier cases where conciseness scoring may misalign with domain norms.
• Systematic paper of trade-offs between reasoning diversity and minimum-length outputs to avoid pathological shortcuts in highly-compressed composed reasoning.
• Investigating CRM architectures that jointly model correctness, conciseness, and other qualitative properties (e.g., persuasiveness, transparency).

Conclusion

This work systematically diagnoses and resolves common collapse pathologies in RLVR-trained LLMs tasked with mathematical reasoning. Through the construction of an LLM-augmented conciseness reward model and a dependency-aware reward function, the proposed method achieves an improved trade-off between reasoning accuracy and efficiency. CRF’s explicit structure facilitates both practical deployment and theoretical understanding, and experimental evidence demonstrates superior performance and compatibility across datasets and architectures.