DRER: Dynamic Reasoning Efficiency Reward

• DRER is a reinforcement learning reward framework that assigns fine-grained, token-level rewards to improve LLM formal reasoning and computational efficiency.
• It integrates dynamic length regulation with reasoning quality assessments to produce concise, interpretable chain-of-thought outputs.
• Empirical evaluations reveal significant gains in accuracy and a 75% reduction in token consumption on complex multi-hop deductive reasoning benchmarks.

Dynamic Reasoning Efficiency Reward (DRER) is a plug-and-play reinforcement learning reward framework devised to enhance the formal reasoning ability, interpretability, and computational efficiency of LLMs by tightly coupling reward attribution to the practical contribution and efficiency of intermediate chain-of-thought (CoT) tokens. DRER departs from traditional outcome-based reward shaping by integrating token-level reasoning quality assessments and dynamic, instance-specific length regulation, leading to more concise, interpretable, and performant reasoning chains (He et al., 7 Sep 2025).

1. Motivation and Conceptual Overview

Conventional RL reward functions for LLM reasoning tasks, such as those employed in mathematical and program synthesis benchmarks, assign rewards only to the final answer's correctness and possibly format validity. These global signals neither distinguish effective from ineffective intermediate reasoning steps nor offer any direct control over logical depth or computational cost. As a result, models frequently generate verbose or shallow CoTs, with no explicit optimization axis for balancing reasoning depth, informativeness, and efficiency.

The DRER framework addresses these deficiencies by:

• Assigning fine-grained reasoning quality rewards that provide direct token-level credit for steps that measurably increase the likelihood (confidence) of the correct answer,
• Introducing dynamic length advantages that modulate advantage functions based on instance-specific validation-derived response length intervals and difficulty,
• Enabling training dynamics that optimize for both accuracy and computational efficiency of reasoning, rather than optimizing purely for end-task performance.

This approach enables models to produce reasoning trajectories that are both logically beneficial for the final answer and cost-effective (in terms of token budget and computational resource).

2. Technical Formulation

2.1 Reasoning Quality Reward

For each input instance, let $x$ denote the prompt and $a^\star$ the ground-truth answer tokens with total answer length $T_a$. The model generates a CoT-augmented context $x_{\text{CoT}}$ and a baseline context without reasoning $x_{\text{NoCoT}}$. The key steps are as follows:

• Compute average log-likelihoods for the ground-truth answer tokens conditioned on the CoT and on the no-CoT context:

$$\ell_{\text{CoT}} = \frac{1}{T_a} \sum_{t=1}^{T_a} \log \pi_\theta(a_t^\star \mid x_{\text{CoT}}, c, a_{<t}^\star)$$

$$\ell_{\text{NoCoT}} = \frac{1}{T_a} \sum_{t=1}^{T_a} \log \pi_\theta(a_t^\star \mid x_{\text{NoCoT}}, a_{<t}^\star)$$

• Calculate the margin in answer likelihood:

$$\Delta(x) = \ell_{\text{CoT}} - \ell_{\text{NoCoT}}$$

• Assign the reasoning quality reward as a squashed function:

$R_q = \tanh(\Delta(x))$

• Form the full reward:

$R = R_{\text{task}} + \lambda_q R_q$

where $R_{\text{task}}$ is the traditional task reward (e.g., answer correctness) and $\lambda_q$ is a hyperparameter controlling the balance.

This implements token-level, trajectory-sensitive reward shaping that directly incentivizes only those steps in the chain that enhance the confidence in the final answer.

2.2 Dynamic Length Advantage

Length regularization avoids degenerate solutions where the model generates excessively long or uncharacteristically short reasoning traces. After periodic validation, length percentiles $L_{\min}^{(d)}$ and $L_{\max}^{(d)}$ are computed for each dynamic difficulty bucket $d$. For an effective answer token length $\ell_i$ in bucket $d$:

$$g_i = \exp\left( -\frac{ \max\{0, L_{\min}^{(d)} - \ell_i,\, \ell_i - L_{\max}^{(d)} \} }{\tau} \right)$$

where $\tau$ is a temperature. The advantage function for policy gradient RL is then modulated as:

$\hat{A}_i = g_i \cdot A_i$

Advantage decay penalizes samples falling outside the trusted length interval, stabilizing training and preventing pathological verbosity or truncation.

3. Empirical Performance and Application Domains

DRER was empirically validated using a 7B-parameter LLM (Qwen2.5-7B-Instruct-1M) trained on the LogicTree dataset—a dynamically constructed deductive reasoning benchmark featuring multi-hop logical deduction tasks up to 8 levels in depth (He et al., 7 Sep 2025). Integration of DRER with GRPO or DAPO RL optimization yielded the following outcomes:

• With only 400 RL training steps, the model achieved GPT-o3-mini-level accuracy (from 7% to ~60%) on LogicTree.
• The average confidence in CoT-augmented answers increased by 30% over initial model states.
• Token consumption for reasoning traces was reduced by up to 75% compared to standard RL or SFT baselines.
• Qualitatively, DRER-trained models concentrated probability mass over correct answers more tightly, generated more concise and interpretable CoT, and showed robust generalization to diverse logical reasoning datasets (e.g., MMLU-redux, ZebraLogic, AIME24).

These results highlight that DRER not only improves formal deductive reasoning accuracy but also achieves substantial computational savings by shaping the answer generation process at a fine granularity.

4. Generalization and Analytical Properties

DRER improves the expressive transparency of LLMs by tightly aligning internal reasoning quality with external task outcomes. By rewarding only reasoning steps that enhance downstream answer likelihood, it reduces the risk of models “overexplaining” for the sake of reward hacking (as occurs when final answer correctness alone is reinforced). The dynamic length advantage regularization prevents degenerate behavior (e.g., excessively verbose or minimal outputs), and enables robust transfer of reasoning policies across tasks of different logical depth or structure.

Experiments across additional benchmarks (AIME24, ProntoQA, ZebraLogic) show that the improvements of DRER on CoT reasoning efficiency and accuracy generalize beyond its primary training domain, indicating the approach’s versatility for formal and semi-formal logical tasks.

5. Practical Considerations and Limitations

DRER can be integrated with standard RL algorithms implementing per-token reward and advantage signals, such as A2C, PPO, DAPO, and GRPO. It does not require significant architectural changes but does rely on accurate log-likelihoods for both CoT and answer tokens. This can present computational overheads, particularly with large models or for long validation datasets, due to the need for paired with/without-CoT rollouts and token-level reward computations.

Another limitation is the need for reliable percentile estimation of desirable response lengths for each difficulty bucket, which must be recalibrated as training progresses or when switching domains.

6. Implications and Future Research

By bridging the gap between outcome-aligned and process-aligned reward shaping, DRER demonstrates that the efficiency and interpretability of LLM reasoning can be systematically improved. Potential research extensions include:

• Incorporating richer logical structures (higher-order, multimodal reasoning),
• Reducing computational cost of token-level reward calculation through approximation or parallelization,
• Integrating human evaluation signals to guide both CoT quality assessment and dynamic length regularization,
• Deploying DRER-enhanced models in formal education, automated theorem proving, specification verification, and safety-critical domains requiring both model transparency and efficiency.

DRER thus marks a concrete advancement in the controlled optimization of LLM reasoning, moving beyond black-box outcome rewards to fully process-aware, dynamically constrained, and efficiency-centric RL frameworks.