Step-wise Rubric Rewards for LLM Reasoning

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in LLMs, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%.

• The paper introduces SRaR, a framework that assigns step-specific rubric rewards to address misalignment in LLM reasoning.
• It employs a decoupled advantage estimator to integrate outcome-based and step-wise signals, boosting accuracy on diverse mathematical benchmarks.
• Empirical results demonstrate improved Faithful Reasoning Rates and reduced self-correction loops, highlighting the framework's potential for complex reasoning tasks.

Detailed Summary of "Step-wise Rubric Rewards for LLM Reasoning" (2605.17291)

Introduction

The paper "Step-wise Rubric Rewards for LLM Reasoning" introduces the Step-wise Rubrics as Rewards (SRaR) framework, advancing the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm for LLMs. Traditional RLVR models, relying on outcome-based rewards, inadequately supervise intermediate reasoning steps, often leading to inconsistencies in verification and self-correction. Recent approaches like Rubrics as Rewards (RaR) introduced rubric-based supervision but aggregated these rubrics into scalar values, often misattributing feedback across steps. SRaR innovates by providing step-level rubric rewards, using an LLM judge to associate specific rubrics with individual reasoning steps, thereby addressing misalignment issues in learning signals.

Figure 1: Overview of SRaR. Top: The SRaR training framework. An LLM judge attributes each rubric item to a specific reasoning step; per-step rewards are normalized across rollouts and combined with outcome rewards via a decoupled advantage estimator. Bottom: The three-stage contrastive rubric distillation pipeline that generates instance-specific rubrics from correct and flawed paths.

Methodology

SRaR operates on three pivotal strategies: step-attributed rubric judging, per-step token assignment with normalization, and a decoupled advantage estimator. The framework delineates rubric items into SUGGEST, PITFALL, and BONUS categories, attributing these to precise reasoning steps rather than global responses. This specificity permits accurate feedback alignment by broadcasting appropriate rewards to the tokens within each step, thereby facilitating finer-grained learning signals. Additionally, a distinct advantage calculation separates rubric impacts from traditional outcome-based signals, averting potential distortions from rubric noise.

Figure 2: Motivation for step-wise rubric reward. (a) RaR applies a single scalar rubric reward to the entire response, so every step receives the same signal; SRaR decomposes the rubric into per-step rewards aligned with each reasoning step. (b) Diagnostic study on 1,000...

Empirical Evaluation

Empirical evaluations underscore SRaR's superior efficacy across diverse mathematical benchmarks, including AIME, AMC, MATH, and more. The results display significant accuracy improvements: SRaR outperforms RaR by 3.57 points on Qwen3-8B and by 2.75 points on Qwen3-32B. Notably, SRaR enhances the Faithful Reasoning Rate, indicating improved internal consistency in reasoning processes, and substantially reduces the incidence of self-correction looping—a prevalent downfall associated with scalar rubric aggregation.

Figure 3: (Left) Correlation between the average number of reasoning steps per benchmark and SRaR's accuracy improvement over RaR... Accuracy vs. problem difficulty on Olympiad Bench...

Analysis and Discussion

Comprehensive analyses reveal that SRaR's design elements align reasoning process quality with outcome accuracy. A diagnostic study affirms that step-wise rewards enhance intermediate step correctness, reflected in improved Faithful Reasoning Rates and reduced Misaligned Trajectory Rates. Furthermore, step-wise credit assignment not only drives reasoning efficacy but also diminishes destructive behaviors such as uncontrolled self-correction.

Figure 4: Training dynamics of DAPO, RaR, and SRaR on Qwen3-32B-Non-Thinking...

Conclusion

SRaR emerges as a significant refinement in rubric-based reinforcement learning for LLM reasoning, delivering precise, step-oriented feedback that aligns closely with desired reasoning trajectories. By efficiently addressing key limitations of prior models, this framework demonstrates clear potential for broader, scalable implications in complex reasoning tasks. The introduction of an extensive rubric dataset and the effective step-wise reward approach pave the path for future exploration into multi-step reasoning domains.

The research acknowledges certain constraints, including potential domain-specific adaptations for non-mathematical reasoning and the dependence on robust LLM judges for initial rubric evaluations. Future trajectories promise to expand SRaR's applicability, with potential enhancements in automated reasoning support and cross-domain generalization.