Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning (2510.24320v1)

Abstract: Training critiquing LLMs to assess and provide feedback on model outputs is a promising way to improve LLMs for complex reasoning tasks. However, existing approaches typically rely on stronger supervisors for annotating critique data. To address this, we propose Critique-RL, an online RL approach for developing critiquing LLMs without stronger supervision. Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly. We first reveal that relying solely on indirect reward signals from the actor's outputs for RL optimization often leads to unsatisfactory critics: while their helpfulness (i.e., providing constructive feedback) improves, the discriminability (i.e., determining whether a response is high-quality or not) remains poor, resulting in marginal performance gains. To overcome this, Critique-RL adopts a two-stage optimization strategy. In stage I, it reinforces the discriminability of the critic with direct rule-based reward signals; in stage II, it introduces indirect rewards based on actor refinement to improve the critic's helpfulness, while maintaining its discriminability via appropriate regularization. Extensive experiments across various tasks and models show that Critique-RL delivers substantial performance improvements. For example, it achieves a 9.02% gain on in-domain tasks and a 5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential.

• The paper presents a two-stage RL framework that disentangles and optimizes discriminability and helpfulness in critiquing language models.
• Empirical results show up to 9.02% improvement on in-domain tasks, highlighting enhanced accuracy, generalization, and compute efficiency over baseline methods.
• Ablation studies confirm that dual-stage reward signals and KL regularization are essential for balancing error detection and constructive feedback.

Critique-RL: Two-Stage Reinforcement Learning for Training Critiquing LLMs

Motivation and Problem Formulation

The paper addresses the challenge of scalable oversight in LLMs, specifically the development of critiquing models that can both assess the correctness of model outputs (discriminability) and provide constructive feedback (helpfulness) without relying on stronger supervisors or oracle verifiers. Existing approaches either depend on expensive human annotation or prompt engineering with external verifiers, both of which are limited in scalability and generalization. The authors propose Critique-RL, a two-stage RL framework that explicitly disentangles and optimizes discriminability and helpfulness in critiquing models.

Two-Player Actor-Critic Interaction and RL Formulation

Critique-RL operates in a two-player setting: the actor generates a response to a query, the critic evaluates and provides feedback, and the actor refines its response based on the critique. The RL optimization is performed on the critic, with the actor model fixed. The reward signals for the critic are derived from the correctness of the actor's original and refined responses, as determined by an oracle verifier during training.

Figure 1: Left: Illustration of the actor-critic interaction and refinement process. Right: Overview of Critique-RL and comparison with baseline RL, highlighting the two-stage optimization.

Policy gradient methods (e.g., REINFORCE) are used to optimize the critic's policy $\pi_\phi$, maximizing expected rewards over sampled critiques and actor refinements. The reward functions are carefully designed to target discriminability and helpfulness separately.

Empirical Analysis of Reward Signal Design

The authors conduct extensive preliminary experiments to analyze the effects of different reward shaping strategies:

• Indirect rewards based on actor refinement correctness ($r_\text{refine}$, $r_\Delta$) tend to improve helpfulness but fail to optimize discriminability, resulting in conservative critics that rarely suggest changes.
• Correction-focused rewards ($r_\text{correction}$) improve error correction but can degrade performance on originally correct responses, leading to aggressive critics.

  Figure 2: Training dynamics showing that indirect reward signals lead to suboptimal discriminability and unstable optimization, while Critique-RL achieves stable improvements in both discriminability and helpfulness.

The analysis reveals that indirect outcome-based rewards are insufficient for training effective critique models, as they do not directly incentivize accurate discrimination between correct and incorrect responses.

Two-Stage Critique-RL: Methodology

To address the identified challenges, Critique-RL introduces a two-stage RL process:

• Stage I (Discriminability Optimization): The critic is trained with direct, rule-based rewards that measure alignment between its judgment and the oracle verifier. The objective is to maximize

$$\mathbb{E}_{c \sim \pi_\phi^{\text{Stage-I}}(\cdot|x,y)} [r_\text{dis}(x,y,c) - \beta \text{KL}(\pi_\phi^{\text{SFT}} || \pi_\phi^{\text{Stage-I}})]$$

where $r_\text{dis}$ is an indicator of correct discrimination.

• Stage II (Helpfulness Optimization with Discriminability Regularization): The critic is further trained to maximize the helpfulness of its feedback (as measured by the correctness of actor refinements), while maintaining discriminability via regularization:

$$\mathbb{E}_{c, y'} [r_\text{refine} + \beta_1 r_\text{dis} - \beta_2 \text{KL}(\pi_\phi^{\text{Stage-I}} || \pi_\phi^{\text{Stage-II}})]$$

This prevents the critic from becoming overly aggressive or conservative.

Experimental Results and Analysis

Critique-RL is evaluated on multiple mathematical reasoning datasets (MATH, GSM8K, AQUA, SVAMP, TheoremQA) and model architectures (Qwen2.5-3B/7B/72B, Llama3.2-3B, DeepSeek-R1-Distill-Qwen-7B). Key findings include:

• Substantial Gains: Critique-RL achieves up to 9.02% improvement on in-domain tasks and 5.70% on out-of-domain tasks for Qwen2.5-7B, outperforming SFT, STaR, Retroformer, and CTRL baselines in both accuracy and discriminability.
• Generalization: The trained critic generalizes to unseen tasks and actor models, demonstrating modularity and transferability.
• Iterative Refinement: Multiple rounds of critique-refinement yield further improvements, with each iteration increasing both accuracy and discriminability.
• Compute Efficiency: Critique-RL is more compute-efficient than parallel response sampling, raising the performance ceiling as inference compute scales.

  Figure 3: Inference compute scaling for Critique-RL, showing improved performance ceiling and compute efficiency compared to baselines.

  

  Figure 4: Refine compute scaling for Critique-RL and SFT critic, demonstrating higher sampling efficiency for Critique-RL.

Ablation and Qualitative Analysis

Ablation studies confirm the necessity of both stages and the regularization terms. Removing discriminability rewards or KL regularization in Stage II leads to significant drops in both accuracy and discriminability. Qualitative examples illustrate that Critique-RL-trained critics can identify errors and provide actionable feedback, whereas SFT critics often fail to detect errors or suggest meaningful refinements.

Figure 5: Example where Critique-RL enables the actor to correct an initially incorrect response via detailed critique.

Figure 6: Example showing the advantage of two-stage RL: Stage I critic detects errors but provides low-quality suggestions, while Stage II critic offers constructive feedback leading to correct refinement.

Implications and Future Directions

Critique-RL provides a scalable framework for training critiquing models that can be flexibly applied to diverse tasks and actor models. The explicit disentanglement and joint optimization of discriminability and helpfulness address key limitations of prior RL-based and SFT-based approaches. The modularity of the critic enables weak-to-strong transfer and supports iterative self-improvement paradigms.

The approach is extensible to open-ended tasks (e.g., summarization) by integrating reward models or AI feedback for supervision. The findings suggest that scalable oversight in LLMs can be achieved without reliance on expensive human annotation or external verifiers, paving the way for more autonomous and reliable language agents.

Conclusion

Critique-RL introduces a principled two-stage RL framework for training critiquing LLMs, achieving strong empirical gains in both discriminability and helpfulness across multiple datasets and architectures. The method's modularity, generalization, and compute efficiency make it a promising foundation for scalable oversight and self-improvement in LLMs. Future work may explore integration with more complex reward models, extension to multi-agent settings, and application to broader domains beyond mathematical reasoning.