Critique Reinforcement Learning

• Critique Reinforcement Learning (CRL) is a family of RL methods where natural language feedback is treated as a trainable signal to guide policy refinement.
• The approach employs a two-stage training paradigm that first optimizes the critic’s discriminability and then its helpfulness to enhance downstream performance.
• Variants like Critique-Coder, ICRL, and ECHO extend CRL by jointly training critic and actor, addressing challenges such as verifier dependency and feedback staleness.

Searching arXiv for the core Critique-RL paper and closely related critique-RL variants to ground the article in the cited literature. Critique Reinforcement Learning (CRL) is a family of reinforcement-learning methods in which natural-language critique becomes an optimized object of learning rather than a fixed supervisory artifact. In these systems, a model generates an answer, trajectory, or candidate solution; a critic then produces feedback, diagnosis, or an explicit judgment; and reinforcement signals are assigned according to whether that critique correctly evaluates the original output, enables successful refinement, or improves later unassisted performance. Across recent formulations, CRL is used to train a separate critic for mathematical reasoning, to mix critique judgments into coder-model post-training, to co-evolve policy and critic in open-world agents, and to internalize critique into a solver’s critique-free policy. The common thread is that critique is treated as an action with downstream utility, not merely as explanatory text (Xi et al., 28 Oct 2025, Ruan et al., 26 Sep 2025, Li et al., 11 Jan 2026, Lin et al., 13 May 2026, Cao et al., 2024, Tang et al., 20 Jul 2025).

1. Conceptual framing and task structure

The canonical CRL setting is a two-player paradigm. Given an input $x$, an actor or solver $\pi_\theta$ generates an initial response $y$. A critic $\pi_\phi$ or $C_\psi$ then observes the task and the initial response, producing a textual critique $c$. The actor subsequently refines its answer to $y'=\pi_\theta(x,y,c)$, or, in agentic settings, refines an initial trajectory into a new trajectory conditioned on the critique. The critic is therefore judged not only by what it says, but by what its feedback causes the policy to do next (Xi et al., 28 Oct 2025, Li et al., 11 Jan 2026).

Recent papers divide critique quality into two separable properties. "Critique-RL" formalizes discriminability as the ability to determine whether the original response is correct, and helpfulness as the ability to provide constructive, actionable feedback that improves refinement. This decomposition is central because indirect outcome-based rewards alone can improve helpfulness while leaving discriminability weak, creating an optimization bottleneck in which the critic becomes either overly conservative or overly aggressive (Xi et al., 28 Oct 2025). A related formulation in "RefCritic" makes the same distinction in different terms: a critic must issue a correct judgment and provide feedback that measurably improves the policy’s refined answer (Tang et al., 20 Jul 2025).

Other CRL variants retain the same basic structure but modify the locus of optimization. "Critique-Coder" conditions on a given $(q,s)$ pair and asks the model to emit a critique terminating in an explicit binary judgment $c\in\{\texttt{True},\texttt{False}\}$ serialized in \conclusion{}; the reward depends solely on whether that final judgment matches ground truth (Ruan et al., 26 Sep 2025). "ICRL" uses a shared backbone with role-specific prompts for a solver and a critic, explicitly aiming to convert critique-induced success into critique-free solver ability (Lin et al., 13 May 2026). "ECHO" generalizes the pattern to open-world agents, where the critic produces multiple diagnoses for an initial trajectory and is rewarded according to the gain induced after refinement (Li et al., 11 Jan 2026).

2. Core optimization objectives

A central design question in CRL is how to turn language feedback into a reinforcement signal. In the separate-critic formulation of "Critique-RL", the critic policy is optimized with a REINFORCE-style estimator. Before two-stage shaping, the basic objective is the expectation of a reward over sampled critiques and refinement outcomes:

$$\mathbb{E}_{c\sim\pi_{\phi}(\cdot|x,y),\,y'\sim\pi_\theta(x,y,c)} \big[r_c(x,y,c,y')\big].$$

The paper studies several indirect rewards, including refinement correctness $\pi_\theta$0, improvement over the original $\pi_\theta$1, and a correction-focused shaping reward. Indirect-only training, however, does not consistently improve discriminability (Xi et al., 28 Oct 2025).

The decisive intervention in "Critique-RL" is a direct rule-based discriminability reward,

$\pi_\theta$2

where $\pi_\theta$3 extracts the critic’s correctness judgment from the critique. Stage I maximizes this reward with KL regularization to the supervised initialization. Stage II adds the indirect refinement reward while preserving discriminability through a weighted $\pi_\theta$4 term and KL regularization to the Stage I critic. This design decouples “reliable judge” from “useful adviser,” then recombines them (Xi et al., 28 Oct 2025).

Other CRL systems instantiate the same principle differently. In "Critique-Coder", the critique reward is purely judgmental:

$\pi_\theta$5

This signal is mixed with standard code-generation reward inside GRPO, with 20% of the RL data substituted by CRL data (Ruan et al., 26 Sep 2025). In "RefCritic", the combined critic reward is $\pi_\theta$6, where $\pi_\theta$7 is judgment correctness and $\pi_\theta$8 is the average pass rate of refined samples, conditioned on the initial answer being wrong and the critic correctly judging it wrong (Tang et al., 20 Jul 2025). In "ICRL", the critic reward is tied directly to solver improvement:

$\pi_\theta$9

This makes critique quality entirely downstream-facing (Lin et al., 13 May 2026).

A distinct but related line appears in "Beyond Sparse Rewards", where critique is converted into dense token- or span-level intrinsic reward. There, a frozen critic labels positive or negative spans, and the policy is optimized with per-step reward

$y$0

The emphasis is not judgment correctness of the critique itself, but temporal credit assignment for text generation (Cao et al., 2024).

3. Two-stage critic training and the discriminability–helpfulness split

The most explicit articulation of CRL as critic training appears in "Critique-RL". The paper argues that optimizing only on actor refinement outcomes creates two characteristic failure modes. A critic may become overly conservative, minimizing changes and reducing $y$1 while failing to increase $y$2, or overly aggressive, encouraging changes that increase $y$3 but harm $y$4. In both cases, discriminability stagnates or worsens during RL, which explains why indirect-only baselines yield only marginal final gains (Xi et al., 28 Oct 2025).

To overcome this, the method uses a two-stage schedule. Stage I reinforces discriminability using the direct oracle-matching reward and KL regularization to the SFT critic. Stage II introduces refinement-based reward to optimize helpfulness while preserving discriminability through both an explicit $y$5 term and KL regularization to the Stage I policy. The implementation uses online policy gradient with RLOO, on-policy sampling, no separate value model, and no entropy regularization; stability is attributed to KL constraints and RLOO baselines (Xi et al., 28 Oct 2025).

"RefCritic" reaches a closely related conclusion from the perspective of long-chain-of-thought critics. It reports that supervised fine-tuning yields high “critique accuracy” yet often produces superficial analyses, brittle reasoning, and vague advice. Its remedy is again reward decomposition: first optimize judgment quality, then incorporate refinement success. The training schedule sets $y$6 for 600 steps and then $y$7 for 300 steps, effectively moving from judgment-only reward to dual reward. The resulting critics produce dramatically longer outputs—approximately from 500 to 3500 tokens for Qwen and from 3000 to 8000 for R1—and these longer critiques correlate with better ProcessBench performance and stronger refinement gains (Tang et al., 20 Jul 2025).

This line of work suggests that CRL is not merely “RL for critique text.” It is a structured optimization program in which correctness assessment and corrective utility are separable objectives that must be staged or jointly regularized. A plausible implication is that critique models fail less because they cannot generate fluent analysis than because standard post-training does not bind critique language to verifiable evaluative and corrective consequences (Xi et al., 28 Oct 2025, Tang et al., 20 Jul 2025).

4. Joint training, co-evolution, and critique internalization

A second branch of CRL rejects the assumption of a fixed actor–critic division. "ICRL" trains solver and critic jointly from a shared backbone and targets what it calls the internalization problem: critique may transiently induce correct behavior, but the same model may fail again once critique is removed. The paper attributes this to distribution shift between critique-conditioned and critique-free behavior. Its solution combines two mechanisms. First, a distribution-calibration re-weighting ratio

$y$8

reconditions revised trajectories under the solver’s original prompt and down-weights tokens that depend heavily on critique. Second, role-wise group advantage estimation normalizes solver and critic rewards separately to stabilize joint optimization (Lin et al., 13 May 2026).

This design changes the objective of CRL. The critic is still rewarded by downstream gain, but the solver is trained to absorb only those critique-guided improvements compatible with its own critique-free distribution. The result is explicitly not dependence on feedback at inference time, but transfer from “success with critique” to “success without critique.” Empirically, the paper reports that $y$9 increases over training, which it interprets as growing compatibility between revised tokens and the solver’s unassisted policy (Lin et al., 13 May 2026).

"ECHO" addresses a different joint-training pathology: stale feedback. In open-world on-policy RL for agents, the policy’s trajectory distribution changes over time, so a frozen or offline critic becomes miscalibrated in granularity and usefulness. ECHO therefore updates policy and critic in lockstep through dual-track GRPO. Its critic reward is a saturation-aware gain,

$\pi_\phi$0

which increases the signal for last-mile improvements near the reward ceiling. The critic emits multiple diagnoses for an initial trajectory, the policy produces multiple refinements, and both tracks are updated from group-relative advantages computed over the same on-policy batch (Li et al., 11 Jan 2026).

Together, these systems extend CRL beyond the fixed-critic setting. They treat critique as a non-stationary control signal whose quality depends on synchronization with the current policy and on whether the policy can internalize critique-induced behavior. This suggests that in long-horizon or self-improving regimes, the central challenge is not only generating actionable critique, but keeping critique aligned with a moving target policy and with the target prompt distribution (Li et al., 11 Jan 2026, Lin et al., 13 May 2026).

5. Domain-specific instantiations

A notable property of CRL is its portability across domains, although each domain changes the available verifier and the semantics of a “good” critique. The following summary captures the main instantiations discussed in papers.

System	Domain	Reward focus
Critique-RL	Mathematical reasoning and summarization	Direct discriminability reward plus indirect refinement reward
RefCritic	Mathematical reasoning with long CoT critic	Judgment correctness and refinement pass rate
Critique-Coder	Code generation and judgment	Binary correctness of final critique label, mixed with RL
ICRL	Agentic tasks and mathematics	Solver gain after critique plus critique internalization
ECHO	Open-world agents	Saturation-aware gain induced by critique
RELC	Text generation	Dense token/span rewards derived from critique

In code generation, "Critique-Coder" recasts a portion of standard RLVR as judgment-level RL. Instead of only generating code and receiving pass-rate reward, the policy also learns to read a question–solution pair and produce a critique concluding with \conclusion{T} or \conclusion{F}. The authors report that substituting 20% of standard RL data with CRL data is best overall: 0% CRL yields AVG 45.7, 50% yields 46.2, 100% yields 44.5, and 20% yields 47.2 for Qwen3-4B. Fully replacing RL with CRL therefore degrades performance, and the paper explicitly characterizes CRL as a complement rather than a replacement (Ruan et al., 26 Sep 2025).

In text generation, "RELC" uses a frozen critic LM to identify spans responsible for desirable or undesirable behavior in sentiment control, detoxification, and summarization. Tokens in positive or factual spans receive positive intrinsic reward; toxic or erroneous spans receive negative reward. The method is therefore critique-guided RL without online critic learning. It is still part of the CRL family in the broader sense that critique supplies the reinforcement signal, but its critic is not itself optimized by RL (Cao et al., 2024).

In mathematical reasoning, "RefCritic" and "Critique-RL" emphasize verifiable correctness via rule-based math answer matching. Both papers also discuss generalization beyond math: "Critique-RL" reports a CNN/DailyMail summarization setting using a reward model for both $\pi_\phi$1 and a soft $\pi_\phi$2, while "RefCritic" tests out-of-domain transfer to GPQA and LiveCodeBench (Xi et al., 28 Oct 2025, Tang et al., 20 Jul 2025). The domain shift is consequential because open-ended domains require alternative verifiers, which alters both the reliability of discriminability rewards and the interpretation of “correctness.”

6. Empirical patterns and evaluation practice

CRL papers evaluate critique systems with a mixture of judgment metrics, refinement metrics, and downstream task scores. "Critique-RL" offers the clearest metricization. Acc@Dis measures whether the critic’s correctness judgment matches the oracle label for the original answer. Acc@Refine measures accuracy after refinement. $\pi_\phi$3 measures net refinement gain. $\pi_\phi$4 records the rate at which correct responses become incorrect after refinement, and $\pi_\phi$5 records the rate at which incorrect responses become correct. This metric set is designed specifically to separate discriminability from helpfulness (Xi et al., 28 Oct 2025).

Representative results support the value of that separation. For Qwen2.5-7B, "Critique-RL" reports a 9.02% in-domain gain and a 5.70% out-of-domain gain. On in-domain tasks, it reports MATH Acc 58.40 with $\pi_\phi$6 12.66 and Acc@Dis 85.20, GSM8K Acc 87.72 with $\pi_\phi$7 12.05 and Acc@Dis 90.43, and AQuA Acc 65.75 with $\pi_\phi$8 2.36 and Acc@Dis 78.09. Ablations show that removing Stage I or Stage II reduces both Acc@Refine and Acc@Dis, and Stage II without discrimination terms harms both (Xi et al., 28 Oct 2025).

"RefCritic" reports one-round critique-and-refine gains of +6.8 on AIME25 for Qwen2.5-14B and +7.2 for DeepSeek-R1-Distill-Qwen-14B. On ProcessBench, despite training only with solution-level rewards, RefCritic-Qwen-14B reaches average F1 68.0 and RefCritic-R1-14B reaches 77.1, outperforming step-level supervised approaches such as DeepCritic-7B and ThinkPRM-14B (Tang et al., 20 Jul 2025). "Critique-Coder-8B" reaches 60.8 on LiveCodeBench (v5), surpassing DeepCoder-14B at 60.6 and GPT-o1 at 59.5, while also improving general reasoning on BBEH (Ruan et al., 26 Sep 2025).

Joint-training variants report a different style of gain. "ICRL" shows average improvements of 6.4 points over GRPO on agentic tasks and 7.0 points on mathematical reasoning; on ALFWorld and WebShop, its learned 8B critic is reported as comparable to 32B frozen critics while using substantially fewer tokens (Lin et al., 13 May 2026). "ECHO" reports that for Qwen3-4B, WebShop improves from 82.37 to 90.03, ALFWorld from 87.50 to 91.25, SciWorld from 79.14 to 82.88, and DeepSearch from 33.25 to 47.25 relative to GRPO, with additional ablations showing large degradations when the critic is frozen (Li et al., 11 Jan 2026).

A recurring empirical claim across the literature is that critique quality cannot be inferred from fluent rationale alone. A critic may produce plausible analysis while failing either to discriminate correctly or to induce beneficial refinements. Accordingly, CRL evaluation tends to pair language-quality judgments with outcome-based measurements such as pass rate, success rate, reward gain, MSE/MAE reduction, or majority-vote filtering effects (Xi et al., 28 Oct 2025, Tang et al., 20 Jul 2025, Lin et al., 13 May 2026).

7. Limitations, controversies, and open questions

One persistent limitation is verifier dependence. "Critique-RL" relies on an oracle $\pi_\phi$9 for correctness, such as answer matching on math datasets. The paper notes that for open-ended tasks such as summarization, one must replace this oracle with a reward model or AI feedback (Xi et al., 28 Oct 2025). "Critique-Coder" similarly derives ground-truth labels from execution and a pass-rate threshold of 80%, chosen to mitigate timeouts causing false negatives (Ruan et al., 26 Sep 2025). "ECHO" depends on an external evaluator $C_\psi$0 for both baseline and refinement scoring, and explicitly warns that if $C_\psi$1 is noisy or miscalibrated, both critic and policy may optimize evaluator artifacts (Li et al., 11 Jan 2026).

A second controversy concerns whether CRL should optimize critique text quality or only terminal judgments. "Critique-Coder" rewards only the final label and acknowledges that rationale quality is reward-invariant, which may encourage minimal justifications unless CRL is mixed with RL (Ruan et al., 26 Sep 2025). By contrast, "RefCritic", "Critique-RL", and "ICRL" treat critique usefulness as inseparable from downstream revision utility, thereby operationalizing language quality through induced gains rather than human-style preference judgments (Xi et al., 28 Oct 2025, Tang et al., 20 Jul 2025, Lin et al., 13 May 2026).

A third issue is frozen versus evolving critics. Fixed critics simplify training and can provide dense reward, as in RELC, but both ECHO and ICRL argue that static critics become stale or induce dependence on critique-conditioned behavior. Their results indicate that co-evolution or shared-backbone joint optimization can materially improve performance, but at the cost of additional compute, more complicated variance control, and the possibility of critic–solver co-adaptation around evaluator quirks (Cao et al., 2024, Li et al., 11 Jan 2026, Lin et al., 13 May 2026).

Finally, CRL still faces unresolved questions of calibration, robustness, and inference-time use. "Critique-RL" notes that formal calibration remains open even though discriminability optimization reduces aggressive incorrect suggestions (Xi et al., 28 Oct 2025). "Critique-Coder" reports that a parallel self-critique aggregation procedure did not improve performance, indicating limited self-critical selection despite improved judgment (Ruan et al., 26 Sep 2025). "ICRL" limits self-improvement rounds to $C_\psi$2 for efficiency, and identifies asynchronous training and scaling to larger backbones as open areas (Lin et al., 13 May 2026). These results collectively imply that CRL has established critique as a trainable reinforcement-learning object, but not yet a solved one: the field now centers on how to verify critique, preserve calibration under distribution shift, and turn critique-induced success into robust, durable capability.