RefCritic: Refinement-Aware Critique Training

• RefCritic is a training framework for LLM critic modules that delivers actionable refinement feedback beyond simple correctness judgments.
• It integrates a cold-start supervised phase with dual-reward reinforcement learning to optimize both judgment accuracy and feedback usefulness.
• Empirical results demonstrate significant improvements in policy pass rates and process-level evaluations across various benchmarks.

Searching arXiv for RefCritic and closely related critique-training papers. RefCritic is a training framework for LLM critic modules that seeks to optimize critique not merely as correctness labeling, but as a source of actionable feedback for downstream refinement. Introduced in “RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback” (Tang et al., 20 Jul 2025), it is designed for settings in which a policy model first produces a solution and a separate critic then evaluates that solution, emits a correctness judgment, and provides suggestions intended to improve a revised answer. The framework is motivated by the claim that conventional supervised fine-tuning for critics often yields superficially correct judgments without producing critiques that materially help refinement. RefCritic therefore combines a cold-start supervised phase with reinforcement learning under dual rule-based rewards: one for instance-level judgment correctness and one for refinement success. Within the broader literature on critique-capable LLMs, it belongs to a shift from treating critique as an auxiliary textual byproduct toward treating it as a first-class optimization target (Tang et al., 20 Jul 2025, Yu et al., 27 Jun 2025, Ruan et al., 26 Sep 2025).

1. Concept and problem formulation

RefCritic addresses a specific mismatch in critique-model training: a critic may classify a solution as correct or incorrect with high apparent accuracy while still providing shallow, misleading, or non-actionable reasoning (Tang et al., 20 Jul 2025). The paper argues that this is especially problematic as reasoning model outputs become longer and more intricate, making direct human inspection harder and increasing reliance on machine critics.

The framework formalizes the critic as a model that takes a problem $x$ and an initial solution $y_0$ from a policy model, then outputs three components:

$C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$

where $z$ is a long chain-of-thought reasoning trace, $\hat{c} \in \{0,1\}$ is a correctness judgment, and $f$ is actionable refinement feedback (Tang et al., 20 Jul 2025). This structure distinguishes RefCritic from critique systems that emit only a scalar verdict or only free-form commentary.

The central problem is not simply solution verification. Rather, it is the joint optimization of two coupled abilities: discriminative judgment over whether the original solution is correct, and generative production of feedback that enables a policy model to revise an incorrect solution into a correct one (Tang et al., 20 Jul 2025). This framing aligns RefCritic with a broader line of work that treats critique as distinct from generation and correction, rather than assuming that strong generators automatically become strong critics (Lin et al., 2024).

A related implication emerges in comparison with CriticBench’s “GQC reasoning” decomposition into generation, critique, and correction (Lin et al., 2024). RefCritic effectively targets the interface between the “Q” and “C” stages: it trains the critic so that its output is useful not just for judging but for changing subsequent model behavior. This suggests a stricter notion of critique quality than pure classification accuracy.

2. Critique model architecture and long chain-of-thought design

A defining feature of RefCritic is its use of a long chain-of-thought critic (Tang et al., 20 Jul 2025). The paper presents output length growth during RL training as evidence that critics can become more detailed and reflective when optimized beyond supervised judgment imitation. After RL, average output length reportedly grows from about 500 tokens to 3500 tokens for RefCritic-Qwen and from about 3000 tokens to 8000 tokens for RefCritic-R1 (Tang et al., 20 Jul 2025).

This long-CoT design is not an incidental formatting choice. The paper’s thesis is that deeper critique requires extended reasoning traces capable of error identification, verification, and suggestion generation. In that respect, RefCritic differs from binary-judgment critique training schemes such as Critique Reinforcement Learning in Critique-Coder, where the reward is based solely on whether the final judgment label matches ground truth (Ruan et al., 26 Sep 2025). RefCritic retains the importance of correctness labels, but adds a second optimization target tied to the downstream utility of feedback.

The backbones used in the paper are Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B (Tang et al., 20 Jul 2025). These models serve both as policy models that generate solutions and as bases for critic training. The choice of reasoning-oriented backbones is consistent with the paper’s emphasis on intricate mathematical solutions and long-form critique.

The long-CoT critic can also be situated relative to other structured or refinement-aware critics. RCO optimizes critics by rewarding critiques that induce better refinements, but it is framed around critique utility estimated from actor preferences rather than around long reasoning traces (Yu et al., 27 Jun 2025). CRITIC-R1, in contrast, imposes explicit fields such as verdict, error location, reason, and fix for retrieval-augmented generation (Xiao et al., 28 May 2026). RefCritic is less schema-constrained than CRITIC-R1, but similarly motivated by the need for actionable diagnosis rather than superficial critique.

3. Training methodology and dual-reward reinforcement learning

RefCritic uses a two-stage training procedure (Tang et al., 20 Jul 2025). The first stage is cold-start supervised fine-tuning with rejection sampling, intended to produce a format-compliant initial critic. The second stage is reinforcement learning with dual rule-based rewards, implemented using GRPO (Tang et al., 20 Jul 2025).

The RL setup begins with a policy model generating an initial solution $y_0$ for problem $x$. Given a ground-truth answer $a$, correctness is defined as:

$c = \mathbb{I}[y_0 = a] \in \{0,1\}$

The critic then outputs $y_0$0, and the policy model produces $y_0$1 refined solutions conditioned on the critique:

$y_0$2

for $y_0$3 (Tang et al., 20 Jul 2025).

The optimization objective is:

$y_0$4

The first reward, instance-level judgment correctness, is

$y_0$5

The second reward, refinement accuracy, is

$y_0$6

Thus refinement reward is only available when the original solution is wrong and the critic correctly identifies it as wrong (Tang et al., 20 Jul 2025).

The scalar $y_0$7 balances the two rewards. The paper uses a staged schedule: first train with $y_0$8 for 600 steps, then continue with $y_0$9 for 300 more steps (Tang et al., 20 Jul 2025). This schedule is motivated by cost: refinement sampling is expensive, so the framework first improves verdict accuracy cheaply and only then incorporates refinement feedback.

This methodology places RefCritic within a broader family of refinement-aware critique optimization methods. RCO likewise argues that critique should be supervised by the improvement it induces in refined responses rather than by critique preference alone (Yu et al., 27 Jun 2025). RefCritic differs in using explicit rule-based rewards over refined correctness rather than the Critique Utility estimate defined as a fractional win rate of refinement outputs over the original response (Yu et al., 27 Jun 2025). The shared premise is that critique usefulness is causally downstream of refinement outcomes, not fully observable from critique text alone.

4. Data construction, supervision regime, and evaluation protocol

The training data for RefCritic is built from NuminaMath-1.5 (Tang et al., 20 Jul 2025). The paper filters about 120K high-quality math problems from the original 900K by exact-string deduplication, semantic deduplication using gte-multilingual-base embeddings with cosine similarity threshold above 0.95 removed, filtering of unsolvable, proof, and multiple-choice problems using Qwen2.5-72B-Instruct, and difficulty balancing via eight sampled solutions per problem to exclude items where all attempts succeed or all fail (Tang et al., 20 Jul 2025).

For critic training, the authors sample 8 responses per problem, keep at most one correct and one incorrect response per problem, and filter incomplete generations (Tang et al., 20 Jul 2025). Initial critique data is generated by stronger models and then filtered for correctness, instruction compliance, and leakage risk (Tang et al., 20 Jul 2025). This pipeline is intended to ensure that the cold-start SFT phase yields a critic that already conforms to the intended task structure before RL begins.

The evaluation covers both in-domain and out-of-domain settings. Main math benchmarks are AIME24, AIME25, and OlympiadBench; out-of-domain benchmarks are LiveCodeBench and GPQA-Diamond; process-level evaluation is performed on ProcessBench (Tang et al., 20 Jul 2025). Since RefCritic outputs natural language critiques rather than explicit step indices, the paper uses Qwen2.5-14B-Instruct as an extractor to map critiques to the earliest erroneous step index for ProcessBench, with F1 as the metric (Tang et al., 20 Jul 2025).

Two evaluation settings are central. In “Majority Vote with Critique,” the critic filters out solutions judged incorrect and majority voting is applied to the remaining solutions, denoted $C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$0 (Tang et al., 20 Jul 2025). In “Refinement after Critique,” the policy produces one solution, the critic evaluates it, and if it is judged incorrect the policy revises based on the critique; the final metric is reported as $C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$1 (Tang et al., 20 Jul 2025).

This evaluation structure closely matches the paper’s conception of the critic as both selector and refiner. It also resonates with findings from CriticBench that critique and correction are related but imperfectly aligned capabilities (Lin et al., 2024). RefCritic explicitly evaluates both functions rather than collapsing them into a single scalar judgment benchmark.

5. Empirical results and process-level behavior

The paper reports consistent improvements for refinement after critique on AIME25. RefCritic-Qwen-14B improves policy Pass@1 by 6.8%, and RefCritic-R1-14B improves policy Pass@1 by 7.2% (Tang et al., 20 Jul 2025). Similar gains are reported on AIME24 and OlympiadBench, including 9.9% for Qwen-based supervision and 2.6% for R1-based supervision on Olympiad (Tang et al., 20 Jul 2025).

Under majority voting with critique, RefCritic yields an average 3.6 percentage point improvement at 64 samples on AIME25 over no critique (Tang et al., 20 Jul 2025). The gains increase with the number of samples, which the paper interprets as evidence that RefCritic becomes more valuable when used as a filter over larger candidate pools. The table highlighted in the paper includes Qwen2.5-14B moving from 23.3 to 24.4 on AIME25 at 64 samples, and R1-Qwen-14B moving from 62.0 to 68.1 (Tang et al., 20 Jul 2025).

Out-of-domain transfer is also reported. On LiveCodeBench, RefCritic-R1-14B improves $C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$2 from 51.0 to 54.1, and RefCritic-Qwen-14B improves from 18.9 to 22.9 (Tang et al., 20 Jul 2025). On GPQA, RefCritic-R1-14B improves $C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$3 from 61.6 to 65.1, and RefCritic-Qwen-14B improves from 23.3 to 24.3 (Tang et al., 20 Jul 2025). These gains are smaller than the in-domain mathematical improvements, but they suggest nontrivial transfer.

One of the paper’s strongest claims concerns ProcessBench. RefCritic-Qwen-14B achieves average F1 of 68.0 and RefCritic-R1-14B achieves 77.1, outperforming prior critics including some trained with step-level supervision, despite RefCritic itself being trained only with solution-level supervision (Tang et al., 20 Jul 2025). This suggests that long-CoT critics with refinement-aware RL can acquire implicit step-sensitive diagnostic behavior without explicit step labels.

Ablation results further support the dual-reward design. On AIME25 for RefCritic-Qwen-14B, baseline critic performance is 14.5, SFT reaches 15.0, $C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$4 reaches 18.5, $C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$5 reaches 19.5, and the two-stage schedule $C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)$6 reaches 21.2 (Tang et al., 20 Jul 2025). Corresponding gains on AIME24 and OlympiadBench are also largest for the two-stage setup (Tang et al., 20 Jul 2025). The pattern indicates that judgment-only RL is already much stronger than SFT, but refinement-aware reward yields further gains.

6. Position in the critique-learning literature, limitations, and interpretation

RefCritic sits within a rapidly developing research area that treats critique as a distinct learning problem. CriticBench showed that generation, critique, and correction are positively related but do not coincide, and that critique-focused training can substantially improve critique performance (Lin et al., 2024). MetaCritique further argued that critique quality itself should be evaluated along factuality and comprehensiveness dimensions using Atomic Information Units and F1-style aggregation (Sun et al., 2024). RCO reframed critic training around refinement outcomes rather than critique preference (Yu et al., 27 Jun 2025). Critique-Coder introduced Critique Reinforcement Learning for coding, where the model learns to judge whether a proposed solution is correct using a binary label reward (Ruan et al., 26 Sep 2025). RefCritic extends this general movement by emphasizing long chain-of-thought critics and explicit refinement-aware RL (Tang et al., 20 Jul 2025).

Its distinctive claim is that supervised fine-tuning alone does not genuinely improve critique ability, because it optimizes the correctness label rather than the usefulness of the critique (Tang et al., 20 Jul 2025). The paper’s preliminary AIME25 comparison is used to illustrate this: SFT raises critique accuracy substantially, but yields only tiny or inconsistent downstream refinement gains relative to self-critique (Tang et al., 20 Jul 2025). This suggests a broader distinction between “surface critics,” which classify correctly, and critics that provide causally useful feedback. That distinction is also compatible with RCO’s observation that preferred critiques do not always correspond to better refinements (Yu et al., 27 Jun 2025).

The main limitations explicitly acknowledged are high compute cost and domain scope (Tang et al., 20 Jul 2025). Dual-reward RL with refinement sampling is expensive, and the experiments focus mainly on mathematical and logical reasoning, leaving generalization to commonsense or specialized professional domains open (Tang et al., 20 Jul 2025). A plausible implication is that the framework’s main bottleneck is not conceptual but operational: training useful critics requires repeated downstream rollouts to estimate whether feedback genuinely helps.

Another important interpretive point is that RefCritic is trained with solution-level supervision yet shows strong process-level behavior on ProcessBench (Tang et al., 20 Jul 2025). This suggests that detailed process understanding may emerge from refinement-aware optimization without explicit step annotations. However, this remains an empirical claim within the paper’s benchmark regime rather than a general theorem about critique learning.

Overall, RefCritic represents a conception of the critic as a feedback generator whose value is measured by the improvement it induces in the policy model, not only by the correctness of its verdict. In that respect, it marks a transition from critique as static evaluation toward critique as a trainable intervention mechanism (Tang et al., 20 Jul 2025).