---
title: 'RefCritic: Refinement-Aware Critique Training'
url: https://www.emergentmind.com/topics/refcritic
type: topic
---

# RefCritic: Refinement-Aware Critique Training

Searching arXiv for RefCritic and closely related critique-training papers.
RefCritic is a training framework for large language model critic modules that seeks to optimize critique not merely as correctness labeling, but as a source of actionable feedback for downstream refinement. Introduced in “RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback” [2507.15024], it is designed for settings in which a policy model first produces a solution and a separate critic then evaluates that solution, emits a correctness judgment, and provides suggestions intended to improve a revised answer. The framework is motivated by the claim that conventional supervised fine-tuning for critics often yields superficially correct judgments without producing critiques that materially help refinement. RefCritic therefore combines a cold-start supervised phase with reinforcement learning under dual rule-based rewards: one for instance-level judgment correctness and one for refinement success. Within the broader literature on critique-capable language models, it belongs to a shift from treating critique as an auxiliary textual byproduct toward treating it as a first-class optimization target [2507.15024][2506.22157][2509.22824].

## 1. Concept and problem formulation

RefCritic addresses a specific mismatch in critique-model training: a critic may classify a solution as correct or incorrect with high apparent accuracy while still providing shallow, misleading, or non-actionable reasoning [2507.15024]. The paper argues that this is especially problematic as reasoning model outputs become longer and more intricate, making direct human inspection harder and increasing reliance on machine critics.

The framework formalizes the critic as a model that takes a problem $x$ and an initial solution $y_0$ from a policy model, then outputs three components:
$$
C_\theta(x, y_0) \rightarrow (z, \hat{c}, f)
$$
where $z$ is a long chain-of-thought reasoning trace, $\hat{c} \in \{0,1\}$ is a correctness judgment, and $f$ is actionable refinement feedback [2507.15024]. This structure distinguishes RefCritic from critique systems that emit only a scalar verdict or only free-form commentary.

The central problem is not simply solution verification. Rather, it is the joint optimization of two coupled abilities: discriminative judgment over whether the original solution is correct, and generative production of feedback that enables a policy model to revise an incorrect solution into a correct one [2507.15024]. This framing aligns RefCritic with a broader line of work that treats critique as distinct from generation and correction, rather than assuming that strong generators automatically become strong critics [2402.14809].

A related implication emerges in comparison with CriticBench’s “GQC reasoning” decomposition into generation, critique, and correction [2402.14809]. RefCritic effectively targets the interface between the “Q” and “C” stages: it trains the critic so that its output is useful not just for judging but for changing subsequent model behavior. This suggests a stricter notion of critique quality than pure classification accuracy.

## 2. Critique model architecture and long chain-of-thought design

A defining feature of RefCritic is its use of a long chain-of-thought critic [2507.15024]. The paper presents output length growth during RL training as evidence that critics can become more detailed and reflective when optimized beyond supervised judgment imitation. After RL, average output length reportedly grows from about 500 tokens to 3500 tokens for RefCritic-Qwen and from about 3000 tokens to 8000 tokens for RefCritic-R1 [2507.15024].

This long-CoT design is not an incidental formatting choice. The paper’s thesis is that deeper critique requires extended reasoning traces capable of error identification, verification, and suggestion generation. In that respect, RefCritic differs from binary-judgment critique training schemes such as Critique Reinforcement Learning in Critique-Coder, where the reward is based solely on whether the final judgment label matches ground truth [2509.22824]. RefCritic retains the importance of correctness labels, but adds a second optimization target tied to the downstream utility of feedback.

The backbones used in the paper are Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B [2507.15024]. These models serve both as policy models that generate solutions and as bases for critic training. The choice of reasoning-oriented backbones is consistent with the paper’s emphasis on intricate mathematical solutions and long-form critique.

The long-CoT critic can also be situated relative to other structured or refinement-aware critics. RCO optimizes critics by rewarding critiques that induce better refinements, but it is framed around critique utility estimated from actor preferences rather than around long reasoning traces [2506.22157]. CRITIC-R1, in contrast, imposes explicit fields such as verdict, error location, reason, and fix for retrieval-augmented generation [2605.29886]. RefCritic is less schema-constrained than CRITIC-R1, but similarly motivated by the need for actionable diagnosis rather than superficial critique.

## 3. Training methodology and dual-reward reinforcement learning

RefCritic uses a two-stage training procedure [2507.15024]. The first stage is cold-start supervised fine-tuning with rejection sampling, intended to produce a format-compliant initial critic. The second stage is reinforcement learning with dual rule-based rewards, implemented using GRPO [2507.15024].

The RL setup begins with a policy model generating an initial solution $y_0$ for problem $x$. Given a ground-truth answer $a$, correctness is defined as:
$$
c = \mathbb{I}[y_0 = a] \in \{0,1\}
$$
The critic then outputs $(z,\hat{c},f)$, and the policy model produces $m$ refined solutions conditioned on the critique:
$$
y_i \sim P_\phi(y|x, y_0, f)
$$
for $i \in \{1,2,\dots,m\}$ [2507.15024].

The optimization objective is:
$$
\mathcal{J}(\theta) = \mathbb{E}[R_j(c, \hat{c}) + \lambda R_r(c, \hat{c}, a, \{y_i\}_{i=1}^m)]
$$
The first reward, instance-level judgment correctness, is
$$
R_j(c, \hat{c})=
\left\{
\begin{array}{rcl}
1 & & \text{if } c = \hat{c},\\
0 & & \text{otherwise}
\end{array}
\right.
$$
The second reward, refinement accuracy, is
$$
R_r(c, \hat{c}, a, \{y_i\}_{i=1}^m)=
\left\{
\begin{array}{lcl}
\frac{1}{m} \sum\limits_{i=1}^{m} \mathbb{I}[y_i = a]
& & \text{if } c = 0 \text{ and } c = \hat{c} \\
0
& & \text{otherwise}
\end{array}
\right.
$$
Thus refinement reward is only available when the original solution is wrong and the critic correctly identifies it as wrong [2507.15024].

The scalar $\lambda$ balances the two rewards. The paper uses a staged schedule: first train with $\lambda = 0$ for 600 steps, then continue with $\lambda = 1$ for 300 more steps [2507.15024]. This schedule is motivated by cost: refinement sampling is expensive, so the framework first improves verdict accuracy cheaply and only then incorporates refinement feedback.

This methodology places RefCritic within a broader family of refinement-aware critique optimization methods. RCO likewise argues that critique should be supervised by the improvement it induces in refined responses rather than by critique preference alone [2506.22157]. RefCritic differs in using explicit rule-based rewards over refined correctness rather than the Critique Utility estimate defined as a fractional win rate of refinement outputs over the original response [2506.22157]. The shared premise is that critique usefulness is causally downstream of refinement outcomes, not fully observable from critique text alone.

## 4. Data construction, supervision regime, and evaluation protocol

The training data for RefCritic is built from NuminaMath-1.5 [2507.15024]. The paper filters about 120K high-quality math problems from the original 900K by exact-string deduplication, semantic deduplication using `gte-multilingual-base` embeddings with cosine similarity threshold above 0.95 removed, filtering of unsolvable, proof, and multiple-choice problems using Qwen2.5-72B-Instruct, and difficulty balancing via eight sampled solutions per problem to exclude items where all attempts succeed or all fail [2507.15024].

For critic training, the authors sample 8 responses per problem, keep at most one correct and one incorrect response per problem, and filter incomplete generations [2507.15024]. Initial critique data is generated by stronger models and then filtered for correctness, instruction compliance, and leakage risk [2507.15024]. This pipeline is intended to ensure that the cold-start SFT phase yields a critic that already conforms to the intended task structure before RL begins.

The evaluation covers both in-domain and out-of-domain settings. Main math benchmarks are AIME24, AIME25, and OlympiadBench; out-of-domain benchmarks are LiveCodeBench and GPQA-Diamond; process-level evaluation is performed on ProcessBench [2507.15024]. Since RefCritic outputs natural language critiques rather than explicit step indices, the paper uses Qwen2.5-14B-Instruct as an extractor to map critiques to the earliest erroneous step index for ProcessBench, with F1 as the metric [2507.15024].

Two evaluation settings are central. In “Majority Vote with Critique,” the critic filters out solutions judged incorrect and majority voting is applied to the remaining solutions, denoted $\mathrm{Maj}_c@N$ [2507.15024]. In “Refinement after Critique,” the policy produces one solution, the critic evaluates it, and if it is judged incorrect the policy revises based on the critique; the final metric is reported as $\mathrm{Pass}_r@1$ [2507.15024].

This evaluation structure closely matches the paper’s conception of the critic as both selector and refiner. It also resonates with findings from CriticBench that critique and correction are related but imperfectly aligned capabilities [2402.14809]. RefCritic explicitly evaluates both functions rather than collapsing them into a single scalar judgment benchmark.

## 5. Empirical results and process-level behavior

The paper reports consistent improvements for refinement after critique on AIME25. RefCritic-Qwen-14B improves policy Pass@1 by 6.8%, and RefCritic-R1-14B improves policy Pass@1 by 7.2% [2507.15024]. Similar gains are reported on AIME24 and OlympiadBench, including 9.9% for Qwen-based supervision and 2.6% for R1-based supervision on Olympiad [2507.15024].

Under majority voting with critique, RefCritic yields an average 3.6 percentage point improvement at 64 samples on AIME25 over no critique [2507.15024]. The gains increase with the number of samples, which the paper interprets as evidence that RefCritic becomes more valuable when used as a filter over larger candidate pools. The table highlighted in the paper includes Qwen2.5-14B moving from 23.3 to 24.4 on AIME25 at 64 samples, and R1-Qwen-14B moving from 62.0 to 68.1 [2507.15024].

Out-of-domain transfer is also reported. On LiveCodeBench, RefCritic-R1-14B improves $\mathrm{Pass}_r@1$ from 51.0 to 54.1, and RefCritic-Qwen-14B improves from 18.9 to 22.9 [2507.15024]. On GPQA, RefCritic-R1-14B improves $\mathrm{Maj}_c@16$ from 61.6 to 65.1, and RefCritic-Qwen-14B improves from 23.3 to 24.3 [2507.15024]. These gains are smaller than the in-domain mathematical improvements, but they suggest nontrivial transfer.

One of the paper’s strongest claims concerns ProcessBench. RefCritic-Qwen-14B achieves average F1 of 68.0 and RefCritic-R1-14B achieves 77.1, outperforming prior critics including some trained with step-level supervision, despite RefCritic itself being trained only with solution-level supervision [2507.15024]. This suggests that long-CoT critics with refinement-aware RL can acquire implicit step-sensitive diagnostic behavior without explicit step labels.

Ablation results further support the dual-reward design. On AIME25 for RefCritic-Qwen-14B, baseline critic performance is 14.5, SFT reaches 15.0, $\mathrm{RL}_{\lambda=0}$ reaches 18.5, $\mathrm{RL}_{\lambda=1}$ reaches 19.5, and the two-stage schedule $\mathrm{RL}_{\lambda=0} \xrightarrow{after} \mathrm{RL}_{\lambda=1}$ reaches 21.2 [2507.15024]. Corresponding gains on AIME24 and OlympiadBench are also largest for the two-stage setup [2507.15024]. The pattern indicates that judgment-only RL is already much stronger than SFT, but refinement-aware reward yields further gains.

## 6. Position in the critique-learning literature, limitations, and interpretation

RefCritic sits within a rapidly developing research area that treats critique as a distinct learning problem. CriticBench showed that generation, critique, and correction are positively related but do not coincide, and that critique-focused training can substantially improve critique performance [2402.14809]. MetaCritique further argued that critique quality itself should be evaluated along factuality and comprehensiveness dimensions using Atomic Information Units and F1-style aggregation [2401.04518]. RCO reframed critic training around refinement outcomes rather than critique preference [2506.22157]. Critique-Coder introduced Critique Reinforcement Learning for coding, where the model learns to judge whether a proposed solution is correct using a binary label reward [2509.22824]. RefCritic extends this general movement by emphasizing long chain-of-thought critics and explicit refinement-aware RL [2507.15024].

Its distinctive claim is that supervised fine-tuning alone does not genuinely improve critique ability, because it optimizes the correctness label rather than the usefulness of the critique [2507.15024]. The paper’s preliminary AIME25 comparison is used to illustrate this: SFT raises critique accuracy substantially, but yields only tiny or inconsistent downstream refinement gains relative to self-critique [2507.15024]. This suggests a broader distinction between “surface critics,” which classify correctly, and critics that provide causally useful feedback. That distinction is also compatible with RCO’s observation that preferred critiques do not always correspond to better refinements [2506.22157].

The main limitations explicitly acknowledged are high compute cost and domain scope [2507.15024]. Dual-reward RL with refinement sampling is expensive, and the experiments focus mainly on mathematical and logical reasoning, leaving generalization to commonsense or specialized professional domains open [2507.15024]. A plausible implication is that the framework’s main bottleneck is not conceptual but operational: training useful critics requires repeated downstream rollouts to estimate whether feedback genuinely helps.

Another important interpretive point is that RefCritic is trained with solution-level supervision yet shows strong process-level behavior on ProcessBench [2507.15024]. This suggests that detailed process understanding may emerge from refinement-aware optimization without explicit step annotations. However, this remains an empirical claim within the paper’s benchmark regime rather than a general theorem about critique learning.

Overall, RefCritic represents a conception of the critic as a feedback generator whose value is measured by the improvement it induces in the policy model, not only by the correctness of its verdict. In that respect, it marks a transition from critique as static evaluation toward critique as a trainable intervention mechanism [2507.15024].

Source: https://www.emergentmind.com/topics/refcritic