---
title: Policy Refinement by Critique
url: https://www.emergentmind.com/topics/policy-refinement-by-critique
type: topic
---

# Policy Refinement by Critique

Policy refinement by critique is a paradigm in large language model (LLM) training wherein model policies are improved not simply by attaching gradient signals to a scalar reward, but by leveraging explicit, often natural-language, critiques as intermediate artifacts. These critiques can be generated by the model itself or a secondary module ("critic") and are then actively used to guide response refinement, provide supervision to reward models, or structure learning objectives for both policy and critic components. This approach unifies several recent advances in RLHF, supervised distillation, and self-improving systems by incorporating explicit feedback loops that enable models to "think out loud" and self-correct, thus achieving levels of performance and alignment more robust than possible with direct preference modeling alone [2408.11791, 2510.24320, 2507.15024].

## 1. Core Mechanisms and Architectural Patterns

Policy refinement by critique characteristically operates via a multi-stage interaction between a "policy" (actor) and a "critic" (which may itself be a language model or a specialized module).

- **Critique generation:** Following a candidate response $y \sim \pi(\cdot|x)$ to a prompt $x$, a critic model produces a free-form, structured, or chain-of-thought critique $c$, which may include both discriminative evaluations and actionable suggestions.
- **Refinement:** The policy model conditions on $(x, y, c)$ to generate an improved response $y' \sim \pi(\cdot|x,y,c)$.
- **Feedback integration:** These $(x, y, c, y')$ tuples feed into supervised, preference-based, or reinforcement learning objectives, with the ultimate aim of shaping the policy to be more aligned, accurate, or helpful on future rollouts.

This loop can be instantiated in both supervised (SFT or distillation) and RL settings, and supports both one-off and iterative improvement protocols [2505.11628, 2512.05387, 2502.09183].

## 2. Formal Training Objectives and Algorithms

Several formalizations underlie contemporary critique-based methods:

- **Critique-out-Loud (CLoud):** Models combine a language modeling head to produce $c \sim p(c|x,y;\theta)$ and a reward head predicting $r_\theta(x,y,c)$, training on a loss
  $$
  \mathcal{L}_{\rm CLoud} = \mathcal{L}_{\rm RM} + \lambda \mathcal{L}_{\rm SFT}
  $$
  where $\mathcal{L}_{\rm SFT}$ is cross-entropy on critiques and $\mathcal{L}_{\rm RM}$ is pairwise Bradley–Terry preference loss over $(x, y^+, c^+), (x, y^-, c^-)$ [2408.11791].

- **Critique-RL:** A two-stage RL objective. First, the critic $\pi_\phi$ is optimized for discriminability with reward $r_{\rm dis}(x,y,c)$ matching a verifier. Then the critic is tuned for helpfulness via actor refinement rewards $r_{\rm refine}(x,y,c,y')$, regularized to preserve discrimination:
  $$
  J_{\rm help}(\phi) = \mathbb{E}[ r_{\rm refine} + \beta_1 r_{\rm dis} - \beta_2 \mathrm{KL}[\pi_\phi^{\rm Stage\text{-}I}, \pi_\phi] ]
  $$
  The actor is supervised to refine $y$ into $y'$ given $c$, but is not updated during the RL training of the critic [2510.24320].

- **RefCritic:** Critic is RL-trained with both instance-level correctness reward $r_{\rm correct}$ and refinement accuracy $r_{\rm refine}$ for actionable feedback. Only the critic is updated; the policy is refined online via injected critiques at test/inference time [2507.15024].

- **Critique-guided methods in SFT/distillation:** CGD uses teacher-generated explanatory critiques as conditioning signals, training the student to map $(x, y', c) \rightarrow \hat y$, where $y'$ is its pre-finetuning response and $c$ the teacher's critique [2505.11628]. SCRPO collects $(x, y^+, y^-)$ preference triples—where $y^+$ is the refined response after critique/refinement and $y^-$ is the original—and optimizes a DPO-style ranking loss [2512.05387].

- **RCO (Refinement-oriented Critique Optimization):** Critic models are updated such that the critiques they generate maximize the probability that the actor's refinement (conditioned on the critique) will be strictly preferred over the original, as determined by an external judge. The utility $\mathrm{CU}$ of a critique $c$ quantifies the expected quality improvement downstream, and forms the reward for critic learning [2506.22157].

## 3. Empirical Gains, Benchmark Outcomes, and Ablations

Policy refinement by critique consistently outperforms direct reward modeling or imitation learning across established reasoning, code, and summarization benchmarks:

| Method (Dataset/Model)            | Baseline                    | Critique-Refinement       | Gain          | Source        |
|-----------------------------------|-----------------------------|---------------------------|--------------|---------------|
| RewardBench (pairwise acc, 8B)    | Classic RM 72.5%            | CLoud 77.2%               | +4.65 pp     | [2408.11791]  |
| AIME25 (pass@1, Qwen-14B)         | 14.4%                       | RefCritic 21.2%           | +6.8 pp      | [2507.15024]  |
| PersonaFeedback (Qwen2.5-7B/PPO)  | 53.1%                       | Critique-Post-Edit 64.1%  | +11.0 pp     | [2510.18849]  |
| MATH (Acc@Refine, Qwen2.5-7B)     | 45.74% (no critic)          | Critique-RL 58.40%        | +12.66 pp    | [2510.24320]  |
| BBH (accuracy, LLaMA-2-7B)        | 39.67                       | RCO 48.06                 | +8.39        | [2506.22157]  |
| HumanEval (RefineCoder-DS-6.7B)   | 73.8 (iter 0)               | 75.0 (iter 3, +1.2)       | +1.2         | [2502.09183]  |

Ablations demonstrate that omitting the critique-refine step (e.g., D_refine in [2503.16024], or selective critic module in [2502.09183]) significantly degrades performance, highlighting the causal role of actionable critiques. Self-consistency and critique-sampling at inference can further boost accuracy for short-horizon tasks [2408.11791]; refinement-based supervision is robust to different judge models and operates across model sizes [2506.22157].

## 4. Critique Types, Utility Estimation, and Training Strategies

- **Types of critiques:** Approaches variously use binary judgments, fine-grained or fact-level explanations [2512.05387], multi-dimensional critiques (helpfulness, personalization, naturalness) [2510.18849], or long chain-of-thought traces [2507.15024].
- **Utility of critiques:** The primary criterion is observed downstream improvement of actor outputs; utility can be explicitly quantified via Critique Utility (CU) or other paired-preference metrics [2506.22157].
- **Supervision signals:** Both direct (correctness matching, explicit refinement success) and indirect (DPO/ranking objectives, offline preference data) signals are used.
- **Critique generation:** Training may rely on manually curated traces, distillation from stronger models, self-generated critiques, or online RL-based optimization [2512.05387, 2505.11628, 2507.15024].
- **Policy updating/coupling:** In some frameworks, only the critic is updated and the actor is refined at inference (RefCritic, RCO); in others, actor and critic are updated jointly, or the actor is iteratively SFTed to incorporate feedback [2503.16024, 2512.05387].

## 5. Extensions, Limitations, and Open Challenges

- **Extensions:** Proposed directions include learning multi-objective reward heads to decompose aspects of critique [2408.11791], integrating human-written rubrics [2408.11791], explicit mixture-of-experts for different sub-tasks [2408.11791], and extending to multi-turn dialogue and agent chains [2408.11791], as well as interactive online RLHF loops with critique-guided PPO updates [2510.24320].
- **Limitations:** Effectiveness of on-policy self-generated critiques depends on the initial critic's quality; off-policy or externally sourced critiques can degrade learning. Critique sampling and self-consistency are effective only for short-horizon reasoning or summarization tasks. Additional computational overhead from decoding critiques may impact efficiency [2408.11791]. Single-round refinement may limit performance versus multi-turn iterative refinement; actor models may remain sub-optimal if not co-trained with critiques [2506.22157].
- **Generalization:** The critique-refinement loop has been validated in code (RefineCoder), math (Critique-GRPO, RefCritic), instruction following, dialogue, and summarization (SCRPO). A plausible implication is that any domain where outputs are amenable to actionable feedback and subsequent revision can benefit from this paradigm [2502.09183, 2512.05387].

## 6. Comparison with Classical RLHF and Distillation Protocols

The critique-refinement paradigm systematically addresses key failure modes of reward hacking and imitation learning found in direct RLHF and vanilla SFT/distillation:

- **Superficial alignment:** Scalar reward models can be gamed (reward hacking) and do not provide actionable error signals; critiques articulate missing reasoning and style flaws explicitly [2510.18849].
- **Imitation without understanding:** SFT on correct responses alone leads to high entropy but shallow understanding [2505.11628]; critique-guided learning reduces uncertainty and sharpens the model's posterior over correct outputs.
- **Exploration/exploitation:** Critique-driven updates favor rare but high-quality refinements over random exploration; longer or higher-entropy outputs alone do not yield better improvements [2506.03106].
- **Format preservation:** Critique-based distillation mitigates format drift and overfitting compared to critique-only fine-tuning [2505.11628].

In summary, policy refinement by critique integrates explanatory feedback as a central learning signal, structurally outperforming direct reward modeling or imitation by enabling LLMs to internalize both "what to do" and "why," resulting in measurable advances in alignment, faithfulness, and reasoning. The approach is characterized by explicit feedback loops, composite supervision, and strong empirical validation across tasks [2408.11791, 2503.16024, 2507.15024].

Source: https://www.emergentmind.com/topics/policy-refinement-by-critique