Critique-guided Improvement (CGI)

• Critique-guided improvement (CGI) is a framework that leverages natural language critique to identify errors and iteratively refine model outputs.
• CGI systems employ generate–critique–revise loops where solver and critic roles interact to optimize performance across domains like code generation and mathematical reasoning.
• Empirical studies show that actionable, optimized feedback significantly improves model outcomes, though challenges remain in internalization and evaluation.

Critique-guided improvement (CGI) denotes a family of methods in which a model’s outputs are evaluated through natural-language critique and the resulting feedback is used to revise outputs, optimize critics, or internalize corrections into the underlying policy. Across recent work on language-model reasoning, code generation, agentic control, safety evaluation, multimodal understanding, and scientific problem solving, CGI is presented as an alternative to purely scalar supervision because critique can specify why an output is wrong and how it should change, rather than only assigning a reward or preference score (Lin et al., 13 May 2026, Zhang et al., 3 Jun 2025, Yang et al., 20 Mar 2025).

1. Conceptual foundations

A central distinction in the CGI literature is between critique-conditioned behavior and critique-guided improvement of the underlying policy. ICRL formalizes this as the difference between acting under $\pi(y \mid q, c)$, where success depends on explicit critique $c$, and improving the critique-free policy $\pi(y \mid q)$ so that performance persists after critique is removed (Lin et al., 13 May 2026). In this formulation, “internalization” means that the model absorbs correction patterns that were previously supplied externally.

This same shift appears in supervised settings. Critique Fine-Tuning replaces the usual supervised target “generate the correct response” with “generate a critique of a noisy response,” training on triples $(x,y,c)$ and maximizing $\log P_\theta(c \mid [x;y])$ rather than $\log P_\theta(r \mid x)$ (Wang et al., 29 Jan 2025). The result is a model that has learned to analyze solution quality rather than merely imitate canonical answers. In prompt optimization, CriSPO applies the same logic at the instruction level: an LLM generates multi-aspect critiques and suggestions comparing outputs to references, and a separate optimizer uses those critiques to rewrite the prompt (He et al., 2024).

This suggests that CGI is best understood not as a single algorithm but as a supervision principle: critique is treated as a structured intermediate object that can shape inference, optimization, or distillation.

2. Canonical loop and role structure

Most CGI systems instantiate a generate–critique–revise loop. In ICRL, for a query $q$, the solver first produces a critique-free trajectory

$\tau_1 \sim \pi^\mathcal{S}_\theta(\cdot \mid q),$

the critic then generates feedback for failed trajectories

$$c_i \sim \pi^\mathcal{C}_\theta(\cdot \mid q, \tau_i),$$

and the solver produces a revised attempt

$$\tau_{i+1} \sim \pi^\mathcal{S}_\theta(\cdot \mid q, c_i).$$

The full session is

$c$0

ICRL uses a shared backbone for solver and critic, differing only by role prompts, so that critique improvement and solver improvement update the same parameter set (Lin et al., 13 May 2026).

Agentic CGI adopts the same structure with environment actions. “The Lighthouse of Language” defines an actor model that samples candidate actions, a critic that produces structured feedback over Contribution, Feasibility, Efficiency, Overall Grading, and Suggested Revision, and an actor refinement step that conditions on the critique set before executing the final action (Yang et al., 20 Mar 2025). ProCrit applies an analogous draft–critique–revise protocol to multimodal sarcasm detection: a proposal agent generates multi-perspective reasoning and a label, a critic outputs a score $c$1 and targeted natural-language feedback, and the proposal revises from scratch under that feedback (Xu et al., 20 May 2026).

Some systems add a third evaluative role. SCALAR uses an Actor–Critic–Judge pipeline for graduate quantum field theory and string theory problems: the Actor proposes a solution, the Critic reviews it with access to a reference answer, and an independent Judge scores the transcript and determines pass/fail or early stopping (Niarchos et al., 7 May 2026). SAFETY-J likewise separates generation from evaluation by producing a safety label followed by a critique, then using downstream meta-evaluation to improve the evaluator itself (Liu et al., 2024).

3. Learning the critic

A defining question in CGI is how the critic is trained so that its feedback is both accurate and actionable. Recent work spans direct supervision, self-validation, reinforcement learning, and automated meta-evaluation.

SCRIT develops a self-evolving critic for mathematical reasoning without human or stronger-model critique supervision. It defines a critique as a structured object $c$2 containing step-wise critiques $c$3, a global conclusion $c$4 about correctness and first error step, and a correction $c$5. Synthetic critiques are generated by a contrastive critic conditioned on a reference correct solution, and then filtered by self-validation:

$c$6

meaning the correction is independently classified as fully correct with no error step (Tang et al., 10 Jan 2025). This turns correction success into a binary quality signal on the critique itself.

Critique-RL makes the critic’s two roles explicit: discriminability and helpfulness. Stage I trains discriminability with a direct rule-based reward

$c$7

so the critic learns to judge whether the actor’s original answer is correct. Stage II then adds indirect reward from actor refinement,

$c$8

while regularizing toward the Stage I policy so that helpfulness improves without collapsing discriminability (Xi et al., 28 Oct 2025).

CTRL trains a code critic through reinforcement learning against a fixed generator. For a problem–solution pair $c$9, the critic samples a textual critique $\pi(y \mid q)$0, the generator revises code conditioned on $\pi(y \mid q)$1, and the critic is optimized to maximize the probability that the revised code passes unit tests. The RL stage uses Group Relative Policy Optimization, with group-normalized advantages computed from pass/fail outcomes of multiple critiques for the same initial program (Xie et al., 5 Feb 2025).

SAFETY-J introduces an automated meta-evaluation benchmark for critiques themselves. Human reference critiques are decomposed into Atomic Information Units, and critique quality is scored by AIU-level precision, recall, and

$\pi(y \mid q)$2

Those scores are then converted into preference pairs for iterative Direct Preference Optimization, yielding successive models $\pi(y \mid q)$3 through $\pi(y \mid q)$4 (Liu et al., 2024). In math reasoning, AutoMathCritique builds a 76,321-example critique dataset using flawed-path generation, GPT-4o step-level critique, and Monte Carlo filtering that keeps a critique when its refinement success rate exceeds $\pi(y \mid q)$5 (Xi et al., 2024).

Across these systems, a recurring principle is that critique quality is not treated as fluency alone. It is grounded in correction outcomes, verdict accuracy, or meta-evaluated semantic coverage.

4. Internalization, distribution shift, and refinement dynamics

A central technical problem in CGI is that successful revised trajectories are typically sampled under critique-conditioned distributions rather than the target critique-free policy. ICRL makes this issue explicit: revised attempts come from $\pi(y \mid q)$6, so naïvely optimizing on them biases learning toward a policy that implicitly assumes critique is present. To correct this, ICRL defines a token-level distribution-calibration ratio

$\pi(y \mid q)$7

and clips it with $\pi(y \mid q)$8 in the final GRPO objective (Lin et al., 13 May 2026). Tokens that remain plausible under the critique-free prompt are transferred more strongly; tokens that depend heavily on critique are down-weighted.

Critique-GRPO addresses a related issue by jointly learning from initial responses and critique-guided refinements, while using a shaping function on refined trajectories to emphasize low-probability tokens that encode genuinely novel corrective behavior. Its analysis shows that higher entropy does not always guarantee efficient learning from exploration, and longer responses do not necessarily lead to more effective exploration (Zhang et al., 3 Jun 2025).

The internalization question also appears in distillation settings. In frontend code generation, a fully automated critic-in-the-loop pipeline with a vision-LLM improves rendered webpage quality over multiple critique cycles, and LoRA fine-tuning on the best critique-guided outputs captures 25% of the gains from the best critic-in-the-loop solution without a significant increase in token counts (Sansford et al., 7 Apr 2026). The same paper notes that the distilled model captures only part of the benefit, indicating that some improvements remain instance-specific and still require live visual critique. Critique Fine-Tuning exhibits an even stronger form of offline internalization: models trained only to generate critiques solve math problems better under direct inference, while single-pass or two-stage self-critique inference on those same models is slightly worse than direct solving (Wang et al., 29 Jan 2025).

This suggests that CGI has two distinct operating modes. In one, critique remains an online control signal. In the other, critique-generated corrections are absorbed into the base model, but only to the extent that they are compatible with its native policy and training distribution.

5. Empirical landscape across domains

The empirical record for CGI is broad and heterogeneous, but the aggregate pattern is consistently positive when critics are trained or calibrated rather than used as static prompts. In ICRL, jointly training solver and critic from a shared backbone yields average gains of 6.4 points over GRPO on agentic tasks and 7.0 points on mathematical reasoning, and the learned 8B critic is reported as comparable to 32B critics while using substantially fewer tokens (Lin et al., 13 May 2026). Critique-GRPO reports average pass@1 improvements of approximately 4.5% over supervised fine-tuning approaches and 5% over RL-based approaches across eight mathematical, STEM, and general reasoning tasks (Zhang et al., 3 Jun 2025). Critique Fine-Tuning reports 4–10% gains across six mathematical reasoning benchmarks, and states that Qwen2.5-Math-CFT matches the performance of SimpleRL while using 140× less compute (Wang et al., 29 Jan 2025).

In code generation, the strongest reported improvements arise when critiques are optimized against execution outcomes. CTRL reports up to 106.1% relative improvements across challenging code generation benchmarks, and its critics improve pass rates while maintaining low regression under iterative critique–revision (Xie et al., 5 Feb 2025). In frontend generation, a critic-in-the-loop system that renders webpages and asks a visual critic to analyze screenshots yields up to 17.8% performance increase over three refinement cycles on WebDev Arena tasks (Sansford et al., 7 Apr 2026).

Multimodal and safety-oriented CGI show the same pattern. ProCrit improves multimodal sarcasm detection from 81.1 to 83.1 F1 on MMSD2.0 and from 78.4 to 80.7 F1 on MMSD when moving from draft-only reasoning to critic-guided revision, while self-revision without feedback is worse than using no revision at all (Xu et al., 20 May 2026). SAFETY-J reports that feeding its critiques to a generator increases the safety rate of revised outputs from 0.42 for the original responses to 0.78 with the most refined critique model $\pi(y \mid q)$9 (Liu et al., 2024).

Scientific and agentic settings expose both the promise and the variability of CGI. “The Lighthouse of Language” reports that an 8B actor+critic system reaches an average score of 74.20 across WebShop, Science World, and TextCraft, outperforming stronger closed-source and specialized agent baselines in its setup (Yang et al., 20 Mar 2025). SCALAR shows that multi-turn dialogue improves over single-shot attempts throughout on quantum field theory and string theory problems, but that the value of critique strategy depends strongly on the Actor–Critic pairing: constructive feedback matters most clearly in asymmetric settings such as a lightweight Haiku Actor with a stronger Sonnet Critic (Niarchos et al., 7 May 2026).

6. Limitations, misconceptions, and open problems

The CGI literature is explicit that critique is not automatically beneficial. ICRL argues that frozen critics plateau as the solver improves, because a static critic cannot adapt to new failure modes or exploit reinforcement signals about which critiques actually improve downstream performance (Lin et al., 13 May 2026). SCRIT notes that self-validation depends on domains with verifiable answers; its own formulation is grounded in mathematics, where corrections can be judged against objective final answers, and the paper identifies domain restriction as a core limitation (Tang et al., 10 Jan 2025).

A common misconception is that harsher or more frequent critique should always help. SCALAR directly contradicts this: in same-family Actor–Critic settings, strict and adversarial feedback are not beneficial, while lenient feedback is sometimes favored; in asymmetric settings, constructive feedback improves mean-score outcomes more clearly (Niarchos et al., 7 May 2026). ProCrit shows the same boundary condition from another angle: self-revision without external feedback is worse than retaining the original draft, whereas revision with an RL-refined critic is best (Xu et al., 20 May 2026). In frontend generation, refinement without critic improves only about 1.2–1.5%, far below the gains from the full visual critic-in-the-loop system (Sansford et al., 7 Apr 2026).

Another misconception is that more exploration, longer reasoning, or more critique rounds necessarily improve learning. Critique-GRPO explicitly reports that higher entropy does not always guarantee efficient learning from exploration and that longer responses do not necessarily lead to more effective exploration (Zhang et al., 3 Jun 2025). CATPO finds that critique-guided healing fails on the subset of dead-wrong reasoning trees that are fundamentally beyond the model’s current capability, even when the critique correctly identifies the error; in those cases refined continuations repeat similar mistakes (Singh et al., 6 Jun 2026). Critique-RL identifies the same structural tension in critic training: optimizing helpfulness alone produces critics that are too conservative or too aggressive, while discriminability deteriorates unless it is explicitly preserved (Xi et al., 28 Oct 2025).

Evaluation itself is a persistent difficulty. SAFETY-J relies on AIU-based meta-evaluation with LLM entailment judgments, and SCALAR shows that absolute Judge scores and some strategy comparisons remain Judge-dependent even when the sign of multi-turn improvement is stable (Liu et al., 2024, Niarchos et al., 7 May 2026). Open problems recur across papers: measuring “true internalization,” extending CGI beyond domains with verifiable rewards, scaling multi-round and multi-agent critique, integrating human or external critiques, and designing asynchronous or more compute-efficient training systems for long critique–revision rollouts (Lin et al., 13 May 2026).

Taken together, the literature presents CGI as a technically coherent but nontrivial paradigm. Critique can function as a process-level supervision channel, a learned verifier, an optimization target, a distillation signal, or an online control policy. Its strongest results occur when feedback is explicitly trained to be actionable, when off-distribution critique-conditioned behavior is controlled, and when evaluation can distinguish useful criticism from merely plausible text.