---
title: Generate–Critique–Refine Loop in AI
url: https://www.emergentmind.com/topics/generate-critique-refine-loop
type: topic
---

# Generate–Critique–Refine Loop in AI

The **Generate–Critique–Refine loop** is an iterative pattern in which a model or agent first produces a candidate artifact, then critiques that artifact against task-specific criteria, external evidence, or both, and finally produces a revised candidate conditioned on the critique. Recent work instantiates this pattern for reasoning traces, plans, personas, frontend code, personalized reviews, activity diagrams, educational feedback, design critiques, and even hidden states inside an image editor, with critics that may be self-based, cross-model, algorithmic, visually grounded, or externally supervised [2606.09637][2501.14492][2604.21611].

## 1. Formal structure and operational variants

A canonical formulation separates an **actor** or **generator** from a **critic**. In Verbal Process Supervision (VPS), an initial reasoning trajectory is generated as $\tau_0 \sim \pi_\theta(\cdot \mid x)$, a supervisor produces critique $c_r = \mathcal{C}(\tau_r, x)$, and the actor regenerates a revised trajectory as $\tau_{r+1} \sim \pi_\theta(\cdot \mid x, \tau_r, c_r)$, repeating up to a round budget $R$ or until a stopping criterion is met [2604.21611]. RealCritic makes the same pattern explicit in three settings: **self-critique**, where the same model generates and critiques its own solution; **cross-critique**, where one model critiques another model’s solution; and **iterative critique**, where critique and correction are repeated over multiple rounds [2501.14492].

The stopping rule is a defining part of the loop. In PerGent, the loop “runs for a **user-defined number of rounds** or stops earlier when the critic indicates that no major issues remain,” and the orchestrator terminates execution when an agent signals completion or a **user-defined maximum number of round-robin cycles** is reached [2606.09637]. Other systems use fixed caps such as **three refinement cycles** for frontend code generation, **$T=5$** iterations for PerFine, or **$k=10$** self-improvement iterations for intrinsic planning self-critique [2604.05839][2510.24469][2512.24103].

The literature also shows that the loop need not always be an outer multi-pass cycle. “Inline Critic Steers Image Editing” embeds critique directly inside a single forward pass of a frozen image editor: a learnable critic token predicts intermediate-layer error structure and then steers later hidden states before denoising finishes [2605.12724]. By contrast, “Critique Fine-Tuning” treats critique primarily as a **training objective** and states that “during inference time, the trained model simply generates the response without involving any critique or refinement process” [2501.17703].

## 2. Components, orchestration, and control flow

A recurring design pattern is explicit orchestration. PerGent uses a centralized **orchestrator**, a **generator agent**, and a **critic agent**. The orchestrator controls turn order with a round-robin policy, stores the shared conversation state in `orchestrator.history`, and passes each agent a prompt built from `role`, `instructions`, `actionList`, `memory`, and `history`. The generator produces an initial persona when `history` is empty and otherwise refines; the critic evaluates the latest persona and emits either feedback or an approval declaration [2606.09637]. This makes the loop stateful rather than stateless prompt chaining.

Other systems add control modules before or around critique. DeCRIM introduces a **decomposer** that rewrites a multi-constraint instruction into an enumerated list of explicit constraints and then lets a critic decide which constraints failed before refinement is invoked [2410.06458]. DCR inserts a **Detect** stage before critique so that only errorful sentences are critiqued and refined, rather than forcing the critique model to both verify and explain everything [2407.02397]. REFINE decomposes feedback into components such as current-state diagnosis, task-level next steps, strategy-level guidance, self-regulated learning support, and praise, and then regenerates **only the failing components** identified by a human-aligned LLM judge [2603.29142].

Orchestration also determines how harmful refinements are contained. PerFine uses a **knockout strategy** that compares the current and previous drafts and retains the stronger one across rounds [2510.24469]. The frontend critic-in-the-loop framework maintains a **best-so-far** candidate across the refinement trajectory rather than assuming the final cycle is optimal [2604.05839]. LADEX restarts generation if a candidate still fails critique after **five refinement iterations**, and it reports that no input required more than one restart [2509.03463]. These variants show that refinement is often treated as a search process over a trajectory of candidates rather than a monotonic improvement operator.

## 3. Forms of critique and sources of supervision

The content of the critique varies sharply across domains. In math reasoning, DeepCritic trains a critic to perform **deliberate step-wise critiques** with **multi-perspective verifications** and **in-depth critiques of initial critiques for each reasoning step**, returning the first erroneous step or \(-1\) if all steps are correct [2505.00662]. AutoMathCritique similarly targets **step-level natural-language feedback** and the **first error location**, and then uses that critique both at test time and in self-improvement training [2411.16579]. VPS requires the supervisor to label each step as **correct — do not revise**, **partially correct — describe the issue**, or **incorrect — give a targeted correction**, and to emit **CONVERGED** when everything is correct [2604.21611].

Several papers distinguish critique from scalar scoring. RealCritic argues that critique should be evaluated through a **closed-loop critique-and-correction framework** rather than by judging critique text in isolation [2501.14492]. CLoud reward models implement a **candidate response $\rightarrow$ generated critique $\rightarrow$ scalar reward** pipeline, showing that critique can mediate scoring even when explicit refinement is replaced by selection [2408.11791]. CFT goes further and trains directly on supervision of the form $([x;y], c)$, where the model critiques a **query–noisy-response pair** instead of imitating a final answer [2501.17703].

Grounding sources also differ. PerGent conditions both generator and critic on **external resources and target persona description**, where the resources include **25 interview transcripts, 28 surveys, and 47 job postings** across ten roles [2606.09637]. In intrinsic planning self-critique, the critic is scaffolded by domain rules: for each action it must take the action and its preconditions from the domain definition, verify whether the preconditions are met, and apply the action to provide the resulting state [2512.24103]. PerFine grounds critique in a retrieved user profile and asks for structured feedback on **Tone Consistency**, **Vocabulary Match**, **Sentence Structure**, and **Topic Relevance** [2510.24469]. Frontend code generation uses a **vision-language model** as a visual critic over rendered screenshots, followed by a code critic that translates visual observations into actionable code-oriented recommendations [2604.05839]. UI design critique generation likewise iteratively refines both design comments and the corresponding bounding boxes that ground each comment in the screenshot [2412.16829].

A broader pattern is that effective critiques are usually more than verdicts. DeepCritic’s long-form critiques, DCR’s error-span-plus-fix feedback, PerFine’s profile-grounded JSON feedback, and CLoud’s critique-before-score architecture all treat critique as a semantically rich object rather than a scalar label [2505.00662][2407.02397][2510.24469][2408.11791].

## 4. Empirical evidence across domains

Reported results consistently show that critique-conditioned refinement can improve quality, but the magnitude depends on the task, the critic, and the grounding signal.

| Domain | Reported effect | Citation |
|---|---|---|
| Industrial persona authoring | **96.90% approval**, **3.10% edit**, **0% remove**, **0% add** | [2606.09637] |
| Symbolic planning (Blocksworld 3–5) | **49.8±4.0%** to **85.5±2.8%**; **89.3±2.5%** with self-consistency | [2512.24103] |
| Frontend code generation | up to **17.8%** increase over three refinement cycles | [2604.05839] |
| Reasoning with VPS | **94.9%** on GPQA Diamond at **$R=4$**; weak-actor rescue from **11.7–26.7%** to **63.3–90.0%** on AIME 2025 | [2604.21611] |
| Activity diagram extraction | up to **86.37%** correctness and **88.56%** completeness, with **fewer than five LLM calls on average** | [2509.03463] |
| Inline image editing | **GEdit-Bench 7.89**, **+9.4** gain on RISEBench over the same backbone, and **KRIS-Bench 81.92** | [2605.12724] |

Other domains show the same pattern. DeCRIM improves Mistral’s performance by **7.3% on RealInstruct** and **8.0% on IFEval**, and with strong feedback open-source LLMs with DeCRIM can outperform GPT-4 on both benchmarks [2410.06458]. DeepCritic raises Qwen2.5-7B-Instruct from **74.00** to **77.20** on MATH500 and raises Qwen2.5-72B-Instruct from **77.00** to **82.00** under critique-based refinement [2505.00662]. PerFine reports **GEval gains of +7–13%** over PGraphRAG, with steady improvements over **3–5 refinement iterations** and gains that scale with critic size [2510.24469]. REFINE reports that judge-guided regeneration significantly improves feedback quality, while its interactive agent produces responses comparable to a state-of-the-art closed-source model [2603.29142]. CLoud reward models improve RewardBench pairwise preference classification accuracy by **4.65** points for Llama-3-8B and **5.84** points for Llama-3-70B, and also improve Best-of-N selection on ArenaHard [2408.11791]. In multimodal UI critique, iterative visual prompting reduces the gap from human performance by **50%** for one rating metric [2412.16829].

## 5. Evaluation paradigms, misconceptions, and failure modes

A central methodological dispute concerns how critique quality should be measured. RealCritic argues that open-loop evaluation is misleading because critique is open-ended, and it reports that about **30% of low-quality critiques are erroneously classified as high-quality** by verdict-style evaluation; among critiques judged low-quality by humans, **26.0%** still had verdicts matching the ground truth [2501.14492]. The paper therefore scores critique by whether it produces a **better corrected answer**. This position is echoed by systems that evaluate the whole loop against expert edits, reference artifacts, or downstream task correctness rather than stylistic plausibility alone.

A second misconception is that high approval implies completeness. PerGent explicitly warns against this: although PerGent reached **96.90% approval** in in-situ expert review, its comparison with pre-LLM expert personas showed average **full preservation 0.424** and **partial preservation 0.678**, meaning a nontrivial amount of expert-authored content was still missing [2606.09637]. The paper’s “Lesson 2” states that strong expert approval of a rich draft should not be mistaken for evidence of completeness.

A third misconception is that self-critique or extra rounds automatically help. RealCritic finds that, in self-critique, classical LLMs may even underperform relative to their baseline capabilities, and only **o1-mini** shows positive average self-critique improvement among the tested models [2501.14492]. CFT reaches the same caution from the training side: its critique-trained models “currently lack the ability to perform self-critique,” and direct inference outperforms their tested self-critique variants [2501.17703]. VPS reports **non-monotonic** round scaling across $R \in \{1,2,3,4\}$ and attributes degradation to **over-correction**, **compounding errors**, and **critique drift** [2604.21611]. Frontend code generation shows monotonic mean gains but also emphasizes **performance regressions and the diminishing returns of the critique**; this is exactly why the algorithm keeps the best candidate found across the trajectory [2604.05839].

Failure modes are strongly domain dependent. VPS reports that performance degrades when decisive errors are not **linguistically expressible**, especially in code synthesis, and therefore motivates **hybrid verbal–executable methods** [2604.21611]. RealCritic shows that specialized STEM multiple-choice tasks such as GPQA and MMLU-STEM are particularly dangerous for critique-refine loops, with large C→I degradations in some settings [2501.14492]. By contrast, intrinsic self-critique becomes much more effective in symbolic planning because the prompt can force the model to verify **action preconditions** and **state transitions** under explicit domain rules [2512.24103].

## 6. Recurrent design patterns and research directions

Across the literature, several design patterns recur. One is **role separation**: actor and critic are distinct in PerGent, VPS, CGI, DeepCritic, AutoMathCritique, and many evaluation setups, even when the same base model family is reused [2606.09637][2604.21611][2503.16024][2505.00662][2411.16579]. Another is **statefulness**: successful industrial and agentic systems preserve conversation history, prior drafts, critiques, or failed attempts rather than issuing isolated prompts [2606.09637][2512.24103]. A third is **schema or component awareness**: personas use a fixed four-section template, REFINE judges pedagogical components separately, LADEX separates structural and alignment constraints, and PerFine critiques exactly four personalization dimensions [2606.09637][2603.29142][2509.03463][2510.24469].

The literature also repeatedly narrows the critic’s job before asking it to refine. DCR adds **Detect** so critique focuses only on flagged sentences [2407.02397]. DeCRIM adds **Decompose** so critique is constraint-wise rather than holistic [2410.06458]. LADEX replaces LLM structural critique with deterministic checks whenever the property is formally machine-verifiable, and this alone improves correctness by **17.81%** and completeness by **13.24%** over LLM-only checks [2509.03463]. REFINE preserves passing feedback components and regenerates only the failing ones [2603.29142]. A plausible implication is that critique becomes substantially more useful when verification, localization, and revision scope are disentangled.

Another recurring theme is that critique ability is itself a trainable capability. CFT shows that learning to critique noisy responses can outperform answer imitation by **4–10%** on math benchmarks [2501.17703]. DeepCritic uses deliberate step-wise critiques followed by reinforcement learning to produce stronger feedback for later refinement [2505.00662]. AutoMathCritique turns incorrect trajectories into critique-conditioned self-improvement data, improving exploration efficiency and solution diversity [2411.16579]. CGI similarly trains the actor not only to solve tasks but also to **utilize** critiques, and its ablations show that removing critique-conditioned training data causes the largest performance drop [2503.16024]. CLoud adds a cautionary systems lesson: the downstream scorer must be trained on the **self-generated critique distribution** it will actually consume, because off-policy oracle critiques substantially degrade performance [2408.11791].

Research directions in the current papers are unusually concrete. VPS argues for **hybrid verbal–executable supervision** when errors are not fully observable in language [2604.21611]. Frontend code generation points to richer **process distillation** rather than sequence-level distillation of only the final refined output [2604.05839]. PerFine highlights **dynamic stopping criteria** and **dynamic retrieval updates across iterations** [2510.24469]. LADEX suggests domain-specific support such as **RAG** for semantic alignment critique in specialized industrial texts [2509.03463]. Inline Critic points toward a broader class of systems in which critique happens **within** a forward pass rather than after an external draft is complete [2605.12724].

Taken together, the current literature supports a precise characterization: effective Generate–Critique–Refine systems are typically **stateful, schema-constrained, evidence-aware, and evaluated in closed loop**; they work best when the critic is stronger, better grounded, or more formal than the actor on the failure modes that matter most.

Source: https://www.emergentmind.com/topics/generate-critique-refine-loop