---
title: Generate-Critic Architecture Overview
url: https://www.emergentmind.com/topics/generate-critic-architecture
type: topic
---

# Generate-Critic Architecture Overview

Generate-critic architecture denotes a family of systems in which one component generates candidate outputs and another component evaluates those outputs and feeds back signals for revision, selection, or policy improvement. In the cited literature, the generated object may be a token distribution, a reasoning path, a control command, a scientific model, a 3D layout, a story plan, or a full story, while the critic may emit scalar values, HJB residuals, empirical p-values, ontology-grounded constraint violations, pairwise preferences, or natural-language critiques [2212.10938] [2411.18203] [2411.06590] [2604.13035]. The architecture therefore names a recurrent systems pattern rather than a single algorithm: a generator proposes, a critic assesses, and the resulting feedback modifies either decoding, iterative refinement, or parameter updates.

## 1. Conceptual scope and recurring roles

Across domains, the generator is the component that produces the next candidate state of the artifact, and the critic is the component that estimates its quality under task-specific criteria. In controlled text generation, a frozen GPT-2 language model proposes token probabilities and a learned value-like critic reshapes them at decoding time [2212.10938]. In multimodal reasoning, a Reasoner VLM generates a reasoning path and answer, while a Critic VLM produces constructive natural-language feedback that is appended to the next prompt [2411.18203]. In scientific model criticism, a model generator proposes a probabilistic program and CriticAL proposes executable summary statistics whose significance is validated by hypothesis testing [2411.06590]. In humanoid loco-manipulation, locomotion and arm policies act as generators of motor commands, while unified or dual critics estimate long-horizon value under mixed or disjoint rewards [2606.11891].

| Domain | Generator | Critic |
|---|---|---|
| Controlled text generation | Frozen GPT-2 LM | Value-like critic trained from reward models [2212.10938] |
| Multimodal reasoning | Reasoner VLM | DPO-trained critique VLM [2411.18203] |
| Scientific modeling | Probabilistic model / LLM scientist | Test-statistic generator plus hypothesis-testing verifier [2411.06590] |
| Humanoid loco-manipulation | Locomotion and arm actors | Unified or dual value critics [2606.11891] |
| 3D indoor scene synthesis | Layout generator | SceneCritic grounded in SceneOnto [2604.13035] |
| Long story generation | Plan or text leader | Collective critics and leader selection [2410.02428] |

This breadth makes clear that “critic” is not restricted to the scalar value function of classical actor-critic RL. In some systems it remains a value estimator; in others it is a natural-language evaluator, a symbolic checker, or a statistically grounded verifier. A plausible implication is that the architecture is best defined by information flow—generation followed by structured evaluation—rather than by any single training objective.

## 2. Structural forms of critic guidance

One recurrent form is **decoding-time guidance with a frozen generator**. CriticControl keeps the language model frozen, trains only the critic, and reweights token probabilities by a value ratio
\[
\alpha(x_t, x_{1:t-1}, a) = \frac{V_\phi(x_{1:t}, a)}{V_\phi(x_{1:t-1}, a)},
\]
so that the adjusted distribution becomes
\[
\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).
\]
This couples sequence-level reward learning to token-level control without updating LM parameters [2212.10938].

A second form is **iterative natural-language critique**. Critic-V separates a Reasoner VLM from a Critic VLM, uses DPO to train the critic on preference-ranked critiques, and updates the Reasoner not in parameter space but by prompt concatenation: the critique is appended as “Reflection on former answer” and the Reasoner answers again [2411.18203]. DeepCritic applies a related pattern to mathematical reasoning, but at the level of individual reasoning steps: the critic emits a deliberate critique for each step, assigns a binary step judgment, and identifies the first erroneous step [2505.00662]. IF-CRITIC uses a checklist generator to decompose an instruction into constraints and then produces, for each constraint, an explanation \(e_k\) and binary judgment \(j_k\), yielding a critique
\[
C = \bigcup_{k=1}^{n}(e_k, j_k).
\]
The resulting scalar reward is the average followed-constraint rate
\[
r_i = \frac{1}{n}\sum_{k=1}^{n} j_{ik},
\]
which is then used in DPO or GRPO for the generator [2511.01014].

A third form is **symbolic or statistical criticism**. CriticAL does not directly trust an LLM’s verbal critique. Instead, the LLM proposes summary-statistic functions, model-generated samples induce an empirical null distribution, and only statistically significant discrepancies are surfaced as critiques [2411.06590]. SceneCritic likewise avoids rendered-view judging by operating directly on floor-plan layouts through SceneOnto, jointly verifying semantic, orientation, and geometric coherence and returning object-level and relationship-level violations [2604.13035].

A fourth form is **control-theoretic value criticism**. In the Actor–Critic–Identifier design for automated EVs, the actor generates motor torque, the identifier learns unknown dynamics, and the critic evaluates the policy through the HJB residual
\[
\delta_{\mathrm{HJB}} = \hat J_x\,\hat F(x,\hat u) + \ell(x,\hat u),
\]
which drives both actor and critic updates [2602.09140]. In humanoid loco-manipulation, the architectural question is whether a single critic should estimate the combined value of locomotion and reaching, or whether dual critics should separately estimate \(V_{\text{loco}}\) and \(V_{\text{arm}}\) under disjoint rewards [2606.11891].

## 3. Training paradigms and feedback channels

The literature instantiates several distinct training routes for critics. One route is **temporal-difference value learning from non-differentiable rewards**. CriticControl treats the frozen LM as the actor, the next token as the action, and the sequence-level reward as \(P(a \mid x_{1:T})\); the critic is then trained from LM-generated trajectories using TD-style targets and Generalized Advantage Estimation [2212.10938]. A related RL pattern appears in continuous-control work: SDPC couples a continuous soft critic \(Q(s,a)\) with decomposed discrete generators, and either trains decomposed actors with SAC-style objectives or trains decomposed Q-networks whose Boltzmann policies act as generators [2308.10203].

A second route is **preference optimization over critiques**. Critic-V trains its multimodal critic with DPO over preferred and disfavored critiques constructed from Rule-based Reward and GPT-4o evaluation, using a 29,012-example critique-VQA dataset [2411.18203]. IF-CRITIC also uses DPO, but at the constraint level rather than the whole-example level: chosen and rejected critiques differ only on the segments where the judgment is wrong, which localizes the preference signal to the exact constraint units under dispute [2511.01014].

A third route is **SFT followed by RL for critique generation itself**. DeepCritic first constructs 4.5K long-form deliberate critiques, each containing multi-perspective step-wise critiques and critique-of-critique refinements, then performs reinforcement learning using either PRM800K human labels or Monte Carlo sampling-based correctness estimation from Numina-derived data [2505.00662]. MultiCritique likewise builds critique data through multiple generator agents and a GPT-4 meta-critic, then trains with SFT and a subsequent RL stage supported by multi-agent preference filtering [2410.15287].

A fourth route is **verification-grounded critic construction**. CriticAL uses an LLM only to generate candidate discrepancy statistics; hypothesis testing, posterior predictive sampling, and Bonferroni correction decide which critiques survive [2411.06590]. SceneCritic is even more explicit: its scale, co-occurrence, orientation, and overlap checks are analytic functions grounded in SceneOnto statistics rather than learned neural judgments [2604.13035].

These routes correspond to different feedback channels. Some critics return scalar values for gradient updates, some return token-level or action-level weights, some return executable diagnostics, and some return natural-language explanations that are consumed by another generator. This suggests that “critic signal” is an architectural interface, not merely a reward.

## 4. Domain-specific realizations

Controlled language generation provides a canonical frozen-generator design. CriticControl addresses topic control, sentiment control, and detoxification by using GPT-2 variants as frozen actors, reward models such as BART-Large-MNLI, DistilBERT, and BERT toxicity classifiers, and a critic trained from LM trajectories; at inference, top-\(k\) or top-\(p\) candidates are reweighted using critic-derived factors [2212.10938].

Multimodal reasoning introduces a natural-language critic with no shared parameters. Critic-V initializes the critic from Qwen2-VL-7B, lets the Reasoner be any VLM such as Qwen2-VL-7B, DeepSeek-VL-7B, or LLaVA-v1.5-7B, and closes the loop by feeding critiques back as prompt updates rather than by changing model weights [2411.18203].

Scientific modeling turns the critic into a verifier. CriticAL takes dataset metadata, a symbolic model such as a Stan or PyMC program, and posterior predictive samples, then proposes test statistics, computes empirical p-values, and emits only statistically grounded natural-language criticisms. In the larger loop, an LLM scientist revises the model using the statistic code and the verbal critique [2411.06590].

Robotics and control expose critic architecture as a first-class design variable. The EV Actor–Critic–Identifier architecture adds an identifier that approximates unknown dynamics \(g(x)\), letting actor and critic compute gradients through \(\hat F(x,u)\) without explicit analytical plant models [2602.09140]. In humanoid loco-manipulation, unified and dual critics are compared directly under a sequential curriculum on the Unitree G1, making the critic itself the experimental variable rather than a fixed backend [2606.11891].

Creative systems often make the critic semantic and discursive rather than scalar. CritiCS uses collective critics in two stages: CRPLAN refines story plans with three critics focused on “Original theme and background setting,” “Unusual story structure,” and “Unusual ending,” while CRTEXT refines story sentences with “Image” and “Voice” critics [2410.02428]. Artism formalizes a dual-engine variant in which AIDA acts as a multi-agent generator of artworks, texts, and social dynamics, and the Ismism Machine acts as a critic that decomposes and recombines semantic units, generates new “isms,” and feeds critique back into the shared corpus [2512.15710]. A planner–actor–critic version also appears in agent-augmented 3D modeling, where the Planner decomposes modeling goals, the Actor executes Blender tool calls, and the Critic evaluates screenshots and execution summaries before the next planning round [2601.05016].

## 5. Empirical behavior and comparative findings

The empirical record consistently shows that critic design changes final system behavior rather than merely improving diagnostics. In controlled text generation, CriticControl is reported to generate more coherent and well-controlled texts than previous methods, to improve fluency and diversity over weighted-decoding baselines, and to demonstrate superior generalization ability in zero-shot settings [2212.10938]. In multimodal reasoning, Critic-V achieves highest performance on 5 of 8 benchmarks, often surpassing GPT-4V, with especially large gains on MathVista and MathVerse; the ablations further show that a DPO-trained critic is markedly stronger than prompt-only or self-refine baselines [2411.18203].

In scientific model criticism, CriticAL’s critiques are preferred by human and LLM judges for transparency, actionability, and tailoredness, and CriticAL-guided revisions improve upon human-designed models on real-world datasets under ELPD-LOO comparisons [2411.06590]. In instruction-following evaluation, IF-CRITIC beats strong LLM-as-a-Judge baselines, including Deepseek-R1 and o4-mini, and the reward signals derived from its per-constraint judgments produce substantial gains in generator optimization under lower computational overhead than strong LLM critic baselines [2511.01014].

The strongest direct architectural comparison appears in humanoid loco-manipulation. Dual-critic policies reach targets 3.5\(\times\) faster, with 6.5 versus 22.6 simulation steps, achieve 2\(\times\) higher throughput at 14.3 versus 7.0 validated reaches per 1,000 steps, and attain higher validated reach rates, 65.2% versus 53.8%, than the unified-critic policy; additional anti-gaming reward mechanisms do not improve beyond the architectural change alone [2606.11891]. This is unusually explicit evidence that critic architecture can dominate reward engineering.

Evaluation quality itself also depends on critic form. SceneCritic aligns substantially better with human judgments than VLM-based evaluators, while VLM scores over rendered scenes show strong view sensitivity and prompt sensitivity [2604.13035]. At the same time, in the refinement test bed, image-based VLM criticism is the most effective modality for semantic and orientation correction, whereas text-only LLMs can outperform VLMs on semantic layout quality as generators or reasoning systems [2604.13035]. DeepCritic offers a related split between evaluation and improvement: its 7B critique model significantly outperforms existing LLM critics, including GPT-4o and same-sized DeepSeek-R1-distill models, on error identification benchmarks, and its detailed feedback improves downstream refinement of generator solutions [2505.00662].

## 6. Design trade-offs, misconceptions, and open directions

A common misconception is that a critic must be a single scalar value model. The literature does not support that restriction. Critics can be value functions, HJB residual estimators, DPO-trained critique generators, symbolic ontology checkers, or statistically grounded verifiers [2602.09140] [2411.18203] [2411.06590] [2604.13035]. A second misconception is that the generator must always be updated jointly with the critic. CriticControl keeps the LM frozen and performs all steering at decoding time, while Critic-V modifies the Reasoner only through prompt augmentation [2212.10938] [2411.18203].

Another central trade-off concerns **signal richness versus stability**. Natural-language critics provide localized, interpretable feedback, but they can hallucinate or propagate critique errors. CriticAL addresses this by embedding proposed discrepancies in hypothesis testing, and SceneCritic addresses it by replacing rendered-view judging with symbolic constraints [2411.06590] [2604.13035]. Conversely, purely symbolic critics are stable and interpretable but domain-bound; SceneCritic depends on SceneOnto, and CriticAL assumes simulable models and executable statistics.

The literature also shows that **critic granularity matters**. Unified critics can entangle objectives: in humanoid RL, a unified critic trained on summed rewards suppresses decisive arm behavior through competing locomotion gradients, whereas dual critics isolate credit assignment [2606.11891]. IF-CRITIC similarly argues that whole-response judging is too coarse for instruction following, and therefore decomposes the task into explicit constraint checklists before critique [2511.01014]. DeepCritic reaches the same conclusion for math reasoning by critiquing one reasoning step at a time rather than only the final answer [2505.00662].

Several open directions are explicit in the cited work. CriticControl suggests offline RL techniques for critic training from logged LM outputs and highlights cost issues for GPT-3-scale models [2212.10938]. The EV Actor–Critic–Identifier paper suggests a two-mode architecture for traction versus regen/coasting and calls for hardware-in-the-loop tests [2602.09140]. CriticAL points toward broader scientific model classes and stronger multi-step critique–repair loops [2411.06590]. Artism and CritiCS imply that multi-critic and semantically discursive critics can be extended to other cultural and design domains [2512.15710] [2410.02428].

Taken together, these works define generate-critic architecture as a general systems principle for controlled generation, reasoning, control, and evaluation: a generator proposes structured candidates, a critic supplies task-aligned information unavailable from likelihood alone, and the loop between them determines whether the system merely produces outputs or can deliberately improve them.

Source: https://www.emergentmind.com/topics/generate-critic-architecture