Generate-Critic Architecture Overview

• Generate-Critic Architecture is a systems paradigm where a generator proposes candidate outputs and a critic evaluates them, enabling iterative refinements.
• It encompasses multiple guidance forms, including decoding-time weighting, iterative natural-language critique, statistical verification, and control-theoretic approaches.
• The design choices in generate-critic systems significantly impact performance in domains like controlled text generation, multimodal reasoning, scientific modeling, and robotics.

Generate-critic architecture denotes a family of systems in which one component generates candidate outputs and another component evaluates those outputs and feeds back signals for revision, selection, or policy improvement. In the cited literature, the generated object may be a token distribution, a reasoning path, a control command, a scientific model, a 3D layout, a story plan, or a full story, while the critic may emit scalar values, HJB residuals, empirical p-values, ontology-grounded constraint violations, pairwise preferences, or natural-language critiques (Kim et al., 2022, Zhang et al., 2024, Li et al., 2024, Sengupta et al., 14 Apr 2026). The architecture therefore names a recurrent systems pattern rather than a single algorithm: a generator proposes, a critic assesses, and the resulting feedback modifies either decoding, iterative refinement, or parameter updates.

1. Conceptual scope and recurring roles

Across domains, the generator is the component that produces the next candidate state of the artifact, and the critic is the component that estimates its quality under task-specific criteria. In controlled text generation, a frozen GPT-2 LLM proposes token probabilities and a learned value-like critic reshapes them at decoding time (Kim et al., 2022). In multimodal reasoning, a Reasoner VLM generates a reasoning path and answer, while a Critic VLM produces constructive natural-language feedback that is appended to the next prompt (Zhang et al., 2024). In scientific model criticism, a model generator proposes a probabilistic program and CriticAL proposes executable summary statistics whose significance is validated by hypothesis testing (Li et al., 2024). In humanoid loco-manipulation, locomotion and arm policies act as generators of motor commands, while unified or dual critics estimate long-horizon value under mixed or disjoint rewards (Yardımcı, 10 Jun 2026).

Domain	Generator	Critic
Controlled text generation	Frozen GPT-2 LM	Value-like critic trained from reward models (Kim et al., 2022)
Multimodal reasoning	Reasoner VLM	DPO-trained critique VLM (Zhang et al., 2024)
Scientific modeling	Probabilistic model / LLM scientist	Test-statistic generator plus hypothesis-testing verifier (Li et al., 2024)
Humanoid loco-manipulation	Locomotion and arm actors	Unified or dual value critics (Yardımcı, 10 Jun 2026)
3D indoor scene synthesis	Layout generator	SceneCritic grounded in SceneOnto (Sengupta et al., 14 Apr 2026)
Long story generation	Plan or text leader	Collective critics and leader selection (Bae et al., 2024)

This breadth makes clear that “critic” is not restricted to the scalar value function of classical actor-critic RL. In some systems it remains a value estimator; in others it is a natural-language evaluator, a symbolic checker, or a statistically grounded verifier. A plausible implication is that the architecture is best defined by information flow—generation followed by structured evaluation—rather than by any single training objective.

2. Structural forms of critic guidance

One recurrent form is decoding-time guidance with a frozen generator. CriticControl keeps the LLM frozen, trains only the critic, and reweights token probabilities by a value ratio

$$\alpha(x_t, x_{1:t-1}, a) = \frac{V_\phi(x_{1:t}, a)}{V_\phi(x_{1:t-1}, a)},$$

so that the adjusted distribution becomes

$$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$

This couples sequence-level reward learning to token-level control without updating LM parameters (Kim et al., 2022).

A second form is iterative natural-language critique. Critic-V separates a Reasoner VLM from a Critic VLM, uses DPO to train the critic on preference-ranked critiques, and updates the Reasoner not in parameter space but by prompt concatenation: the critique is appended as “Reflection on former answer” and the Reasoner answers again (Zhang et al., 2024). DeepCritic applies a related pattern to mathematical reasoning, but at the level of individual reasoning steps: the critic emits a deliberate critique for each step, assigns a binary step judgment, and identifies the first erroneous step (Yang et al., 1 May 2025). IF-CRITIC uses a checklist generator to decompose an instruction into constraints and then produces, for each constraint, an explanation $e_k$ and binary judgment $j_k$, yielding a critique

$C = \bigcup_{k=1}^{n}(e_k, j_k).$

The resulting scalar reward is the average followed-constraint rate

$r_i = \frac{1}{n}\sum_{k=1}^{n} j_{ik},$

which is then used in DPO or GRPO for the generator (Wen et al., 2 Nov 2025).

A third form is symbolic or statistical criticism. CriticAL does not directly trust an LLM’s verbal critique. Instead, the LLM proposes summary-statistic functions, model-generated samples induce an empirical null distribution, and only statistically significant discrepancies are surfaced as critiques (Li et al., 2024). SceneCritic likewise avoids rendered-view judging by operating directly on floor-plan layouts through SceneOnto, jointly verifying semantic, orientation, and geometric coherence and returning object-level and relationship-level violations (Sengupta et al., 14 Apr 2026).

A fourth form is control-theoretic value criticism. In the Actor–Critic–Identifier design for automated EVs, the actor generates motor torque, the identifier learns unknown dynamics, and the critic evaluates the policy through the HJB residual

$$\delta_{\mathrm{HJB}} = \hat J_x\,\hat F(x,\hat u) + \ell(x,\hat u),$$

which drives both actor and critic updates (Faghihian et al., 9 Feb 2026). In humanoid loco-manipulation, the architectural question is whether a single critic should estimate the combined value of locomotion and reaching, or whether dual critics should separately estimate $V_{\text{loco}}$ and $V_{\text{arm}}$ under disjoint rewards (Yardımcı, 10 Jun 2026).

3. Training paradigms and feedback channels

The literature instantiates several distinct training routes for critics. One route is temporal-difference value learning from non-differentiable rewards. CriticControl treats the frozen LM as the actor, the next token as the action, and the sequence-level reward as $P(a \mid x_{1:T})$; the critic is then trained from LM-generated trajectories using TD-style targets and Generalized Advantage Estimation (Kim et al., 2022). A related RL pattern appears in continuous-control work: SDPC couples a continuous soft critic $$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$0 with decomposed discrete generators, and either trains decomposed actors with SAC-style objectives or trains decomposed Q-networks whose Boltzmann policies act as generators (Zhang et al., 2023).

A second route is preference optimization over critiques. Critic-V trains its multimodal critic with DPO over preferred and disfavored critiques constructed from Rule-based Reward and GPT-4o evaluation, using a 29,012-example critique-VQA dataset (Zhang et al., 2024). IF-CRITIC also uses DPO, but at the constraint level rather than the whole-example level: chosen and rejected critiques differ only on the segments where the judgment is wrong, which localizes the preference signal to the exact constraint units under dispute (Wen et al., 2 Nov 2025).

A third route is SFT followed by RL for critique generation itself. DeepCritic first constructs 4.5K long-form deliberate critiques, each containing multi-perspective step-wise critiques and critique-of-critique refinements, then performs reinforcement learning using either PRM800K human labels or Monte Carlo sampling-based correctness estimation from Numina-derived data (Yang et al., 1 May 2025). MultiCritique likewise builds critique data through multiple generator agents and a GPT-4 meta-critic, then trains with SFT and a subsequent RL stage supported by multi-agent preference filtering (Lan et al., 2024).

A fourth route is verification-grounded critic construction. CriticAL uses an LLM only to generate candidate discrepancy statistics; hypothesis testing, posterior predictive sampling, and Bonferroni correction decide which critiques survive (Li et al., 2024). SceneCritic is even more explicit: its scale, co-occurrence, orientation, and overlap checks are analytic functions grounded in SceneOnto statistics rather than learned neural judgments (Sengupta et al., 14 Apr 2026).

These routes correspond to different feedback channels. Some critics return scalar values for gradient updates, some return token-level or action-level weights, some return executable diagnostics, and some return natural-language explanations that are consumed by another generator. This suggests that “critic signal” is an architectural interface, not merely a reward.

4. Domain-specific realizations

Controlled language generation provides a canonical frozen-generator design. CriticControl addresses topic control, sentiment control, and detoxification by using GPT-2 variants as frozen actors, reward models such as BART-Large-MNLI, DistilBERT, and BERT toxicity classifiers, and a critic trained from LM trajectories; at inference, top-$$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$1 or top-$$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$2 candidates are reweighted using critic-derived factors (Kim et al., 2022).

Multimodal reasoning introduces a natural-language critic with no shared parameters. Critic-V initializes the critic from Qwen2-VL-7B, lets the Reasoner be any VLM such as Qwen2-VL-7B, DeepSeek-VL-7B, or LLaVA-v1.5-7B, and closes the loop by feeding critiques back as prompt updates rather than by changing model weights (Zhang et al., 2024).

Scientific modeling turns the critic into a verifier. CriticAL takes dataset metadata, a symbolic model such as a Stan or PyMC program, and posterior predictive samples, then proposes test statistics, computes empirical p-values, and emits only statistically grounded natural-language criticisms. In the larger loop, an LLM scientist revises the model using the statistic code and the verbal critique (Li et al., 2024).

Robotics and control expose critic architecture as a first-class design variable. The EV Actor–Critic–Identifier architecture adds an identifier that approximates unknown dynamics $$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$3, letting actor and critic compute gradients through $$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$4 without explicit analytical plant models (Faghihian et al., 9 Feb 2026). In humanoid loco-manipulation, unified and dual critics are compared directly under a sequential curriculum on the Unitree G1, making the critic itself the experimental variable rather than a fixed backend (Yardımcı, 10 Jun 2026).

Creative systems often make the critic semantic and discursive rather than scalar. CritiCS uses collective critics in two stages: CRPLAN refines story plans with three critics focused on “Original theme and background setting,” “Unusual story structure,” and “Unusual ending,” while CRTEXT refines story sentences with “Image” and “Voice” critics (Bae et al., 2024). Artism formalizes a dual-engine variant in which AIDA acts as a multi-agent generator of artworks, texts, and social dynamics, and the Ismism Machine acts as a critic that decomposes and recombines semantic units, generates new “isms,” and feeds critique back into the shared corpus (Liu et al., 17 Dec 2025). A planner–actor–critic version also appears in agent-augmented 3D modeling, where the Planner decomposes modeling goals, the Actor executes Blender tool calls, and the Critic evaluates screenshots and execution summaries before the next planning round (Gao et al., 8 Jan 2026).

5. Empirical behavior and comparative findings

The empirical record consistently shows that critic design changes final system behavior rather than merely improving diagnostics. In controlled text generation, CriticControl is reported to generate more coherent and well-controlled texts than previous methods, to improve fluency and diversity over weighted-decoding baselines, and to demonstrate superior generalization ability in zero-shot settings (Kim et al., 2022). In multimodal reasoning, Critic-V achieves highest performance on 5 of 8 benchmarks, often surpassing GPT-4V, with especially large gains on MathVista and MathVerse; the ablations further show that a DPO-trained critic is markedly stronger than prompt-only or self-refine baselines (Zhang et al., 2024).

In scientific model criticism, CriticAL’s critiques are preferred by human and LLM judges for transparency, actionability, and tailoredness, and CriticAL-guided revisions improve upon human-designed models on real-world datasets under ELPD-LOO comparisons (Li et al., 2024). In instruction-following evaluation, IF-CRITIC beats strong LLM-as-a-Judge baselines, including Deepseek-R1 and o4-mini, and the reward signals derived from its per-constraint judgments produce substantial gains in generator optimization under lower computational overhead than strong LLM critic baselines (Wen et al., 2 Nov 2025).

The strongest direct architectural comparison appears in humanoid loco-manipulation. Dual-critic policies reach targets 3.5$$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$5 faster, with 6.5 versus 22.6 simulation steps, achieve 2$$\pi'_\theta(x_t \mid x_{1:t-1}, a) = \alpha(x_t, x_{1:t-1}, a)\,\pi_\theta(x_t \mid x_{1:t-1}).$$6 higher throughput at 14.3 versus 7.0 validated reaches per 1,000 steps, and attain higher validated reach rates, 65.2% versus 53.8%, than the unified-critic policy; additional anti-gaming reward mechanisms do not improve beyond the architectural change alone (Yardımcı, 10 Jun 2026). This is unusually explicit evidence that critic architecture can dominate reward engineering.

Evaluation quality itself also depends on critic form. SceneCritic aligns substantially better with human judgments than VLM-based evaluators, while VLM scores over rendered scenes show strong view sensitivity and prompt sensitivity (Sengupta et al., 14 Apr 2026). At the same time, in the refinement test bed, image-based VLM criticism is the most effective modality for semantic and orientation correction, whereas text-only LLMs can outperform VLMs on semantic layout quality as generators or reasoning systems (Sengupta et al., 14 Apr 2026). DeepCritic offers a related split between evaluation and improvement: its 7B critique model significantly outperforms existing LLM critics, including GPT-4o and same-sized DeepSeek-R1-distill models, on error identification benchmarks, and its detailed feedback improves downstream refinement of generator solutions (Yang et al., 1 May 2025).

6. Design trade-offs, misconceptions, and open directions

A common misconception is that a critic must be a single scalar value model. The literature does not support that restriction. Critics can be value functions, HJB residual estimators, DPO-trained critique generators, symbolic ontology checkers, or statistically grounded verifiers (Faghihian et al., 9 Feb 2026, Zhang et al., 2024, Li et al., 2024, Sengupta et al., 14 Apr 2026). A second misconception is that the generator must always be updated jointly with the critic. CriticControl keeps the LM frozen and performs all steering at decoding time, while Critic-V modifies the Reasoner only through prompt augmentation (Kim et al., 2022, Zhang et al., 2024).

Another central trade-off concerns signal richness versus stability. Natural-language critics provide localized, interpretable feedback, but they can hallucinate or propagate critique errors. CriticAL addresses this by embedding proposed discrepancies in hypothesis testing, and SceneCritic addresses it by replacing rendered-view judging with symbolic constraints (Li et al., 2024, Sengupta et al., 14 Apr 2026). Conversely, purely symbolic critics are stable and interpretable but domain-bound; SceneCritic depends on SceneOnto, and CriticAL assumes simulable models and executable statistics.

The literature also shows that critic granularity matters. Unified critics can entangle objectives: in humanoid RL, a unified critic trained on summed rewards suppresses decisive arm behavior through competing locomotion gradients, whereas dual critics isolate credit assignment (Yardımcı, 10 Jun 2026). IF-CRITIC similarly argues that whole-response judging is too coarse for instruction following, and therefore decomposes the task into explicit constraint checklists before critique (Wen et al., 2 Nov 2025). DeepCritic reaches the same conclusion for math reasoning by critiquing one reasoning step at a time rather than only the final answer (Yang et al., 1 May 2025).

Several open directions are explicit in the cited work. CriticControl suggests offline RL techniques for critic training from logged LM outputs and highlights cost issues for GPT-3-scale models (Kim et al., 2022). The EV Actor–Critic–Identifier paper suggests a two-mode architecture for traction versus regen/coasting and calls for hardware-in-the-loop tests (Faghihian et al., 9 Feb 2026). CriticAL points toward broader scientific model classes and stronger multi-step critique–repair loops (Li et al., 2024). Artism and CritiCS imply that multi-critic and semantically discursive critics can be extended to other cultural and design domains (Liu et al., 17 Dec 2025, Bae et al., 2024).

Taken together, these works define generate-critic architecture as a general systems principle for controlled generation, reasoning, control, and evaluation: a generator proposes structured candidates, a critic supplies task-aligned information unavailable from likelihood alone, and the loop between them determines whether the system merely produces outputs or can deliberately improve them.