Co-Evolving Critic: Adaptive Feedback

• Co-evolving critic is a dynamic evaluation mechanism that continuously adapts its feedback to align with evolving actor outputs in reinforcement and multimodal systems.
• It leverages synchronized updates, memory evolution, and iterative critique—using techniques like maturity-aware rewards and critic distillation—to optimize performance.
• Practical implementations in GUI grounding, physical reasoning, and creative generation have demonstrated significant gains in accuracy and ranking, alongside challenges like convergence.

A co-evolving critic is a critic mechanism whose supervision does not remain fixed while the rest of the system changes. In the strongest formulations, the critic and the actor, proposer, evaluator, or environment model are updated in tandem, so that the critic stays aligned with the current error distribution or search regime. In weaker formulations, the critic remains parametrically fixed but still alters the generation trajectory through stateful iterative feedback, revised prompts, or evolving external memory. Recent work spans synchronized policy–critic co-optimization in open-world reinforcement learning, proposer–critic mutual reinforcement for GUI grounding, evaluator evolution under non-stationary utilities, critic distillation under policy shift in environment co-design, and inference-time critique loops in multimodal reasoning and creative generation (Li et al., 11 Jan 2026, Wang et al., 23 Apr 2026, Iacob et al., 24 Jun 2026, Li et al., 5 Nov 2025, Zhang et al., 2024).

1. Conceptual scope and variants

Current work supports three recurrent interpretations of co-evolving criticism. In the strongest sense, the critic is itself part of the learning system and is updated on the current distribution induced by the actor or proposer. This is explicit in ECHO, where the critic is optimized according to the downstream gains caused by its own critiques on current policy rollouts, and in the GUI-grounding Propose-then-Critic framework, where proposer diversity and critic discrimination are jointly optimized through maturity-aware adaptive co-evolutionary reinforcement learning (Li et al., 11 Jan 2026, Wang et al., 23 Apr 2026). A related strong form appears in the Red Queen Gödel Machine, where evaluators are promoted, frozen within epochs, and replaced at checkpoints under anchor-based selection, so the utility itself evolves across search epochs (Iacob et al., 24 Jun 2026).

An intermediate sense appears when the critic is not directly co-trained with the actor, but is continuously refreshed from the actor’s current learning signal or shares parameters with a policy-like model. Diffusion Co-Design distills a current MAPPO critic into an environment critic so that guided diffusion remains aligned with the current policy under policy shift (Li et al., 5 Nov 2025). PhyCritic first improves physical reasoning and then trains a self-referential judge that generates its own internal answer before judging candidates, so critic competence and policy-like physical reasoning improve together within one multimodal model (Xiong et al., 11 Feb 2026). Table-Critic replaces parameter co-training with memory evolution: its Judge, Critic, Refiner, and Curator operate over a self-evolving template tree that accumulates successful critique patterns across episodes (Yu et al., 17 Feb 2025).

In the weakest sense, the critic changes only contextually. Critic-V uses an offline-trained external critic to refine a prompt-conditioned reasoner at inference time, but the critic itself does not adapt online (Zhang et al., 2024). The planner–actor–critic 3D modeling framework likewise uses iterative critique as revision_context for replanning, yet the Critic remains a prompt-based module rather than a jointly optimized value learner (Gao et al., 8 Jan 2026). Collective Critics for story generation and SCALAR for theoretical physics both implement iterative or multi-agent critique with fixed-role critics rather than learned co-evolution (Bae et al., 2024, Niarchos et al., 7 May 2026). This distinction matters because many recent systems are “actor–critic-inspired” without instantiating critic evolution in the strict reinforcement-learning sense.

2. Recurrent architectural patterns

Across domains, co-evolving-critic systems repeatedly separate generation from evaluation and then reconnect them through iterative or synchronized feedback. The actor side may produce trajectories, candidate clicks, story drafts, code trajectories, or multimodal reasoning chains; the critic side inspects those outputs using richer context than the actor’s local action state, often including screenshots, ranked candidates, reference solutions, execution summaries, or rollout outcomes. The difference between systems lies in what is allowed to evolve: parameters, prompts, utility functions, memory structures, or all of them.

System	Critic role	Adaptation mechanism
CoPC (Wang et al., 23 Apr 2026)	Ranks $K=5$ rendered click candidates on a screenshot	Proposer and critic are jointly optimized with maturity-aware rewards
ECHO (Li et al., 11 Jan 2026)	Generates multiple diagnoses for one trajectory	Dual-track GRPO updates policy and critic on the same on-policy batch
DiCoDe (Li et al., 5 Nov 2025)	Scores environments for guided diffusion	Environment critic is repeatedly distilled from the current RL critic
PhyCritic (Xiong et al., 11 Feb 2026)	Solves first, then judges candidate responses	Two-stage RLVR couples physical skill warmup with self-referential judging
ProCrit (Xu et al., 20 May 2026)	Scores drafts and gives targeted revision feedback	Alternating RL optimizes proposal and critic via revision outcomes
Table-Critic (Yu et al., 17 Feb 2025)	Localizes first error and critiques reasoning	A self-evolving template tree accumulates critique knowledge

Other systems instantiate adjacent patterns. Critic-R inserts a critic between retrieval and commitment, reading the reasoner’s introspective trace to decide whether retrieved evidence supports the next reasoning step, then converting successful and failed refinement trajectories into retriever supervision (Alam et al., 30 May 2026). The 3D modeling planner–actor–critic system uses persistent object names, scene introspection, screenshots, and revision-aware planning so that critique can be operationalized as stable scene modification rather than destructive regeneration (Gao et al., 8 Jan 2026). In SCALAR, the Actor–Critic–Judge loop is explicitly pedagogical: the Critic sees the reference solution, the Judge scores independently, and the transcript is iteratively revised until pass or early stopping (Niarchos et al., 7 May 2026).

3. Learning signals and optimization

The strongest co-evolving-critic papers are defined less by the presence of natural-language feedback than by how that feedback is optimized. In GUI grounding, the proposer is rewarded for accuracy and coverage while the critic is rewarded for top-1 selection quality and listwise ranking quality. Maturity is tracked by exponential moving averages,

$$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$

and then used to gate the coupled rewards,

$$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$

This produces an adaptive curriculum in which critic maturity unlocks proposer exploration, and proposer maturity unlocks harder ranking training for the critic (Wang et al., 23 Apr 2026).

ECHO makes the critic’s target explicitly hindsight-based. For an initial rollout score $s_o$ and critique-conditioned refined score $s_r$, the critic reward is

$$r_c = \ln \left( \frac{1-s_o+\eta}{1-s_r+\eta} \right).$$

The critic is therefore rewarded not for sounding plausible, but for producing critiques that actually improve the current policy’s refined rollout. Policy and critic are then updated synchronously with dual-track GRPO on the same on-policy batch (Li et al., 11 Jan 2026).

ProCrit uses a similar outcome-grounded logic in a different setting. Its critic reward combines format validity, score alignment, and feedback actionability. The actionability term is

$$r_{\mathrm{act}} = \begin{cases} +1 & \text{if an incorrect draft becomes correct after revision} \ -1 & \text{if revision changes the answer and remains incorrect} \ 0 & \text{otherwise.} \end{cases}$$

The proposal is optimized by a dual-stage GRPO objective over draft and revision candidates, so the critic is explicitly refined according to whether its feedback enables useful revisions (Xu et al., 20 May 2026).

DiCoDe replaces manual or Monte Carlo environment supervision with critic distillation. Its environment critic $\mathcal{V}_\vartheta(\theta)$ is regressed toward targets from the current RL critic,

$$\mathcal{L}_{\text{distill}}(\vartheta,\boldsymbol{\theta}) = \sum_{\theta \in \boldsymbol{\theta}} \left( \mathcal{V}_\vartheta(\theta) - \mathbb{E}_{s_0 \sim \mathcal{P}_\theta}[V_\psi(s_0)] \right)^2.$$

The resulting scalar value gradient is then used in guided diffusion, so environment generation remains coupled to the current policy critic rather than to stale returns (Li et al., 5 Nov 2025).

At the theoretical end, actor–critic with evolving reward treats the critic as tracking a moving target $$\boldsymbol{\omega}^*(\boldsymbol{\varphi}_t,\boldsymbol{\theta}_t)$$. The finite-time bounds are

$$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$0

so the standard $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$1 rate is preserved when reward drift is slow enough, in particular when $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$2 (Hu et al., 14 Oct 2025). This provides a formal model of a critic co-evolving with both policy and reward.

4. Domain-specific instantiations

In embodied and interactive settings, co-evolving critics are used to repair the gap between semantic understanding and precise execution. GUI grounding is framed around the large gap between Pass@1 and Pass@k: the model often contains a correct click among several dispersed proposals, and the critic’s job is to visually discriminate among them after they are rendered back on the screenshot (Wang et al., 23 Apr 2026). Open-world agent learning uses a critic to diagnose current policy failures in WebShop, ALFWorld, SciWorld, and DeepSearch, with the explicit claim that static critics become stale as failure patterns drift over training (Li et al., 11 Jan 2026). Code agents, by contrast, mostly use weaker formulations: a small critic can steer a frozen agent intra-trajectory and materially improve SWE-bench Verified, but the agent itself does not co-adapt, so the paper is better understood as a building block for co-evolution rather than a complete instance of it (Gandhi et al., 20 Jun 2026).

In multimodal reasoning, the contrast between weak and strong critic evolution is especially clear. Critic-V uses a DPO-trained external critic to refine VLM reasoning and improves 23 of 24 model–benchmark comparisons, with Qwen2-VL-7B+Critic-V achieving the best score on 5 of 8 benchmarks, but the critic remains a static learned module supervising a changing prompt-conditioned reasoner (Zhang et al., 2024). PhyCritic moves closer to true co-evolution by coupling physical-skill warmup with self-referential judging; its final model reaches 68.0 on PhyCritic-Bench and 63.9 on CosmosReason1-Bench, while the dependence between self-answer correctness and judgment correctness strengthens from $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$3 to $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$4 across the two stages (Xiong et al., 11 Feb 2026). ProCrit applies the same general principle to multimodal sarcasm detection through self-elicited perspectives, outcome-grounded critique, and revision from scratch (Xu et al., 20 May 2026).

Creative and structured generation systems mostly realize weaker or memory-based forms. The planner–actor–critic 3D modeling architecture uses screenshots, scene metadata, and structured success criteria so that the critic can redirect subsequent planning, but it remains a prompt-based inference-time loop rather than an online learner (Gao et al., 8 Jan 2026). CritiCS uses three plan-level critics, two text-level critics, a leader, and an evaluator to iteratively revise stories, yet the critics are fixed-role LLM instances rather than evolving models (Bae et al., 2024). Table-Critic is closer to a co-evolving critic because its critique competence changes through a self-evolving template tree, allowing critique knowledge to accumulate across table-reasoning episodes (Yu et al., 17 Feb 2025). SCALAR sits between these extremes: it exposes the dynamics of critique in research-level physics, but the critic remains fixed by prompt strategy and model choice (Niarchos et al., 7 May 2026).

5. Empirical effects and failure modes

The empirical case for stronger forms of critic co-evolution is substantial. In GUI grounding, replacing static weighting with maturity-aware co-evolution yields the largest ablation gain: removing dynamic maturity drops Oracle@5 / Top-1 from $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$5 to $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$6, and the full method improves ScreenSpot-Pro from $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$7 Oracle@5 and $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$8 Top-1, OSWorld-G from $$C_p^{(t)} = (1-\alpha) C_p^{(t-1)} + \alpha R_{acc}^{(t)}, \qquad C_c^{(t)} = (1-\alpha) C_c^{(t-1)} + \alpha R_{ndcg}^{(t)},$$9 and $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$0, and I2E-bench from $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$1 and $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$2 (Wang et al., 23 Apr 2026). In ECHO, freezing the critic lowers Qwen3-4B performance from 90.03 to 83.60 on WebShop, from 91.25 to 85.75 on ALFWorld, from 82.88 to 68.58 on SciWorld, and from 47.25 to 40.25 on DeepSearch, directly supporting the stale-critic hypothesis (Li et al., 11 Jan 2026). In RQGM, co-evolving evaluators raise coding held-out pass rate to 71.7 versus 69.9 for HGM-H while using $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$3–$$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$4 fewer tokens, raise paper-writer acceptance rates by $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$5–$$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$6, and raise grader ground-truth accuracy by 9%; the same paper also reports that a strong baseline reviewer over-accepts AI-generated papers at up to $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$7 the human rate, motivating adversarial evaluator evolution (Iacob et al., 24 Jun 2026). Table-Critic improves WikiTQ from 68.3 to 77.2 and TabFact from 89.7 to 92.6 relative to Chain-of-Table while keeping the solution degradation rate far below Critic-CoT (Yu et al., 17 Feb 2025). PhyCritic’s full two-stage pipeline outperforms stage-isolated variants, reaching 68.0 on PhyCritic-Bench versus 64.4 without self-reference and 65.8 without the self-prediction reward (Xiong et al., 11 Feb 2026).

The same literature also documents recurrent failure modes. Static or offline critics can become stale as the actor’s error distribution shifts (Li et al., 11 Jan 2026). Benefits are often non-monotonic: Critic-V improves most benchmarks but drops Qwen2-VL-7B on MMStar from 60.7 to 56.2 (Zhang et al., 2024). Iterative critique can plateau or degrade quality: the planner–actor–critic 3D modeling paper reports that after multiple iterations, “often three,” quality degrades or converges, and the actor does not always incorporate the critic’s feedback (Gao et al., 8 Jan 2026). In code agents, a trained critic can improve resolve rate but also induce “repeated demand,” where near-identical guidance is repeated every $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$8 steps until the agent exhausts its budget (Gandhi et al., 20 Jun 2026). In GUI grounding, rendered markers can slightly occlude tiny interface elements, which directly harms critic discrimination on dense layouts (Wang et al., 23 Apr 2026). SCALAR shows that even when multi-turn dialogue improves over single-shot attempts, projected fixed points remain on the hardest problem, with both DS8B and DS70B showing a fixed-point region near $$R_{Proposer} = R_{acc} + C_c \cdot R_{cov}, \qquad R_{Critic} = R_{top1} + C_p \cdot R_{ndcg}.$$9 on Polchinski 2.7, and strict or adversarial feedback is not beneficial (Niarchos et al., 7 May 2026).

6. Relation to actor–critic RL and unresolved issues

A persistent methodological issue is that many recent systems borrow actor–critic language without instantiating actor–critic optimization. The 3D modeling planner–actor–critic framework explicitly cites actor–critic inspiration yet has no Bellman target, advantage estimate, or online critic update (Gao et al., 8 Jan 2026). Critic-V similarly frames natural-language critique as an actor–critic-style signal, but its implemented method is offline DPO training for the critic plus inference-time prompt revision for the reasoner rather than joint actor–critic learning (Zhang et al., 2024). CritiCS and SCALAR likewise remain iterative critique frameworks with fixed-role or fixed-strategy critics rather than evolving critic learners (Bae et al., 2024, Niarchos et al., 7 May 2026).

By contrast, stronger formulations make critic adaptation part of the objective. MetaPG evolves actor and critic loss functions together as one symbolic genotype and reports improvements over SAC of 4% in performance, 20% in generalizability, and up to 67% lower instability, showing that critic evolution can also occur at the algorithm-design level rather than the trajectory level (Garau-Luis et al., 2022). The evolving-reward actor–critic analysis provides a formal account of when a critic can track a doubly moving target under Markovian sampling (Hu et al., 14 Oct 2025). RQGM shows how evaluator evolution can be made piecewise-stationary through frozen-within-epoch critics, anchor-based replacement, and selective erasure of stale evaluator-dependent records (Iacob et al., 24 Jun 2026).

Several open problems recur across the literature. Many papers still lack critic-specific ablations that separate critic effects from decomposition, human supervision, or tool use (Gao et al., 8 Jan 2026). Critic quality is often actor-relative rather than globally optimal: ECHO explicitly notes that a critique that is ideal in principle but too hard for the current actor to use may receive low reward (Li et al., 11 Jan 2026), and the code-agent critic paper identifies the same issue when a valid diagnosis is repeatedly ignored by the frozen actor (Gandhi et al., 20 Jun 2026). Memory-based systems such as Table-Critic demonstrate that critics can evolve through accumulated templates, but the update mechanism remains procedural rather than formally optimized (Yu et al., 17 Feb 2025). The resulting picture is that co-evolving criticism is no longer a single mechanism but a design space: critics may co-adapt through synchronized RL, evaluator replacement, self-reference, distillation, actionability rewards, or evolving critique memory, and the strength of the term depends on which of those elements are actually learned rather than merely replayed.