---
title: Agentic Critic in AI Systems
url: https://www.emergentmind.com/topics/agentic-critic
type: topic
---

# Agentic Critic in AI Systems

Agentic Critic denotes a class of mechanisms in agentic AI systems in which a model evaluates intermediate actions, reasoning traces, retrieved evidence, drafted outputs, or candidate strategies inside an iterative workflow rather than only scoring a final answer. In recent work, this role appears as a prompted selector over video-reasoning strategies, a process-level intervention module for reinforcement learning, a retrieval sufficiency judge in agentic search, a reflexive verifier for code-grounded documentation repair, a natural-language critic for off-policy language-space learning, and a safety layer that challenges recommendations before human review [2509.07680]. The common pattern is not a single architecture but a family of designs in which critique changes subsequent computation: by selecting among trajectories, rewriting queries, rewinding adverse steps, revising drafts, shaping policy updates, or constraining unsafe outputs.

## 1. Conceptual scope and defining variants

Across the literature, the critic function is attached to different objects. In "CAViAR" [2509.07680], the critic judges sampled *strategies*—tool sequences such as `get_segment` plus `retrieval_qa` plus `find_when`—because multiple plausible action sequences can be produced by the agentic video-reasoning pipeline, but not all are equally reliable. In "ProCeedRL" [2604.02006], the critic operates at the *process* level, monitoring each intermediate interaction step online and intervening when the score falls below a threshold. In "Critic-R" [2606.00590], the critic evaluates whether retrieved documents are satisfactory for the next reasoning step by using the reasoner’s own introspective trace. In "DocSync" [2605.02163], the critic checks whether a generated documentation update is semantically consistent with the changed code. In "Agentic Critical Training" [2603.08706], the policy itself is trained to judge which action is better among alternatives. In "Natural Language Actor-Critic" [2512.04601], the critic outputs natural language rather than scalar values.

This diversity produces several technically distinct notions of critique. Some systems use a *selector* that ranks trajectories or strategies without modifying them directly, as in CAViAR’s prompted LLM selector [2509.07680]. Some use an *interventionist* critic that rewinds and refines actions before the rollout continues, as in ProCeedRL [2604.02006]. Others use a *verification* critic that accepts or rejects outputs and feeds back reasons for revision, as in DocSync [2605.02163] and XAI narrative generation [2603.20003]. Still others attach the critic to *retrieval*, *value estimation*, or *policy refinement* rather than to textual answers themselves [2606.00590].

A concise way to organize the space is by what the critic evaluates and what consequence follows.

| Representative system | What the critic evaluates | What the critic changes |
|---|---|---|
| CAViAR [2509.07680] | Candidate strategy trajectories | Winning strategy selection |
| ProCeedRL [2604.02006] | Intermediate action plus next observation | Rewind and refined action |
| Critic-R [2606.00590] | Retrieval sufficiency for next reasoning step | Query and instruction refinement |
| DocSync [2605.02163] | Draft docstring against code | Reflexive draft revision |
| NLAC [2512.04601] | Natural-language action quality | Refined action distillation |
| Android Coach [2604.07277] | Multiple candidate actions per state | Group-wise policy improvement |

This suggests that “agentic critic” is best treated as an architectural role rather than a fixed algorithm. The role is defined by closed-loop evaluation inside an agent workflow, not merely by the existence of a scalar reward.

## 2. Architectural patterns in agentic-critic systems

A recurring pattern is *adaptive tool orchestration plus post hoc strategy selection*. CAViAR exposes the reasoning agent to `think(thought)`, `get_segment(start, end)`, `find_when(query, video_segment)`, `asr_understanding(question, answer_options)`, `retrieval_qa(question, answer_options, video_segment)`, and `finish(final_answer)`. The large language model acts as a controller that iteratively calls one tool at a time and decides the next tool based on previous outputs. The critic then receives the question, candidate strategies, and examples of trajectories and outcomes, and outputs `Winning Strategies: A, B, ...`. It is instructed that if multiple strategies lead to the same correct final answer, multiple can be selected, but conflicting answers cannot be selected together [2509.07680]. The critic therefore judges the quality of the *action sequence* rather than only the terminal answer.

A second pattern is *online process monitoring and intervention*. ProCeedRL identifies a “vicious circle” in long-horizon exploration: a suboptimal action elicits irrelevant or misleading observations, these are added to the history, and later decisions degrade further. Its process critic evaluates each step in real time,
$$
l_t, c_t = \phi(\tau_t, a_t, s_{t+1}),
$$
and when the score falls below \(l_{\text{th}}\), the method rewinds and invokes a refining policy,
$$
a_t' = \mu(\tau_{t-1}, a_t, l_t, c_t).
$$
The refined action replaces the original one, so the rollout proceeds from a corrected step instead of propagating the flawed one [2604.02006]. Here the critic is not merely advisory; it changes the experienced context.

A third pattern is *evaluation–repair separation*. Critic-R separates satisfaction judgment from query refinement. The critic \(\mathcal{M}_C\) first receives the global query \(Q\), local sub-query \(q_i^{(t)}\), retrieved documents \(D_i^{(t)}\), and introspective trace \(T_{i+1}^{(t)}\), and returns a binary verdict \(\sigma_i^{(t)} \in \{yes, no\}\) and a reason \(r_i^{(t)}\). Only if the verdict is negative does the system enter query refinement mode, producing a refined instruction and refined query [2606.00590]. DocSync uses the same architectural separation in a different domain: the model drafts \(D_{draft}\), the critic returns `IsGood, Reason`, and the reason is appended to the prompt for another pass if needed [2605.02163].

A fourth pattern is *critic as safety or governance layer*. In commercial insurance underwriting, the primary agent performs underwriting analysis and the critic agent acts as a skeptical internal reviewer that cross-checks facts, checks alignment with rules and appetite constraints, identifies missing risk factors, flags contradictions, tests robustness using hypothetical adversarial scenarios, and can force revision before the case reaches the human underwriter [2602.13213]. The system permits one full critique-revision loop and remains explicitly decision-negative: the AI can recommend but cannot make binding underwriting decisions.

A fifth pattern is *critic as coach over unexecuted alternatives*. Android Coach reframes online Android agent RL from Single State Single Action to Single State Multiple Actions. For one collected state \(s_t\), it resamples multiple candidate actions \(\{a_t^1,\dots,a_t^k\}\) and evaluates them with a learned critic \(Q_\phi(s_t,a_t)\) without extra emulator overhead [2604.07277]. The critic therefore coaches policy improvement over hypothetical actions rather than only over executed ones.

## 3. Learning objectives, scoring rules, and formalizations

The literature contains several distinct formalizations of critique. Some critics are prompted selectors with no explicit supervised loss in the provided text. CAViAR’s supplemental material gives no formal critic loss; instead, the critic uses 4 in-context examples per prompt, examples authored to be in-distribution for the target dataset, and a decision rule that selects the strategy most likely to yield a correct answer, selecting all strategies that produce the same final answer when appropriate [2509.07680].

Other systems define explicit reward or value objectives. ACT reframes post-training from action imitation to action judgment. Imitation learning is written as
$$
\mathcal{L}_{\text{IL}(\theta) = -\mathbb{E}_{(s,a)\sim\mathcal{D}_{\text{expert}}}\left[\log \pi_{\theta}(a|s)\right],
$$
whereas ACT rewards correct judgment between expert and alternative actions with
$$
R(s, y) = R_{\text{acc}(a, a^+) + R_{\text{adm}(a, \mathcal{A}_{\text{admissible}}) + R_{\text{fmt}(y).
$$
The appendix specifies \(R_{\text{acc}}\) as exact match reward, \(R_{\text{adm}}\) as partial credit for admissible but non-expert actions, and \(R_{\text{fmt}}\) as a format penalty tied to `<action>...</action>` tags. Optimization uses Group Relative Policy Optimization (GRPO) [2603.08706]. The critic-like capability is learned by rewarding whether the model’s *judgment* is correct, not only whether its action matches the expert.

ProCeedRL extends a DAPO-style group-based RLVR setup with a mixed buffer \(\mathcal{B}\) formed from on-policy rollouts \(\{o_i\}\) and refined demonstration trajectories \(\{d_j\}\). Demonstration steps are off-policy and masked in failed trajectories, and the demonstration loss is reweighted by
$$
\sigma(d_{j,t}) = \pi_\theta(d_{j,t}\mid x,d_{j,<t}) \cdot \left(1 - \pi_\theta(d_{j,t}\mid x,d_{j,<t})\right),
$$
while on-policy steps use weight 1. The objective is a clipped, group-based relative-advantage loss with importance sampling [2604.02006]. In this formulation, the critic is part of a rollout-and-data-collection scheme rather than a separate standalone RL algorithm.

Critic-R formalizes critic-generated supervision for retriever optimization. Retrieved documents accepted by the critic become positives, rejected documents become intra-trajectory hard negatives, and the retriever is trained with the contrastive loss
$$
\mathcal{L} = - \log \frac{\exp\!\big(\mathrm{sim}(q_i, z_i^+)/\tau\big)}{\sum_{z \in \mathcal{Z}_i} \exp\!\big(\mathrm{sim}(q_i, z)/\tau\big)}.
$$
Critic-Embed is initialized from Stella-400M and trained with InfoNCE using \(\tau = 0.02\) [2606.00590].

NLAC replaces scalar value estimation with a natural-language critic \(Q_L^\pi(s_t,a_t)\in\mathcal{V}^*\). A successor model \(M^\pi\) is trained via a language Bellman backup, and an evaluator \(E\) aggregates sampled futures into the final critique. Actor improvement occurs through self-refinement,
$$
a_t^r \sim \pi^r(\cdot \mid s_t, a_t, Q_L^\pi(s_t,a_t)),
$$
followed by a distillation-style projection of the base policy onto the refinement policy, rather than a policy gradient update [2512.04601]. This is one of the clearest formulations in which critique is itself the learning signal.

Android Coach uses a conventional scalar critic but couples it to a Process Reward Model and a group-wise advantage estimator. The process reward model is trained with
$$
\mathcal{L}_{\text{PRM}(\beta)} = - \log P_{\beta}(y \mid I, s_t, a_t),
$$
and the actor update uses the ACLOO leave-one-out advantage
$$
\hat{A}_t^i = Q_\phi(s_t,a_t^i) - \frac{1}{k-1}\sum_{j\neq i} Q_\phi(s_t,a_t^j).
$$
The paper states that this estimator is unbiased and shift-invariant [2604.07277].

Finally, several papers clarify the boundary cases. SeeUPO argues that mainstream backbone RL algorithms cannot simultaneously achieve both critic-free and convergence guarantees in multi-turn scenarios, and proposes a critic-free alternative based on sequential turn-by-turn updates in reverse execution order [2602.06554]. EVOM, by contrast, makes the outer design loop agentic but does not make the runtime critic an autonomous evaluator; the LLM functions as an architecture designer for actor-critic networks rather than as a decision-time critic [2606.26327].

## 4. Empirical domains and reported effects

Agentic critics have been evaluated across long-video reasoning, search, documentation maintenance, embodied and web tasks, GUI control, front-end code generation, XAI narrative generation, theoretical physics reasoning, underwriting, and general agent benchmarks.

In long-video reasoning, CAViAR reports on EgoSchema that SeViLA scores 25.7, LLoVi 57.6, VideoAgent 60.2, VideoTree 66.2, Direct inference 70.6, and CAViAR 72.2, an improvement of 1.6 points over direct inference on the evaluation subset of 500. On 25% of LVBench with GPT-4o-mini, direct inference is 43.2 and CAViAR is 49.0, a 5.8-point gain [2509.07680]. The reported analyses attribute the gains to choosing strategies that properly localize with `get_segment`, preferring transcript-based reasoning when the task is speech-heavy, and avoiding strategies that guess after failing to find exact evidence.

In deep search and embodied planning, ProCeedRL reports for Qwen3-8B: 73.87% on Bamboogle, 29.52% on MuSiQue, 46.42% on Frames, 13.79% on GAIA, and 23.01% on WebWalkerQA, compared with DAPO/Search-R1 at 70.83%, 23.60%, 43.59%, 10.10%, and 19.51%. On AlfWorld, ProCeedRL reaches 51.43% in-distribution and 55.22% out-of-distribution for Qwen3-8B, versus DAPO at 45.23% and 53.24%; ProCeedSFT reaches 57.14% and 58.95% with an additional SFT stage on correct samples collected during rollout [2604.02006].

In retrieval-centric agentic search, Critic-R reports an overall relative improvement of 12.4% for Critic-R-Zero in the multi-hop setting. For retriever comparison, the reported multi-hop averages at \(k=1\) are Stella-400M: 0.3472 / 0.4470, Agentic-R: 0.3670 / 0.4564, and Critic-Embed: 0.3794 / 0.4806. Combining Critic-Embed with Critic-R-Zero yields full Critic-R at 0.3957 / 0.4959 [2606.00590].

In documentation maintenance, DocSync (Final) reports BLEU 0.575, BERTScore F1 0.985, Summary Exact 0.969, and Judge 3.44, compared with CodeT5-base at BLEU 0.193, BERTScore F1 0.880, Summary Exact 0.188, and Judge 1.91. Relative to DocSync (Initial), the final critic pass raises BERTScore F1 from 0.980 to 0.985, Summary Exact from 0.938 to 0.969, and Judge from 3.25 to 3.44, with little BLEU change [2605.02163]. The interpretation given in the paper is that the critic improves semantic faithfulness rather than surface n-gram overlap.

In benchmarked action-judgment training, ACT on Qwen3-8B reports ALFWorld ID/OOD 72.86/72.39 for ACT alone, 91.43/87.31 for IL w/ ACT, and 92.86/88.06 for RL w/ ACT. The paper summarizes average gains of 5.07 points over imitation learning, 4.62 points over reinforcement learning, and 2.42 points over Early Experience [2603.08706]. It also reports improvements on MATH-500 and GPQA-Diamond without reasoning-specific training data.

In Android GUI control, Android Coach improves UI-TARS-1.5-7B from 31.9% to 39.4% on AndroidLab and from 32.8% to 41.1% on AndroidWorld, corresponding to gains of 7.5% and 8.3%. At matched success rates, it attains 1.4x higher training efficiency than Single State Single Action methods PPO and GRPO [2604.07277].

In front-end code generation, ReLook reports a monotone ranking \(\text{ReLook} > \text{Web-RL} > \text{Base Model}\) across ArtifactsBench subsets and attributes the gains to the MLLM-in-the-loop visual critic, the zero-reward rule for invalid renders, and Forced Optimization. On ArtifactsBench-Lite for Qwen2.5-7B-Instruct, the ablation rises from base 21.59 to 24.89 with MLLM reward, to 25.84 with format constraint, and to 27.88 with Forced Optimization [2510.11498].

In XAI narrative generation, the Basic Design, Critic Design, and Critic-Rule Design improve faithfulness across all LLMs, and Claude Sonnet 4.5 on Basic Design reduces unfaithful narratives from 30 to 3 by round 2, which the paper presents as a 90% reduction after three rounds [2603.20003].

In regulated underwriting, the adversarial critique mechanism reduces hallucination rates from 11.3% to 3.8% and increases decision accuracy from 92% to 96% on 500 expert-validated cases. It also raises contradiction detection from 76% to 89%, source traceability from 81% to 96%, guideline compliance from 95% to 98.5%, and risk factor recall from 88% to 95% [2602.13213].

## 5. Failure modes, limitations, and negative results

A major result of the literature is that critique is not automatically beneficial. "Helpful Agent Meets Deceptive Judge" shows that feedback-based workflows are vulnerable when the judge is misleading, biased, or adversarial. The paper introduces a two-dimensional taxonomy over intent and knowledge access, distinguishes constructive, hypercritical, and malicious judges, and distinguishes no-knowledge, parametric-knowledge, and grounded-knowledge judges. It also constructs WAFER-QA with 574 contextual samples and 708 non-contextual samples to evaluate grounded adversarial critiques [2506.03332]. The central empirical finding is that even strong models can switch correct answers after a single round of misleading feedback; for example, GPT-4o on ARC-Challenge drops from 96.5% to 76.0% after one round of baseless criticism, and grounded critiques on WAFER-QA often cause drops over 50% from initial accuracy after one round [2506.03332].

Theoretical-physics reasoning yields a second negative result. SCALAR reports that multi-turn dialogue improves over single-shot attempts throughout, but the mechanism of improvement and the value of prompting choices depend strongly on the Actor–Critic pairing. In asymmetric Actor–Critic settings, such as Haiku Actor with stronger Sonnet Critic, constructive feedback improves mean-score outcomes. In same-family pairings such as DS8B or DS70B with DS70B Critic, strategy effects are weaker, lenient feedback is sometimes favored, and strict and adversarial feedback are not beneficial [2605.06772]. The hardest task, Polchinski 2.7, remains a bottleneck even with larger models and structured critique.

Several papers also document intrinsic failure modes of the critics themselves. CAViAR notes that `retrieval_qa` alone may hallucinate timestamps or answers, `find_when` can return ambiguous ranges, `asr_understanding` may help on audio-rich videos but hurt on purely visual reasoning, and strategies may stop too early and guess [2509.07680]. ProCeedRL states that the method adds computation, roughly \(2\times\) vanilla exploration in their estimates, and does not guarantee improvement because the critic and refiner are themselves LLM-based and can still make mistakes [2604.02006]. Android Coach reports that training the critic from scratch causes divergence and that PRM pretraining is needed for stable critic learning [2604.07277]. DocSync’s remaining failure modes include minor formatting debris, over-elaboration or hallucinated details, and truncation or repetition on long docstrings [2605.02163]. XAI narrative generation identifies extraction mistakes, confusing faithfulness instruction, counterintuitive feature influence, misleading coherence instruction, and generation misalignment as persistent causes of error [2603.20003].

There is also a methodological controversy over whether a critic must exist at all. SeeUPO argues that mainstream methods cannot jointly provide critic-free training and convergence guarantees in multi-turn settings, while ProAct introduces a parameter-free Monte-Carlo Critic to stabilize PPO and GRPO with environment-grounded rollouts [2602.06554]. This suggests that some of the observed benefits attributed to “agentic critics” are entangled with deeper questions about credit assignment, long-horizon variance, and the limits of critic-free optimization.

## 6. Broader landscape: judges, reward models, self-critique, and agency

The broader agentic-AI literature places the agentic critic inside a larger taxonomy of evaluators. "The Hitchhiker's Guide to Agentic AI" treats critics as classical value functions in actor-critic RL, reward models trained with Bradley–Terry preference objectives, human or LLM judges, group-relative baselines in GRPO, verification-based rewards, process reward models, and agent evaluation frameworks [2606.24937]. This wider view is consistent with the empirical papers: some critics are learned value estimators, some are prompted evaluators, some are reward models, and some are explicit judges in multi-agent loops.

At the same time, the conceptual literature cautions against conflating critique mechanisms with full agency. "Critique of Agent Model" distinguishes *agentic* systems, whose competence resides in engineered workflows, from *agentive* systems, whose capabilities arise endogenously. It argues that genuine agency requires internalized structures of goal, identity, decision-making, self-regulation, and learning, rather than external scaffolding alone [2606.23991]. From that perspective, many current agentic critics are components inside larger engineered loops rather than evidence of fully internalized self-regulation.

A balanced synthesis is therefore possible. On one side, agentic critics have shown measurable benefits in long-video reasoning, search, documentation maintenance, agent training, GUI control, front-end code generation, and regulated workflows [2509.07680]. On the other side, critique can destabilize systems when judges are deceptive, when feedback strategies are mismatched to model pairings, or when the critic itself is noisy, miscalibrated, or computationally burdensome [2506.03332]. The literature therefore supports a restrained definition: an agentic critic is an evaluator embedded in an iterative agent workflow whose outputs alter subsequent reasoning, retrieval, editing, or optimization. Whether that evaluator is helpful, harmful, or unnecessary depends on the evaluated object, the intervention mechanism, the training regime, and the robustness of the surrounding system.

Source: https://www.emergentmind.com/topics/agentic-critic