Critique-Augmented Reinforcement Learning

Updated 16 March 2026

Critique-Augmented Reinforcement Learning is a framework that integrates human or model-generated critiques with traditional RL rewards to accelerate learning and enhance interpretability.
It employs diverse techniques like natural language feedback, advantage substitution, and multi-agent critique aggregation to optimize and stabilize policy updates.
Empirical results show significant gains in sample efficiency and performance, with notable improvements in pass rates and error detection across various benchmarks.

Critique-Augmented Reinforcement Learning (CA-RL) denotes a class of reinforcement learning methodologies in which learning is accelerated and/or stabilized by incorporating structured critique, either as natural-language feedback, explicit judgment signals, or auxiliary reward components. In contrast to scalar-only reward frameworks, CA-RL leverages critique to deliver intermediate, interpretable, and highly targeted learning signals. Modern CA-RL encompasses approaches where the critique is produced by humans, static models, or co-evolved agents, and spans reward modeling, policy optimization, and complex multi-agent coordination.

1. Formal Definitions and Core Paradigms

The core distinction of CA-RL lies in the explicit use of critique as a first-class training signal, which may take several forms:

Natural Language Critiques: Structured textual feedback on policy actions or outputs, often including error analysis, suggestions, and judgments (Xie et al., 5 Feb 2025, Zhang et al., 2024, Tang et al., 20 Jul 2025).
Critique as Advantage Estimate: Human or model feedback directly substitutes for the advantage function in the policy gradient (Arumugam et al., 2019).
Critique-Augmented Reward Modeling: Generation and use of critiques as auxiliary channels for reward models, yielding more interpretable and robust preference learning (Yu et al., 2024, Ye et al., 2024, Ankner et al., 2024).
Decoupled Critic-Policy Architectures: Policies and critics are separate, with the critic generating critiques that serve as guidance for subsequent policy updates or refinements (Xie et al., 5 Feb 2025, Tang et al., 20 Jul 2025, Li et al., 11 Jan 2026).

Formally, a CA-RL algorithm augments the Markov Decision Process (MDP) or its extension (e.g., POMDP) such that, in addition to observing classic state, action, and reward triples $(s,a,r)$ , it leverages critique $c$ produced as $c\sim Q_\theta(\cdot|s,a)$ (model-based), or $f_t$ (human). The learning objective then incorporates the downstream impact of critiques, such as maximizing the expected corrected policy reward or directly optimizing for critique alignment with gold labels (Xie et al., 5 Feb 2025, Ruan et al., 26 Sep 2025).

2. Methodological Taxonomy

2.1 Critique as Reward Shaping or Advantage Signal

Intrinsic Reward Decomposition: CA-RL decomposes sparse extrinsic rewards by inserting critique-derived intrinsic rewards at the token, span, or action level, as in RELC for text generation (Cao et al., 2024), or via advantage substitution as in Deep COACH (Arumugam et al., 2019).

2.2 Critique-Generation via Reinforcement Learning

Training Explicit Critic Policies: Separate LLM-based critics are optimized to synthesize effective natural language critiques through reward signals measuring downstream correction or alignment with reference judgments (Xie et al., 5 Feb 2025, Tang et al., 20 Jul 2025, Xi et al., 28 Oct 2025, Zhang et al., 2024). The CTRL framework, for example, formalizes critic training as maximizing post-refinement pass rates using group-relative policy optimization (Xie et al., 5 Feb 2025).

2.3 Joint Generation-Critique Training

Second-Order Rollout: Methods such as GC-RL interleave first-order generation with explicit critique tasks (second-order rollout), assigning separate but coordinated RL objectives to each (Yang et al., 26 Feb 2026). Critiques generated on sampled (q, r) pairs are directly scored and used in RL updates, often with careful filtering to maintain balanced data and avoid skew (Yang et al., 26 Feb 2026).

2.4 Co-Evolving Critic-Policy Loops

On-Policy Co-Evolution: ECHO advances CA-RL by jointly optimizing policy and critic models in a synchronized loop, solving the critic-staleness problem and dynamically adapting critique granularity to evolving policy error patterns (Li et al., 11 Jan 2026).

2.5 Critique-Augmented Reward Modeling

Two-Stage RM Training: Modern reward models synthesize critiques (with or without supervision) and utilize these as explicit additional inputs for scalar reward prediction, leading to interpretable and more robust RMs (Yu et al., 2024, Ye et al., 2024, Ankner et al., 2024). Techniques include joint optimization of critique generation and reward prediction heads and self-consistency decoding for robust scoring.

2.6 Multi-Agent and Multi-LLM Critique Aggregation

Multi-Agent Feedback: Training data can be constructed via the aggregation and meta-summarization of critiques from multiple LLMs, producing high-fidelity SFT and RL datasets annotated with meta-critique severity and cross-agent flaw-detection (Lan et al., 2024).
Actor–Critic–Refiner Pipelines: In long-horizon RL or planning, explicit segregation of generation, critique, and refinement LLMs leads to improved alignment between plans and executable behaviors; critique LLMs score and rank plans for targeted intervention (Fan, 26 Nov 2025).

3. Representative Algorithms and Workflows

Framework	Critique Source	RL Objective	Policy/Critic Decoupling	Notable Innovations
CTRL (Xie et al., 5 Feb 2025)	Learned critic (LLM)	Maximize pass-rate	Yes	Iterative critique, GRPO, code domain
GC-RL (Yang et al., 26 Feb 2026)	Policy (self-critique)	Dual gen./critique	No	Second-order rollout, denoising
Critique-Coder (Ruan et al., 26 Sep 2025)	Policy (binary)	Hybrid RL+CRL	No	Transferability, logical reasoning
RefCritic (Tang et al., 20 Jul 2025)	LLM critic	RL over CoT & refinement	Yes	Long-CoT, dual rewards, math reasoning
Critique-RL (Xi et al., 28 Oct 2025)	LLM critic	Two-stage RL	Yes	Stagewise discriminability/helpfulness
CLoud/Critic-RM (Ankner et al., 2024, Yu et al., 2024)	LLM (self)	RM fine-tuning	N/A	Explicit critique output for reward
ECHO (Li et al., 11 Jan 2026)	LLM critic	Co-evolutionary loop	Yes	Critic–policy synchronization
MultiCritique (Lan et al., 2024)	Multi-agent LLMs	Preference-based PPO	N/A	MARS, analytic critique units

These frameworks implement critique in heterogeneous roles: as reward, as action in an auxiliary space, or as evaluator in a modular LLM pipeline.

4. Empirical Findings, Performance, and Scalability

Reported results demonstrate that introducing critique—whether as standalone RL tasks, augmenting rewards, or generating natural language judgments—substantially enhances both sample efficiency and final task success rates:

CTRL: RL-trained critics double the pass@1 rate on competitive code benchmarks, scaling further with iterative refinement (e.g., 7.88→16.24% on CodeContests in five turns) (Xie et al., 5 Feb 2025).
GC-RL: Joint gen-critique training increases generation accuracy (59.3% vs 56.7% vanilla RL), with further gains in critique accuracy (78.6% vs 73.8%) (Yang et al., 26 Feb 2026).
Critique-Coder: Hybrid RL + CRL achieves higher LiveCodeBench scores (e.g., Qwen3-8B, 60.8% vs 59.6% RL-only) and generalization to logical reasoning tasks (Ruan et al., 26 Sep 2025).
RefCritic: RL-trained long CoT critics yield 4–7 point increases in refinement pass rates and outperform step-level supervised critics on error detection (Tang et al., 20 Jul 2025).
Reward Model Augmentation: Critic-RM and CLoud reward models increase RewardBench accuracy by 3–6 points, add interpretability, and strengthen resistance to overfitting (Yu et al., 2024, Ankner et al., 2024).

Statistical ablations confirm that learned or dynamically-evolved critics outperform static, supervised, or self-critique baselines. Data filtering and denoising are often critical for maintaining the informativeness of critique-derived rewards (Yang et al., 26 Feb 2026).

5. Critique Representation and Technical Design

CA-RL frameworks employ a variety of formalizations for critique. The most common are:

Structured Multi-Part Texts: Critiques are generated with forced structure (e.g., strengths, improvement suggestions, final judgment) (Xie et al., 5 Feb 2025, Xi et al., 28 Oct 2025).
Analytical Critique Units (ACUs): Finer-grained meta-labels describing error location, type, suggested fix, and severity, often used in pipelines with multi-agent critique aggregation (Lan et al., 2024).
Chain-of-Thought Critiques: Multi-step reasoning traces plus explicit correctness labels; reward assignment may be based on both judgment accuracy and downstream refinement efficacy (Tang et al., 20 Jul 2025, Zhang et al., 2024).
Natural-Language Feedback for Reward Modeling: Critiques are concatenated with candidate outputs and directly attended to by reward model heads (Yu et al., 2024, Ankner et al., 2024, Ye et al., 2024).

Encoding and decoding strategies typically flatten both problem and solution context into a text prompt, possibly augmented with synthetic hints, reference responses, or meta-instructions.

6. Limitations, Open Problems, and Future Directions

A number of open challenges affect the design and deployment of CA-RL:

Data Quality and Distributional Effects:

Training critics with imbalanced positive/negative examples leads to skewed predictions; cache filtering or reward re-weighting is required to avoid degenerate solutions (Yang et al., 26 Feb 2026).
Outcome-based rewards for critiques may be noisy; sampling-based denoising alleviates but does not eliminate this (Yang et al., 26 Feb 2026).
Excessive critique data (>50%) can degrade performance, suggesting critique must be a complement to, not a replacement for, generation (Ruan et al., 26 Sep 2025).

Critic Staleness and Pipeline Synchronization:

Offline critics rapidly become misaligned with evolving policy distributions; joint on-policy updates (as in ECHO) resolve staleness but increase computational and algorithmic complexity (Li et al., 11 Jan 2026).

Interpretability and Computational Overhead:

Critique-enhanced rewards add inference steps; multi-sample critique techniques (e.g., CLoud self-consistency) yield only marginal gains in some domains (Ankner et al., 2024).
Large LLM critics add training/inference cost; smaller models risk insufficient discrimination (Xie et al., 5 Feb 2025, Cao et al., 2024).

Generalization and Domain Transfer:

Most implementations are restricted to code or mathematical domains, requiring execution sandboxes or verifiers for reward assignment. Extending CA-RL to open-ended tasks or multi-modal domains necessitates robust proxy reward mechanisms (Xie et al., 5 Feb 2025, Zhang et al., 2024).

Future Research Directions:

Extending critique-augmented RL to larger scales and more domains (e.g. mathematics, open-ended reasoning) (Yang et al., 26 Feb 2026, Xie et al., 5 Feb 2025).
Developing critics with richer output spaces: step-level markers, graded severity, or multi-aspect reasoning (Lan et al., 2024, Yu et al., 2024).
Running joint reward model and critique generation optimization, particularly for data efficiency and robustness in reward learning (Yu et al., 2024, Ye et al., 2024).
Incorporating human-in-the-loop or hybrid human/machine feedback, especially in high-stakes domains (Zhang et al., 2024).
Co-evolving policy, critic, and reward models end-to-end to realize self-improving RLHF pipelines (Li et al., 11 Jan 2026, Yu et al., 2024).

7. Relationship to Classic Actor–Critic and RLHF Methods

CA-RL generalizes classic actor–critic schemas by reconceptualizing the critic: from estimating a value function to delivering dense, interpretable critique or even direct policy-correction instructions. In reward modeling for RLHF, CA-RL frameworks transition from direct scalar regression to “think out loud” reasoning, improving transparency and sample efficiency (Ankner et al., 2024). In off-policy RL, innovations like Critic-Guided Action Redistribution explicitly utilize the critic’s Q-values to resample or reweight actions, provably improving or matching actor-only expected rewards (Huang et al., 2022).

Across settings, the structural decoupling of generation and critique, explicit reward shaping via critique, and multi-agent feedback aggregation are central mechanisms by which CA-RL advances beyond previous RL pipelines. Recent empirical results demonstrate that these mechanisms yield significant and robust gains in both final task accuracy and alignment with human-preferred behaviors.