Language-Space Critic Learning

• Language-Space Critic Learning is a framework that uses natural language–based critique to shape rewards and guide agent policies in text generation and interactive settings.
• It employs pretrained language models as critics to provide fine-grained, step‐wise feedback, enhancing sample efficiency and aligning outputs with target objectives.
• Applications span reinforcement learning, dialogue systems, agentic planning, and robotics, demonstrating measurable gains in performance, safety, and interpretability.

Language-space critic learning refers to a family of methods in which an agent’s progress, behavior, or generation is evaluated—and improved—by utilizing critique, guidance, or reward signals expressed in natural language or in the native representational space of LLMs, instead of only using scalar values, binary labels, or hand-crafted task-specific reward functions. Pioneered to address challenges in alignment, sample efficiency, and credit assignment across both text generation and interactive agentic settings, these frameworks leverage large pretrained LLMs as critics to provide dense, structured, or step-wise feedback that can be fed back into policy optimization, supervised updates, or iterative refinement cycles. The paradigm is applicable across settings including reinforcement learning from human feedback (RLHF), offline reinforcement learning, agentic planning, error diagnosis, and robotic control, and subsumes both scenario-specific and general-purpose variants.

1. Formalization and Key Principles

Language-space critic learning is typically instantiated within Markov Decision Process (MDP) or partially observable MDP (POMDP) formulations where both the state and action spaces are defined in terms of language (token sequences, text descriptions, or multi-modal embeddings). The defining characteristic is the use of LLMs operating in semantic space to mediate the critic signal—either as dense reward, episodic verdict, step-level critique, or trajectory evaluation.

For example, in autoregressive text generation under RL,

• State $s_t = (prompt, y_0, \ldots, y_{t-1})$, action $a_t = y_t$
• Sparse extrinsic reward $r^{{ex}}_t$ is only assigned at final output
• Intrinsic language-derived reward $r^{{in}}_t$ is computed for intermediate steps or spans via a critic LM, leading to a shaped objective:

$$J^{\mathrm{RELC}}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T} \gamma^t \left( \alpha_1 r_t^{ex} + \alpha_2 r_t^{in} \right) \right]$$

[$2401.07382$].

In sequential, interactive agents, the critic model may be tasked with generating a natural-language assessment or refinement signal (e.g., “The last API call failed because parameter X was malformed; next, try...”), which is then consumed by the actor policy to inform updated action selection or by the training process to select high-quality demonstration data without relying on sparse or environment-defined scalar rewards [$2411.19547$].

The multi-modal extension includes vision-language-action critics that, given language goals and observation pairs, predict scalar progress and done signals, again leveraging semantic similarity and grounding between instruction and observed state changes [$2509.15937$].

2. Core Architectures and Algorithms

Language-space critic learning admits diverse architectural variants, sharing a critic-actor decomposition but diverging in critic-output format, update mechanism, and agent-critic interaction. Across current literature, the following general patterns are observed:

Critic Output Type	Agent Update Mechanism	Main Tasks/Domains
Token/span-level scores	Policy gradient with shaped reward	Text generation, RLHF
Trajectory verdict	Iterative supervised fine-tuning	API composition, dialogue agents
Natural-language critique	Iterative action refinement via prompt	Planning, reasoning agents
Scalar progress delta	PPO advantage computed on dense rewards	Real-world robotics
Q-value in language space	Offline RL with action re-ranking	Interactive environments

In RELC (Cao et al., 2024), the critic is a frozen LLM (e.g., GPT-3.5-turbo or Llama 2) prompted with task description and policy outputs, returning natural-language critiques mapping to intrinsic reward at the token or span level; these are integrated into the RL optimization loop via standard policy gradient methods.

In weak-supervision frameworks [$2411.19547$], the critic scores agent-generated trajectories (“Success probability” in $[0,1]$) and top-ranked outputs are mined as pseudo-positive examples for further supervised fine-tuning, iterating over multiple rounds.

Critique-guided improvement frameworks (Yang et al., 20 Mar 2025, Hong et al., 4 Dec 2025) employ a generative or discriminative natural-language critic, often trained by supervised learning on expert-annotated critique corpora, to provide multi-faceted feedback (e.g., contribution, feasibility, revision suggestions). The agent is updated by conditioning on these critiques in subsequent action generations (in-episode or off-policy), or by distilling improved decisions into its core policy via maximum-likelihood or hybrid RL-SFT approaches.

Vision-language-action critics (e.g., VLAC (Zhai et al., 19 Sep 2025)) unify action and critic heads within a single transformer backbone, issuing progress deltas, done signals, or direct action tokens as part of a shared token stream; dense step-wise rewards replace sparse environmental success signals, enabling rapid transfer across robotic skills.

Offline RL settings (Retrospex (Xiang et al., 17 May 2025)) use a compact language-based critic network (GRU over task, state, and action tokens) trained to compute Q-values, which are then used to rescore sampled actions, interpolating between LLM action-likelihood and value-based assessment.

3. Critique-to-Reward and Integration Mechanisms

The core technical challenge is mapping the critic’s natural-language output to a form that can be ingested by the agent update procedure. Paper-specific paradigms include:

• Span or token mapping: Parse output of the form “Span X–Y: [label/score]” and apply per-token shaping reward $a_t = y_t$0 via a mapping $a_t = y_t$1, where $a_t = y_t$2 is the critic-assigned score or label to that span/tokens [$a_t = y_t$3].
• Trajectory-level filtering: Use the critic’s binary or scalar success probability to select the top $a_t = y_t$4 of sampled trajectories, treating them as weakly labeled data for subsequent policy updates [$a_t = y_t$5].
• Iterative revision: Prompt the actor with a batch of candidate actions, elicit individual natural-language critiques per action, and then condition the next action on the set of critiques, allowing the actor to directly “read” and integrate the feedback into refined decision-making (Yang et al., 20 Mar 2025).
• Bellman language backup: For actor-critic in language, generate successor predictions and evaluate future reward as a natural-language “optimality” assessment, aligning the actor’s future behavior with these critiques (Hong et al., 4 Dec 2025).
• RL–supervised hybrid: Use deliberate stepwise, multi-perspective critiques as pseudo-labels for SFT followed by reinforcement learning with rewards defined as successful error detection or refinement (Yang et al., 1 May 2025).

Correct parsing and calibration of the critic’s feedback (e.g., via mapping functions or prompt standardization) are critical to ensure that the shaped reward or guidance is well-aligned with final task objectives and that intrinsic rewards do not mislead the policy due to critic-policy mismatch.

4. Empirical Results and Benchmarking

Language-space critic learning methods consistently demonstrate improved sample efficiency, performance, and stability compared to scalar-reward RL or pure imitation approaches across a range of domains:

• RELC (Cao et al., 2024): Achieves +15 points in sentiment control and halves toxicity in detoxification relative to PPO; outperforms PPO on human preference evaluation and matches it in ROUGE for summarization.
• Weakly supervised API-bank (Gong et al., 2024): Agents scaled from ∼10–16% baseline accuracy to 47–50%, nearly matching the 51.6% of GPT-4, and substantially outperforming other open-source models; critic precision ≈70%, recall ≈97%, which is sufficient for effective filtering via percentile selection.
• VLAC (Zhai et al., 19 Sep 2025): Lifts real-robot success rates from ≈30% to ≈90% with only 200 episodes; further 50% sample efficiency increase with human-in-the-loop; success persists across scaling to many robots.
• Retrospex (Xiang et al., 17 May 2025): Offline Q-learning critic yields consistent >+3–9 point improvements in success rate versus imitation-only LLM agents, with dynamic rescoring outperforming static mixtures or critic ablation.
• CGI (Yang et al., 20 Mar 2025): Critique-guided improvement with an 8B critic yields 74.2% aggregate performance across three hard reasoning environments and surpasses GPT-4o, AgentLM-70B, iterative SFT, and Reflexion baselines.
• Critique-RL (Xi et al., 28 Oct 2025): Two-stage critic RL achieves +12 points over SFT in in-domain accuracy, +5–9 points out-of-domain, with ablation confirming the need for discrimination and refinement rewards.

Notably, actor-critic variants in language space show more stable and sample-efficient convergence than classical RL, and natural-language critics are often superior to token-level or scalar regression-based critics in both interpretability and efficacy.

5. Theoretical Rationale, Strengths, and Limitations

Language-space critic learning provides several theoretical and practical benefits:

• Dense credit assignment: Intermediate feedback circumvents the endemic problem of sparse rewards in long-horizon tasks, markedly accelerating credit discovery in sequential generation or manipulation.
• Alignment with model priors: Leveraging LLMs as critics exploits their pretraining on explanation, reasoning, and error diagnosis tasks, creating instruction-following and generalization capacity in constructing reward or critique.
• Generalizability: Natural-language critics can express multi-dimensional task success or failure (e.g., factuality, coverage, reasoning, formatting), reducing the dependence on bespoke reward engineering.
• Plug-and-play supervision: Methods require minimal additional annotation or domain-specific labeling, often learning entirely from model-prompted, synthetic, or weakly labeled examples.

However, the approach is not without limitations:

• Critic capacity: Meaningful critiques require a sufficiently strong critic; smaller or poorly aligned critics may yield misleading feedback, impeding policy progress.
• Inference overhead: Critic inference (especially for large LLMs or multi-pass critique-refinement cycles) can introduce non-trivial latency and computational expense, which must be balanced against learning gains.
• Critic-policy mismatch: When using frozen critics, policy improvement can degrade the utility or relevance of future critiques; periodic critic updates or distillation of lightweight reward models are potential remedies.
• Reward misalignment risk: As with any reward-shaping scheme, improperly calibrated critic outputs (e.g., bad span mapping, untested edge cases) may cause detrimental exploration or policy collapse.

6. Extensions and Research Directions

Active research in language-space critic learning investigates several promising directions:

• Joint critic–policy training: Alternating or co-training the critic and actor, potentially with hybrid RL–SFT objectives, to reduce mismatch as the actor’s competence increases.
• Multi-modal and real-world tasks: Extending to vision-language-action settings, using video-language critics for robotics [$a_t = y_t$6, $a_t = y_t$7], or to symbolic reasoning and code synthesis.
• Critique format automation: Learning to generate or parse critiques in arbitrary formats, reducing handcrafted prompt engineering and moving toward critic self-supervision and automated feedback.
• Human-in-the-loop and safety: Merging model-based critique with human interventions for high-stakes, opaque, or safety-critical applications.
• Distillation to efficient models: Compressing LLM critic judgments into smaller, low-latency reward or value models suitable for large-scale or on-device deployment.
• Meta-critique and self-improvement: Developing critics capable of evaluating and refining their own critiques and reasoning chains, as in deliberate or meta-critique frameworks [$a_t = y_t$8].

7. Representative Implementations

Method	Critic Output	Domain(s)	Update Mechanism	Empirical Highlights
RELC (Cao et al., 2024)	Token/span reward	Text generation	PPO w/dense RL	+15 sentiment, –50% toxicity, ↑ preference
Weak-Superv. (Gong et al., 2024)	Trajectory verdict	API agents	Iterative SFT	50% accuracy (GPT-4: 51.6%); open-source 6B
VLAC (Zhai et al., 19 Sep 2025)	Scalar progress/done	Real-robotics	PPO (unified)	30%→90% success (200 episodes, 4 tasks)
Retrospex (Xiang et al., 17 May 2025)	Q-value (language state-action)	Text agents	Offline IQL, rescoring	+3–9% SR; no LLM parameter update
CGI (Yang et al., 20 Mar 2025)	Natural-language critique	Agentic reasoning	Critique-guided SFT loop	SOTA, surpasses GPT-4 on benchmarks
Critique-RL (Xi et al., 28 Oct 2025)	Nat. lang. critique	Reasoning models	Two-stage RL	+12% in-domain, +9% OOD; robust improvement
DeepCritic (Yang et al., 1 May 2025)	Multi-persp. math crit	Stepwise math solns	SFT→RL (PRM800K, MC-Numina)	F1=67 (vs GPT-4o 58); deliberate error-finding

• “Beyond Sparse Rewards: Enhancing Reinforcement Learning with LLM Critique in Text Generation” [$a_t = y_t$9]
• “Training Agents with Weakly Supervised Feedback from LLMs” [$r^{{ex}}_t$0]
• “A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning” [$r^{{ex}}_t$1]
• “Retrospex: Language Agent Meets Offline Reinforcement Learning Critic” [$r^{{ex}}_t$2]
• “Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics” [$r^{{ex}}_t$3]
• “The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement” [$r^{{ex}}_t$4]
• “Critique-RL: Training LLMs for Critiquing through Two-Stage Reinforcement Learning” [$r^{{ex}}_t$5]
• “Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space” [$r^{{ex}}_t$6]
• “DeepCritic: Deliberate Critique with LLMs” [$r^{{ex}}_t$7]