RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks (2511.01758v1)

Abstract: Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to prohibitively high verification costs and incomplete assessments of a response, making reinforcement learning (RL) post-training with rubric-based rewards difficult to scale. This problem is exacerbated by the fact that often the best way to combine these rubrics into one single reward is also highly prompt-specific. We propose Reinforcement Learning with Adversarial Critic (RLAC), a post-training approach that addresses these challenges via dynamic rubric verification. Our approach employs a LLM as a critic that dynamically identifies only the most likely failure modes (e.g., a factual error or unhandled edge case), which are then verified by an external validator to optimize both generator and critic jointly. By training both the generator and the critic, this game enhances the critic's error detection and the generator's output quality while reducing required verifications. Our experiments demonstrate that RLAC improves factual accuracy in text generation and correctness in code generation, while also outperforming exhaustive verification and reward model methods. We show that dynamic critics are more effective than fixed critics, showcasing the potential of RLAC for scaling RL post-training to free-form generation tasks.

• The paper introduces RLAC, a novel min-max framework that leverages an adversarial critic to enable scalable, prompt-specific optimization for free-form generation tasks.
• It demonstrates significant improvements in factual accuracy and verification efficiency by dynamically adapting the critic and utilizing external validation.
• Empirical results on text and code generation tasks show reduced verification calls and higher FactScore performance compared to traditional RL and reward-model methods.

RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks

Introduction

The RLAC framework addresses the challenge of post-training LLMs for open-ended generation tasks where outputs must satisfy diverse, often implicit, and prompt-specific rubrics. Traditional RL post-training approaches either rely on exhaustive rubric enumeration, which is computationally prohibitive, or reward-model methods, which are susceptible to reward hacking and misalignment. RLAC introduces a dynamic, adversarial game between a generator and a learned critic, with an external validator providing ground-truth supervision. This paradigm enables scalable, verifiable, and prompt-specific optimization for free-form generation tasks.

Figure 1: RLAC contrasts with enumerative verification and reward-model methods by training a critic to propose likely-wrong facts, which are then externally validated, yielding efficient, adversarial supervision.

RLAC Framework

Formalization

RLAC reformulates the RL objective for free-form generation as a min-max game. The generator $\pi^g$ produces outputs $a$ for prompts $s$, and the critic $\pi^c$ proposes rubrics $c$ (verifiable properties) where the output is likely to fail. The external validator checks $R(s, a, c)$, a binary function indicating rubric satisfaction. The optimization is:

$$\pi^g{} = \arg\max_{\pi}~ \min_{\pi^c}~ \mathbb{E}_{s \sim \mathcal{S}} \left[ \mathbb{E}_{a \sim \pi(\cdot|s)} \mathbb{E}_{c \sim \pi^c(\cdot|s,a)} \left[ R(s, a, c) \right] \right]$$

This formulation bypasses the need to enumerate all rubrics, focusing verification on the most adversarially selected failure mode.

Practical Implementation

• Generator: Fine-tuned LLM producing candidate outputs for each prompt.
• Critic: Fine-tuned LLM proposing rubrics (e.g., atomic facts or test cases) likely to be violated.
• Validator: External process (e.g., FactScore for factuality, code execution for programming) providing binary feedback.

Both generator and critic are updated via the DPO objective, using positive/negative samples derived from validator feedback. RLAC is agnostic to the underlying RL algorithm, supporting PPO, GRPO, or DPO.

Experimental Results

Factual Text Generation

RLAC was evaluated on Wikipedia biography generation, using FactScore as the validator. Compared to FactTune-FS (exhaustive verification) and ArmoRM (reward-model), RLAC achieved higher factual accuracy with dramatically fewer verification calls.

• Qwen3-8B, 8-sentence generation: RLAC reached FactScore 0.889 vs. FactTune-FS 0.867, with only 76,800 verification calls (5.7× fewer).
• RLAC scales efficiently with output length, maintaining verification quality while reducing cost.

  Figure 2: RLAC demonstrates superior training dynamics, verification efficiency, and exploration compared to FactTune-FS and ArmoRM on Qwen3-8B.

RLAC avoids reward hacking, as evidenced by monotonic FactScore improvements correlated with KL divergence from the base model, unlike ArmoRM.

Ablation Studies

• Noisy Validator: RLAC performance degrades below baseline, confirming the necessity of reliable external validation.
• Static Critic: Detection accuracy drops as the generator exploits fixed patterns, yielding minimal gains (+0.6%) compared to adversarial critic (+1.8%).

  Figure 3: Validator outcomes for suspicious facts proposed by the critic; static critics quickly lose supervision quality, while adversarial critics sustain learning pressure.

  

  Figure 4: Static critic's detection accuracy degrades as the generator exploits its patterns, while adversarial critic maintains robust supervision and performance gains.

Code Generation

RLAC was tested on HumanEval, MBPP, BigCodeBench, and LiveCodeBench using Qwen2.5-Coder-7B models. Despite using only 9% of the training data, RLAC outperformed AceCoder-RM (reward model) and AceCoder-Rule (enumerative RL):

• Qwen2.5-Coder-7B-Instruct: RLAC achieved average score 56.6, surpassing AceCoder-Rule (55.9) and AceCoder-RM (55.3).
• RLAC required only 192k validator calls vs. 7.86M for AceCoder-Rule (97.5% reduction).

RLAC is robust to noisy validation, maintaining performance where reward-model methods degrade due to reward hacking.

Implementation Considerations

• Validator Design: RLAC's effectiveness depends on reliable, domain-specific validators. For factuality, FactScore is used; for code, reference solutions and test-case execution are required.
• Critic Adaptation: Dynamic, adversarial critic training is essential to prevent generator exploitation and maintain supervision quality.
• Resource Efficiency: RLAC reduces verification cost by focusing on high-risk failure modes, enabling practical scaling to long-form or complex tasks.
• Algorithm Choice: RLAC is compatible with any RL policy optimization method; DPO is preferred for stability and simplicity.

Implications and Future Directions

RLAC generalizes to any domain where outputs must satisfy multiple, potentially unenumerable rubrics, including story generation, scientific writing, and agent planning. By adversarially selecting the most informative rubric for verification, RLAC makes RL post-training feasible for tasks previously intractable due to combinatorial rubric explosion or lack of universal reward functions.

Potential future developments include:

• Integration with more sophisticated validators (e.g., retrieval-augmented or multi-modal).
• Extension to multi-agent or multi-critic settings for richer supervision.
• Application to safety-critical domains where exhaustive verification is impossible.

Conclusion

RLAC introduces a scalable, adversarial RL post-training paradigm for free-form generation tasks, leveraging a learned critic and external validator to provide prompt-specific, verifiable, and efficient supervision. Empirical results demonstrate superior factuality and correctness with reduced verification cost, and ablation studies confirm the necessity of dynamic critic adaptation and reliable validation. RLAC is broadly applicable to open-ended generation tasks and represents a practical solution to the challenge of scalable RL post-training in the presence of diverse evaluation rubrics.