LLaVA-Critic-R1: Unified Multimodal RL Model

• The paper introduces a unified critic-policy architecture that leverages RL on preference-labeled data to simultaneously evaluate and generate multimodal responses.
• It employs a 'think-then-answer' format with GRPO optimization to enforce output regularity and accurate preference selection.
• Empirical results show significant gains across 26 visual reasoning benchmarks, achieving state-of-the-art performance on several complex tasks.

LLaVA-Critic-R1 is a multimodal vision-LLM that unifies critic and policy model roles within a single architecture by leveraging reinforcement learning (RL) on preference-labeled critic datasets. Historically, critic models have been exclusively trained to evaluate outputs by assigning scalar scores or pairwise preferences and have been dissociated from policy models, which generate those outputs. LLaVA-Critic-R1 challenges this entrenched separation, transforming critic data into a verifiable RL task and directly fine-tuning a generative base model so that it can both evaluate and generate responses. Empirically, this approach produces a model that matches or surpasses specialized reasoning VLMs on visual reasoning, understanding, and complex multimodal benchmarks, while also supporting advanced test-time self-critique and promoting robust self-improvement mechanisms (Wang et al., 31 Aug 2025).

1. Model Architecture and Reinforcement Learning Training

LLaVA-Critic-R1 is built atop a strong multimodal base architecture (notably Qwen-2.5-VL-7B for principal experiments, with extensions to models such as ThinkLite-VL-7B). Unlike conventional approaches that maintain distinct architectures for critics (evaluators) and policy models (generators), LLaVA-Critic-R1 collapses this dichotomy via RL on preference judgment tasks. The architecture is enforced to produce output in a "think-then-answer" format:

• The "^^^^1^^^^" mandates that the model first emits an internal reasoning chain, delimited by > ...</think>, followed by a final discrete answer encapsulated in LaTeX-style \boxed{...} syntax. > > - The RL pipeline utilizes preference-labeled critic data, discarding superfluous chain-of-thought (CoT) rationales in the dataset and presenting the system only with the image, question, and two candidate responses. The prompt then requires a verifiable selection of the superior (or equally adequate) response. > > The RL objective couples preference accuracy and format adherence: > > $$r = \alpha \cdot r_{\text{pref}} + (1 - \alpha) \cdot r_{\text{format}}$$ > > where $r_{\text{pref}}$ is set to 1 if the predicted preference matches the ground truth, and $r_{\text{format}}$ is 1 only if the model conforms exactly to the enforced template. Experiments select $\alpha = 0.9$ to bias reward toward correct preference selection while maintaining output regularity. The RL update employs Group Relative Policy Optimization (GRPO), ensuring the model's output distribution remains close to the base model by incorporating a Kullback-Leibler penalty. > > ## 2. Preference Data Reorganization and Training Signal > > Traditional critic datasets for preference learning—such as instruction-following or pairwise human feedback—typically associate each data point with explanations or evaluative rationales. However, these rationales are often ambiguous, inject annotator biases, and limit machine verifiability. > > LLaVA-Critic-R1 strips away these justifications and presents only the core tuple: > > - Image > > - Question > > - Candidate Response 1 > > - Candidate Response 2 > > The RL training prompt then precisely requires the model to: > > 1. Reason internally (<think>...);

1. Output a final, machine-verifiable judgment (\boxed{...} syntax), selecting the preferred, non-preferred, or an "equal" outcome.

This restructuring allows for the preference label to be used as a direct reward signal, eliminating ambiguity and the need for auxiliary teacher-forcing or distillation from stronger models. The model is thus trained end-to-end for both critic and generative roles, without supervision on rationale content.

3. Performance Across Visual Reasoning and Understanding Benchmarks

LLaVA-Critic-R1's architecture and training regime deliver robust gains across a diverse suite of 26 established benchmarks, spanning:

• Perceptual/general VQA (Blink, HallusionBench_Image, MMVP, MMHal, MMBench, RealWorldQA)
• Complex image reasoning (MathVista, MathVision, MathVerse, MMMU, EMMA, Blind, V*, VisuLogic, ZeroBench)
• Chart understanding (ChartQA, OCRBench, AI2D, Charxiv_Reasoning)
• Video reasoning (MMVU, VideoMMMU)
• VLM-agent and reward learning (OSWorld, Online-Mind2Web, VLRewardBench, MM-RLHF)

In aggregate, LLaVA-Critic-R1 delivers an average policy gain of +5.7% over its Qwen-2.5-VL-7B base. LLaVA-Critic-R1+, which applies the method to ThinkLite-VL-7B, achieves state-of-the-art (SoTA) performance of 71.9 on MMMU at the 7B parameter scale. Moreover, via test-time self-critique (inference-time candidate selection using internal critic ability), performance further increases by an average of +13.8% on five representative reasoning tasks.

These results demonstrate that RL training on reorganized critic data not only produces a top-tier evaluator but also enhances generative reasoning, supporting the claim that critic models can be repurposed as strong policy models.

4. Unified Critic-Policy Role and Self-Critique at Inference

A distinctive feature of LLaVA-Critic-R1 is the unification of critic and policy functions:

• The model is capable of both evaluating external (or self-generated) candidate outputs (as a critic) and generating new, high-quality outputs in the same "think-then-answer" format (as a policy).
• At inference, self-critique is realized by generating multiple candidate trajectories, recursively judging them using the internal critic module, and selecting the optimal answer via a tournament of pairwise comparisons (akin to best-of-N sampling). This mechanism is compute-intensive (typically best-of-128 is reported) but obviates the need for external majority voting or third-party verification.

This internal self-critique enables the system to iteratively refine or select its own responses—functionality aligned with the goal of bootstrapped, self-improving vision-language agents.

5. Comparison to Previous Critic and Policy Models

The traditional division between critic and policy roles is entrenched in earlier approaches:

• Supervised fine-tuning (SFT) on critic data leads to models that provide shallow binary verdicts, rarely transferring improvements to generation quality.
• Policy models (trained for response generation) generally do not benefit from preference-data-driven training focused on evaluation.

LLaVA-Critic-R1's RL-based pipeline, using only the preference label as a verifiable reward, allows direct transfer of evaluative quality into policy performance. The empirical result—substantial gains in generative accuracy and state-of-the-art scores on multimodal reasoning benchmarks—corroborates the effectiveness of this unified approach.

In comparison to recent reinforcement-learning-based multimodal models, such as SophiaVL-R1 (Fan et al., 22 May 2025) (which introduces explicit thinking rewards to guide reasoning mid-process), LLaVA-Critic-R1 demonstrates that even without auxiliary rationale supervision, optimizing for verifiable preference yields both strong evaluation and generation. It is distinguished from models such as RefCritic (Tang et al., 20 Jul 2025) (which integrates refinement feedback rewards in RL to improve not just judgments but actionable guidance) by the simplicity and verifiability of its preference-driven RL signal.

6. Applications, Future Directions, and Research Implications

LLaVA-Critic-R1's dual-role architecture supports a range of applications:

• Visual Q&A and complex reasoning, where both assessment and answer generation are critical.
• Interactive vision-language agents capable of autonomous self-improvement, e.g., via iterative self-critique or tournament-style candidate selection.
• Evaluation and reinforcement learning pipelines, as its critic signals can be adopted for preference optimization or in-model reward modeling.

Future research may focus on:

• Optimizing the tradeoff between critic and policy performance via mixed or curriculum learning strategies, adjusting the composition of training data to balance evaluation and generative objectives.
• Extending RL objectives with richer reward formulations, potentially integrating human rationales in a controlled, verifiable fashion.
• Designing more efficient test-time self-critique strategies that maintain selection accuracy with lower computational cost.

This paradigm indicates a shift toward scalable, self-improving multimodal systems, where evaluation and generation are intertwined, and critic data is directly leveraged to build robust, high-performing vision-language agents. The unified approach and empirical outcomes position LLaVA-Critic-R1 as a foundational step toward dissolving the separation between evaluative and generative components in multimodal modeling (Wang et al., 31 Aug 2025).