Self-Improving VLM Judges Without Human Annotations (2512.05145v1)

Abstract: Effective judges of Vision-LLMs (VLMs) are crucial for model development. Current methods for training VLM judges mainly rely on large-scale human preference annotations. However, such an approach is costly, and the annotations easily become obsolete as models rapidly improve. In this work, we present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data. Our method is iterative and has three stages: (1) generate diverse multimodal instruction-response pairs at varying quality levels, (2) generate reasoning traces and judgments for each pair, removing the ones that do not match our expected quality levels, and (3) training on correct judge answers and their reasoning traces. We evaluate the resulting judge on Multimodal RewardBench and VL-RewardBench across domains: correctness, preference, reasoning, safety, and visual question-answering. Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on VL-RewardBench, often outperforming much larger models including Llama-3.2-90B, GPT-4o, and Claude 3.5 Sonnet, with particularly strong gains in general, hallucination, and reasoning dimensions. The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.

• The paper introduces a self-supervised approach that iteratively refines VLM judges using synthetic preference pairs.
• The methodology leverages majority voting and synthetic data filtering to achieve significant accuracy gains, with up to a 40.5% improvement on VLRB metrics.
• The study demonstrates that an 11B parameter VLM judge can rival larger industry models, though challenges remain in safety and generalization.

Self-Improving Vision-LLM Judges Without Human Annotations

Introduction and Motivation

Vision-LLMs (VLMs) have established themselves as critical platforms for multimodal content generation and evaluation. Central to advancing practical usability and alignment is the training of robust reward models (judges) capable of automatic output assessment. Existing paradigms for VLM judge construction depend heavily on large-scale human preference annotation or distillation from proprietary, closed-source models—a process that is both expensive and susceptible to rapid obsolescence given the pace of model advancement. "Self-Improving VLM Judges Without Human Annotations" (2512.05145) introduces an alternate data-centric paradigm: self-supervised judge training via strategically constructed synthetic preference pairs, obviating the need for any human preference supervision.

Methodology: Iterative Synthetic Self-Improvement

The proposed framework consists of a three-stage iterative self-improvement loop: synthetic preference pair generation, judge output filtering with reasoning trace sampling, and judge model fine-tuning. This architecture leverages the VLM's own generations and the current judge's evaluations to drive progressive self-refinement without any external annotation or teacher model.

Figure 1: Schematic of the iterative synthetic data generation and judge self-training pipeline for both open- and closed-solution tasks.

Synthetic Data Generation

Task types are split by answer structure: open-ended (e.g., captions, reasoning) and closed-ended (e.g., multiple choice, numerical, short-form responses).

• Open-ended tasks: The system generates an original response and then synthesizes a "degraded" version by explicitly injecting semantic errors (e.g., altering object attributes, swapping relations, or modifying quantitative details), creating synthetic preference pairs with known label orientation.
• Closed-ended tasks: The model samples multiple candidate responses; majority voting designates the most common answer as the "preferred" response, paired with a randomly sampled alternative. This exploits the baseline model's own response consistency, avoiding ground-truth label dependence.

Iterative Data Curation and Model Training

After each synthetic batch, the previous-iteration judge model is tasked with evaluating the new preference pairs, generating natural language reasoning traces and binary verdicts. Only cases where the model's verdict aligns with the synthetic preference are retained, and positional bias is mitigated by requiring correct judgments in both response orderings. The curated set of (image, question, pair, reasoning trace, binary label) samples is then used for supervised fine-tuning of the judge model, and the process repeats.

Experimental Results and Quantitative Analysis

Experiments center on Llama-3.2-11B as the base judge, evaluated on VL-RewardBench (Li et al., 26 Nov 2024) and Multimodal RewardBench (Yasunaga et al., 20 Feb 2025). The iterative self-supervised process yields substantial improvements:

• VLRB overall accuracy increases from 0.38 to 0.54 after 4 iterations, a 40.5% relative gain.
• MMRB overall accuracy increases from 0.50 to 0.54, a 7.5% gain.
• On VLRB general instruction following, the 11B judge outperforms both Llama-3.2-90B and Claude-3.5 Sonnet, achieving scores of 0.50+.
• For hallucination detection and VQA, the model approaches or exceeds much larger systems, despite starting from a much weaker baseline.

  Figure 2: Iterative judge training improves VLRB/MMRB metrics, matching or exceeding large closed models after 4 iterations.

Ablation analyses show that majority voting for synthetic data filtering regularly outperforms ground-truth-based filtering (when both are held to equivalent data sizes), suggesting that model-consistency signals are highly effective supervision, and may even more robustly surface reasoning pathologies missed by correctness-only selection.

Figure 3: Majority voting for preference pair construction yields consistent improvement over gold-label filtering in reasoning and VQA.

Dimension-Specific Gains, Limitations, and Failure Modes

Performance improvement is not uniform across metrics. VLRB General and Hallucination detection benefit most from iterative self-training, with VLRB General seeing a 69% jump and Hallucination 41%. However, reasoning, safety, and general evaluation on some benchmarks show minimal or non-monotonic improvement. Layered analysis reveals:

Figure 4: Large improvement is concentrated on VLRB General and Hallucination; MMRB Safety and General show little advance.

• Synthetic data methodology (detail alternation for general/hallucination; majority voting for closed-form VQA) is most effective in domains structurally similar to the synthetic errors injected.
• Safety evaluation is limited by a lack of explicit toxic/biased data in self-generated samples—a natural deficit of the generative pipeline.
• Diminishing returns are observed after 3-4 iterations, highlighting an inherent limit in self-improvement absent external signals or increased task/image diversity.

Additional analysis of reasoning trace quality across iterations demonstrates not only higher verdict accuracy, but a marked improvement in the granularity and correctness of natural language rationales, supporting the claim that self-improvement cannot be solely attributed to overfitting to superficial signals.

Implications and Future Directions

This work demonstrates that VLM judge models can be iteratively improved using only the model's generative and evaluative capabilities, bypassing human annotation and large-teacher distillation entirely. The strong result that an 11B parameter VLM judge can rival or outperform industry-scale models on established reward benchmarks contradicts the commonly held conviction that human or external teacher signals are indispensable for high-fidelity evaluator training.

From a theoretical angle, these findings reinforce the plausibility of self-bootstrapping reward models in multimodal tasks, highlighting the signal sufficiency of model-consistency heuristics (e.g., majority voting) even in the absence of ground-truth supervision. In practical terms, this opens pathways for efficient, scalable training of reward models for novel, data-scarce, or privacy-restricted domains—so long as the underlying generative models possess sufficient base proficiency for meaningful synthetic contrast generation.

Outstanding open problems include robust safety evaluation under adversarial content, domain generalization to high-diversity distribution shifts, and the integration of self-improving frameworks with mixture-of-experts and routing architectures that can address performance variance across judgment dimensions.

Conclusion

The methods introduced in "Self-Improving VLM Judges Without Human Annotations" push forward the frontier in unsupervised, scalable reward model training for multimodal systems. Iterative self-correction via synthetic preference generation and reasoning-based filtering allows compact VLMs to achieve evaluation performance comparable to closed, much larger models without reliance on costly or quickly outdated human preference signals. The results suggest substantial autonomy for future VLM reward model development, though further research is warranted for safety-critical and highly complex generalization tasks.

Citation: "Self-Improving VLM Judges Without Human Annotations" (2512.05145).