Online Self-Calibration Against Hallucination in Vision-Language Models

Abstract: Large Vision-LLMs (LVLMs) often suffer from hallucinations, generating descriptions that include visual details absent from the input image. Recent preference alignment methods typically rely on supervision distilled from stronger models such as GPT. However, this offline paradigm introduces a Supervision-Perception Mismatch: the student model is forced to align with fine-grained details beyond its perceptual capacity, learning to guess rather than to see. To obtain reliable self-supervision for online learning, we identify a Generative-Discriminative Gap within LVLMs, where models exhibit higher accuracy on discriminative verification than open-ended generation. Leveraging this capability, we propose \textbf{O}nline \textbf{S}elf-\textbf{CA}lib\textbf{R}ation (OSCAR), a framework that integrates Monte Carlo Tree Search with a Dual-Granularity Reward Mechanism to construct preference data and iteratively refines the model via Direct Preference Optimization. Extensive experiments demonstrate that OSCAR achieves state-of-the-art performance on hallucination benchmarks while improving general multimodal capabilities.

• The paper introduces the OSCAR framework, using MCTS-guided decoding and dual-granularity rewards to mitigate hallucination in vision-language models.
• It identifies a supervision-perception mismatch where teacher-driven fine-tuning leads to overfitting and increased ungrounded content generation.
• Empirical results demonstrate that OSCAR significantly reduces hallucination rates and improves model faithfulness across multiple benchmarks.

Online Self-Calibration Against Hallucination in Vision-LLMs: A Detailed Analysis

Introduction and Motivation

Large Vision-LLMs (LVLMs) have propelled progress in multimodal AI, yet persistent hallucination—generating content unanchored in the visual input—remains a significant deployment barrier, particularly for high-stakes tasks requiring factual consistency. This paper identifies two intertwined root causes undermining hallucination mitigation in LVLMs:

1. Supervision-Perception Mismatch: Existing preference alignment approaches typically distill supervision from teacher models with stronger visual representations (e.g., GPT-4V, Qwen3-VL), compelling weaker student models to mimic fine-grained details which they cannot reliably perceive. This leads to overfitting to teacher priors and greater hallucination, as the student model compensates with language-based inferences instead of grounded perception.
2. Generative-Discriminative Gap: The authors observe that hallucination-prone LVLMs display substantially higher accuracy in discriminative tasks (e.g., verifying the presence of specific objects) than in open-ended generative tasks. This discrepancy suggests underutilized latent verification capabilities that can be leveraged for improved self-supervision.

These insights motivate a paradigm shift away from offline teacher-driven distillation toward an online, self-calibrated learning scheme aligned with the model's own perceptual bounds.

Figure 1: Supervision-Perception Mismatch. Teacher supervision with fine-grained visual details compels the student LVLM to hallucinate ungrounded content.

Empirical Evidence of Supervision-Perception Mismatch

The paper presents controlled experiments where LLaVA-1.5-7B is fine-tuned with captions distilled from a stronger Qwen3-VL-8B-Instruct teacher. Despite the teacher's superior faithfulness metrics, the student’s hallucination rate—in both the CHAIR and AMBER generative/faithfulness evaluations—increases monotonically with the amount of distilled data. Notably, the performance consistently degrades, even as more teacher data is used.

Figure 2: Fine-tuning with stronger teacher-distilled data leads to increased hallucination rates and degraded faithfulness, confirming the supervision-perception gap.

Characterizing the Generative-Discriminative Gap

Through quantitative and qualitative investigations, the authors demonstrate that the same LVLM can hallucinate objects (e.g., “clock on the wall”) during open-ended captioning, yet correctly deny the presence of these objects when queried in a discriminative, verification setting. Applying self-querying to “clean” output captions already reduces object hallucination metrics (e.g., CHAIR_S and CHAIR_I) by a substantial margin, empirically confirming the gap.

Figure 3: A model prone to hallucinate in generation can accurately reject hallucinated objects in discriminative verification, revealing exploitable verification capacity.

The OSCAR Framework

The core methodological contribution is Online Self-CAlibRation (OSCAR), which operationalizes online preference optimization tailored to each model's perceptual limits. OSCAR introduces:

• MCTS-Guided Generation: Rather than local, greedy or beam decoding (which propagates early mistakes and fails to consider token-level risks for downstream hallucination), OSCAR employs Monte Carlo Tree Search (MCTS) to explore sentence-level trajectories with non-local lookahead. MCTS not only samples plausible candidate tokens, but also performs rollouts to assess the impact of early choices on faithfulness and overall response quality.
• Dual-Granularity Reward Mechanism:
  • Process Reward (Node-level): Each candidate sentence is verified by the model itself, using discriminative prompts to score the likelihood that no non-existent objects are referenced.
  • Gated Outcome Reward (Trajectory-level): If the entire response passes a global faithfulness check (no hallucinated objects as per gold labels), it is further scored for logical consistency, fluency, and redundancy using an automatic quality evaluator. Otherwise, its contribution is gated off (zero reward).
  • 

  • Figure 4: OSCAR structure—Left: MCTS explores possible responses. Middle: Preference pairs are extracted at two granularity levels for Direct Preference Optimization. Right: The reward mechanism integrates per-sentence and per-trajectory faithfulness checks.

• Preference Pair Construction and DPO: From each MCTS search tree, OSCAR derives both trajectory-level (best/worst paths) and sibling-level (pathwise comparisons at each step) preference pairs. These are used to finetune the LVLM via Direct Preference Optimization (DPO), incrementally aligning the policy to favor more faithful completions conditioned on its actual perceptual capacity.

Experimental Evaluation

Hallucination Mitigation

Across standard hallucination evaluation benchmarks—Object-HalBench, AMBER, MM-VET, and POPE—OSCAR consistently demonstrates lower hallucination rates (e.g., reducing CHAIR_S for LLaVA-1.5-7B from 49.0 to 27.6 and for LLaVA-1.5-13B from 44.8 to 5.4) and higher faithfulness/fidelity scores than strong baselines, including methods that exploit teacher distillation, beam-search-based self-improvement, or post-hoc hallucination correction. AMBER and Cog metrics indicate improved factuality without sacrificing expressiveness or general multimodal capability.

Iterative Self-Improvement

A defining property of OSCAR is its ability to support continuous self-improvement. Performance metrics improve monotonically over three DPO iterations, validating the hypothesis that online, on-policy preference data coupled with model-aligned reward signals allows for progressively sharper self-calibration. Unlike offline learning, this approach does not saturate or collapse under additional iterations.

Ablation Analysis

Component-wise ablation shows that each OSCAR element—the process reward, trajectory reward, and MCTS search—is individually necessary. Using all three in conjunction yields the lowest hallucination rates, demonstrating their synergistic effect.

Supervision Data Analysis

Fine-tuning with teacher-distilled or self-generated (greedy decoding) data fails to reduce hallucinations, sometimes increasing them due to the aforementioned mismatch. In contrast, using OSCAR-extracted preference pairs for supervised finetuning produces the best results, underscoring the value of online, model-aware supervision.

Qualitative Inspection

OSCAR outputs are qualitatively more grounded (fewer hallucinations), more concise, and show improved fluency/redundancy characteristics compared to their offline/preference-aligned or baseline LLaVA counterparts.

Figure 5: OSCAR's outputs exhibit fewer hallucinations and improved linguistic quality relative to LLaVA-1.5 (hallucinations are highlighted in red, grounded statements in green).

Implications and Theoretical Impact

OSCAR contributes a new principled approach to preference optimization in LVLMs under epistemic uncertainty about perception. By leveraging each model’s own discriminative strengths, hallucination mitigation can be achieved without reliance on potentially misaligned teacher supervision, circumventing the traditional trade-off between faithfulness and expressiveness. MCTS as the data-construction backbone further opens new avenues for non-local decoding and reward propagation in generative modeling, aligned with trends in self-improvement for LLMs and multimodal models.

Practically, these findings signal a transition toward model-internal alignment for hallucination-sensitive applications in safety-critical domains, where external teacher signals may not be available or trustworthy. Theoretically, the distinction between generative and discriminative model capabilities elucidates inductive biases inherent in current autoregressive LVLMs, suggesting further benefit in dynamically managing inference paradigms at training and deployment.

Future developments are likely to privilege scalable online preference optimization, hybrid verification-generation pipelines, and the integration of richer discriminative signals (e.g., cross-modal entailment, object-level grounding) into multimodal alignment.

Conclusion

This work identifies the critical limitations of prevalent hallucination mitigation techniques in LVLMs, reframing the problem as one of alignment between supervisory signal and inherent perceptual capabilities. The OSCAR framework systematically leverages the generative-discriminative gap and incorporates non-local policy improvement via MCTS and dual-granularity rewards, resulting in state-of-the-art faithfulness on both generative and discriminative benchmarks. These insights are expected to inform the design of future robust, self-aligned multimodal models.