ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models

Abstract: Large vision-LLMs (LVLMs) often hallucinate, generating content that the input image does not support. Preventing such content during decoding calls for a candidate-specific measure of how strongly the image supports the token under consideration. The model's visual-token states offer a natural source of this evidence because projecting each state through the output head reveals which vocabulary items that position favors. These position-wise readouts cannot be pooled directly because their probability magnitudes are not comparable across visual positions. Vocabulary ranks provide a scale-invariant basis for pooling, but tokens still differ systematically in their typical rank-based evidence. We propose ReWEIGH, a training-free decoding intervention that aggregates these ranks across visual positions and compares each candidate with a token-specific reference estimated from unlabeled images. At inference, ReWEIGH caches the image evidence during prefill and applies a bounded penalty only to candidates that fall below their reference. On four 7B backbones, ReWEIGH reduces hallucinated object mentions by up to 21.3% while largely preserving or improving descriptive and general performance. With evidence cached, the average added latency is 1.33% per token, and the reductions extend across six architecture families to 32B parameters.

• The paper introduces ReWEIGH, a training-free decoding method that uses dense mean reciprocal rank and token-specific calibration to measure image support for candidate tokens and apply bounded, suppression-only penalties.
• Across four 7B vision-language backbones, ReWEIGH cuts CHAIR-I hallucination rates by 10.3%–21.3% relative to greedy decoding while preserving or improving F1, and it improves AMBER, MMHal-Bench, and MM-Vet results.
• The method adds only 1.33% per-token latency with cached evidence and 0.31% peak-memory overhead, but requires white-box access and remains backbone-, language-, and calibration-specific.

Overview and Motivation

Large vision-LLMs (LVLMs) generate fluent text that is not always supported by the input image, and existing training-free decoding interventions trade off computational cost against grounding specificity. Contrastive methods such as VCD, M3ID, ICD, and CODE require auxiliary forward or decoding passes under altered conditioning; attention- and uncertainty-based methods (OPERA, PAI, MemVR) rely on signals that indicate where the model routes information or how confident it is, but neither signal directly measures whether the image supports a particular candidate token. The paper's authors observe that hallucinated object mentions persist even in the top quartile of output confidence — 13.9% of object-mention onsets in that quartile are hallucinated on LLaVA-1.5-7B, with a positive median evidence deficit at those onsets versus zero for grounded onsets — establishing that confidence alone does not subsume a candidate-specific visual-support measure.

The proposed method, ReWEIGH (Reference-calibrated Weighting via Ordinal Evidence for Image-Grounded Hallucination Mitigation), exploits the logit-lens observation that projecting each visual-token hidden state through the LM head yields a vocabulary readout indicating which words that position favors. The central contribution is a two-stage analysis of why naive use of these readouts fails, followed by a bounded, training-free decoding intervention built on the resulting diagnostics.

Two Obstacles: Scale Dependence and Token Identity

Scale-invariant aggregation. Probability magnitudes from visual-position readouts are not comparable across positions because each position's output distribution has different sharpness. The paper quantifies this nuisance directly: the image-median p90/p10 ratio of the rank-10 probability across visual positions is $13.8\times$ for LLaVA-1.5-7B and $7.8\times$ for Qwen2.5-VL-7B, and affine scale equalization reduces these only to $11.0\times$ and $4.2\times$ — leaving most dispersion intact (67–95% survives equalization across measured depths). Under an order-preserving rescaling stress test, probability-derived suppression strengths shift by a mean absolute 0.045–0.046 while rank-derived strengths remain exactly unchanged. ReWEIGH therefore aggregates evidence through dense mean reciprocal rank (DMRR), $DMRR_I(v)=\frac{1}{|P|}\sum_{j\in P}1/rank_j(v)$, which depends only on within-position vocabulary orderings.

Token-specific interpretation. Ranks make positions comparable but not vocabulary items interchangeable. An image-balanced variance decomposition attributes 66.4% of DMRR variability to differences between tokens rather than images for the same token (ICC analyses on all four backbones confirm substantial token-identity components). A single global reference therefore conflates unusually low evidence for one token with typical evidence for another. Token-specific calibration medians fit on 500 unlabeled MS COCO images reduce out-of-sample reference error by 92% (LLaVA-1.5) and 91% (Qwen2.5-VL) relative to a global reference, and outperform all 1,000 random reassignments of references to tokens ($p = 1/1001$ under the permutation null).

Method

ReWEIGH follows a Measure–Register–Intervene workflow. Measure computes DMRR once during prefill from the readout at a selected layer $\ell$, applying final output normalization and vocabulary projection to each visual-position hidden state. Register runs the unmodified model greedily on unlabeled calibration images, collecting candidate-conditioned observations of $DMRR_I(v)$ for tokens in a top-$p=0.9$ prefix (2–50 candidates). Each registered token stores its median reference $b(v)$ alongside a pooled normalization scale $7.8\times$0. Registration uses an order-statistic range over the token's observations as a conservative stability diagnostic — explicitly not a coverage-guaranteed confidence interval, since observations repeat within trajectories — and abstains when moving the reference between endpoints changes the normalized edit by $7.8\times$1. Intervene applies a suppression-only, bounded penalty:

$7.8\times$2

for candidates in both the current top-$7.8\times$3 set and the registered set; all other logits pass unchanged. The lower clip makes the update one-sided, and the upper clip prevents degeneration. Because image-level evidence is cached at prefill, each decoding step requires only candidate selection and an $7.8\times$4 edit.

Main Results

On four 7B backbones (LLaVA-1.5, Qwen2.5-VL, InstructBLIP, LLaVA-NeXT), ReWEIGH reduces CHAIR$7.8\times$5 by 10.3%–21.3% relative to greedy decoding while preserving or improving F1 — e.g., LLaVA-1.5 improves CHAIR$7.8\times$6 from 15.61 to 12.67 with F1 rising from 80.66 to 80.85 — so gains do not come from suppressing object mentions wholesale. On AMBER generation it lowers CHAIR and hallucination rate for all four backbones and raises the combined score; notably, under the paper's shading rule ReWEIGH is the only compared method that never degrades any baseline metric by more than 5%, with its largest regression a 2.1% relative drop in AMBER coverage. It also improves MMHal-Bench quality scores and MM-Vet accuracy across all backbones. The reductions extend to 11 models spanning six architecture families from 7B to 32B, reducing CHAIR$7.8\times$7 and CHAIR$7.8\times$8 for every model, though each model is calibrated and tuned separately.

Inference cost is modest: 1.33% added per-token latency with cached evidence (2.40% including online DMRR computation at prefill) and 0.31% peak-memory overhead.

Component and Mechanistic Analysis

The ablations isolate each design decision. Token-specific calibration matters: global references raise CHAIR$7.8\times$9 from 44.8 to 58.4 on LLaVA-1.5, and shuffled references (which preserve the marginal value distribution) degrade similarly, isolating the token-to-reference correspondence. Image specificity matters: rotating the cached evidence vector across examples recovers less than half of the CHAIR$11.0\times$0 reduction and drops F1 below base decoding, even though the intervention fires more often (87.8% vs. 80.4%). Bounding matters: unbounded updates lower CHAIR$11.0\times$1 superficially (39.2) but induce repetition collapse — 765 bound-escape events, maximum single-object repetition of 122, F1 collapsing to 69.09 — while bidirectional updates degrade all metrics. Removing abstention has negligible aggregate effect (it touches ~3.5% of candidate positions), so registration functions as a conservative safeguard rather than a source of gains.

Two mechanistic findings deserve emphasis. First, standalone detection quality does not predict mitigation: the probability readout achieves higher emitted-object hallucination AUROC (0.718 vs. 0.679 on LLaVA-1.5) and shows no local disadvantage in raw score-to-edit controls, yet under a matched mean edit budget the calibrated rank intervention lowers CHAIR$11.0\times$2 from 50.0 to 44.8 and nets 131 hallucinated mentions removed versus 56. Second, allocation analysis locates the difference: the rank-derived edit field places more of its switches before structural precursor tokens (+2.36 pp selectivity vs. +0.33 pp for lexical content tokens on LLaVA-1.5). These analyses are descriptive and do not establish a single causal mediator.

Robustness

Ten calibration images already recover half of the CHAIR$11.0\times$3 reduction, and 100 match the deployed configuration's CHAIR$11.0\times$4 with F1 above base decoding; the method also tolerates corpus shift, matching COCO-calibration performance when calibrated on GQA. Registration-rule contrasts show the reference-stability rule performs comparably to fixed minimum-count rules (within 1.2 CHAIR$11.0\times$5 points across backbones) while avoiding a fixed external cutoff. Per-position z-scored readouts do not reproduce the ordinal intervention's behavior, supporting the ordinal choice beyond simple standardization.

Limitations and Open Questions

The method requires white-box access to visual-token hidden states, output normalization, and the vocabulary head, precluding closed-API deployment. Operating points (readout layer, $11.0\times$6, calibration table) are backbone-specific; whether these transfer across models without additional calibration remains untested despite the breadth of the scaling study. Because the corrective signal derives entirely from internal representations, ReWEIGH cannot inject facts absent from the image or backbone, and its benchmarks do not test knowledge-dependent factual errors; combining ordinal evidence with retrieval or external verification is left open. All calibration, development, and evaluation are in English, and cross-language behavior — particularly where tokenization and morphology differ — is untested. Calibration corpora may encode frequency and cultural biases that token-specific references could reproduce.

Conclusion

ReWEIGH demonstrates that vocabulary-rank aggregation over visual-token states, interpreted against reliably estimated token-specific references, provides an effective and cheap interface between LVLM internal representations and decoding. The paper's careful counterfactual and matched-budget controls distinguish which components carry the improvement — ordinal aggregation plus token-calibrated, image-specific, bounded suppression — and candidly separate detection quality from mitigation utility, a distinction likely relevant to future work on representation-guided decoding.