HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models (2511.12693v1)

Abstract: Vision-LLMs (VLMs) enable open-ended visual question answering but remain prone to hallucinations. We present HEDGE, a unified framework for hallucination detection that combines controlled visual perturbations, semantic clustering, and robust uncertainty metrics. HEDGE integrates sampling, distortion synthesis, clustering (entailment- and embedding-based), and metric computation into a reproducible pipeline applicable across multimodal architectures. Evaluations on VQA-RAD and KvasirVQA-x1 with three representative VLMs (LLaVA-Med, Med-Gemma, Qwen2.5-VL) reveal clear architecture- and prompt-dependent trends. Hallucination detectability is highest for unified-fusion models with dense visual tokenization (Qwen2.5-VL) and lowest for architectures with restricted tokenization (Med-Gemma). Embedding-based clustering often yields stronger separation when applied directly to the generated answers, whereas NLI-based clustering remains advantageous for LLaVA-Med and for longer, sentence-level responses. Across configurations, the VASE metric consistently provides the most robust hallucination signal, especially when paired with embedding clustering and a moderate sampling budget (n ~ 10-15). Prompt design also matters: concise, label-style outputs offer clearer semantic structure than syntactically constrained one-sentence responses. By framing hallucination detection as a geometric robustness problem shaped jointly by sampling scale, prompt structure, model architecture, and clustering strategy, HEDGE provides a principled, compute-aware foundation for evaluating multimodal reliability. The hedge-bench PyPI library enables reproducible and extensible benchmarking, with full code and experimental resources available at https://github.com/Simula/HEDGE .

• The paper introduces HEDGE—a domain-agnostic pipeline for hallucination detection in VQA by measuring dense geometric entropy in response distributions.
• It employs novel sampling and clustering methods, integrating NLI and embedding-based techniques to assess uncertainty with metrics like SE, RadFlag, and VASE.
• Experimental results on medical benchmarks highlight prompt dependency, computational trade-offs, and guide scalable evaluation for vision-language models.

HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-LLMs

Framework Overview

The paper introduces HEDGE, a domain-agnostic pipeline for hallucination detection in Vision-LLMs (VLMs) focused on visual question answering (VQA). HEDGE operationalizes hallucination estimation by measuring the stability of semantic clusters in the response distribution when subjected to controlled visual perturbations. The pipeline includes answer generation using both clean and noise-augmented images, semantic clustering via either Natural Language Inference (NLI) or embedding-based methods, and computation of uncertainty metrics such as Semantic Entropy (SE), RadFlag, and Vision-Amplified Semantic Entropy (VASE).

Figure 1: Overview of the HEDGE framework’s pipeline for VLM hallucination detection using answer sampling, clustering, and entropy metrics.

Methodological Innovations

HEDGE reframes hallucination detection from the conventional log-probability and token-level uncertainty metrics into dense geometric entropy measurement in multimodal representation space. By introducing sampling of answers under varying temperature settings and controlled image distortions, the framework assesses how locally stable the model’s response manifolds are regarding input variability. This geometric robustness perspective leverages both logical and geometric perspectives for semantic grouping: NLI-based clustering for entailment-driven similarity, and embedding-based clustering utilizing sentence transformer encodings.

To operationalize these principles, the framework establishes a standardized pipeline:

• Sampling: Generate baseline and diverse answers using low/high-temperature sampling from clean and perturbed images.
• Clustering: Apply NLI-based entailment clustering or embedding-based geometric grouping to sampled responses.
• Metrics: Compute SE, RadFlag, and VASE over semantic clusters, quantifying uncertainty and robustness to input noise.
• Prompting Regimes: Evaluate hallucination tendencies across varying linguistic verbosity: minimal-label, clinical-phrase, default, and one-sentence prompts.

Experimental Evaluation

Experiments are performed on two medical VQA benchmarks (VQA-RAD and KvasirVQA-x1) using three representative VLM architectures (LLaVA-Med, Med-Gemma, Qwen2.5-VL). The hallucination detection is quantitatively benchmarked by ROC-AUC scores against binary hallucination labels generated by adjudicator LLMs.

Key empirical results:

• Model Ranking: Qwen2.5-VL consistently yields superior separation of hallucinated responses, followed by LLaVA-Med and Med-Gemma.
• Prompt Dependency: Default and clinical-phrase prompts produce the most reliable uncertainty signals; minimal-label prompts favor strong models but degrade for weaker ones; one-sentence prompts consistently underperform.
• Clustering Regime: NLI clustering is robust for longer, verbose answers, especially for LLaVA-Med and Qwen2.5-VL. Embedding clustering excels for concise, label-style responses, particularly in moderate sampling regimes.
• Sampling Scale: ROC-AUC gains saturate beyond $n \approx 10$–15 (number of sampled answers per condition). VASE remains robust even at low sampling budgets; SE requires $n \gtrsim 10$ to approach optimal detection.
• Metric Hierarchy: VASE dominates in robustness-oriented settings, SE is competitive with label-style prompts, and RadFlag is effective in small-sample regimes.

  Figure 2: Effect of sampling scale and visual perturbations on ROC-AUC for Qwen2.5-VL across multiple prompt configurations and clustering methods.

Scalability Analysis

The computational costs for NLI and embedding-based clustering are contrasted. NLI clustering incurs $\mathcal{O}(n^2 T_{\text{NLI}})$ complexity and significant GPU memory consumption, scaling poorly with number of samples and perturbations. Embedding-based clustering achieves scalability with a stable, low memory footprint and only quadratic dot-product costs for clustering.

Figure 3: Processing time and GPU memory usage for NLI and embedding clustering as sampling scale increases; embedding clustering remains computationally efficient.

Implications

Practical Implications

HEDGE delivers a standardized protocol for reproducible hallucination benchmarking across multimodal architectures and datasets. Metrics derived are actionable for model selection, prompt design, and deployment planning in safety-critical VQA applications such as clinical decision support.

• Efficient Detection: Embedding clustering paired with VASE metric is recommended for high-throughput, scalable evaluation.
• Prompting Guidance: Concise but expressive prompts optimize detection performance; minimal answers require question concatenation for disambiguation.
• Sampling Budget Optimization: Sampling scale $n \approx 10$–15 suffices; increasing beyond this yields diminishing returns.

Theoretical Implications

Defining hallucination as geometric instability in multimodal manifolds provides a new conceptual anchor for reliability assessment in VLMs. It suggests that architectures with dense tokenization and unified fusion (e.g., Qwen2.5-VL) are intrinsically more robust against hallucination due to smoother response manifolds. The clustering methodology facilitates research into disentangling semantic, architectural, and data-induced uncertainty sources.

Future Directions

HEDGE’s modular approach invites extensions into longer clinical reports, non-medical VQA domains, and multilingual settings. Potential enhancements include standardized preprocessing across models, hybrid semantic clustering (fusing NLI and embedding graphs), and improvements in ground-truth adjudication reliability.

Conclusion

HEDGE provides a principled, compute-aware foundation for hallucination detection in VLMs. Through geometric entropy quantification in response distributions, it exposes how model architectures, prompting conventions, sampling scale, and clustering methods collectively shape multimodal reliability. Its robust pipeline and accompanying hedge-bench toolkit enable reproducible, extensible benchmarking, positioning HEDGE as an infrastructure for structured robustness analysis rather than static metric-based evaluation. Future multimodal systems can leverage these insights for systematically improving reliability, interpretability, and safety in open-ended VQA tasks.