VHBench-10: Hallucination Benchmark for LVLMs

• VHBench-10 is a vision-centric benchmark that systematically diagnoses hallucinations in LVLMs using a detailed hierarchical taxonomy.
• It employs paired factual and hallucinated captions to isolate specific visual perception errors for targeted evaluation of encoder performance.
• Empirical findings reveal that different visual encoders exhibit distinct hallucination profiles, informing design strategies and multi-expert fusion approaches.

VHBench-10 is a vision-centric benchmark for diagnosing hallucinations in Large Vision-LLMs (LVLMs). Introduced in "Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-LLMs," it comprises approximately 10,000 samples organized into ten fine-grained hallucination categories and is designed to evaluate how visual encoders affect hallucination behavior across distinct perceptual competencies (Wang et al., 17 Sep 2025). In this context, object hallucination denotes cases in which an LVLM describes objects or attributes not actually present in an image. The benchmark’s central contribution is a hierarchical taxonomy grounded in detection, segmentation, localization, and classification, allowing hallucination analysis at a substantially finer granularity than coarse object-presence tests.

1. Conceptual Scope

VHBench-10 was motivated by the observation that hallucinations in LVLMs are not a single failure mode. The benchmark treats hallucinations as outcomes of failures in different visual sub-tasks, including detection, segmentation, localization, and classification. This framing is explicitly vision-centric: rather than only asking whether a generated statement is true or false, VHBench-10 is structured to identify which aspect of visual perception is implicated in the error (Wang et al., 17 Sep 2025).

The benchmark is intended for systematic comparison of visual encoders within LVLMs. The underlying hypothesis is that different encoders, trained under different paradigms, acquire distinct inductive biases and therefore exhibit different hallucination profiles. VHBench-10 operationalizes that hypothesis by decomposing hallucination into ten fine-grained categories and measuring error rates category by category.

The paper further situates this design choice within mainstream computer vision by stating that the four core competencies of detection, segmentation, localization, and classification cover 81% of dataset annotations in mainstream computer vision. This makes the taxonomy diagnostically aligned with established perception tasks rather than with a purely linguistic notion of hallucination.

2. Taxonomy of Hallucination Types

VHBench-10 introduces a hierarchical taxonomy with four high-level tasks and ten fine-grained hallucination categories. The categories are intended to isolate a single perceptual failure mode per sample, enabling precise error attribution rather than aggregate hallucination scoring (Wang et al., 17 Sep 2025).

High-level task	Fine-grained category	Diagnostic focus
Detection	Category	Incorrect object presence
Detection	Counting	Incorrect object number
Detection	Occlusion	Over-assertion from partial view
Segmentation	Text	Incorrect text reading
Segmentation	Shape	Incorrect object shape
Localization	Absolute Positioning	Incorrect left/right or similar placement
Localization	Relative Positioning	Incorrect spatial relation
Classification	Color	Incorrect color attribution
Classification	Action	Incorrect state or action
Classification	Relative Interaction	Incorrect interpersonal or social interaction

The benchmark’s examples make the distinctions concrete. “Category” refers to incorrectly identifying the presence of an object not in the image, such as describing “A man surfing” in a beach-and-sea image. “Counting” refers to miscounting objects, such as describing two cats when the image contains three. “Occlusion” captures cases where a model infers a whole object from a partial view, for example describing a complete car when only tires are visible. “Text” addresses reading or transcription failures, “Shape” covers geometric misdescription, “Absolute Positioning” and “Relative Positioning” separate coordinate-space errors from relational spatial errors, and “Color,” “Action,” and “Relative Interaction” cover distinct classes of classification error.

This taxonomy is the benchmark’s defining methodological choice. It makes hallucination categories correspond to recognizable vision competencies, which allows per-capability diagnosis rather than treating hallucination as a monolithic property.

3. Dataset Construction and Sample Structure

VHBench-10 is built from images drawn from LLaVA-ReCap-118K. The construction process begins with 2000 images selected as the base set. For each image-caption pair, the benchmark applies 10 GPT-4o-mini prompts, described as a “unified hallucination prompt,” with each prompt designed to inject a specific error type from the taxonomy (Wang et al., 17 Sep 2025).

Each benchmark sample is a ternary $(I, R, H)$, where $I$ is the image, $R$ is the factual caption, and $H$ is a matched hallucinated caption. The hallucinated caption is constructed so that it contains exactly one targeted error unrelated to other aspects of the caption. Not every image yields all ten hallucination types; only relevant hallucination prompts are generated. The examples given include omitting text hallucination when text is not applicable and omitting counting hallucination when only single objects are present.

The resulting dataset contains approximately 10,000 ternaries distributed across the ten hallucination classes. The benchmark is therefore neither a simple binary hallucination corpus nor a free-form caption set. Its structure is explicitly paired and controlled: for each image, a factual caption and a minimally perturbed hallucinated counterpart are evaluated against one another. This paired design is what makes the benchmark suitable for fine-grained error testing.

4. Evaluation Protocol

The benchmark uses an Error Test Protocol based on language-model perplexity. For each sample $(I, R, H)$, the model is evaluated on two inputs: the image paired with the factual caption, and the same image paired with the hallucinated caption. The prompt template is:

'<image>\nDescribe the image: <caption>'

For both inputs, perplexity is computed over the output. A hallucination error is recorded when the model assigns lower perplexity, and thus higher probability, to the hallucinated caption than to the factual caption (Wang et al., 17 Sep 2025). The criterion is:

$$\text{If } PPL(I+R) > PPL(I+H) \implies \text{Model error}$$

Error rates are then reported per hallucination category and per visual encoder. The paper states that the visual encoder in the LVLM can be varied systematically, including CLIP, DINOv2, ConvNext, EVA-02, SAM, Vary, or hybrid configurations. This is central to the benchmark’s intended use: VHBench-10 is not only a dataset for aggregate robustness scoring, but also an instrument for controlled studies of encoder-dependent failure modes.

Methodologically, the use of paired factual and hallucinated captions with a single isolated error makes the evaluation more diagnostic than open-ended caption comparison. It measures whether the model prefers the hallucinated interpretation over the factual one under a controlled perturbation regime.

5. Empirical Findings and Analytical Uses

Evaluations on VHBench-10 indicate that visual encoders exhibit distinct hallucination characteristics. The paper reports that CLIP shows lower error rates on global perception, specifically Category/Detection, while struggling with fine-grained attributes. DINOv2 is reported to perform better on color, action, and attribute recognition, but not as strongly on object presence. Vary outperforms other encoders on text-related hallucinations, attributed to pre-training for text recognition. SAM excels on segmentation errors such as object shapes and boundaries. A hybrid model, VisionWeaver, is reported to provide the lowest error rates across all ten categories (Wang et al., 17 Sep 2025).

These results support several forms of analysis enabled by the benchmark. First, VHBench-10 exposes encoder-task matching: hallucination failures can be related to encoder pre-training bias rather than only to downstream language behavior. Second, it enables multi-expert fusion analysis. The paper states that naive feature fusion, including addition or concatenation, does not guarantee improvement across all tasks and can worsen some hallucination rates. Third, the benchmark is used to motivate and validate VisionWeaver, described as a Context-Aware Routing Network that employs global visual features to generate routing signals and dynamically aggregate features from multiple specialized experts.

The benchmark also supports per-class, per-encoder heatmaps and radar-plot style profiling of hallucination vulnerabilities. This diagnostic use is important because it shifts hallucination analysis from broad claims about a model being more or less hallucinatory to a structured account of which visual competencies are failing. A plausible implication is that benchmark-driven architecture design can be guided by the mismatch between encoder specialization and hallucination category.

6. Relation to Earlier Benchmarks and Naming Ambiguity

VHBench-10 is presented as a response to the limitations of earlier hallucination benchmarks, which the paper characterizes as coarse-grained, often centered on object existence or broad attribute and relation errors, and frequently implemented through binary queries such as whether an object is present. The paper contrasts VHBench-10 with prior benchmarks including POPE, HallusionBench, and AMBER, arguing that these do not isolate which visual competency—such as counting, color, text perception, spatial reasoning, or action understanding—is failing (Wang et al., 17 Sep 2025).

The benchmark’s distinctiveness lies in three linked properties: a taxonomy grounded in core vision sub-tasks, single-error isolation in each sample, and systematic per-class evaluation across alternative visual encoders. These design choices make it possible to diagnose failure modes that are obscured by coarse aggregate hallucination scores.

The designation “VHBench-10” can also be a source of confusion because of its superficial resemblance to “VHDLBench” terminology in hardware-design benchmarking. In "VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation," however, “VHBench-10” or any ten-problem subset is not explicitly named or discussed; the paper states that all key results refer to the full VHDLBench benchmark rather than to a benchmark called VHBench-10 (Shen et al., 11 Jun 2026). In current arXiv usage, VHBench-10 refers to the LVLM hallucination benchmark introduced for fine-grained vision-centric analysis, not to a VHDL-generation subset.