FINER-CompreCap: Fine-Grained Hallucination Benchmark

• The paper introduces FINER-CompreCap, a benchmark that detects false-positive hallucinations in multimodal models using fine-grained negative queries.
• The methodology relies on human-annotated scene graphs and entropy-filtered, LLM-generated negatives to construct controlled multiple-choice questions.
• Empirical results reveal significant performance drops on relational and 'What' queries, underscoring challenges in robust visual-semantic verification.

Searching arXiv for the referenced FINER benchmark paper to ground the article in the cited source. arXiv search query: (Xiao et al., 18 Mar 2026) FINER MLLMs Hallucinate under Fine-grained Negative Queries FINER-CompreCap is a fine-grained negative-query benchmark for multimodal LLMs (MLLMs) introduced in “FINER: MLLMs Hallucinate under Fine-grained Negative Queries” (Xiao et al., 18 Mar 2026). It is built on the CompreCap dataset of COCO images and is designed to probe false-positive hallucinations under subtle semantic mismatches rather than coarse image-level errors. The benchmark uses human-annotated scene graphs and carefully filtered LLM-generated negative counterparts to construct paired multiple-choice questions across multi-object, multi-attribute, multi-relation, and “What” settings, enabling evaluation of whether a model can simultaneously recognize genuinely present content and reject plausibly confusable but absent alternatives (Xiao et al., 18 Mar 2026).

1. Dataset basis and benchmark motivation

FINER-CompreCap uses 560 COCO images from CompreCap, whose authors supply human-annotated scene graphs listing objects, attributes, and relations (Xiao et al., 18 Mar 2026). Within the FINER framework, the benchmark is intended to study hallucinations that arise when fine-grained mismatches co-occur with genuinely present elements in the same image. This focus distinguishes it from evaluations centered on coarse image-related questions, because the error mode of interest is not merely failure to recognize an object class but failure to reject an almost-correct description that differs in one object, attribute, or relation.

The construction relies on the scene-graph decomposition of each image into objects (OBJ), attributes (ATTR), and relations (REL). FINER-CompreCap then turns these scene-graph elements into queryable textual units. Attribute sentences are rewritten into “with {attr}” phrases, and relations are parsed into subject–predicate–object triplets (Xiao et al., 18 Mar 2026). This establishes a controlled symbolic substrate from which both positive and negative questions can be generated.

A plausible implication is that the benchmark operationalizes hallucination as a failure of discriminative grounding under minimal semantic perturbation. In this formulation, the key challenge is not open-ended captioning fluency but robust visual-semantic verification at scene-graph granularity.

2. Construction pipeline and negative generation

For every scene-graph element—object, attribute, or relation—the benchmark authors prompt Qwen3-14B to propose four semantically plausible but incorrect alternatives (Xiao et al., 18 Mar 2026). These alternatives serve as negative counterparts. Because such negatives can inadvertently correspond to content that is actually present in the image, the benchmark applies an explicit filtering stage to reduce “false negatives.”

The filtering procedure forms five-choice multiple-choice questions with Qwen2.5-VL-72B as a visual discriminator and computes the output entropy

$H(p)=-\sum_{i=1}^{5} p_i \log p_i$

which is Eq. 2 in the supplement (Xiao et al., 18 Mar 2026). Misclassifications with entropy below a human-verified threshold $\theta$ are flagged and re-generated until they pass the filter. This entropy-based visual consistency check is used to ensure that negatives are not actually present in the image.

Using template-based composition, each image yields positive and negative multiple-choice questions in four settings: Multi-obj, Multi-attr, Multi-rel, and Wh (“What” questions) (Xiao et al., 18 Mar 2026). Positive MCQs ask about a combination of true scene-graph elements. Negative MCQs replace exactly one element with its negative variant, while distractors are drawn from the other negatives. The result is a paired-query protocol in which the semantic delta between positive and negative instances is tightly controlled.

This design suggests that FINER-CompreCap targets a specific failure regime: models may succeed on globally plausible interpretations while failing under single-element corruption within otherwise grounded descriptions.

3. Query schema and task format

FINER-CompreCap defines four query types (Xiao et al., 18 Mar 2026). Multi-obj questions take the form “Can you see X and Y and Z …?”, where the positive query mentions $k$ true objects and the negative query replaces one of the $k$ with a negative object. Multi-attr follows the same pattern but with attribute phrases such as “with a red stripe.” Multi-rel likewise uses relation phrases such as “sitting on the table.” Wh questions select one object in a predicate context and ask “What is …?” or “What is the dog standing under?”, with positive and negative forms differing by whether the conditioning attribute is correct or incorrect.

For the yes/no style object, attribute, and relation MCQs, the template is (Xiao et al., 18 Mar 2026):

“Can you see {X} in this image? A. Yes, I can see {Y}. B. No, but I can see {Z₁}. C. No, but I can see {Z₂}. D. No, but I can see {Z₃}. E. No, but I can see {Z₄}.”

For $q^+$, ${X}={Y}$ is the true phrase and ${Z_j}$ are corrupted phrases. For $q^-$, ${X}={Y}$ is the corrupted phrase, one ${Z_j}$ is the true phrase, and the others are corrupted (Xiao et al., 18 Mar 2026). In effect, the correct answer flips from affirmation to correction when one scene-graph component is replaced.

The example instances illustrate the intended granularity. A positive Multi-obj query may ask “Can you see cat and door frame?”, while the negative counterpart asks “Can you see cat and pillar?”, where “door frame” is the true phrase and “pillar” is the corrupted one (Xiao et al., 18 Mar 2026). A positive Multi-rel query may ask “Can you see cat lying on sofa and cat next to coffee table?”, while the negative replaces one relation with “cat under sofa” (Xiao et al., 18 Mar 2026). In Wh form, a positive question may ask “What is the dog with brown fur standing under?” with “table” as the correct answer, whereas the negative changes the conditioning phrase to “yellow fur” and the correct option becomes a correction sentence indicating that the dog is not with yellow fur but with brown fur (Xiao et al., 18 Mar 2026).

The schema is notable because the negative example is not globally nonsensical. Instead, it preserves most of the grounded context and corrupts exactly one constituent. That makes the task a controlled probe of fine-grained semantic verification rather than generic visual question answering.

4. Scale, statistics, and evaluation protocol

FINER-CompreCap covers 560 images and yields 6,300 Multi-obj MCQs, 3,338 Multi-attr MCQs, 4,280 Multi-rel MCQs, and 3,166 Wh MCQs (Xiao et al., 18 Mar 2026). Before filtering negatives, the scene-graph counts are 3,505 objects, 4,509 attributes, and 3,494 relations. During negative generation, 320 object negatives, 414 attribute negatives, and 173 relation negatives were regenerated (Xiao et al., 18 Mar 2026). These counts reflect the extent to which the filtering process detected candidate negatives that were insufficiently reliable.

The primary metric is paired accuracy. For $\theta$0 image-question pairs $\theta$1 and a model $\theta$2, with indicator $\theta$3 if the model’s answer is correct and $\theta$4 otherwise, paired accuracy is defined as

$\theta$5

Under this metric, a model only receives credit on a pair if it answers both the positive and the corresponding negative correctly (Xiao et al., 18 Mar 2026). This sharply penalizes models that can recognize present content but fail to reject nearly matching absent content.

The entropy used for negative filtering is the same quantity employed in construction:

$\theta$6

In the benchmark design, this entropy is not an end metric for model comparison but a quality-control device for the generation pipeline (Xiao et al., 18 Mar 2026).

A plausible implication is that paired accuracy measures robustness to local semantic negation more strictly than standard single-question accuracy. Success requires both positive recognition and negative discrimination, which aligns closely with the benchmark’s stated concern with false-positive hallucinations.

5. Baseline results and empirical difficulty profile

The benchmark reports paired accuracy percentages for several MLLMs across the four settings (Xiao et al., 18 Mar 2026).

Model	Multi-obj	Multi-attr	Multi-rel	Wh
Random guess	4.0	4.0	4.0	4.0
LLaVA-1.6 (7B)	25.3	13.0	7.6	15.3
Qwen2.5-VL (7B)	69.2	62.5	30.1	28.9
InternVL-3.5 (14B)	74.5	68.1	47.0	21.8
Gemini-2.5-Flash	75.7	77.3	77.8	58.2

The reported pattern is that performance systematically degrades as granularity and relational complexity increase (Xiao et al., 18 Mar 2026). Multi-rel and Wh questions are identified as the hardest settings, and even strong 14B models drop below 50% on Wh (Xiao et al., 18 Mar 2026). The analysis further states that, in terms of failure rate, the ordering is Multi-rel $\theta$7 Wh $\theta$8 Multi-attr $\theta$9 Multi-obj (Xiao et al., 18 Mar 2026).

The paper also reports a granularity effect: as the number of objects, attributes, or relations $k$0 grows, paired accuracy falls sharply, especially for attributes and relations (Xiao et al., 18 Mar 2026). In addition, positional bias is observed: when the single negative element is inserted at different positions in a three-element list, model accuracy fluctuates by up to 15% (Xiao et al., 18 Mar 2026). This indicates that some of the observed failures are not solely due to visual grounding limits but are entangled with prompt-structure sensitivity.

Human performance on a 20-question subset per setting averages above 90% paired accuracy (Xiao et al., 18 Mar 2026). The benchmark therefore highlights a gap between human and model robustness under subtle negations. This suggests that the error mode being measured is not inherently ambiguous to human annotators, but remains unresolved for current MLLMs.

6. FINER-Tuning and benchmark significance

The same work introduces FINER-Tuning, which leverages Direct Preference Optimization on FINER-inspired data (Xiao et al., 18 Mar 2026). Finetuning frontier MLLMs on 160 K preference tuples yields up to +24.2% gain on Multi-rel for InternVL-14B, and +10.7% on Wh (Xiao et al., 18 Mar 2026). After tuning, InternVL-14B reaches 80.0, 78.9, 71.2, and 30.1 percent on Multi-obj, Multi-attr, Multi-rel, and Wh respectively (Xiao et al., 18 Mar 2026).

The paper further states that finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2% gains on hallucinations from the FINER benchmarks, while simultaneously improving performance on eight existing hallucination suites and enhancing general multimodal capabilities across six benchmarks (Xiao et al., 18 Mar 2026). Within the scope of FINER-CompreCap, this indicates that the benchmark is not only diagnostic but also usable as supervision for targeted hallucination mitigation.

FINER-CompreCap is therefore significant in two linked senses. First, it supplies a controlled evaluation regime for fine-grained negative queries grounded in scene-graph structure. Second, it provides training signals that appear to reduce the specific hallucination behaviors it exposes. A plausible implication is that the benchmark occupies an intermediate position between structured visual reasoning tasks and preference-based robustness tuning for MLLMs.

7. Interpretation, scope, and common misconceptions

A common misconception is to treat FINER-CompreCap as a generic visual question answering benchmark. Its actual target is narrower and more specific: it systematically probes false-positive hallucinations under fine-grained negative queries, using carefully filtered LLM-generated negatives and MCQ templates (Xiao et al., 18 Mar 2026). The task is not primarily open-ended generation quality, nor is it a standard captioning benchmark, even though it is built on CompreCap data.

Another possible misunderstanding is that failure on the benchmark merely reflects rare-object recognition difficulty. The reported results do not support such a simple account. Performance drops with granularity and relational complexity, positional bias can shift accuracy by up to 15%, and Multi-rel and Wh queries remain difficult even for strong models (Xiao et al., 18 Mar 2026). This indicates that the benchmark stresses compositional and contrastive grounding, not only lexical recognition.

The benchmark also should not be read as showing that MLLMs fail uniformly across all settings. Multi-obj accuracy is substantially higher than Multi-rel and Wh for most reported models, and Gemini-2.5-Flash reaches 77.8 on Multi-rel and 58.2 on Wh, outperforming the other listed baselines on the latter settings (Xiao et al., 18 Mar 2026). The empirical picture is therefore heterogeneous: MLLMs can exhibit strong performance on some fine-grained queries while remaining brittle under subtle negations and relational perturbations.

Overall, FINER-CompreCap defines a high-resolution evaluation regime for hallucination analysis in MLLMs. By grounding both positive and negative queries in scene graphs, enforcing negative filtering through entropy-based visual consistency checks, and scoring with paired accuracy, it turns fine-grained semantic mismatch into a measurable robustness problem (Xiao et al., 18 Mar 2026). The benchmark’s main contribution lies in showing that hallucination persists even when most of a query is correct and visually grounded, and in making that failure mode quantitatively accessible for both evaluation and finetuning.