FINER-DOCCI: Fine-Grained Hallucination Benchmark

• FINER-DOCCI is a benchmark that assesses multimodal models' ability to reject subtly false, image-grounded queries using paired multiple-choice questions.
• It is constructed from DOCCI's dense human-written captions, repurposed to generate fine-grained negative queries targeting objects, attributes, and relations.
• By employing a rigorous construction pipeline and FINER-Tuning, the benchmark exposes compositional, binding, and relation sensitivity failures in MLLMs.

Searching arXiv for FINER-DOCCI and DOCCI to ground the article in the cited papers. FINER-DOCCI is a multiple-choice hallucination benchmark for multimodal LLMs (MLLMs), introduced as part of the FINER suite—“FIne-grained NEgative queRies”—to test whether models can correctly reject subtle, image-grounded false premises (Xiao et al., 18 Mar 2026). It is built from DOCCI images and captions and is designed around a specific failure mode: a query that is mostly consistent with the image but contains one fine-grained contradiction in an object, attribute, relation, or false-premise “what” question. In this sense, FINER-DOCCI occupies the intersection of dense image description and hallucination evaluation: it inherits its source material from DOCCI, a dataset of long, human-authored descriptions intended to distinguish visually similar images, and repurposes that substrate into paired positive and negative multiple-choice questions for large-scale evaluation of fine-grained visual rejection (Xiao et al., 18 Mar 2026, Onoe et al., 2024).

1. Definition and position within the FINER suite

FINER-DOCCI is one of two benchmarks introduced in FINER, alongside FINER-CompreCap (Xiao et al., 18 Mar 2026). The suite targets hallucination under fine-grained negative queries rather than coarse image-level mistakes. The benchmark is organized into four settings:

• Multi-obj: multiple-object existence queries, with one object corrupted
• Multi-attr: multiple-attribute queries, with one attribute corrupted
• Multi-rel: multiple-relation queries, with one relation corrupted
• Wh: “what” questions with a false premise caused by an incorrect attribute of an existing object

The distinction between the two FINER benchmarks is structural. FINER-CompreCap is described as the smaller, more controlled benchmark built from CompreCap, which already provides human-annotated scene graphs. FINER-DOCCI is the larger-scale benchmark built from DOCCI, which has 5,000 human-written dense captions but no scene graphs, requiring the authors to derive scene-graph-like structure from captions before question generation (Xiao et al., 18 Mar 2026). The paper states that FINER-CompreCap supports more precise evaluation, whereas FINER-DOCCI is intended to test whether dense captions can be used to synthesize scene-graph supervision beyond COCO categories and images, enabling open-set, larger-scale hallucination evaluation.

A central conceptual claim of the benchmark is that prior hallucination benchmarks such as POPE, DASH, and AMBER mostly probe single-object presence or isolated single attributes or relations, often on COCO-based data, and therefore do not adequately test the case in which most of a query is true and only one fine-grained component is false (Xiao et al., 18 Mar 2026). FINER-DOCCI was introduced to isolate that harder regime at scale.

2. Relationship to DOCCI and source data

FINER-DOCCI is built from DOCCI. DOCCI—“Descriptions of Connected and Contrasting Images”—is a dataset of 14,847 annotated images paired with long human-written English descriptions, with the abstract rounding this to “15k” (Onoe et al., 2024). Its descriptions average 135.9 words and were explicitly designed to distinguish each image from related or similar images, with curation centered on challenges such as spatial relations, counting, text rendering, world knowledge, viewpoint, and optical effects (Onoe et al., 2024). These properties make DOCCI an especially suitable substrate for FINER-DOCCI, whose task depends on constructing nearly-correct but subtly false queries from dense image-grounded text.

Within FINER-DOCCI itself, the source subset is stated more narrowly: DOCCI contains 5,000 images, each paired with a detailed human-annotated caption, and these are used for benchmark construction (Xiao et al., 18 Mar 2026). Because DOCCI does not provide scene graphs, FINER-DOCCI begins by extracting a positive scene graph for each image. This scene graph contains objects, attributes, and relations and is explicitly described as non-panoptic and caption-derived.

DOCCI’s original design is relevant because its descriptions are already discriminative among similar images. The DOCCI paper reports a pivot-versus-distractor identification study in which 97.1% of groups had all three annotators identify the correct pivot image, with Fleiss’ $\kappa = 0.98$, indicating that the descriptions are sufficiently specific to support fine-grained discrimination (Onoe et al., 2024). This suggests that DOCCI is unusually well matched to FINER-DOCCI’s objective of constructing false premises that remain visually close to the truth. That implication is interpretive, but it follows directly from the source datasets’ respective design goals.

3. Construction pipeline

The construction process begins with positive scene-graph extraction from DOCCI captions (Xiao et al., 18 Mar 2026). Because there are no ground-truth scene graphs, the authors use a multi-stage pipeline to derive objects, attributes, and relations.

In the first stage, Gemini-2.0-Flash is used to extract objects and attributes from the caption. The model is instructed to copy phrases verbatim from the caption and to avoid inventing new entities or attributes. In the second stage, candidate object pairs are examined for explicitly stated relations: Gemini-2.0-Flash must either return the exact relation phrase from the caption or return nothing if no relation is explicitly mentioned. Relation extraction is then subjected to an additional validation step. Qwen2.5-VL-72B is used as a binary visual classifier to judge whether the relation holds in the image, and Gemini is queried again about whether the relation is explicitly supported by the caption. If both models disagree with the proposed relation, the relation is discarded. Humans also verify a subset of 400 relation samples, and the joint process filters out 1,771 relations. The paper characterizes the pipeline as conservative and as prioritizing precision over recall (Xiao et al., 18 Mar 2026).

After a positive scene graph is obtained, negative counterparts are generated for each positive object, attribute, and relation. For each positive element, four distinct negatives are produced using Gemini-2.0-Flash. These negatives are intended to be semantically plausible but incorrect. Because some generated negatives may accidentally correspond to content genuinely present in the image, the authors use Qwen2.5-VL-72B as a visual discriminator to filter false negatives. Given a positive phrase $x$ and four candidate negatives $\{x_j^{-}\}_{j=1}^{4}$, a five-way candidate set is formed:

$$\mathcal{C}(x) = \{x, x_1^{-}, x_2^{-}, x_3^{-}, x_4^{-}\}.$$

Qwen2.5-VL-72B predicts a distribution

$$p = (p_1, \dots, p_5), \qquad \sum_{i=1}^{5} p_i = 1,$$

and the entropy

$H(p) = - \sum_{i=1}^{5} p_i \log p_i$

is used to identify problematic low-entropy misclassifications, which are then regenerated iteratively (Xiao et al., 18 Mar 2026).

The paper reports separate entropy thresholds and regeneration counts for FINER-DOCCI scene-graph negatives:

Type	$\theta$	Count / regenerated
Obj	0.8	24,528 / 3,242
Attr	0.4	52,911 / 2,827
Rel	0.8	15,342 / 2,143

The resulting scene-graph quality is evaluated using InternVL3.5-8B as a binary classifier. For FINER-DOCCI, the reported positive scene-graph accuracies are 96.1% for objects, 88.3% for attributes, and 85.1% for relations. For the generated negatives, accuracies are 89.5% for objects, 88.3% for attributes, and 82.8% for relations (Xiao et al., 18 Mar 2026). The paper presents these numbers as evidence that the extracted scene graphs and generated negatives are sufficiently reliable for large-scale benchmarking, while also emphasizing that the benchmark is not fully human-curated.

4. Question design, paired evaluation, and benchmark statistics

FINER-DOCCI is formulated as a paired multiple-choice task (Xiao et al., 18 Mar 2026). Each negative question $q^{-}$ is paired with a corresponding positive question $q^{+}$. For Multi-obj, Multi-attr, and Multi-rel, the fixed template is:

• Question: “Can you see $\{X\}$ in this image?”
• Options:
  • A. “Yes, I can see $x$0 in this image.”
  • B–E. “No, but I can see $x$1 in this image.”

In a positive MCQ, the question asks about the true phrase $x$2 and the correct answer is the confirming “Yes” option. In the corresponding negative MCQ, the question asks about a corrupted phrase, the “Yes” option incorrectly confirms the corruption, and the correct answer is a “No, but…” option that restores the true phrase. Wh questions are constructed from a relation triplet $x$3, with one object serving as answer target and the other as contextual anchor; the negative version injects an incorrect attribute into the context object, and the correct answer becomes a correction sentence rather than an object name.

This design is evaluated with paired accuracy, defined as

$x$4

where a pair counts as correct only if both the positive and the corresponding negative MCQ are answered correctly (Xiao et al., 18 Mar 2026). This metric is intended to prevent degenerate strategies such as always choosing affirmative or always choosing negative responses. The main reported random-guess baseline is 4.0%.

The main paper reports the following benchmark totals for FINER-DOCCI:

Setting	MCQs
Multi-obj	10,000
Multi-attr	28,630
Multi-rel	11,542
Wh	20,944

The supplementary further states that these totals arise from paired MCQs and provides pair counts by entity count: Multi-obj has pairs for $x$5 to $x$6; Multi-attr for $x$7 to $x$8; Multi-rel for $x$9 to $\{x_j^{-}\}_{j=1}^{4}$0; and Wh has 10,472 pairs (Xiao et al., 18 Mar 2026). FINER-DOCCI also supports up to 6 objects, up to 5 attributes, and up to 3 relations per question. Post hoc human correction covers 200 MCQs per setting, and the 3-relation subset receives additional human cleaning, resulting in 199 improved paired MCQs (Xiao et al., 18 Mar 2026).

The paper does not provide train/validation/test splits for FINER-DOCCI, because the benchmark is used for evaluation rather than model training (Xiao et al., 18 Mar 2026).

5. Difficulty profile and error modes

FINER-DOCCI is designed to expose a particular class of hallucination: false-positive acceptance of a query that is almost correct (Xiao et al., 18 Mar 2026). The authors’ central hypothesis is that MLLMs fail when a small contradiction is embedded inside otherwise true image content. In a motivating yes/no study, accuracy for InternVL3.5-14B on FINER-DOCCI reportedly falls from about 58% at granularity level 1 to about 15% by levels 6–7 as more true details are combined with one subtle contradiction (Xiao et al., 18 Mar 2026).

The benchmark’s challenge lies in the co-occurrence of true and false premises. A negative query often mentions many things that are genuinely visible; only one element is wrong. The paper states that the benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image (Xiao et al., 18 Mar 2026). In Multi-obj this may take the form of one wrong object among several otherwise correct ones. In Multi-attr, one attribute is corrupted while the rest of the description remains visually grounded. In Multi-rel, the predicate is altered while the entity set remains largely correct. In Wh, a real object is queried under a false premise, and answering the object name without rejecting that premise is counted as an error.

The supplementary examples emphasize concrete failure modes. In Multi-obj, models may fail to identify a fine-grained concept such as “macbook.” In Multi-attr, questions target subtle details such as “the white note on the back driver’s side window” or “the cat with perked-up ears.” In Multi-rel, a model may hallucinate that a dog is behind the fence although it is clearly in front of the fence. In Wh, models may answer the object name instead of correcting an anomalous attribute premise (Xiao et al., 18 Mar 2026).

The paper summarizes the new failure modes exposed by FINER-DOCCI as failure under composition, binding failures, relation sensitivity failures, and false-premise answerability failures (Xiao et al., 18 Mar 2026). A plausible implication is that FINER-DOCCI probes not only visual recognition but also compositional grounding and rejection behavior under semantically entangled input conditions.

6. Empirical results and FINER-Tuning

FINER-DOCCI is reported to be difficult even for strong models (Xiao et al., 18 Mar 2026). The paper evaluates LRV-V2, LLaVA-RLHF, RLHF-V, OPA-DPO, RLAIF-V, LLaVA-1.6, Qwen2.5-VL, InternVL-3.5 at several scales, and Gemini-2.5-Flash. Paired accuracy on FINER-DOCCI is as follows:

Model	Multi-obj	Multi-attr	Multi-rel	Wh
Random Guess	4.0	4.0	4.0	4.0
LRV-V2 13B	6.3	5.4	6.1	5.2
LLaVA-RLHF 13B	7.3	3.0	5.1	5.3
RLHF-V 13B	13.2	7.2	8.1	7.0
OPA-DPO 13B	8.1	5.5	8.3	8.0
RLAIF-V 12B	46.5	31.7	32.4	19.4
LLaVA-1.6 7B	10.1	12.3	8.2	13.3
+ FINER-Tuning	26.4	29.4	24.7	18.5
Qwen2.5-VL 7B	48.7	47.5	36.7	23.4
+ FINER-Tuning	49.8	52.2	43.4	28.0
InternVL-3.5 8B	58.1	54.3	41.8	16.8
+ FINER-Tuning	62.6	60.1	52.7	23.7
InternVL-3.5 14B	58.6	55.9	41.4	15.6
+ FINER-Tuning	65.9	65.0	57.0	23.0
InternVL-3.5 38B	62.3	64.8	54.2	36.6
Gemini-2.5-Flash*	64.4	64.5	56.7	49.6

The paper emphasizes several patterns. First, Wh questions challenge all models, and Multi-rel is also difficult (Xiao et al., 18 Mar 2026). Second, performance worsens as the number of queried entities increases. Third, prior hallucination-mitigation methods transfer poorly to this setting, with several models performing near random in some categories. Human paired accuracy on a subset of 20 MCQs per setting is much higher: 92.5 for Multi-obj, 95.0 for Multi-attr, 90.0 for Multi-rel, and 90.0 for Wh (Xiao et al., 18 Mar 2026).

To address the failure mode, the paper proposes FINER-Tuning, a Direct Preference Optimization (DPO) procedure trained on positive queries that should be accepted and negative fine-grained queries that should be rejected (Xiao et al., 18 Mar 2026). The DPO objective is given as

$\{x_j^{-}\}_{j=1}^{4}$1

$\{x_j^{-}\}_{j=1}^{4}$2

$\{x_j^{-}\}_{j=1}^{4}$3

with $\{x_j^{-}\}_{j=1}^{4}$4 and one epoch of LoRA-based training (Xiao et al., 18 Mar 2026). Applied to FINER-DOCCI, the tuning yields substantial gains. For InternVL-3.5-14B, Multi-rel rises from 41.4 to 57.0, and for LLaVA-1.6-7B, Multi-attr rises from 12.3 to 29.4 (Xiao et al., 18 Mar 2026). The paper also states that FINER-Tuning improves performance on eight existing hallucination benchmarks and raises average performance on six general multimodal benchmarks, suggesting that the gains are not simply due to blanket refusal behavior.

7. Limitations, disambiguation, and significance

The authors explicitly state that FINER-DOCCI is not noise-free (Xiao et al., 18 Mar 2026). It is not fully human-curated; it relies on caption-to-scene-graph extraction, LLM-generated negatives, MLLM-based filtering, and limited human verification. The supplementary states that FINER-DOCCI prioritizes scale over per-sample fidelity. Question construction is rule-based and may be less natural than real user queries. Some ambiguity remains, especially for attributes and relations. Multi-rel questions contain at most three relations, and evaluation is confined to multiple-choice selection rather than open-ended generation. These constraints are integral to the benchmark’s methodology rather than accidental limitations.

A further issue is terminological disambiguation. The acronym “FINER” is also used by unrelated arXiv papers in security and astronomy. One paper presents FINER as a framework for explainable risk detection classifiers in malware, vulnerability, and security analysis (He et al., 2023). Another uses FINER to denote the Far-Infrared Nebular Emission Receiver project for the Large Millimeter Telescope (Kang et al., 2024). Neither paper mentions DOCCI. Thus, in current arXiv usage, “FINER-DOCCI” refers specifically to the benchmark introduced in “FINER: MLLMs Hallucinate under Fine-grained Negative Queries” (Xiao et al., 18 Mar 2026).

Within multimodal evaluation, FINER-DOCCI is best understood as a benchmark for false-premise rejection under dense, fine-grained visual semantics. DOCCI contributes the long human-authored descriptive substrate and contrastive image design (Onoe et al., 2024), while FINER contributes the paired MCQ formalization, the positive/negative scene-graph construction pipeline, and the focus on hallucination when a query is almost—but not entirely—correct (Xiao et al., 18 Mar 2026). This suggests that FINER-DOCCI is not merely another hallucination benchmark, but a benchmark for a narrower and more demanding regime: compositional rejection under subtle contradiction.