POPE: Benchmark for Object-Presence Probing

Updated 17 March 2026

POPE is a benchmark evaluation protocol that assesses LVLM hallucination by querying binary object presence, reducing ambiguities in object detection.
It employs random, popular, and adversarial negative sampling strategies to systematically evaluate object-specific hallucinations in image captioning and VQA.
POPE’s metrics, grounded in binary classification theory, enhance model reliability and standardize comparisons of visual grounding across LVLMs.

POPE (Polling-Based Object Presence Evaluation) is a benchmark and evaluation protocol designed to measure object hallucination in large vision-LLMs (LVLMs) and related multimodal systems. It addresses the limitations of caption-parsing metrics by reframing hallucination evaluation as a binary object presence question-answering task across systematically sampled object candidates per image. POPE has become the principal reference point for quantifying object-specific hallucinations in image captioning and visual question answering, offering a prompt-stable, closed-world, and easily extensible evaluation framework.

1. Motivation and Foundational Principle

LVLMs frequently generate mention of objects absent from the input image, impairing trustworthiness and reliability in downstream vision-language tasks. This "object hallucination" is especially prevalent when models are exposed to strong language priors or receive ambiguous visual inputs. Prior metrics, including CHAIR (Caption Hallucination Assessment with Image Relevance), depend upon parsing free-form captions and are highly sensitive to prompt wording, generation style, and exact-match heuristics, resulting in brittle or misleading measurements. POPE formalizes the evaluation of object hallucination as a binary classification task: for each image $x$ and a candidate object $o$ , the model is queried with "Is there a $<$ object $>$ in the image?" and must answer "Yes" or "No" according to the ground-truth object presence, sidestepping the ambiguities of text generation (Li et al., 2023).

2. POPE Framework and Technical Protocol

For each test image $x$ , the ground-truth object set $GT(x)$ is extracted (typically from human annotations in datasets like MSCOCO, or alternatively from off-the-shelf segmentation tools such as SEEM). From a predefined object vocabulary $V$ , a negative set $Neg(x)$ is constructed using one of three sampling strategies:

Random: uniform sampling of objects not in $GT(x)$ .
Popular: top- $k$ most frequent objects in the corpus, excluding $GT(x)$ .
Adversarial: top- $k$ objects most frequently co-occurring with any $o \in GT(x)$ .

Each evaluation instance for image $x$ is a set of $l$ binary questions, each formatted as $q(o) =$ "Is there a $<$ object $>$ in the image?", with target label $a(o) =$ Yes if $o \in GT(x)$ , No otherwise. The model predictions $\hat{y}(o)$ are collected and compared to $a(o)$ .

Pseudocode specifying POPE's canonical workflow is as follows (Li et al., 2023):

Input: images {x_j}, object vocabulary V, polling size l
For each image x:
  GT = extract_true_objects(x)
  For each strategy S ∈ {Random, Popular, Adversarial}:
    Neg = sample_negatives(GT, V, S, l/2)
    O = GT ∪ Neg
    For each object o in O:
      q = format("Is there a %s in the image?", o)
      pred = LVLM.answer(x, q)
      label = “Yes” if o∈GT else “No”
      record (pred, label)
Compute metrics over all (pred, label) pairs

3. Metrics and Analytical Mechanisms

POPE's main metrics are grounded in standard binary classification theory. Let $TP$ , $FP$ , $TN$ , $FN$ denote the usual counts over all pooled questions:

Accuracy:

$\mathrm{Accuracy} = \frac{TP + TN}{TP + FP + TN + FN}$

Precision and Recall ("Yes"-class):

$\mathrm{Precision} = \frac{TP}{TP + FP}, \quad \mathrm{Recall} = \frac{TP}{TP + FN}$

F1 Score:

$F_1 = \frac{2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$

POPE also supports prompt robustness analysis (F1 variance under reformatted templates), hit-ratio analyses (e.g., $HR_a@k$ for quantifying the rate of hallucination involving the most frequent or adversarially sampled objects), and yes-ratio statistics to reflect class-balance or response bias.

Unlike CHAIR, POPE provides closed-form outputs, robustly sidesteps caption parsing, and enables direct evaluation on unannotated datasets via segmentation-extracted $GT(x)$ (Li et al., 2023, Park et al., 8 Dec 2025). Leading models such as LLaVA, InstructBLIP, and MiniGPT-4 have been benchmarked under this protocol.

4. Empirical Analyses and Model Behavior

A standard POPE benchmark comprises 500 validation images from MSCOCO, each evaluated on six binary object presence questions per image (yielding $3,000$ pairs per negative-sampling split). Representative F1 scores (Random/Popular/Adversarial) for open models include:

Model	Random	Popular	Adversarial
mPLUG-Owl	68.06	66.79	66.82
LLaVA	68.65	67.72	66.98
MiniGPT-4	78.86	72.21	71.37
InstructBLIP	89.29	83.45	78.45

Some models exhibit "Yes" response rates approaching 97-100% for randomly sampled negatives, indicating severe over-confidence (Li et al., 2023). POPE outperforms CHAIR in prompt stability: altering the question format changes POPE's F1 by $\approx 0.78$ (std), compared to $3.22$ for CHAIR $_I$ . Substitution of segmentation-based $GT(x)$ (such as SEEM) for human labels preserves overall model ranking and statistical trends across both MSCOCO and VQA datasets.

Integration with downstream tasks such as VQA reveals that higher POPE F1 is predictive of VQA task accuracy, albeit not monotonically. The items most frequently hallucinated correspond to the most prevalent or co-occurring entities in the training instructions (for instance, "person," "cell phone," or "dining table").

5. Extensions, Annotation Critique, and RePOPE

POPE's reliance on MSCOCO annotations introduces vulnerability to label errors propagating from the underlying dataset. Re-annotation studies, as in RePOPE (Neuhaus et al., 22 Apr 2025), reveal that 9.3% of positive POPE pairs are actually false (object absent), with an additional 13.8% deemed ambiguous and thus removed. Negative pairs have a false negative rate of 1.7% and 4.3% ambiguous. Correction procedures involve dual expert labeling with consensus adjudication; ambiguous examples are excluded, and misannotations are flipped.

Variant	RePOPE–POPE=Yes	RePOPE–POPE=No
Random	76.9%/9.3%/13.8%	0.3%/98.4%/1.3%
Popular	76.9%/9.3%/13.8%	2.6%/93.0%/4.4%
Adversarial	76.9%/9.3%/13.8%	2.2%/90.5%/7.3%

Model rankings shift visibly: for example, while InternVL2.5-26B ranks highest on original POPE (F1=90.1%), after re-annotation, Ovis2-4B and Ovis2-8B both ascend to the top (F1=94.2%, 94.1%), and InternVL2.5-26B drops to 10th (Neuhaus et al., 22 Apr 2025). False positives nearly double on the Random split, and F1 dynamics shift due to the removal of ambiguous or incorrect ground truth. This underscores the sensitivity of model comparisons to annotation fidelity. Recommendations include routine re-annotation, explicit reporting of balanced-accuracy, and the adoption of stratified hard-negative sampling to improve discriminative power and benchmark robustness.

6. POPE in Methodological Advances: The Case of SAVE

Recent work, such as the SAVE framework (Park et al., 8 Dec 2025), leverages POPE not only as a diagnostic but also as an integral probe in representation learning. SAVE constructs a large, balanced binary object-presence probing set–with 10,000 queries drawn from both genuine and GPT-3.5-suggested hallucinated objects–to identify SAE (Sparse Autoencoder) latent features most predictive of grounded, visually accurate answers. Feature attribution is based on separation scores between correct ( $\mathcal{X}_{\mathrm{correct}}$ ) and hallucinated ( $\mathcal{X}_{\mathrm{hallu}}$ ) activations:

$s_j = f^\mathrm{correct}_j - f^\mathrm{hallu}_j, \quad j^* = \arg\max_j s_j$

Steering the model along decoder direction $W_\mathrm{dec}[j^*,:]$ at inference systematically reduces POPE-measured hallucination. For the LLaVA-1.6 architecture, reported post-steering F1 scores are:

Random: F1 = 92.71% (vs. 92.16% baseline)
Popular: F1 = 89.74% (vs. 89.66%)
Adversarial: F1 = 86.52% (vs. 86.32%)

Further analysis shows that the intervention does not simply bias the model's yes/no tendencies; it increases cross-attention to image tokens by 37% (from 0.19 to 0.26), correlating with improved visual grounding and reduced language-prior drift in the output tokens (Park et al., 8 Dec 2025).

7. Recommendations and Future Directions

Best practices articulated by both the original POPE and subsequent RePOPE include:

Employing both random and adversarial negative sampling to assess model robustness to priors and co-occurrence effects.
Utilizing multiple question templates to verify prompt insensitivity.
Relying on robust automated segmentation for GT extraction when human labels are unavailable.
Systematically re-annotating test labels with mechanisms for ambiguity handling and evaluation exclusion.
Reporting class-specific TPR/TNR and balanced-accuracy to combat dataset saturation and enable fairer model comparison.

A plausible implication is that continued refinement of object-presence probing benchmarks, attention to annotation quality, and integration with feature attribution pipelines will not only yield more reliable hallucination measurements but also stimulate methods that tie multimodal generation more tightly to visual evidence, directly advancing the reliability of LVLMs (Li et al., 2023, Neuhaus et al., 22 Apr 2025, Park et al., 8 Dec 2025).