POPE: Polling-based Object Probing Evaluation

• Polling-based Object Probing Evaluation (POPE) is a binary framework that measures object hallucination in vision-language models by asking yes/no questions.
• It employs negative sampling strategies—random, popular, and adversarial—to rigorously assess model performance and error rates.
• POPE enhances evaluation stability and enables fine-grained analysis, with extensions for attribute and 3D scene assessments.

Polling-based Object Probing Evaluation (POPE) is a closed-ended, binary evaluation framework specifically designed to measure object-level hallucination in large vision-LLMs (LVLMs) and vision-LLMs (VLMs). POPE addresses limitations of prior generative caption-based benchmarks by directly probing for object presence using “yes/no” questions, providing increased robustness, stability, and fine-grained error analysis. The protocol and its variants have influenced the development of more sophisticated, granular, and domain-adapted hallucination assessment frameworks.

1. Formal Definition and Evaluation Protocol

POPE defines object hallucination as a model’s assertion that an object exists in an image when it is not present, measured via a binary classification setup. For each image $I$ and object label $o$ from a pre-defined vocabulary, POPE constructs a query:

$$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$

Given an image $I$, a set of ground-truth present objects $O^+$ (from dataset annotations or automated segmentation) and an equal number of absent objects $O^-$ are compiled. Each object $o \in O^+ \cup O^-$ receives a corresponding $q(o)$. The LVLM is prompted with $(I, q(o))$, producing a response $a \in \{\text{Yes}, \text{No}\}$, interpreted as:

• $o$0 if $o$1,
• $o$2 if $o$3.

The ground-truth label $o$4 indicates true object presence. Model performance is measured using:

• True Positives (TP): $o$5
• False Positives (FP): $o$6 (hallucinations)
• False Negatives (FN): $o$7
• True Negatives (TN): $o$8

Aggregated metrics:

• $o$9
• $$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$0
• $$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$1
• $$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$2
• Hallucination Rate (HR): $$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$3

This polling-based, binary format circumvents the need for syntactic parsing and mitigates ambiguity introduced by varying model generation styles (Li et al., 2023, Neuhaus et al., 22 Apr 2025).

2. Negative Sampling Strategies and Experimental Design

POPE incorporates three negative sampling regimes for selecting absent object probes, each designed to stress-test distributions of hallucinations:

• Random Sampling: Uniform selection from objects not present in $$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$4.
• Popular Sampling: Selection of the most frequent corpus objects not present in $$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$5.
• Adversarial Sampling: For each present object, the absent object with the highest historical co-occurrence is chosen.

These regimes balance the number of positive and negative probes, ensuring robust estimation of both hallucination and detection rates. The canonical POPE setup on MSCOCO employs 500 images, three positive and three negative probes per image, yielding 3,000 “Yes” and 3,000 “No” pairs per split (Li et al., 2023, Neuhaus et al., 22 Apr 2025).

Evaluation proceeds in five steps:

1. Image/object extraction (annotations or automatic segmentation).
2. Negative object sampling using the chosen strategy.
3. Construction of binary probe questions.
4. Model polling and answer collection.
5. Calculation of all standard binary metrics and hallucination rates.

3. Comparative Perspective and Methodological Impact

POPE’s closed-ended probing alleviates sensitivity to prompt rephrasings and model output styles, which previously confounded generative benchmarks such as CHAIR. For example, POPE achieves F1 stability ($$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$6) across prompt phrasings, compared to significantly higher sensitivity for CHAIR (up to $$q(o) = \text{“Is there a } <o> \text{ in the image?”}$$7) (Li et al., 2023). POPE is also extensible to datasets lacking human object annotations by substituting segmentation-based object or attribute discovery, as demonstrated with the SEEM segmenter.

Subsequent works have generalized the polling-based approach:

• H-POPE extends POPE to a hierarchical framework, systematically quantifying both object- and attribute-level hallucinations (e.g., color, material), revealing that attribute hallucination rates are substantially higher (Pham et al., 2024).
• 3D-POPE adapts POPE for 3D scene grounding in Large 3D Vision-LLMs (3D-LLMs), providing a direct assessment of hallucination in volumetric settings (Yang et al., 2024).

4. Annotation Quality and Benchmark Reliability

The POPE protocol’s efficacy is contingent upon accurate, unambiguous labels for object presence/absence. Systematic re-evaluation of the benchmark (RePOPE) identified a notable imbalance: while only 1.7% of “No” pairs (absent object probes) were actually present, 9.3% of “Yes” pairs (present object probes) were incorrectly labeled, with an additional 13.8% in ambiguous boundary categories. This label noise can significantly shift both absolute metrics (e.g., F1, precision) and relative model rankings (Neuhaus et al., 22 Apr 2025).

Improvements incorporated by RePOPE include:

• Multi-reviewer annotation with explicit “Ambiguous” labeling and exclusion of such pairs.
• Release of corrected labels and explanatory statistics.
• Recommendations, such as stratified hard-negative selection and complementing POPE with fine-grained benchmarks (e.g., DASH-B), to further improve benchmarking validity (Neuhaus et al., 22 Apr 2025).

5. Key Experimental Findings

POPE-based analysis across recent LVLMs revealed pronounced variation in hallucination and detection metrics both by model and sampling regime. On MSCOCO (random negatives):

Model	F1 Score (%)	Accuracy (%)	Yes-rate (%)
InstructBLIP	89.3	88.7	55
MiniGPT-4	78.9	77.8	54.8
LLaVA/MultiModal-GPT	68	50–55	>95

Under popular/adversarial negative sampling, F1 scores systematically decline by 5–10 points, confirming increased proneness to hallucination for common or correlated absent objects.

Stability tests highlight prompt-agnosticism, and cross-dataset checks with automated segmenters (A-OKVQA, GQA) preserve ranking and trends.

Subsequent analysis reveals, for instance, that InstructBLIP achieves object-level accuracy of 85.8% and attribute-level accuracy of 72.2% on H-POPE, underlining the increased difficulty of fine-grained hallucination assessment (Pham et al., 2024).

6. Domain-Specific Extensions: Hierarchical, Attribute, and 3D Probing

• H-POPE (Pham et al., 2024): Incorporates coarse (object existence) and fine (attribute) levels by polling both “Is there a <object> in the image?” and “Is the <object> of <attribute> in the image?”, using datasets such as MSCOCO (objects) and LSA (attributes). Evaluation reveals strong “yes” bias for objects and increased hallucination rates for attributes, especially under adversarial sampling.
• 3D-POPE (Yang et al., 2024): Transfers POPE’s protocol to 3D-LLMs, querying “Is there a <object> in the given 3D scene?” and calculating metrics across random, popular, and adversarial sampling regimes. Experiments demonstrate that pre-training on large-scale 3D-language data (3D-GRAND) sharply reduces hallucination, measured by higher precision and reduced Yes-rate compared with baseline 3D-LLMs.

Variant	Context	Metrics Assessed
POPE	2D images/objects	Existence hallucination
H-POPE	2D images + attributes	Object- & attribute-level hallucination
3D-POPE	3D scenes/objects	Volumetric existence hallucination

This progression demonstrates the flexibility and extensibility of the polling-based paradigm for hierarchical and multi-domain hallucination quantification.

7. Implications, Limitations, and Best Practices

POPE provides stable, interpretable, and flexible evaluation of object hallucination in multimodal models, exposing both overall model bias and specific failure modes, such as over-attribution to common or co-occurring objects. However, label quality and ambiguity can impact benchmark validity and lead to erroneous conclusions if not managed rigorously (Neuhaus et al., 22 Apr 2025).

Recommended practices include exhaustive expert re-annotation, ambiguity curation, stratified (especially adversarial) negative sampling, public release of corrected labels, and complementing binary probing with generative benchmarks and fine-grained attributes. The polling-based paradigm has become the foundation for multi-tiered benchmarks and domain-general frameworks assessing model consistency, grounding, and hallucination, both in 2D and 3D vision-language research (Li et al., 2023, Pham et al., 2024, Yang et al., 2024, Neuhaus et al., 22 Apr 2025).