KnowDR-REC Benchmark for Knowledge-Driven REC

• KnowDR-REC is a benchmark for referring expression comprehension that combines visual cues with factual, temporal, and relational knowledge to localize target persons.
• The methodology uses paired positive and adversarial negative samples generated via fine-grained temporal knowledge graph perturbations to rigorously test model robustness.
• The benchmark employs conditional evaluation metrics that separate textual reasoning from visual grounding, highlighting models’ anti-hallucination and uncertainty handling.

Searching arXiv for the benchmark and closely related REC work to ground the article in current papers. KnowDR-REC is a benchmark for referring expression comprehension (REC) with real-world knowledge. In this setting, a model receives a single image $I$ and a declarative referring expression $e$, and must localize the target person by outputting a bounding box $b=(x,y,w,h)$; in negative settings, the correct output is abstention or a binary decision “no,” depending on protocol. The benchmark is designed to move REC beyond appearance- and spatial-only grounding by requiring fine-grained multimodal reasoning over text and image, including factual, temporal, relational, and role-based knowledge. It introduces hard negative samples through structured temporal knowledge graph perturbations and coarse image-text mismatches, and it adds conditional evaluation metrics intended to separate textual reasoning from visual grounding performance (Jin et al., 12 Aug 2025).

1. Conceptual scope and research motivation

KnowDR-REC is situated within REC, a task traditionally framed as detecting a target object in a single image from a natural-language expression. The benchmark begins from the observation that standard REC has focused primarily on intra-image cues such as appearance and spatial relations, and that popular benchmarks such as RefCOCO, RefCOCO+, and RefCOCOg are largely saturated. The central claim of KnowDR-REC is that many real-world referring expressions cannot be resolved from intra-image evidence alone because they depend on external knowledge, including temporal, cultural, biographical, organizational, and geographical information (Jin et al., 12 Aug 2025).

The benchmark therefore defines a knowledge-driven variant of REC. Its target expressions encode factual, temporal, relational, and role-based knowledge, including references to laureates, team memberships, event roles, nationality, uniforms, flags, logos, and historical or event-specific time points. A representative example in the benchmark description is an expression such as “the flag bearer wearing the Chinese delegation uniform from the 2024 Paris Olympics,” which requires linking visual evidence to an event, a time reference, and a role. This departs from standard REC formulations in which short expressions such as “the woman in red” are often sufficient.

A common misconception is that KnowDR-REC is merely a harder version of standard person grounding. The benchmark is narrower and more specific than that description suggests: it is constructed to test whether a model can integrate external real-world knowledge with visual grounding, and its diagnostic design explicitly probes robustness, hallucination, and the coupling between text understanding and localization (Jin et al., 12 Aug 2025).

2. Task formulation and benchmark design

The core task takes as input a single image $I$ and a declarative referring expression $e$, often rewritten from question-answering datasets into sentences beginning with “The person …”. The output is a bounding box for the target person. In negative settings, the correct behavior is refusal or “no,” rather than forced localization. This design is central to the benchmark’s anti-hallucination objective (Jin et al., 12 Aug 2025).

KnowDR-REC is defined by three features. First, it is built upon real-world knowledge and therefore requires fine-grained multimodal reasoning across text and image. Second, it includes elaborately constructed negative samples via fine-grained expression editing. These negatives are formed by parsing expressions into temporal knowledge graph quadruples,

$(s, r, o, t),$

and perturbing exactly one component to create a fluent but factually incompatible expression. Third, it introduces three evaluation metrics intended to expose the model’s internal reasoning process rather than only its final box prediction.

The negative-sample design is especially important. Fine-grained negatives alter one element of the underlying fact structure, such as the subject, relation, object, or time, while preserving surface fluency. Coarse negatives are produced by pairing an expression with an unrelated image and manually verifying that no described entity exists in the image. The result is a benchmark in which failure cannot be explained simply by lexical mismatch or poor image quality; instead, the benchmark tests whether a model can detect subtle semantic inconsistency and avoid hallucinated grounding (Jin et al., 12 Aug 2025).

This design also clarifies what counts as “knowledge-driven” in the benchmark. The expressions include attributes such as uniforms, flags, logos, and emblems; relations such as “member of,” “winner of,” “flag bearer of,” “captain of,” and “born in”; temporal qualifiers such as event year or award year; and titles such as “the 2022 Nobel laureate in Physics.” A plausible implication is that KnowDR-REC is not primarily a benchmark for generic object localization, but for entity disambiguation under multimodal factual constraints.

3. Dataset construction and corpus characteristics

KnowDR-REC is constructed from four QA datasets: ComplexWebQuestions, HotpotQA, KQA Pro, and MetaQA. The benchmark retains only questions involving person entities, because these support rich attributes, roles, and biographical facts. After manual filtering for uniqueness of referent and ambiguity removal, GPT-4o is used to transform questions into declarative, grounding-oriented sentences starting with “The person…”, while preserving factual content (Jin et al., 12 Aug 2025).

Positive samples are then created by retrieving candidate images from Wikipedia or Wikimedia Commons for the referenced people. The images are filtered for visual richness using three criteria: at least 4 distinct individuals, instances from more than 2 categories, and resolution greater than $300 \times 300$. YOLOv8 is used to retain visually rich, multi-person scenes. Annotation proceeds along two independent paths: in one, annotators consult public descriptions such as appearance, clothing, and hair; in the other, annotators provide the person’s name and the image to GPT-4o to generate detailed appearance descriptions and then localize the target. Only samples for which both paths identify the same instance are retained.

Component	Value	Notes
Expression corpus	2,537	Rewritten and filtered referring expressions
Positive triplets	1,042	Image–expression–bounding-box sets
Evaluation regime	Zero-shot	No official train/validation/test splits reported

Negative samples are constructed in two ways. Fine-grained textual negatives are produced from temporal knowledge graph edits that minimally change the expression while making it factually incompatible with the image. Coarse negatives are formed by randomly shuffling images and expressions, then manually verifying that no described entity appears in the paired image. Negative sample counts are not enumerated in the text, but the benchmark states that both fine-grained TKG-edited texts and coarse mismatches are included and balanced with positives in the instruction-guided setting (Jin et al., 12 Aug 2025).

The current benchmark focuses on person entities in visually complex, multi-person scenes. This is an explicit design choice rather than an accidental bias. The authors note that person entities were chosen because they provide rich online information and visually distinctive cues, but this also limits generalization to other knowledge-rich categories such as landmarks, artworks, inventions, and natural scenes.

4. Evaluation metrics and diagnostic protocol

KnowDR-REC extends standard grounding evaluation with metrics that distinguish reasoning success from localization success. It reports visual grounding accuracy at $\alpha \in \{0.50, 0.75, 0.90\}$, textual reasoning accuracy $\mathrm{Acc}_{\text{Text}}$, and two conditional grounding accuracies: $$A_{50}^{\text{match}} = P(B \mid N), \qquad A_{50}^{\text{mismatch}} = P(B \mid \bar{N}),$$ where $e$0 is the event that $e$1 and $e$2 is the event that textual reasoning is correct (Jin et al., 12 Aug 2025).

$e$3 is computed by extracting the model’s entity resolution from Chain-of-Thought output and matching it against ground truth using a character-specific lexicon. The two conditional grounding metrics stratify grounding accuracy by whether the model reasoned about the text correctly. Their purpose is diagnostic rather than purely evaluative: they reveal whether a model localizes correctly because it understood the expression, or whether it arrives at a plausible box despite incorrect textual reasoning.

Negative samples are evaluated under two protocols. In the Basic Setting, the key metric is Error Rate, defined as the proportion of negatives for which the model wrongly outputs any bounding box. In the Instruction-guided Setting, positives and negatives are balanced and the model answers a yes/no existence question; the reported metric is binary accuracy. Together, these protocols distinguish between forced-box REC behavior and explicit refusal behavior (Jin et al., 12 Aug 2025).

This metric design addresses a recurrent problem in multimodal evaluation: a model can appear competent under plain $e$4 while still failing entity resolution. KnowDR-REC makes that failure visible. A common misconception is that grounding accuracy alone measures understanding; the benchmark’s conditional metrics are introduced precisely to show that it does not.

5. Experimental results and empirical findings

KnowDR-REC evaluates 16 state-of-the-art multimodal models in zero-shot, closed-book conditions, using no external retrieval or internet access. The model set includes closed-source generalist MLLMs such as Gemini 2.5 Flash, Gemini 2.5 Pro, Grok-2 Vision, Qwen-VL-Plus, and Qwen-VL-Max; open-source generalist MLLMs such as Qwen2.5-VL in multiple sizes and SPHINX variants; and open-source specialist grounding-oriented models such as VLM-R1, GroundingGPT, Ferret, and REESEEK (Jin et al., 12 Aug 2025).

On positive samples, no model achieves consistently strong performance across both textual reasoning and visual grounding. Selected results illustrate the pattern. Qwen-VL-Max reports $e$5, $e$6, $e$7, $e$8, $e$9, and $b=(x,y,w,h)$0. Qwen2.5-VL-72B reports 71.0, 63.8, 45.9; $b=(x,y,w,h)$1; $b=(x,y,w,h)$2; and $b=(x,y,w,h)$3. Gemini 2.5 Flash reports a markedly different pattern: $b=(x,y,w,h)$4 but $b=(x,y,w,h)$5, indicating strong text resolution relative to weaker localization.

The benchmark’s main diagnostic finding is a decoupling between textual understanding and visual grounding. Across models, $b=(x,y,w,h)$6, showing that correct textual reasoning improves localization. For Qwen-VL-Max, the contrast is 80.4 versus 59.0. Yet many models still localize plausible targets when textual reasoning fails, as indicated by high $b=(x,y,w,h)$7. This is one of the benchmark’s central observations: models can ground without genuinely resolving the expression (Jin et al., 12 Aug 2025).

Results on negative samples reinforce the same conclusion. In the Basic Setting, many models exhibit very high Error Rate: Gemini 2.5 Pro scores 97.1, Grok-2 Vision 83.5, Qwen2.5-VL-3B 100.0, VLM-R1 100.0, and GroundingGPT 100.0. Even comparatively better results remain high, such as Qwen-VL-Plus at 52.5 and Qwen-VL-Max at 70.7. In the Instruction-guided Setting, some closed-source models improve substantially, with Gemini 2.5 Flash at 82.0, Gemini 2.5 Pro at 79.4, and Grok-2 Vision at 78.5, while many open-source models remain near chance, including Qwen-VL-Plus at 51.7 and Qwen-VL-Max at 57.4.

The benchmark also reports that many models achieve substantial $b=(x,y,w,h)$8 even on adversarial negative pairs when evaluated against the positive ground-truth boxes. Examples include Qwen-VL-Plus at 59.6, Qwen-VL-Max at 56.9, and Qwen2.5-VL-72B at 61.9. The benchmark interprets this as evidence of shortcut-driven localization unrelated to textual correctness. This suggests that some MLLMs rely on memorized instance-level bindings or visual priors rather than text-conditioned reasoning.

6. Interpretation, limitations, and practical significance

KnowDR-REC is intended as both a benchmark and a diagnostic instrument. Its use of Chain-of-Thought outputs, text-level entity matching, and conditional grounding metrics provides a structured view of internal failure modes, including hallucinated localization despite incorrect textual inference, prompt-sensitive refusal behavior, and over-reliance on visual priors. In this respect, the benchmark is not limited to ranking models by accuracy; it is designed to isolate where reasoning pipelines fail (Jin et al., 12 Aug 2025).

Several limitations are explicitly acknowledged. The current benchmark focuses on person entities, and explicit per-type distributions or sub-benchmarks are not reported. Although the benchmark includes factual, temporal, and relational knowledge, knowledge-type coverage is described qualitatively rather than with a full numeric breakdown. Potential annotation noise may remain despite dual-path verification, and GPT-4o-based rewriting and TKG extraction may introduce subtle artifacts. No official train/validation/test splits or training recipes are reported, because the benchmark is currently used for zero-shot, closed-book evaluation.

The benchmark’s practical recommendations follow from these findings. It advises evaluating both $b=(x,y,w,h)$9 and $I$0 so that localization is not conflated with understanding; using $I$1 and $I$2 to diagnose text-vision decoupling; and testing with fine-grained negative samples to assess anti-hallucination and refusal behavior. For system design, it recommends combining curated knowledge sources such as Wikipedia or knowledge bases with vision encoders and textual decoders, enforcing consistency checks between textual reasoning and predicted boxes, and calibrating refusal and uncertainty (Jin et al., 12 Aug 2025).

KnowDR-REC is available through a public repository and is released with explicit licensing guidance for source datasets, Wikimedia Commons images, and benchmark JSON. Its benchmark JSON, excluding original content, is released under CC0 1.0 Universal. The benchmark therefore occupies a specific position in multimodal evaluation: it is a zero-shot testbed for knowledge-intensive visual grounding that exposes the gap between apparent localization competence and genuine multimodal reasoning.