ORIC: Incongruous Context Object Recognition

• ORIC is a benchmark that tests LVLMs on object recognition in images where context misleads, focusing on misidentification and hallucination errors.
• It employs LLM-guided sampling for unexpected present objects and CLIP-guided sampling for visually plausible absent ones from the MSCOCO validation set.
• Results show notable performance drops compared to standard benchmarks, highlighting the challenges of relying on scene-level priors in object detection.

Searching arXiv for ORIC and closely related context-robust object recognition work. Object Recognition in Incongruous Context Benchmark (ORIC) is a benchmark for evaluating Large Vision-LLMs (LVLMs) in scenarios where object-context relationships deviate from expectations. It studies recognition under two forms of contextual incongruity: objects that are genuinely present but contextually unexpected, and objects that are contextually plausible but in fact absent. In this setting, ORIC focuses on two failure modes—object misidentification and hallucination—and formulates evaluation as a balanced binary recognition problem over image-question pairs. The benchmark is built from MSCOCO validation images and is designed to measure how strongly models rely on scene-level priors rather than visual evidence when context is misleading (Li et al., 19 Sep 2025).

1. Definition and scope

ORIC defines “object recognition in incongruous context” as recognition when the relationship between an object and its surrounding scene defies common-sense expectations. The benchmark distinguishes two types of incongruity. The first is unexpected presence: objects that truly appear in an image but are unusual for that scene, such as a train in an office. The second is expected absence: objects that are plausible given the scene but do not actually appear, such as a sports ball on a baseball field when there is no ball. These two cases correspond to the benchmark’s two target error modes: failure to recognize a present object, and hallucination of an absent but contextually suggested object (Li et al., 19 Sep 2025).

The formal object inventory for an image is written as

$$\mathcal{O} = \{ o_i = (n_i, \{B_{ij}\}_{j=1}^{m_i}) \}_{i=1}^N,$$

where $n_i$ is the category name and $B_{ij}$ are the bounding boxes for that object. Each benchmark question specifies a target object and a binary ground-truth label $y \in \{\text{yes}, \text{no}\}$, indicating whether the queried object is actually present. A model prediction $\hat{y}$ is then evaluated as binary classification. In this formulation, misidentification is a false negative: $y=\text{yes}$ but $\hat{y}=\text{no}$. Hallucination is a false positive: $y=\text{no}$ but $\hat{y}=\text{yes}$ (Li et al., 19 Sep 2025).

ORIC differs from standard object recognition and detection benchmarks in that its questions are deliberately constructed to maximize contextual incongruity rather than merely test whether the object is visible. It also differs from hallucination benchmarks that focus primarily on semantic consistency or language priors: ORIC jointly measures missed recognition and hallucination within a single balanced binary task, and it does so by selecting objects whose relationship to the surrounding scene is specifically adversarial from the standpoint of contextual expectations (Li et al., 19 Sep 2025).

2. Dataset construction and sampling strategy

ORIC is built on 1,000 images sampled from the MSCOCO validation set. From these images, the benchmark constructs 2,000 binary questions, comprising 1,000 “yes”-label questions and 1,000 “no”-label questions. Presence and absence labels are derived from COCO instance annotations: a question is labeled “yes” if at least one instance of the queried category appears in the image, and “no” otherwise (Li et al., 19 Sep 2025).

To separate focal objects from scene-defining context, ORIC partitions the objects in each image into regions of interest (ROI) and non-ROI objects using area statistics. For each object $o_i$, the covered area is

$n_i$0

Objects are then divided by the median area $n_i$1: $n_i$2

$n_i$3

ROIs are therefore typically smaller objects, whereas non-ROIs are larger objects that more strongly define scene context (Li et al., 19 Sep 2025).

The benchmark uses two distinct selection procedures. For positive questions, ORIC applies LLM-guided sampling to choose present objects that are contextually unexpected. For each ROI object, GPT-4o is prompted with the list of non-ROI background objects and asked whether the target object can be inferred from those background objects using textual priors or common-sense co-occurrence. This yields a rejection function

$n_i$4

Objects for which the LLM answers “no” are treated as contextually incongruous despite being truly present, and a subset of these is used to form “yes” questions (Li et al., 19 Sep 2025).

For negative questions, ORIC uses CLIP-guided sampling to choose absent objects that are visually plausible given the scene. For each query image $n_i$5, a CLIP-based ViT encoder retrieves the most visually similar image $n_i$6 from a candidate pool by minimizing cosine distance

$n_i$7

For each candidate absent object $n_i$8, the text prompt

$n_i$9

is scored against $B_{ij}$0 using

$B_{ij}$1

High-scoring absent objects are used to form “no” questions, because they are strongly suggested by the context even though they do not appear (Li et al., 19 Sep 2025).

The resulting benchmark is intentionally skewed toward difficult local recognition. For “yes” questions, ORIC contains 44.8% small, 41.2% medium, and 14.0% large objects, compared with 27.6% small, 34.9% medium, and 37.4% large in POPE. Human checking of 300 questions found a 2% error rate, attributed mainly to inaccurate labels inherited from COCO and occasional cases where context was not truly incongruous (Li et al., 19 Sep 2025).

3. Task formulation and metrics

Each ORIC instance is posed as a binary visual question such as “Is there a train in the image?” or “Does the image contain train?” The task is to answer yes or no based on the image. The dataset is balanced, so macro recall equals accuracy when averaged over the two classes (Li et al., 19 Sep 2025).

Evaluation is defined in terms of the standard confusion matrix. For the “yes” class,

$B_{ij}$2

$B_{ij}$3

For the “no” class,

$B_{ij}$4

$B_{ij}$5

The macro averages are

$B_{ij}$6

$B_{ij}$7

$B_{ij}$8

The benchmark also reports Yes Proportion

$B_{ij}$9

which functions as a direct indicator of how often a model tends to answer “yes,” and thus how prone it is to hallucination (Li et al., 19 Sep 2025).

Because LVLMs often return free-form answers rather than a single token, ORIC uses a two-stage answer matching procedure. First, simple heuristic parsing attempts to extract an explicit yes/no response. If that fails, GPT-4o-0806 is used only as a normalization tool to map the response to the closest option among “yes” and “no.” The evaluation is therefore described as LLM-free at evaluation time, in the sense that correctness is not judged by an LLM; the helper model is used only for answer normalization (Li et al., 19 Sep 2025).

A conceptual implication is that ORIC measures recognition under contextual pressure rather than reasoning over complex instructions. This helps explain why chain-of-thought prompting does not consistently improve performance: the dominant failure mode is low-level visual grounding under misleading scene priors rather than textual decomposition of a complex query. This interpretation is supported by the reported chain-of-thought results, but it remains an interpretation of the published findings rather than an explicit theorem (Li et al., 19 Sep 2025).

4. Experimental setting and benchmark results

ORIC evaluates 18 LVLMs and 2 open-vocabulary detection models. The LVLM set includes GPT-4o-2024-08-06, LLaVA-v1.6-Vicuna-13B, Janus-Pro-7B, InternVL3-9B, Qwen2.5-VL-7B-Instruct, Phi-3.5-Vision-Instruct, GLM-4v-9B, VILA1.5-13B, BLIP-3, InternLM-XComposer2.5-7B, Kimi-VL-A3B-Instruct, SmolVLM2-2.2B-Instruct, Molmo-7B-D-0924, Llama-3.2-11B-Vision, Chameleon-7B, Fuyu-8B, Emu3-Chat, and EVE-7B-HD-v1.0. The detector baselines are Grounding DINO 1.5 Pro and OWLv2. Experiments use a single NVIDIA H100 GPU, with temperature set to 0 and maximum output length of 1,024 tokens for LVLMs (Li et al., 19 Sep 2025).

For LVLMs, each question is asked in four semantically equivalent forms: “Is there {object} in the image?”, “Does the image contain {object}?”, “Have you noticed {object} in the image?”, and “Can you see {object} in the image?” Results are averaged across the four prompts. A zero-shot chain-of-thought prompt variant is also tested. For Grounding DINO 1.5 Pro, candidate object names are given as a dot-separated prompt, and the model answers “yes” if any detection exceeds confidence 0.25. OWLv2 uses CLIP-style prompts of the form “an image of {object}” (Li et al., 19 Sep 2025).

The strongest reported ORIC results are InternVL3-9B with $y \in \{\text{yes}, \text{no}\}$0, GPT-4o with 75.45, Janus-Pro-7B with 74.83, and LLaVA-v1.6-13B with 74.37. Open-vocabulary detectors perform competitively but somewhat below the best LVLMs: Grounding DINO 1.5 Pro reaches 72.48, and OWLv2 reaches 72.02. Lower-performing models include Fuyu-8B with 34.16 and EVE-7B-HD-v1.0 with 51.59 (Li et al., 19 Sep 2025).

A central result is ORIC’s comparison with POPE, a widely used hallucination benchmark with the same number of balanced binary questions. Performance drops are substantial. For example, Janus-Pro-7B declines from 87.00 on POPE to 74.83 on ORIC; GPT-4o-0806 from 86.75 to 75.45; InternVL3-9B from 88.68 to 76.87; Emu3-Chat from 86.65 to 64.78; and Grounding DINO 1.5 Pro from 84.99 to 72.48. These results show that ORIC is considerably harder than POPE, largely because its “yes” and “no” examples are selected to be visually similar while differing in the presence or absence of a contextually critical object (Li et al., 19 Sep 2025).

The benchmark also exposes asymmetries between hallucination and missed detection. Some models are conservative and suffer low “yes” recall, thereby misidentifying present objects. Others answer “yes” far too often and hallucinate absent objects. For example, Chameleon-7B has Yes Proportion = 99.28 and “no” recall = 0.82, indicating near-constant hallucination. Grounding DINO 1.5 Pro attains “yes” recall = 91.70 but performs worse on “no” cases, indicating a trade-off between aggressive detection and false positives. At the other extreme, Llama-3.2-11B-Vision reports “yes” recall = 0.00 under the stated setup (Li et al., 19 Sep 2025).

Object scale remains a major factor. On ORIC “yes” questions, recall drops across all size categories relative to POPE, and the degradation is especially pronounced for small objects. For InternVL3-9B, recall falls from 82.29/90.43/96.34 on POPE to 63.63/77.61/86.45 on ORIC for small, medium, and large objects respectively. This indicates that contextual incongruity compounds the already difficult problem of fine-grained small-object recognition (Li et al., 19 Sep 2025).

5. Contextual incongruity as a research problem

ORIC is part of a broader line of work showing that context can both help and hinder recognition. A psychophysics-oriented benchmark based on MSCOCO demonstrated that congruent context enhances recognition while incongruent context consistently impairs performance for both humans and CATNet, a biologically inspired context-aware model. In that work, the core incongruent-context condition is created by pasting objects into semantically mismatched scenes, and both humans and models follow the same qualitative ordering: congruent > minimal > incongruent (Zhang et al., 2019). ORIC extends this general problem to LVLMs and recasts it as a large-scale binary recognition benchmark centered on misidentification and hallucination (Li et al., 19 Sep 2025).

A distinct but related direction studies out-of-context detection through explicit contextual relations such as co-occurrence, relative size, and spatial position. That line of work introduces a graph contextual reasoning network (GCRN) and a large-scale dataset generated by adding out-of-context objects to COCO images, with the goal of identifying objects that violate contextual relations (Acharya et al., 2022). ORIC differs in task formulation: it does not ask which object is out of context, but whether a queried object is present under conditions where contextual expectations are adversarial (Li et al., 19 Sep 2025). A plausible implication is that these two paradigms are complementary: one emphasizes recognition under misleading context, the other explicit contextual anomaly detection.

Recent work on feature attribution likewise shows that context manipulations affect both accuracy and explanation. On ImageNet-9 and ImageNet-CS, context change produces much larger performance drops than context perturbation, and misclassified examples exhibit higher context attribution volume than correctly classified examples. The same study reports non-trivial context attribution even in “no-information” backgrounds such as black or noise-only context, suggesting that attribution methods may overstate context relevance in some cases (Adhikari et al., 2024). This resonates with ORIC’s core motivation: contextual incongruity is not merely a nuisance variable but a structured source of recognition error, especially when models treat scene priors as substitutes for visual evidence (Li et al., 19 Sep 2025).

A further extension appears in the COOCO dataset, which investigates scene-object congruency and referential communication. COOCO manipulates semantic relatedness between object and scene and shows that models rely more on context when object evidence is degraded, while context can become a distractor when the target is incongruous. Attention analysis there finds increased target focus in mid-level layers during successful categorization under moderate noise (Merlo et al., 27 Jun 2025). Although COOCO is framed around referential communication rather than binary presence recognition, it is closely aligned with the ORIC problem: both aim to characterize how multimodal systems balance local object evidence against scene-level priors when the two disagree.

6. Significance, limitations, and extensions

ORIC’s main contribution is methodological. By combining LLM-guided sampling for contextually unexpected present objects and CLIP-guided sampling for contextually plausible absent objects, it constructs a benchmark in which contextual priors are systematically adversarial. The ablation study shows that LLM-guided positives mainly reduce yes-recall, whereas CLIP-guided negatives mainly reduce no-recall. For example, under the reported ablation, InternVL3-9B drops from 80.88 / 97.83 on random sampling to 68.83 / 81.75 with LLM-guided positives, and Grounding DINO 1.5 Pro undergoes a 32.45-point drop in “no”-recall under CLIP-guided negatives. This establishes that the benchmark difficulty is not incidental but induced by the selection strategy itself (Li et al., 19 Sep 2025).

The benchmark also provides a concrete empirical account of contextual bias in LVLMs. ORIC argues that LVLMs heavily rely on scene-level priors learned from large-scale image-text co-occurrence. Such priors can suppress recognition of truly present but contextually unexpected objects and induce hallucination of absent objects that “fit” the scene. The qualitative examples reported in the paper—a train missed in an office scene, or a sports ball hallucinated on a baseball court—are canonical demonstrations of this failure mode (Li et al., 19 Sep 2025).

Several limitations are stated explicitly. ORIC is built entirely on MSCOCO validation images, so its scope is bounded by COCO’s object categories and everyday scenes. Some noise remains from inherited annotations, and the notion of contextual incongruity is operationalized through GPT-4o and CLIP, both of which carry their own biases. The task itself is binary and therefore narrower than richer forms of recognition involving attributes, relations, or open-vocabulary description (Li et al., 19 Sep 2025).

These limitations suggest several extensions. One direction is broader dataset coverage, including domains such as medical imaging, industrial inspection, or robotics, where context priors may differ sharply from COCO-style scenes. Another is explicit modeling of when to trust context and when to rely on local evidence. A conceptually related development is the multi-rationale explainable object recognition benchmark, which augments images with multiple textual rationales and proposes contrastive conditional inference (CCI) to condition classification on rationale embeddings rather than prompt concatenation. That framework is described as conceptually aligned with ORIC-style questions because it encourages decisions grounded in object-specific features instead of global scene context (Rasekh et al., 19 Aug 2025). This suggests a possible route toward ORIC-style systems that are not merely more accurate, but more explicit about why they rejected misleading background cues.

In the broader literature, ORIC occupies a specific place: it is not a general detection benchmark, not purely a hallucination benchmark, and not only an attribution study. It is a deliberately adversarial test of whether a model can recognize objects when context is wrong in precisely the ways that large-scale multimodal pretraining encourages models to trust. Its reported results show that even top LVLMs remain substantially vulnerable in this regime, and that contextual incongruity remains a distinct obstacle for reliable multimodal perception (Li et al., 19 Sep 2025).