---
title: 'ORIC: Incongruous Context Object Recognition'
url: https://www.emergentmind.com/topics/object-recognition-in-incongruous-context-benchmark-oric
type: topic
---

# ORIC: Incongruous Context Object Recognition

Searching arXiv for ORIC and closely related context-robust object recognition work.
Object Recognition in Incongruous Context Benchmark (ORIC) is a benchmark for evaluating Large Vision-Language Models (LVLMs) in scenarios where object-context relationships deviate from expectations. It studies recognition under two forms of contextual incongruity: objects that are genuinely present but contextually unexpected, and objects that are contextually plausible but in fact absent. In this setting, ORIC focuses on two failure modes—object misidentification and hallucination—and formulates evaluation as a balanced binary recognition problem over image-question pairs. The benchmark is built from MSCOCO validation images and is designed to measure how strongly models rely on scene-level priors rather than visual evidence when context is misleading [2509.15695].

## 1. Definition and scope

ORIC defines “object recognition in incongruous context” as recognition when the relationship between an object and its surrounding scene defies common-sense expectations. The benchmark distinguishes two types of incongruity. The first is **unexpected presence**: objects that truly appear in an image but are unusual for that scene, such as a train in an office. The second is **expected absence**: objects that are plausible given the scene but do not actually appear, such as a sports ball on a baseball field when there is no ball. These two cases correspond to the benchmark’s two target error modes: failure to recognize a present object, and hallucination of an absent but contextually suggested object [2509.15695].

The formal object inventory for an image is written as
\[
\mathcal{O} = \{ o_i = (n_i, \{B_{ij}\}_{j=1}^{m_i}) \}_{i=1}^N,
\]
where \(n_i\) is the category name and \(B_{ij}\) are the bounding boxes for that object. Each benchmark question specifies a target object and a binary ground-truth label \(y \in \{\text{yes}, \text{no}\}\), indicating whether the queried object is actually present. A model prediction \(\hat{y}\) is then evaluated as binary classification. In this formulation, **misidentification** is a false negative: \(y=\text{yes}\) but \(\hat{y}=\text{no}\). **Hallucination** is a false positive: \(y=\text{no}\) but \(\hat{y}=\text{yes}\) [2509.15695].

ORIC differs from standard object recognition and detection benchmarks in that its questions are deliberately constructed to maximize contextual incongruity rather than merely test whether the object is visible. It also differs from hallucination benchmarks that focus primarily on semantic consistency or language priors: ORIC jointly measures missed recognition and hallucination within a single balanced binary task, and it does so by selecting objects whose relationship to the surrounding scene is specifically adversarial from the standpoint of contextual expectations [2509.15695].

## 2. Dataset construction and sampling strategy

ORIC is built on **1,000 images** sampled from the **MSCOCO validation set**. From these images, the benchmark constructs **2,000 binary questions**, comprising **1,000 “yes”-label questions** and **1,000 “no”-label questions**. Presence and absence labels are derived from COCO instance annotations: a question is labeled “yes” if at least one instance of the queried category appears in the image, and “no” otherwise [2509.15695].

To separate focal objects from scene-defining context, ORIC partitions the objects in each image into **regions of interest (ROI)** and **non-ROI** objects using area statistics. For each object \(o_i\), the covered area is
\[
A_i = \mathrm{area}\Bigl(\bigcup_{j=1}^{m_i} B_{ij}\Bigr).
\]
Objects are then divided by the median area \(P_{50}(A)\):
\[
\mathcal{O}_{\text{ROI}} = \{ o_{(i)} \mid A_{(i)} < P_{50}(A) \},
\]
\[
\mathcal{O}_{\text{nonROI}} = \{ o_{(i)} \mid A_{(i)} \ge P_{50}(A) \}.
\]
ROIs are therefore typically smaller objects, whereas non-ROIs are larger objects that more strongly define scene context [2509.15695].

The benchmark uses two distinct selection procedures. For positive questions, ORIC applies **LLM-guided sampling** to choose present objects that are contextually unexpected. For each ROI object, GPT-4o is prompted with the list of non-ROI background objects and asked whether the target object can be inferred from those background objects using textual priors or common-sense co-occurrence. This yields a rejection function
\[
f(o) =
\begin{cases}
1, & \text{if } \text{LLM}(o, \mathcal{O}_{\text{nonROI}}) = \text{``no''},\\
0, & \text{otherwise}.
\end{cases}
\]
Objects for which the LLM answers “no” are treated as contextually incongruous despite being truly present, and a subset of these is used to form “yes” questions [2509.15695].

For negative questions, ORIC uses **CLIP-guided sampling** to choose absent objects that are visually plausible given the scene. For each query image \(I_q\), a CLIP-based ViT encoder retrieves the most visually similar image \(I'\) from a candidate pool by minimizing cosine distance
\[
D(I_q, I_i) = 1 - \frac{\mathbf{e}_q \cdot \mathbf{e}_i}{\|\mathbf{e}_q\| \|\mathbf{e}_i\|}.
\]
For each candidate absent object \(n_i\), the text prompt
\[
T_i = \text{``an image contains } n_i \text{''}
\]
is scored against \(I'\) using
\[
\mathrm{CLIPScore}(I', T_i)
= \hat{f}_I^\top \hat{f}_{T_i}
= \frac{f_I^\top f_{T_i}}{\|f_I\|\,\|f_{T_i}\|}.
\]
High-scoring absent objects are used to form “no” questions, because they are strongly suggested by the context even though they do not appear [2509.15695].

The resulting benchmark is intentionally skewed toward difficult local recognition. For “yes” questions, ORIC contains **44.8% small**, **41.2% medium**, and **14.0% large** objects, compared with **27.6% small**, **34.9% medium**, and **37.4% large** in POPE. Human checking of **300 questions** found a **2% error rate**, attributed mainly to inaccurate labels inherited from COCO and occasional cases where context was not truly incongruous [2509.15695].

## 3. Task formulation and metrics

Each ORIC instance is posed as a binary visual question such as “Is there a train in the image?” or “Does the image contain train?” The task is to answer **yes** or **no** based on the image. The dataset is balanced, so macro recall equals accuracy when averaged over the two classes [2509.15695].

Evaluation is defined in terms of the standard confusion matrix. For the “yes” class,
\[
\text{Precision}_{\text{yes}} = \frac{\text{TP}}{\text{TP} + \text{FP}},\quad
\text{Recall}_{\text{yes}} = \frac{\text{TP}}{\text{TP} + \text{FN}},
\]
\[
F1_{\text{yes}} =
2 \times
\frac{\text{Precision}_{\text{yes}} \times \text{Recall}_{\text{yes}}}
{\text{Precision}_{\text{yes}} + \text{Recall}_{\text{yes}}}.
\]
For the “no” class,
\[
\text{Precision}_{\text{no}} = \frac{\text{TN}}{\text{TN} + \text{FN}},\quad
\text{Recall}_{\text{no}} = \frac{\text{TN}}{\text{TN} + \text{FP}},
\]
\[
F1_{\text{no}} =
2 \times
\frac{\text{Precision}_{\text{no}} \times \text{Recall}_{\text{no}}}
{\text{Precision}_{\text{no}} + \text{Recall}_{\text{no}}}.
\]
The macro averages are
\[
\text{Precision}_{\text{macro}} =
\frac{\text{Precision}_{\text{yes}} + \text{Precision}_{\text{no}}}{2},
\]
\[
\text{Recall}_{\text{macro}} =
\frac{\text{Recall}_{\text{yes}} + \text{Recall}_{\text{no}}}{2}
= \text{Accuracy},
\]
\[
F1_{\text{macro}} =
\frac{F1_{\text{yes}} + F1_{\text{no}}}{2}.
\]
The benchmark also reports **Yes Proportion**
\[
\text{Yes Proportion} =
\frac{\text{TP} + \text{FP}}
{\text{TP} + \text{FP} + \text{TN} + \text{FN}},
\]
which functions as a direct indicator of how often a model tends to answer “yes,” and thus how prone it is to hallucination [2509.15695].

Because LVLMs often return free-form answers rather than a single token, ORIC uses a two-stage answer matching procedure. First, simple heuristic parsing attempts to extract an explicit yes/no response. If that fails, GPT-4o-0806 is used only as a normalization tool to map the response to the closest option among “yes” and “no.” The evaluation is therefore described as **LLM-free at evaluation time**, in the sense that correctness is not judged by an LLM; the helper model is used only for answer normalization [2509.15695].

A conceptual implication is that ORIC measures recognition under contextual pressure rather than reasoning over complex instructions. This helps explain why chain-of-thought prompting does not consistently improve performance: the dominant failure mode is low-level visual grounding under misleading scene priors rather than textual decomposition of a complex query. This interpretation is supported by the reported chain-of-thought results, but it remains an interpretation of the published findings rather than an explicit theorem [2509.15695].

## 4. Experimental setting and benchmark results

ORIC evaluates **18 LVLMs** and **2 open-vocabulary detection models**. The LVLM set includes GPT-4o-2024-08-06, LLaVA-v1.6-Vicuna-13B, Janus-Pro-7B, InternVL3-9B, Qwen2.5-VL-7B-Instruct, Phi-3.5-Vision-Instruct, GLM-4v-9B, VILA1.5-13B, BLIP-3, InternLM-XComposer2.5-7B, Kimi-VL-A3B-Instruct, SmolVLM2-2.2B-Instruct, Molmo-7B-D-0924, Llama-3.2-11B-Vision, Chameleon-7B, Fuyu-8B, Emu3-Chat, and EVE-7B-HD-v1.0. The detector baselines are Grounding DINO 1.5 Pro and OWLv2. Experiments use a single NVIDIA H100 GPU, with temperature set to 0 and maximum output length of 1,024 tokens for LVLMs [2509.15695].

For LVLMs, each question is asked in four semantically equivalent forms: “Is there {object} in the image?”, “Does the image contain {object}?”, “Have you noticed {object} in the image?”, and “Can you see {object} in the image?” Results are averaged across the four prompts. A zero-shot chain-of-thought prompt variant is also tested. For Grounding DINO 1.5 Pro, candidate object names are given as a dot-separated prompt, and the model answers “yes” if any detection exceeds confidence 0.25. OWLv2 uses CLIP-style prompts of the form “an image of {object}” [2509.15695].

The strongest reported ORIC results are **InternVL3-9B** with **\(F1_{\text{macro}} = 76.87\)**, **GPT-4o** with **75.45**, **Janus-Pro-7B** with **74.83**, and **LLaVA-v1.6-13B** with **74.37**. Open-vocabulary detectors perform competitively but somewhat below the best LVLMs: **Grounding DINO 1.5 Pro** reaches **72.48**, and **OWLv2** reaches **72.02**. Lower-performing models include **Fuyu-8B** with **34.16** and **EVE-7B-HD-v1.0** with **51.59** [2509.15695].

A central result is ORIC’s comparison with **POPE**, a widely used hallucination benchmark with the same number of balanced binary questions. Performance drops are substantial. For example, **Janus-Pro-7B** declines from **87.00** on POPE to **74.83** on ORIC; **GPT-4o-0806** from **86.75** to **75.45**; **InternVL3-9B** from **88.68** to **76.87**; **Emu3-Chat** from **86.65** to **64.78**; and **Grounding DINO 1.5 Pro** from **84.99** to **72.48**. These results show that ORIC is considerably harder than POPE, largely because its “yes” and “no” examples are selected to be visually similar while differing in the presence or absence of a contextually critical object [2509.15695].

The benchmark also exposes asymmetries between hallucination and missed detection. Some models are conservative and suffer low “yes” recall, thereby misidentifying present objects. Others answer “yes” far too often and hallucinate absent objects. For example, **Chameleon-7B** has **Yes Proportion = 99.28** and **“no” recall = 0.82**, indicating near-constant hallucination. **Grounding DINO 1.5 Pro** attains **“yes” recall = 91.70** but performs worse on “no” cases, indicating a trade-off between aggressive detection and false positives. At the other extreme, **Llama-3.2-11B-Vision** reports **“yes” recall = 0.00** under the stated setup [2509.15695].

Object scale remains a major factor. On ORIC “yes” questions, recall drops across all size categories relative to POPE, and the degradation is especially pronounced for small objects. For **InternVL3-9B**, recall falls from **82.29/90.43/96.34** on POPE to **63.63/77.61/86.45** on ORIC for small, medium, and large objects respectively. This indicates that contextual incongruity compounds the already difficult problem of fine-grained small-object recognition [2509.15695].

## 5. Contextual incongruity as a research problem

ORIC is part of a broader line of work showing that context can both help and hinder recognition. A psychophysics-oriented benchmark based on MSCOCO demonstrated that **congruent context enhances recognition** while **incongruent context consistently impairs performance** for both humans and CATNet, a biologically inspired context-aware model. In that work, the core incongruent-context condition is created by pasting objects into semantically mismatched scenes, and both humans and models follow the same qualitative ordering: **congruent > minimal > incongruent** [1911.07349]. ORIC extends this general problem to LVLMs and recasts it as a large-scale binary recognition benchmark centered on misidentification and hallucination [2509.15695].

A distinct but related direction studies out-of-context detection through explicit contextual relations such as co-occurrence, relative size, and spatial position. That line of work introduces a **graph contextual reasoning network (GCRN)** and a large-scale dataset generated by adding out-of-context objects to COCO images, with the goal of identifying objects that violate contextual relations [2202.05930]. ORIC differs in task formulation: it does not ask which object is out of context, but whether a queried object is present under conditions where contextual expectations are adversarial [2509.15695]. A plausible implication is that these two paradigms are complementary: one emphasizes recognition under misleading context, the other explicit contextual anomaly detection.

Recent work on feature attribution likewise shows that context manipulations affect both accuracy and explanation. On ImageNet-9 and ImageNet-CS, **context change** produces much larger performance drops than context perturbation, and misclassified examples exhibit higher context attribution volume than correctly classified examples. The same study reports non-trivial context attribution even in “no-information” backgrounds such as black or noise-only context, suggesting that attribution methods may overstate context relevance in some cases [2411.02833]. This resonates with ORIC’s core motivation: contextual incongruity is not merely a nuisance variable but a structured source of recognition error, especially when models treat scene priors as substitutes for visual evidence [2509.15695].

A further extension appears in the **COOCO** dataset, which investigates scene-object congruency and referential communication. COOCO manipulates semantic relatedness between object and scene and shows that models rely more on context when object evidence is degraded, while context can become a distractor when the target is incongruous. Attention analysis there finds increased target focus in mid-level layers during successful categorization under moderate noise [2506.22274]. Although COOCO is framed around referential communication rather than binary presence recognition, it is closely aligned with the ORIC problem: both aim to characterize how multimodal systems balance local object evidence against scene-level priors when the two disagree.

## 6. Significance, limitations, and extensions

ORIC’s main contribution is methodological. By combining **LLM-guided sampling** for contextually unexpected present objects and **CLIP-guided sampling** for contextually plausible absent objects, it constructs a benchmark in which contextual priors are systematically adversarial. The ablation study shows that LLM-guided positives mainly reduce **yes-recall**, whereas CLIP-guided negatives mainly reduce **no-recall**. For example, under the reported ablation, **InternVL3-9B** drops from **80.88 / 97.83** on random sampling to **68.83 / 81.75** with LLM-guided positives, and **Grounding DINO 1.5 Pro** undergoes a **32.45-point** drop in “no”-recall under CLIP-guided negatives. This establishes that the benchmark difficulty is not incidental but induced by the selection strategy itself [2509.15695].

The benchmark also provides a concrete empirical account of contextual bias in LVLMs. ORIC argues that LVLMs heavily rely on scene-level priors learned from large-scale image-text co-occurrence. Such priors can suppress recognition of truly present but contextually unexpected objects and induce hallucination of absent objects that “fit” the scene. The qualitative examples reported in the paper—a train missed in an office scene, or a sports ball hallucinated on a baseball court—are canonical demonstrations of this failure mode [2509.15695].

Several limitations are stated explicitly. ORIC is built entirely on **MSCOCO validation images**, so its scope is bounded by COCO’s object categories and everyday scenes. Some noise remains from inherited annotations, and the notion of contextual incongruity is operationalized through GPT-4o and CLIP, both of which carry their own biases. The task itself is binary and therefore narrower than richer forms of recognition involving attributes, relations, or open-vocabulary description [2509.15695].

These limitations suggest several extensions. One direction is broader dataset coverage, including domains such as medical imaging, industrial inspection, or robotics, where context priors may differ sharply from COCO-style scenes. Another is explicit modeling of when to trust context and when to rely on local evidence. A conceptually related development is the **multi-rationale explainable object recognition benchmark**, which augments images with multiple textual rationales and proposes **contrastive conditional inference (CCI)** to condition classification on rationale embeddings rather than prompt concatenation. That framework is described as conceptually aligned with ORIC-style questions because it encourages decisions grounded in object-specific features instead of global scene context [2508.14280]. This suggests a possible route toward ORIC-style systems that are not merely more accurate, but more explicit about why they rejected misleading background cues.

In the broader literature, ORIC occupies a specific place: it is not a general detection benchmark, not purely a hallucination benchmark, and not only an attribution study. It is a deliberately adversarial test of whether a model can recognize objects when context is wrong in precisely the ways that large-scale multimodal pretraining encourages models to trust. Its reported results show that even top LVLMs remain substantially vulnerable in this regime, and that contextual incongruity remains a distinct obstacle for reliable multimodal perception [2509.15695].

Source: https://www.emergentmind.com/topics/object-recognition-in-incongruous-context-benchmark-oric