---
title: 'COCO-OOD: A COCO-Based OOD Benchmark'
url: https://www.emergentmind.com/topics/coco-ood
type: topic
---

# COCO-OOD: A COCO-Based OOD Benchmark

Searching arXiv for COCO-OOD-related papers to ground the article in current literature.
COCO-OOD denotes a family of COCO-centered out-of-distribution evaluation settings rather than a single universally standardized task. In the literature, the term spans at least three distinct formulations: natural distribution-shift benchmarking for COCO-trained object detectors, semantic OOD detection on COCO-derived classification tasks, and cross-dataset or external-dataset OOD evaluation using COCO as either the in-distribution or out-of-distribution source. This plurality is explicit in the relevant research. “COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts” defines a COCO-compatible benchmark for object detection under six natural shifts while preserving the original 80 COCO categories [2307.12730]. “Using Semantic Information for Defining and Detecting OOD Inputs” uses COCO as a case study to argue that OOD should be defined relative to intended task semantics rather than the empirical training-image distribution [2302.11019]. “COOD: Concept-based Zero-shot OOD Detection” uses MS-COCO as the in-distribution dataset in a multi-label zero-shot OOD recognition setting with external OOD datasets [2411.13578]. This suggests that “COCO-OOD” is best treated as an umbrella term covering multiple benchmark philosophies, all anchored to COCO but differing in task, ontology, and the meaning assigned to “out-of-distribution.”

## 1. Historical emergence and scope

The modern use of COCO as an OOD reference point arose from the central role of MS-COCO in object detection and segmentation research. One line of work treats COCO as the in-distribution training and evaluation domain and asks how detectors behave under natural but non-i.i.d. test inputs. In this formulation, COCO-O preserves the original 80 COCO categories and evaluation protocol while changing the image distribution through web-collected natural shifts such as weather, painting, handmake, cartoon, tattoo, and sketch [2307.12730]. A second line of work treats COCO as a semantically biased training source and argues that standard OOD detectors inherit these biases when they operate only on the empirical training distribution rather than task-relevant semantics [2302.11019]. A third line uses COCO in multi-label recognition, where the challenge is zero-shot OOD detection in the presence of label co-occurrence and semantic dependence, evaluated against external datasets such as Filtered ImageNet22K and Textures [2411.13578].

This diversity is reinforced by adjacent work that is COCO-related but not identical in objective. “Complementary datasets to COCO for object detection” introduces COCO_OI and ObjectNet_D to study generalization beyond standard COCO under natural dataset shift, but it does not formalize explicit OOD detection scores or open-set rejection [2206.11473]. “CD-COCO: A Versatile Complex Distorted COCO Database for Scene-Context-Aware Computer Vision” defines a COCO-derived robustness benchmark under global and local photorealistic distortions rather than semantic OOD [2311.06976]. “Proto-OOD: Enhancing OOD Object Detection with Prototype Feature Similarity” is directly relevant to COCO-OOD in the widespread VOC-vs-COCO unknown-object setting, but it does not introduce a standalone benchmark named COCO-OOD [2409.05466].

A plausible implication is that the phrase “COCO-OOD” should not be interpreted as designating one fixed dataset split. It instead refers to a set of COCO-anchored robustness and OOD formulations spanning detection, recognition, semantic shift, covariate shift, and cross-dataset evaluation.

## 2. Major formulations of COCO-OOD

The literature distinguishes several concrete formulations, each with different semantics, metrics, and failure modes.

| Formulation | Core idea | Representative paper |
|---|---|---|
| Natural distribution shift for detection | Train on standard COCO, test on naturally shifted but COCO-labeled images | COCO-O [2307.12730] |
| Semantic OOD on COCO-derived classification | Define OOD relative to task semantics rather than image-distribution likelihood | Semantic-information framework [2302.11019] |
| Multi-label zero-shot OOD recognition | Use MS-COCO as ID, external datasets as OOD, no retraining | COOD [2411.13578] |

In the natural-shift formulation, the detector is still solving the standard COCO task, and the ontology remains unchanged. COCO-O contains 6,782 images and 26,624 annotated boxes spanning the original 80 COCO categories, with six disjoint OOD test domains: Sketch, Weather, Cartoon, Painting, Tattoo, and Handmake [2307.12730]. OOD here means a change in image distribution, not a change in label space.

In the semantic formulation, OOD is not equivalent to “unlikely under the training set.” The intended distribution $\mathcal{D}_I$ is introduced so that
$$
\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},
$$
and the associated intended set of inputs is
$$
\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.
$$
Under this view, samples with the same task-relevant semantic classes but different nuisance factors are not OOD, whereas samples lacking the relevant semantics are OOD even if their appearance resembles training data [2302.11019].

In the multi-label zero-shot formulation, COCO’s complexity lies in co-occurring labels. COOD uses MS-COCO as the in-distribution dataset and evaluates OOD against Filtered ImageNet22K and Textures. The detector is zero-shot and training-free, using pretrained CLIP plus concept-based label expansion [2411.13578]. OOD is external to COCO, but the difficulty arises from the multi-label structure of the COCO in-distribution side.

A common misconception is that any evaluation involving COCO and a distribution shift constitutes the same benchmark. The papers show otherwise: a held-out COCO-category detection test, a COCO-derived semantic classification experiment, and a COCO-vs-external multi-label recognition benchmark answer different scientific questions.

## 3. COCO-O as a natural OOD benchmark for object detection

COCO-O is the most direct benchmark paper for COCO-trained object detectors under natural distribution shifts. It is test-only and domain-structured, with image and object counts per domain reported as follows: Sketch: 992 images, 3,707 objects; Weather: 961 images, 4,509 objects; Cartoon: 1,996 images, 8,774 objects; Painting: 954 images, 4,879 objects; Tattoo: 918 images, 1,489 objects; Handmake: 961 images, 3,266 objects [2307.12730]. Class mapping is unchanged: COCO-O adopts the original 80 COCO categories directly.

Construction is based on web search over combinations of domain keywords and COCO object names. To reduce iconic single-object bias, the search process mimics COCO’s multi-object style by combining multiple class names, such as “cartoon + dog + car,” and uses multiple search engines when required [2307.12730]. The benchmark therefore changes image distribution while maintaining compatibility with COCO-trained detectors and standard COCO evaluation software.

Evaluation uses standard COCO mean Average Precision averaged over IoU thresholds from 50% to 95%, computed per OOD domain and averaged across the six domains as the overall COCO-O score [2307.12730]. The key robustness metric is Effective Robustness:
$$
ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),
$$
where $mAP_{id}(\cdot)$ is COCO mAP, $mAP_{ood}(\cdot)$ is COCO-O mAP, and 0.45 is the fitted slope from linear regression over 11 classic detectors [2307.12730]. Positive ER indicates better OOD performance than expected from in-distribution accuracy alone.

The benchmark is substantially harder than standard COCO evaluation. On Faster R-CNN with ResNet-50, COCO-O yields a reported $55.7\%$ relative performance drop, corresponding to a decrease from 37.4 COCO mAP to 16.4 COCO-O mAP [2307.12730]. The paper also states that COCO-O has a much larger Fréchet Inception Distance from clean COCO than COCO-C, 132 vs. 41 [2307.12730].

Controlled studies show that backbone choice dominates neck and head modifications for robustness. On Faster R-CNN, ResNet-50 gives COCO 37.4 / COCO-O 16.4 / ER -0.41, while PVTv2-B2 gives 45.6 / 24.9 / +4.33 [2307.12730]. On RetinaNet, ResNet-50 gives 36.5 / 16.6 / +0.18, while PVTv2-B2 reaches 44.6 / 25.2 / +5.10 [2307.12730]. By contrast, neck variants such as PAFPN or NAS-FPN and head modifications such as SABL, FSAF, or FreeAnchor yield limited or negative ER changes [2307.12730].

The paper also reports that end-to-end detection transformers do not automatically improve robustness. DETR has COCO 42.0 but COCO-O only 17.1 with ER -1.82, while Deformable DETR has COCO 44.5 and COCO-O 18.5 with ER -1.49 [2307.12730]. Strong augmentations help, with MixUp singled out as especially important: removing MixUp in YOLOX-S reduces COCO-O mAP from 19.8 to 17.7 [2307.12730]. Large-scale pretraining is beneficial but heterogeneous: ImageNet-21K and “Sup_RSB” improve ER, whereas SwAV hurts robustness in the reported Faster R-CNN study [2307.12730].

Foundation and vision-language models dominate the top end. In the larger SOTA benchmark, EVA achieves COCO 64.2 and COCO-O 57.8 with ER +28.86; GLIP-L gets 48.0 and +24.89; DETA gets 48.5 and +20.15 [2307.12730]. This suggests that the strongest robustness gains are linked less to conventional architectural tweaks than to large-scale pretraining and external data.

## 4. Semantic redefinitions of OOD on COCO

The COCO case study in “Using Semantic Information for Defining and Detecting OOD Inputs” shifts attention from benchmark construction to the definition of OOD itself. The central claim is that detectors tied to the training distribution $\mathcal{D}$ can inherit dataset bias and become impermeable to inputs outside the training distribution but with the same semantic information [2302.11019]. This is especially salient for COCO because its images are rich in context, background, co-occurrence structure, and annotation bias.

The paper formalizes a distinction between the empirical training distribution $\mathcal{D}$ and an intended distribution $\mathcal{D}_I$, then introduces the semantically relevant part of an image through maps
$$
\mathscr{F} := \{F:\mathcal{X}\rightarrow \mathcal{X}_I \cup \{\bot\}\mid x\mapsto F(x)=s,\; C(x)=C(s)\},
$$
where $C:\mathcal{X}\to\mathcal{Y}$ is the oracle classifier and $\bot$ denotes an empty image [2302.11019]. If semantic extraction yields a distribution $\mathcal{D}'$ close to $\mathcal{D}_I$ in Kolmogorov distance, then an estimator of intended distribution exists in the asymptotic sense stated by the theorem in the paper [2302.11019]. The purpose is not a complete detection theory, but support for estimating intended support after removing irrelevant image content.

The practical COCO instantiation uses semantic segmentation. The estimated intended set is written as
$$
\hat{\mathcal{X}_I} := \{x' : x'=\mathcal{N}_s(x),\; x\in\mathcal{S}\},
$$
where
$$
\mathcal{N}_s:\mathcal{X}\to \mathbb{R}^{h\times w\times (|\mathcal{Y}|+1)}.
$$
Each pixel receives a probability vector over the $N=|\mathcal{Y}|$ semantic labels plus one background label, and the foreground is the set of pixels assigned to one of the class labels in $\mathcal{Y}$ [2302.11019].

For a per-pixel class-probability vector $q=(q_1,\dots,q_N,q_{N+1})^\top$, the paper defines
$$
V(q):= \begin{cases} \max_i q_i & \text{if } \arg\max_i q_i \in \{1,\dots,N\},\ 0 & \text{otherwise}. \end{cases}
$$
The baseline semantic score is
$$
BLS(x):= \frac{ \sum_{i\in H}\sum_{j\in W}V(\mathcal{N}_s(x)_{i,j}) }{ \sum_{i\in H}\sum_{j\in W}\mathbbm{1}\!\left(V(\mathcal{N}_s(x)_{i,j})\neq 0\right) }.
$$
ODIN is adapted similarly, yielding the foreground score
$$
ODS(x):= \frac{ \sum_{i\in H}\sum_{j\in W}V(\mathcal{N}_s(\widetilde{x})_{i,j}) }{ \sum_{i\in H}\sum_{j\in W}\mathbbm{1}\!\left(V(\mathcal{N}_s(\widetilde{x})_{i,j})\neq 0\right) }.
$$
Detection is by thresholding:
$$
\text{declare OOD if } ds(t)<\epsilon,\qquad ds\in\{BLS,ODS\}.
$$
This is a confidence-based detector on extracted semantic foreground rather than a density model over semantic features [2302.11019].

The COCO experiment is not standard detection. It is a COCO-derived image classification problem with label space
$$
\mathcal{Y}=\{\text{cup, umbrella, orange, toaster, broccoli, banana, vase, zebra, kite}\}.
$$
The paper uses COCO train and test images “which can be classified” with those labels, but does not fully specify how multi-object scenes are filtered or assigned [2302.11019]. The motivation is cross-dataset generalization to VizWiz. A ResNet18 classifier trained on COCO reaches $74.33\%$ accuracy on COCO test and $68.14\%$ on clear VizWiz, which the authors interpret as evidence that clear VizWiz belongs to the intended distribution $\mathcal{D}_I$ despite shifted image statistics [2302.11019].

Three COCO-relevant test cases are evaluated: in-distribution from clear VizWiz, OOD from VizWiz quality degradation, and OOD from COCO using test images whose class labels are not in $\mathcal{Y}$, specifically $\{\text{traffic light, stop sign, parking meter, fire hydrant}\}$ [2302.11019]. On OOD-from-VizWiz, the segmentation-based method achieves AUROC 98.52 with $ODS$ and 96.75 with $BLS$. For clear VizWiz, AUROC is less than $50\%$, interpreted as an inability to distinguish clear VizWiz from test COCO and therefore desirable under the semantics-aware definition. For OOD-from-COCO, Algorithm 2 with either score achieves AUROC higher by at least $28.86\%$ than existing detectors [2302.11019].

This reframing matters because standard COCO-OOD practice often marks any external dataset or unseen category as OOD. The semantic approach instead separates benign nuisance/domain shift from semantic novelty. External images with the same relevant semantics may be in-distribution, while familiar-domain images with semantically absent labels are OOD [2302.11019].

## 5. Cross-dataset, open-set, and multi-label COCO-centered protocols

A second major usage of COCO-OOD arises when COCO is one side of a cross-dataset open-set or OOD protocol. Proto-OOD evaluates the common Pascal VOC as ID / MS-COCO as OOD setting, following SIREN and VOS-style protocols [2409.05466]. The method uses RT-DETR with ResNet-50 and augments it with a project head, a similarity module, and a negative embedding generator. Projected embeddings $r \in \mathbb{R}^{n \times d}$ are compared against momentum-updated class prototypes
$$
p_c = (\alpha p_c + (1-\alpha) r_c),
$$
and OOD detection is based on similarity between input features and category prototypes [2409.05466].

At test time, the energy is defined as
$$
E(g;r^*,p)= H(r^*,p)\cdot s^*=exp(Cosine\_similarity(r^*,p))\cdot s^*,
$$
with thresholding
$$
g(r^*,p)=\left\{ \begin{aligned} & 1,\quad \quad E(g;r^*,p)>=\gamma\\ & 0,\quad \quad E(g;r^*,p)<\gamma \end{aligned} \right.
$$
although the reduction over multiple prototypes is not fully specified in the paper text [2409.05466]. In the Pascal VOC as ID / MS-COCO as OOD setting under protocol_A, Proto-OOD reports FPR95 $20.98 \pm 1.5$, AUROC $95.23 \pm 0.2$, and mAP 64.32, outperforming the best reported baselines in both FPR95 and AUROC [2409.05466]. The paper also proposes protocol_B, which keeps only the top $K$ predictions per ID image with $K$ equal to the number of annotated objects, together with NMS, to reduce metric contamination from inaccurate ID detections [2409.05466]. This protocol critique is influential for interpreting VOC-vs-COCO OOD numbers.

In the multi-label recognition setting, COOD uses COCO as the in-distribution dataset and external datasets as OOD. The method constructs a base label set $\mathcal{B}$, positive concepts $\mathcal{P}$ from GPT-4-generated features, superclasses, and commonly associated items, and negative concepts $\mathcal{N}$ mined by similarity thresholding from a lexical candidate pool [2411.13578]. For a concept set $\mathcal{S}$ and image $I$, the score uses top-$k$ mean similarity
$$
\mu_k(\mathcal{S}, I) = \frac{1}{k}\sum_{i=1}^k \mathrm{sim}(h,e_i),
$$
and the main scoring function is
$$
S_{\text{ID}(I) = w_{\mathcal{B}\,\mu_k(\mathcal{B},I) - w_{\mathcal{P}\,\mu_k(\mathcal{P},I) - w_{\mathcal{N}\,\mu_k(\mathcal{N},I).
$$
In the ablation form, this is written as
$$
S_{\text{OOD}(I) = \left[\mu_k(\mathcal{B},I)-\mu_k(\mathcal{P},I)\right] + \left[\mu_k(\mathcal{B},I)-\mu_k(\mathcal{N},I)\right].
$$
The paper notes a threshold $\gamma$ for deciding ID versus OOD, but does not fully specify its selection procedure in the visible main text [2411.13578].

For COCO as ID and ImageNet22K as OOD, the best reported COOD result is FPR95 20.37 and AUROC 95.07 with ViT-based CLIP [2411.13578]. For COCO as ID and Textures as OOD, COOD reports FPR95 10.53 and AUROC 97.17 with ResNet-based CLIP, and 21.63 / 94.53 with ViT-based CLIP [2411.13578]. These experiments show that COCO-OOD can also designate a multi-label semantic recognition problem rather than object detection.

## 6. Relation to robustness, complementary datasets, and context-aware OOD

Several COCO-adjacent resources broaden the notion of OOD without being identical to canonical COCO-OOD benchmarks. COCO_OI merges COCO with selected OpenImages data under a shared ontology and contains 1,418,978 training bounding boxes over 380,111 images, plus 41,893 validation bounding boxes over 18,299 images [2206.11473]. ObjectNet_D contains 5,875 bounding boxes across 5,875 images with one annotated object per image from 29 categories in common with COCO [2206.11473]. Evaluating COCO-trained detectors on ObjectNet_D exposes large performance drops: EfficientDet falls from AP 51.2 on COCO to 23.9 on ObjectNet_D, and DetectoRS falls from 49.1 to 20.3 [2206.11473]. The paper interprets lower performance on ObjectNet_D as evidence that object detectors, like recognition models, suffer from domain shift [2206.11473]. This is a closed-set cross-dataset generalization benchmark rather than unknown-object OOD detection.

CD-COCO is a corruption-style robustness benchmark rather than a semantic OOD benchmark. It contains more than 123K images with 80 object classes and standard COCO-like splits of 95K train, 5K validation, and 23K test [2311.06976]. The dataset applies 10 distortion types, including Compression artifact, Contrast changing, Gaussian noise, Global motion blur, Global defocus blur, Fog, Rain, Local Backlight, Local defocus blur, and Local motion blur, with reported image counts for each type [2311.06976]. Distortion generation uses COCO masks, scene semantics, and MiDaS depth. Because the paper does not provide detector result tables, CD-COCO functions primarily as a data resource for non-i.i.d. robustness studies [2311.06976].

A more conceptual extension appears in “Divide and Conquer: Object Co-occurrence Helps Mitigate Simplicity Bias in OOD Detection,” which argues that near-OOD detection benefits from modeling object co-occurrence rather than relying only on entangled image features [2605.07821]. Although it does not evaluate on a benchmark named COCO-OOD or use MS COCO as in-distribution data, its methodological emphasis is highly relevant to COCO-style scenes, where contextual anomalies often take the form of implausible object combinations. OCO represents images by slot-frequency patterns
$$
\mathcal{F}_i = \left\{ \left(c, \sum_{k=1}^{K}\mathbb{I}(c_i^{(k)}=c)\right) \,\bigg|\, c \in \mathcal{U}_i \right\},
$$
divides test images into single, typical, and atypical co-occurrence scenarios, and scores them differently [2605.07821]. A plausible implication is that future COCO-OOD benchmarks may incorporate scene-level contextual relations more explicitly, rather than treating OOD as purely marginal feature shift or unseen labels.

## 7. Conceptual tensions, limitations, and benchmark-design implications

The COCO-OOD literature contains a persistent conceptual tension between distributional and semantic definitions of OOD. COCO-O measures natural shift while preserving the original 80-class COCO ontology [2307.12730]. Semantic-information-based work argues that this is insufficient when domain shift preserves task-relevant semantics and low-level familiarity can mask semantic irrelevance [2302.11019]. COOD frames OOD in a multi-label semantic embedding space but uses external datasets rather than novel label combinations within COCO [2411.13578]. Proto-OOD studies VOC-vs-COCO unknown-object detection and highlights how evaluation protocols alter FPR95 and AUROC substantially [2409.05466].

These differences produce several benchmark-design consequences. First, whether OOD should include same-semantics domain shift is not settled. The semantic-information framework explicitly argues that clear VizWiz images with the same relevant labels should not be treated as OOD even though they differ from COCO in image statistics [2302.11019]. Second, whether OOD should be category-based, dataset-based, or context-based remains application dependent. COCO-O is category-preserving and shift-based [2307.12730], whereas Proto-OOD is category-open and cross-dataset [2409.05466]. Third, robustness gains from large pretrained models may partly reflect exposure to similar imagery during pretraining, complicating the interpretation of “algorithmic robustness” on benchmarks like COCO-O [2307.12730].

Reproducibility is uneven across formulations. The COCO-derived classification subset in the semantic-information paper is underspecified in its filtering of multi-object scenes [2302.11019]. Proto-OOD does not fully enumerate the exact COCO subset used as OOD and inherits some setup from SIREN [2409.05466]. COOD does not fully specify the exact COCO split, prompt templates, or negative-mining details in the visible text [2411.13578]. These limitations do not invalidate the findings, but they do constrain direct cross-paper comparability.

Taken together, the literature indicates that COCO-OOD is not a monolithic benchmark but a research area organized around COCO as a common reference distribution. Its central questions are whether a model trained on COCO remains reliable when images undergo natural domain shift, when semantics change but superficial context remains familiar, when unknown objects appear, or when multi-label semantic structure becomes more complex than the training label space. The strongest unifying theme is that COCO’s contextual richness, annotation bias, and multi-object structure make it especially suitable for studying the gap between i.i.d. performance and deployment robustness [2307.12730][2302.11019].

Source: https://www.emergentmind.com/topics/coco-ood