COCO-OOD: A COCO-Based OOD Benchmark

• COCO-OOD is an umbrella term defining COCO-anchored out-of-distribution evaluations across natural shifts, semantic redefinitions, and multi-label zero-shot recognition.
• It leverages COCO’s rich multi-object scenes and contextual complexity to assess robustness by preserving the original 80-category ontology while exploring diverse OOD settings.
• Robustness is measured using metrics like mAP drop and Effective Robustness, highlighting the impact of backbone choices, large-scale pretraining, and novel evaluation protocols.

Searching arXiv for COCO-OOD-related papers to ground the article in current literature. COCO-OOD denotes a family of COCO-centered out-of-distribution evaluation settings rather than a single universally standardized task. In the literature, the term spans at least three distinct formulations: natural distribution-shift benchmarking for COCO-trained object detectors, semantic OOD detection on COCO-derived classification tasks, and cross-dataset or external-dataset OOD evaluation using COCO as either the in-distribution or out-of-distribution source. This plurality is explicit in the relevant research. “COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts” defines a COCO-compatible benchmark for object detection under six natural shifts while preserving the original 80 COCO categories (Mao et al., 2023). “Using Semantic Information for Defining and Detecting OOD Inputs” uses COCO as a case study to argue that OOD should be defined relative to intended task semantics rather than the empirical training-image distribution (Kaur et al., 2023). “COOD: Concept-based Zero-shot OOD Detection” uses MS-COCO as the in-distribution dataset in a multi-label zero-shot OOD recognition setting with external OOD datasets (Liu et al., 2024). This suggests that “COCO-OOD” is best treated as an umbrella term covering multiple benchmark philosophies, all anchored to COCO but differing in task, ontology, and the meaning assigned to “out-of-distribution.”

1. Historical emergence and scope

The modern use of COCO as an OOD reference point arose from the central role of MS-COCO in object detection and segmentation research. One line of work treats COCO as the in-distribution training and evaluation domain and asks how detectors behave under natural but non-i.i.d. test inputs. In this formulation, COCO-O preserves the original 80 COCO categories and evaluation protocol while changing the image distribution through web-collected natural shifts such as weather, painting, handmake, cartoon, tattoo, and sketch (Mao et al., 2023). A second line of work treats COCO as a semantically biased training source and argues that standard OOD detectors inherit these biases when they operate only on the empirical training distribution rather than task-relevant semantics (Kaur et al., 2023). A third line uses COCO in multi-label recognition, where the challenge is zero-shot OOD detection in the presence of label co-occurrence and semantic dependence, evaluated against external datasets such as Filtered ImageNet22K and Textures (Liu et al., 2024).

This diversity is reinforced by adjacent work that is COCO-related but not identical in objective. “Complementary datasets to COCO for object detection” introduces COCO_OI and ObjectNet_D to study generalization beyond standard COCO under natural dataset shift, but it does not formalize explicit OOD detection scores or open-set rejection (Borji, 2022). “CD-COCO: A Versatile Complex Distorted COCO Database for Scene-Context-Aware Computer Vision” defines a COCO-derived robustness benchmark under global and local photorealistic distortions rather than semantic OOD (Beghdadi et al., 2023). “Proto-OOD: Enhancing OOD Object Detection with Prototype Feature Similarity” is directly relevant to COCO-OOD in the widespread VOC-vs-COCO unknown-object setting, but it does not introduce a standalone benchmark named COCO-OOD (Chen et al., 2024).

A plausible implication is that the phrase “COCO-OOD” should not be interpreted as designating one fixed dataset split. It instead refers to a set of COCO-anchored robustness and OOD formulations spanning detection, recognition, semantic shift, covariate shift, and cross-dataset evaluation.

2. Major formulations of COCO-OOD

The literature distinguishes several concrete formulations, each with different semantics, metrics, and failure modes.

Formulation	Core idea	Representative paper
Natural distribution shift for detection	Train on standard COCO, test on naturally shifted but COCO-labeled images	COCO-O (Mao et al., 2023)
Semantic OOD on COCO-derived classification	Define OOD relative to task semantics rather than image-distribution likelihood	Semantic-information framework (Kaur et al., 2023)
Multi-label zero-shot OOD recognition	Use MS-COCO as ID, external datasets as OOD, no retraining	COOD (Liu et al., 2024)

In the natural-shift formulation, the detector is still solving the standard COCO task, and the ontology remains unchanged. COCO-O contains 6,782 images and 26,624 annotated boxes spanning the original 80 COCO categories, with six disjoint OOD test domains: Sketch, Weather, Cartoon, Painting, Tattoo, and Handmake (Mao et al., 2023). OOD here means a change in image distribution, not a change in label space.

In the semantic formulation, OOD is not equivalent to “unlikely under the training set.” The intended distribution $\mathcal{D}_I$ is introduced so that

$$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$

and the associated intended set of inputs is

$$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$

Under this view, samples with the same task-relevant semantic classes but different nuisance factors are not OOD, whereas samples lacking the relevant semantics are OOD even if their appearance resembles training data (Kaur et al., 2023).

In the multi-label zero-shot formulation, COCO’s complexity lies in co-occurring labels. COOD uses MS-COCO as the in-distribution dataset and evaluates OOD against Filtered ImageNet22K and Textures. The detector is zero-shot and training-free, using pretrained CLIP plus concept-based label expansion (Liu et al., 2024). OOD is external to COCO, but the difficulty arises from the multi-label structure of the COCO in-distribution side.

A common misconception is that any evaluation involving COCO and a distribution shift constitutes the same benchmark. The papers show otherwise: a held-out COCO-category detection test, a COCO-derived semantic classification experiment, and a COCO-vs-external multi-label recognition benchmark answer different scientific questions.

3. COCO-O as a natural OOD benchmark for object detection

COCO-O is the most direct benchmark paper for COCO-trained object detectors under natural distribution shifts. It is test-only and domain-structured, with image and object counts per domain reported as follows: Sketch: 992 images, 3,707 objects; Weather: 961 images, 4,509 objects; Cartoon: 1,996 images, 8,774 objects; Painting: 954 images, 4,879 objects; Tattoo: 918 images, 1,489 objects; Handmake: 961 images, 3,266 objects (Mao et al., 2023). Class mapping is unchanged: COCO-O adopts the original 80 COCO categories directly.

Construction is based on web search over combinations of domain keywords and COCO object names. To reduce iconic single-object bias, the search process mimics COCO’s multi-object style by combining multiple class names, such as “cartoon + dog + car,” and uses multiple search engines when required (Mao et al., 2023). The benchmark therefore changes image distribution while maintaining compatibility with COCO-trained detectors and standard COCO evaluation software.

Evaluation uses standard COCO mean Average Precision averaged over IoU thresholds from 50% to 95%, computed per OOD domain and averaged across the six domains as the overall COCO-O score (Mao et al., 2023). The key robustness metric is Effective Robustness:

$ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$

where $mAP_{id}(\cdot)$ is COCO mAP, $mAP_{ood}(\cdot)$ is COCO-O mAP, and 0.45 is the fitted slope from linear regression over 11 classic detectors (Mao et al., 2023). Positive ER indicates better OOD performance than expected from in-distribution accuracy alone.

The benchmark is substantially harder than standard COCO evaluation. On Faster R-CNN with ResNet-50, COCO-O yields a reported $55.7\%$ relative performance drop, corresponding to a decrease from 37.4 COCO mAP to 16.4 COCO-O mAP (Mao et al., 2023). The paper also states that COCO-O has a much larger Fréchet Inception Distance from clean COCO than COCO-C, 132 vs. 41 (Mao et al., 2023).

Controlled studies show that backbone choice dominates neck and head modifications for robustness. On Faster R-CNN, ResNet-50 gives COCO 37.4 / COCO-O 16.4 / ER -0.41, while PVTv2-B2 gives 45.6 / 24.9 / +4.33 (Mao et al., 2023). On RetinaNet, ResNet-50 gives 36.5 / 16.6 / +0.18, while PVTv2-B2 reaches 44.6 / 25.2 / +5.10 (Mao et al., 2023). By contrast, neck variants such as PAFPN or NAS-FPN and head modifications such as SABL, FSAF, or FreeAnchor yield limited or negative ER changes (Mao et al., 2023).

The paper also reports that end-to-end detection transformers do not automatically improve robustness. DETR has COCO 42.0 but COCO-O only 17.1 with ER -1.82, while Deformable DETR has COCO 44.5 and COCO-O 18.5 with ER -1.49 (Mao et al., 2023). Strong augmentations help, with MixUp singled out as especially important: removing MixUp in YOLOX-S reduces COCO-O mAP from 19.8 to 17.7 (Mao et al., 2023). Large-scale pretraining is beneficial but heterogeneous: ImageNet-21K and “Sup_RSB” improve ER, whereas SwAV hurts robustness in the reported Faster R-CNN study (Mao et al., 2023).

Foundation and vision-LLMs dominate the top end. In the larger SOTA benchmark, EVA achieves COCO 64.2 and COCO-O 57.8 with ER +28.86; GLIP-L gets 48.0 and +24.89; DETA gets 48.5 and +20.15 (Mao et al., 2023). This suggests that the strongest robustness gains are linked less to conventional architectural tweaks than to large-scale pretraining and external data.

4. Semantic redefinitions of OOD on COCO

The COCO case study in “Using Semantic Information for Defining and Detecting OOD Inputs” shifts attention from benchmark construction to the definition of OOD itself. The central claim is that detectors tied to the training distribution $\mathcal{D}$ can inherit dataset bias and become impermeable to inputs outside the training distribution but with the same semantic information (Kaur et al., 2023). This is especially salient for COCO because its images are rich in context, background, co-occurrence structure, and annotation bias.

The paper formalizes a distinction between the empirical training distribution $\mathcal{D}$ and an intended distribution $\mathcal{D}_I$, then introduces the semantically relevant part of an image through maps

$$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$0

where $$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$1 is the oracle classifier and $$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$2 denotes an empty image (Kaur et al., 2023). If semantic extraction yields a distribution $$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$3 close to $$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$4 in Kolmogorov distance, then an estimator of intended distribution exists in the asymptotic sense stated by the theorem in the paper (Kaur et al., 2023). The purpose is not a complete detection theory, but support for estimating intended support after removing irrelevant image content.

The practical COCO instantiation uses semantic segmentation. The estimated intended set is written as

$$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$5

where

$$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$6

Each pixel receives a probability vector over the $$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$7 semantic labels plus one background label, and the foreground is the set of pixels assigned to one of the class labels in $$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$8 (Kaur et al., 2023).

For a per-pixel class-probability vector $$\operatorname{supp}(\mathcal{D}) \subsetneq \operatorname{supp}(\mathcal{D}_I) \subset \mathcal{X},$$9, the paper defines

$$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$0

The baseline semantic score is

$$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$1

ODIN is adapted similarly, yielding the foreground score

$$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$2

Detection is by thresholding:

$$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$3

This is a confidence-based detector on extracted semantic foreground rather than a density model over semantic features (Kaur et al., 2023).

The COCO experiment is not standard detection. It is a COCO-derived image classification problem with label space

$$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$4

The paper uses COCO train and test images “which can be classified” with those labels, but does not fully specify how multi-object scenes are filtered or assigned (Kaur et al., 2023). The motivation is cross-dataset generalization to VizWiz. A ResNet18 classifier trained on COCO reaches $$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$5 accuracy on COCO test and $$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$6 on clear VizWiz, which the authors interpret as evidence that clear VizWiz belongs to the intended distribution $$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$7 despite shifted image statistics (Kaur et al., 2023).

Three COCO-relevant test cases are evaluated: in-distribution from clear VizWiz, OOD from VizWiz quality degradation, and OOD from COCO using test images whose class labels are not in $$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$8, specifically $$\mathcal{X}_I := \{x \in \mathcal{X} : \mathcal{D}_I(x) > \epsilon\}.$$9 (Kaur et al., 2023). On OOD-from-VizWiz, the segmentation-based method achieves AUROC 98.52 with $ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$0 and 96.75 with $ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$1. For clear VizWiz, AUROC is less than $ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$2, interpreted as an inability to distinguish clear VizWiz from test COCO and therefore desirable under the semantics-aware definition. For OOD-from-COCO, Algorithm 2 with either score achieves AUROC higher by at least $ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$3 than existing detectors (Kaur et al., 2023).

This reframing matters because standard COCO-OOD practice often marks any external dataset or unseen category as OOD. The semantic approach instead separates benign nuisance/domain shift from semantic novelty. External images with the same relevant semantics may be in-distribution, while familiar-domain images with semantically absent labels are OOD (Kaur et al., 2023).

5. Cross-dataset, open-set, and multi-label COCO-centered protocols

A second major usage of COCO-OOD arises when COCO is one side of a cross-dataset open-set or OOD protocol. Proto-OOD evaluates the common Pascal VOC as ID / MS-COCO as OOD setting, following SIREN and VOS-style protocols (Chen et al., 2024). The method uses RT-DETR with ResNet-50 and augments it with a project head, a similarity module, and a negative embedding generator. Projected embeddings $ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$4 are compared against momentum-updated class prototypes

$ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$5

and OOD detection is based on similarity between input features and category prototypes (Chen et al., 2024).

At test time, the energy is defined as

$ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$6

with thresholding

$ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$7

although the reduction over multiple prototypes is not fully specified in the paper text (Chen et al., 2024). In the Pascal VOC as ID / MS-COCO as OOD setting under protocol_A, Proto-OOD reports FPR95 $ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$8, AUROC $ER(f) = mAP_{ood}(f)-0.45 \times mAP_{id}(f),$9, and mAP 64.32, outperforming the best reported baselines in both FPR95 and AUROC (Chen et al., 2024). The paper also proposes protocol_B, which keeps only the top $mAP_{id}(\cdot)$0 predictions per ID image with $mAP_{id}(\cdot)$1 equal to the number of annotated objects, together with NMS, to reduce metric contamination from inaccurate ID detections (Chen et al., 2024). This protocol critique is influential for interpreting VOC-vs-COCO OOD numbers.

In the multi-label recognition setting, COOD uses COCO as the in-distribution dataset and external datasets as OOD. The method constructs a base label set $mAP_{id}(\cdot)$2, positive concepts $mAP_{id}(\cdot)$3 from GPT-4-generated features, superclasses, and commonly associated items, and negative concepts $mAP_{id}(\cdot)$4 mined by similarity thresholding from a lexical candidate pool (Liu et al., 2024). For a concept set $mAP_{id}(\cdot)$5 and image $mAP_{id}(\cdot)$6, the score uses top-$mAP_{id}(\cdot)$7 mean similarity

$mAP_{id}(\cdot)$8

and the main scoring function is

$mAP_{id}(\cdot)$9

In the ablation form, this is written as

$mAP_{ood}(\cdot)$0

The paper notes a threshold $mAP_{ood}(\cdot)$1 for deciding ID versus OOD, but does not fully specify its selection procedure in the visible main text (Liu et al., 2024).

For COCO as ID and ImageNet22K as OOD, the best reported COOD result is FPR95 20.37 and AUROC 95.07 with ViT-based CLIP (Liu et al., 2024). For COCO as ID and Textures as OOD, COOD reports FPR95 10.53 and AUROC 97.17 with ResNet-based CLIP, and 21.63 / 94.53 with ViT-based CLIP (Liu et al., 2024). These experiments show that COCO-OOD can also designate a multi-label semantic recognition problem rather than object detection.

6. Relation to robustness, complementary datasets, and context-aware OOD

Several COCO-adjacent resources broaden the notion of OOD without being identical to canonical COCO-OOD benchmarks. COCO_OI merges COCO with selected OpenImages data under a shared ontology and contains 1,418,978 training bounding boxes over 380,111 images, plus 41,893 validation bounding boxes over 18,299 images (Borji, 2022). ObjectNet_D contains 5,875 bounding boxes across 5,875 images with one annotated object per image from 29 categories in common with COCO (Borji, 2022). Evaluating COCO-trained detectors on ObjectNet_D exposes large performance drops: EfficientDet falls from AP 51.2 on COCO to 23.9 on ObjectNet_D, and DetectoRS falls from 49.1 to 20.3 (Borji, 2022). The paper interprets lower performance on ObjectNet_D as evidence that object detectors, like recognition models, suffer from domain shift (Borji, 2022). This is a closed-set cross-dataset generalization benchmark rather than unknown-object OOD detection.

CD-COCO is a corruption-style robustness benchmark rather than a semantic OOD benchmark. It contains more than 123K images with 80 object classes and standard COCO-like splits of 95K train, 5K validation, and 23K test (Beghdadi et al., 2023). The dataset applies 10 distortion types, including Compression artifact, Contrast changing, Gaussian noise, Global motion blur, Global defocus blur, Fog, Rain, Local Backlight, Local defocus blur, and Local motion blur, with reported image counts for each type (Beghdadi et al., 2023). Distortion generation uses COCO masks, scene semantics, and MiDaS depth. Because the paper does not provide detector result tables, CD-COCO functions primarily as a data resource for non-i.i.d. robustness studies (Beghdadi et al., 2023).

A more conceptual extension appears in “Divide and Conquer: Object Co-occurrence Helps Mitigate Simplicity Bias in OOD Detection,” which argues that near-OOD detection benefits from modeling object co-occurrence rather than relying only on entangled image features (Dai et al., 8 May 2026). Although it does not evaluate on a benchmark named COCO-OOD or use MS COCO as in-distribution data, its methodological emphasis is highly relevant to COCO-style scenes, where contextual anomalies often take the form of implausible object combinations. OCO represents images by slot-frequency patterns

$mAP_{ood}(\cdot)$2

divides test images into single, typical, and atypical co-occurrence scenarios, and scores them differently (Dai et al., 8 May 2026). A plausible implication is that future COCO-OOD benchmarks may incorporate scene-level contextual relations more explicitly, rather than treating OOD as purely marginal feature shift or unseen labels.

7. Conceptual tensions, limitations, and benchmark-design implications

The COCO-OOD literature contains a persistent conceptual tension between distributional and semantic definitions of OOD. COCO-O measures natural shift while preserving the original 80-class COCO ontology (Mao et al., 2023). Semantic-information-based work argues that this is insufficient when domain shift preserves task-relevant semantics and low-level familiarity can mask semantic irrelevance (Kaur et al., 2023). COOD frames OOD in a multi-label semantic embedding space but uses external datasets rather than novel label combinations within COCO (Liu et al., 2024). Proto-OOD studies VOC-vs-COCO unknown-object detection and highlights how evaluation protocols alter FPR95 and AUROC substantially (Chen et al., 2024).

These differences produce several benchmark-design consequences. First, whether OOD should include same-semantics domain shift is not settled. The semantic-information framework explicitly argues that clear VizWiz images with the same relevant labels should not be treated as OOD even though they differ from COCO in image statistics (Kaur et al., 2023). Second, whether OOD should be category-based, dataset-based, or context-based remains application dependent. COCO-O is category-preserving and shift-based (Mao et al., 2023), whereas Proto-OOD is category-open and cross-dataset (Chen et al., 2024). Third, robustness gains from large pretrained models may partly reflect exposure to similar imagery during pretraining, complicating the interpretation of “algorithmic robustness” on benchmarks like COCO-O (Mao et al., 2023).

Reproducibility is uneven across formulations. The COCO-derived classification subset in the semantic-information paper is underspecified in its filtering of multi-object scenes (Kaur et al., 2023). Proto-OOD does not fully enumerate the exact COCO subset used as OOD and inherits some setup from SIREN (Chen et al., 2024). COOD does not fully specify the exact COCO split, prompt templates, or negative-mining details in the visible text (Liu et al., 2024). These limitations do not invalidate the findings, but they do constrain direct cross-paper comparability.

Taken together, the literature indicates that COCO-OOD is not a monolithic benchmark but a research area organized around COCO as a common reference distribution. Its central questions are whether a model trained on COCO remains reliable when images undergo natural domain shift, when semantics change but superficial context remains familiar, when unknown objects appear, or when multi-label semantic structure becomes more complex than the training label space. The strongest unifying theme is that COCO’s contextual richness, annotation bias, and multi-object structure make it especially suitable for studying the gap between i.i.d. performance and deployment robustness (Mao et al., 2023, Kaur et al., 2023).