COCO-Search18: Visual Search Benchmark

• COCO-Search18 is a goal-directed human-attention benchmark that shifts focus from free-viewing saliency to task-specific visual search using strict image constraints.
• It compiles 6,202 images, 18 target categories, and nearly 300,000 fixations to analyze both target-present and target-absent scenarios in natural scenes.
• Models benchmarked on COCO-Search18 leverage sequential fixation data and metrics like scanpath ratio and DTW to advance gaze prediction and inverse reinforcement learning research.

COCO-Search18 is a goal-directed human-attention benchmark built on Microsoft COCO for modeling visual search rather than free-viewing saliency. It was introduced to train and evaluate models of where people look when given an explicit object-search task, and was described as the largest dataset of high-quality search fixations in existence at the time of publication. The dataset comprises 10 participants, 18 target-object categories, 6,202 images, and about 300,000 recorded fixations, with model training and evaluation commonly performed on correctly answered target-present trials and on fixation sequences truncated at the first fixation on the target (Yang et al., 2020).

1. Dataset composition and stimulus constraints

COCO-Search18 covers 18 target categories: bottle, bowl, car, chair, clock, cup, fork, keyboard, knife, laptop, microwave, mouse, oven, potted-plant, sink, stop-sign, toilet, and TV. The benchmark was designed around both target-present and target-absent search, with 3,101 target-present images and 3,101 target-absent images drawn from the 2014 MS-COCO dataset. This construction makes the benchmark suitable for studying both successful target guidance and distractor-driven behavior during search (Samiei et al., 2022).

The target-present images were selected under explicit stimulus constraints. They were required to contain exactly one unoccluded instance of the target, with bounding-box area between 1% and 10% of the image, not appearing in the central $1/25$th of the frame, and with an overall aspect ratio between 1.2 and 2.0. Candidate images were further screened by a patch-based classifier to eliminate heavily occluded or ambiguous examples, and clocks showing exclusively digital faces were dropped from the clock class. Target-absent images were chosen to contain no target but at least two sibling objects, discouraging trivial scene-type rejection strategies (Yang et al., 2020).

These constraints are methodologically important because they suppress degenerate shortcuts. A plausible implication is that COCO-Search18 was engineered not merely as an eye-tracking corpus, but as a controlled testbed for top-down search policies under semantically structured scene context.

2. Experimental protocol and recorded behavior

The data were collected from ten student volunteers aged 18–30 with normal or corrected vision. Each participant completed six search sessions over six days; each session lasted roughly two hours and included approximately 500 target-present and 500 target-absent trials randomly interleaved. Within a session, participants searched all images for a single target category. On each trial, a central fixation dot appeared, participants pressed a button to begin, the search image appeared, and they responded “yes/no” via a gamepad; one summary specifies that the images were resized to $1680 \times 1050$ with zero padding during presentation (Yang et al., 2020, Samiei et al., 2022).

Eye movements were recorded at 1,000 Hz using an SR Research EyeLink 1000 tower-mounted tracker. The accompanying JSON records include fixation coordinates, onset time, duration, and reaction time. Across all subjects, 299,037 fixations were recorded. For the original scanpath-prediction setting, only target-present trials with correct responses were retained, and only fixations prior to the first fixation landing on the target were used, yielding 100,232 target-present search fixations for model training and testing (Yang et al., 2020).

This protocol distinguishes COCO-Search18 from conventional saliency datasets. The recorded behavior is explicitly task-conditioned, sequential, and decision-linked, rather than a passive accumulation of fixation density under unconstrained viewing.

3. Annotation structure and derived representations

COCO bounding boxes serve as the ground-truth target locations. A fixation $(x,y)$ is labeled as a target fixation if it falls inside the target bounding box; otherwise it is treated as a distractor fixation. The public release is organized by target category and includes images/, fixations.json, bounding_boxes.json, and splits.txt. The images/ directory contains the original COCO images resized to $320 \times 512$, while fixations.json stores, for each trial and subject, fields such as subject_id, image_id, trial_type, correct, and an array of fixation records $\{x, y, time\_ms\}$. The provided split file specifies a random 70/10/20 train/val/test partition for reproducible experiments (Yang et al., 2020).

To support state representations for sequential search models, the original work also precomputed panoptic segmentation maps for 80 “thing” and 54 “stuff” COCO categories using a Panoptic-FPN with a ResNet-50-FPN backbone. These 134 belief channels form coarse peripheral object-category beliefs and are used in the Dynamic Contextual Belief representation of the inverse-reinforcement-learning model (Yang et al., 2020).

Subsequent work extended the annotation space beyond fixation sequences. For three target categories—bottle, bowl, and car—Samiei and Clark manually labeled the exact masks of human-fixated objects, including both targets and distractors, using the COCO Annotator tool. At training time, these masks are reassembled into a one-hot encode in which channel 0 is the target, channels $1$–$K$ are distractors with $K \leq 20$, and there is an additional background class (Samiei et al., 2022).

4. Benchmark tasks and evaluation methodology

The original benchmark emphasized scanpath prediction and search efficiency. Models were compared up to $T=6$ fixations beyond the initial central fixation using metrics including Target-Fixation Probability AUC, Probability Mismatch, Scanpath Ratio, Sequence Score, and MultiMatch. Among the explicitly defined quantities, the scanpath ratio is

$$\mathrm{Scanpath\ Ratio} = \frac{D(\mathrm{start}, \mathrm{target})} {\sum_{i=1}^{\tau}\|p_i-p_{i-1}\|},$$

where $1680 \times 1050$0 is the index of the fixation on the target; higher values indicate more direct guidance. The same work also described reaction time to target as

$1680 \times 1050$1

This metric suite makes COCO-Search18 unusually rich for evaluating not only where a model looks, but how efficiently and in what order it searches (Yang et al., 2020).

Later work repurposed the dataset for target-conditioned fixation-density prediction. Samiei and Clark constructed ground-truth fixation density maps by placing a Gaussian with $1680 \times 1050$2 px at each fixation coordinate, summing the Gaussians, and normalizing the map to integrate to one. Their evaluation used AUC-Judd, AUC-Borji, shuffled AUC, NSS, KLD, CC, SIM, and IG. For example, Kullback–Leibler divergence was defined as

$1680 \times 1050$3

and normalized scanpath saliency as the average of a zero-mean, unit-standard-deviation normalized prediction map over fixation locations (Samiei et al., 2022).

Cross-task evaluation has added scanpath-alignment measures that do not require gaze supervision at training time. In EVA, COCO-Search18 is used only for evaluation, and alignment is measured with Dynamic Time Warping, ScanMatch, NSS, and AUC. The paper defines

$1680 \times 1050$4

for fixation sequences $1680 \times 1050$5 and $1680 \times 1050$6, where smaller DTW indicates more similar trajectories, and

$1680 \times 1050$7

where larger NSS indicates that predicted fixations fall on human fixations more consistently (Pan et al., 28 Mar 2026).

5. Modeling results and benchmark uses

The dataset was introduced together with an inverse reinforcement learning model for search. That model uses Dynamic Contextual Beliefs,

$1680 \times 1050$8

and is trained with an adversarial imitation-learning objective. On the held-out test set, the IRL model outperformed baseline models in both scanpath similarity and search efficiency: TFP-AUC 4.509, Probability Mismatch 0.987, Scanpath Ratio 0.826, Sequence Score 0.422, and MultiMatch shape/direction/length/position 0.886/0.695/0.866/0.885. The recovered reward maps exhibited target-dependent patterns of object prioritization, such as concentration on laptop and monitor edges for mouse search and on road surfaces and other vehicles for car search (Yang et al., 2020).

Samiei and Clark used COCO-Search18 for two distinct tasks. In fixation-density prediction, they used only the 3,101 target-present images, discarded the first fixation at the image center and incorrect trials, and produced a split of 1,826 train, 324 validation, and 324 test task-image pairs, doubling the training set to 3,652 with horizontal flips. Their two-stream encoder–decoder, trained on images at $1680 \times 1050$9 and target patches at $(x,y)$0, achieved AUC-Judd $(x,y)$1, AUC-Borji $(x,y)$2, sAUC $(x,y)$3, NSS $(x,y)$4, KLD $(x,y)$5, CC $(x,y)$6, SIM $(x,y)$7, and IG $(x,y)$8. In object-level prediction, their fine-tuned Mask-R-CNN achieved overall average $(x,y)$9, $320 \times 512$0, and $320 \times 512$1 over bottle, bowl, and car (Samiei et al., 2022).

Pan et al. used COCO-Search18 as a zero-gaze-supervision transfer benchmark. EVA is trained only on COCO image-classification labels, with no COCO-Search18 fixation or search-task data seen during training, and is then run for $320 \times 512$2 glimpses per image on the COCO-Search18 subset. Under this protocol, EVA-Mobile (pretrained) obtained COCO-Search18 accuracy 16.63%, DTW 280.29, ScanMatch 0.313, NSS 0.307, and AUC 0.714. Gazeformer, which uses gaze supervision, remained stronger overall with DTW 168.39, ScanMatch 0.571, NSS 1.961, and AUC 0.800, but EVA-Mobile was reported as the strongest among methods without any gaze supervision (Pan et al., 28 Mar 2026).

The abstract of SIGN reports a different use of the benchmark: COCO-Search18 is used as a dataset with individual-level fixation patterns collected during search to test whether aggregate gaze-time prediction can also recover plausible gaze patterns. The abstract states that SIGN improves gaze duration prediction over deep-learning benchmarks on both AdGaze3500 and COCO-Search18, and can deliver plausible gaze patterns corresponding to empirical fixation patterns in COCO-Search18 (Ye et al., 29 Jan 2025).

6. Scientific significance and recurrent misunderstandings

COCO-Search18 occupies a distinct position in the visual-attention literature because it was created explicitly to move beyond free-viewing saliency. The original motivation was that free-viewing predictions do not generalize to goal-directed behavior, whereas visual search demands a greater amount of top-down attention control. Subsequent work on fixation-density prediction, distractor segmentation, hard-attention transfer, and aggregate gaze-time modeling has preserved that emphasis on task-conditioned attention, even when the modeling objective shifts from scanpath imitation to saliency prediction or image classification (Yang et al., 2020, Samiei et al., 2022, Pan et al., 28 Mar 2026).

A recurrent misunderstanding is to treat COCO-Search18 as merely another pixel-level saliency benchmark. The dataset does support fixation density maps, but its primary structure is sequential and target-specific: each trial pairs an image with a designated target category, records fixations and response behavior during search, distinguishes target and distractor fixations, and admits metrics for efficiency, alignment, and policy structure. A plausible implication is that models evaluated on COCO-Search18 should be interpreted not only through static heatmaps, but through the dynamics of evidence acquisition and stopping behavior.

Another common reduction is to view the benchmark solely through gaze supervision. The EVA results show that COCO-Search18 can also function as a transfer benchmark for models trained without gaze labels or task-specific fine-tuning, while the original IRL work shows that the same dataset supports explicit reward learning over contextual belief states. Taken together, these uses suggest that COCO-Search18 has become a general benchmark for studying how goals, contexts, uncertainty, and learned policies shape human-like search in natural scenes (Yang et al., 2020, Pan et al., 28 Mar 2026).