COOOL Benchmark for Out-of-Label Hazards

• COOOL is an evaluation-only benchmark for autonomous driving that uses dashcam videos to assess out-of-label hazard detection and driver reaction.
• It challenges systems to differentiate true hazards from nuisance objects using zero-shot and open-set language evaluation protocols.
• Extensions like COOOLER and 2COOOL further enhance semantic evaluation by introducing hazard captioning and incident report generation.

Searching arXiv for papers on the COOOL benchmark and related extensions. COOOL, short for Challenge Of Out-Of-Label, is an evaluation-only benchmark for autonomous driving that targets the novelty problem: hazards that are critical for safe driving but are not included in a model’s closed set of training labels. In the autonomous-driving literature, “out-of-label” denotes rare, safety-critical objects or events such as exotic animals, debris, smoke, or other unusual hazards that fall outside standard taxonomies. COOOL uses dashcam video to evaluate whether a system can determine when the driver reacts, identify which objects are hazardous, and name or describe those hazards under zero-shot or open-set conditions rather than conventional closed-set detection assumptions (AlShami et al., 2024, Picek et al., 2024).

1. Conceptual scope and problem setting

COOOL was introduced to probe a persistent weakness of autonomous-driving perception systems: strong performance on common, in-distribution categories does not guarantee robustness to rare, long-tail, or previously unseen hazards. The benchmark explicitly frames this as out-of-label hazard understanding, aligning it with open-set recognition and out-of-distribution reasoning while grounding the problem in traffic safety rather than generic novelty scoring (AlShami et al., 2024, Picek et al., 2024).

A central distinction in COOOL is between hazards and nuisance objects. Hazards are objects or events that could require driver reaction, including pedestrians, vehicles, animals, and debris; nuisance objects are present in the scene but should not trigger a reaction. This makes the task more demanding than generic anomaly detection, because the system must not only recognize novelty but also decide which object is safety-relevant in context (AlShami et al., 2024).

The benchmark is also defined by an evaluation-centric philosophy. The original release is described as serving exclusively as an evaluation benchmark, with no official train/validation/test split and no recommendation that participants train directly on benchmark labels. This design forces reliance on external pretraining, heuristics, or zero-shot reasoning rather than conventional supervised fitting to an in-domain training set (AlShami et al., 2024, Picek et al., 2024).

Later work extends this core idea rather than replacing it. COOOLER adds explicit open-set natural-language hazard annotations and embedding-based evaluation, while 2COOOL extends the task family from hazard identification and captioning to incident report generation, emphasizing temporally ordered, human-interpretable narratives about safety-critical events (Shriram et al., 18 Apr 2025, Yokoi et al., 14 Oct 2025).

2. Dataset composition and annotation structure

The original benchmark is described as containing over 200 collections of dashcam-oriented videos curated to emphasize rare or novel hazards and corresponding driver reactions. The videos are high-definition and dashcam-perspective; no additional sensor modalities such as LiDAR or radar are specified in the benchmark description, so the benchmark is vision-focused (AlShami et al., 2024).

Published analyses provide more granular statistics. One competition paper describes COOOL as comprising over 200 dashcam videos collected worldwide, with each frame annotated by bounding boxes and unique object IDs, enabling trajectory analysis. Another paper, in the context of the COOOOLER extension, reports 200 short video clips, approximately 10 seconds per clip at 30 fps, and states that the dataset includes annotated bounding boxes for all objects within each scene (Picek et al., 2024, Shriram et al., 18 Apr 2025).

The annotation density is a defining property. The benchmark description reports over 100,000 vehicle annotations, approximately 40,000 animal annotations, and over 10,000 hazard annotations smaller than $50 \times 50$ pixels, stressing early detection of distant or low-resolution hazards. The majority of frames contain 1–5 potential hazards, the maximum observed is 26 potential hazards in a single frame, and at driver-reaction onset there are 1–18 potential hazards, with the majority of reactions occurring when 3 potential hazards are present (AlShami et al., 2024).

The public challenge format exposes structured object annotations. One solution paper states that the organizers provide an annotations_public file with bounding boxes and tracking IDs for two categories: challenge_object and traffic_scene, each with unique tracking IDs. Another benchmark description states that every object is labeled in every frame and that “objects of interest” and “potential driving hazards” are explicitly annotated, together with a Tag system that adds information about vehicle movement and driver decisions (Duong et al., 27 Jan 2025, AlShami et al., 2024).

Hazard semantics in the original benchmark are intentionally open-ended. Hazards are defined broadly as “an animal, person, debris or anything that is blocking the road,” and anomalous examples include debris, animals such as cats, and erratic pedestrian behaviors. This breadth is integral to the benchmark’s open-world character: the task is not merely to recover labels from a fixed ontology, but to recognize and describe unforeseen hazards in traffic scenes (Shriram et al., 18 Apr 2025).

3. Official tasks and evaluation protocols

The current COOOL challenge is organized around three tasks. Systems are given the raw video together with a list of bounding boxes, and must determine when the driver has started to react to a hazard, identify which object or objects in the scene are hazardous, and provide the name or description of the hazard (AlShami et al., 2024, Picek et al., 2024).

A common formalization in the competition literature defines three per-frame or per-sequence accuracies and a macro average: $$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$

$$A_{\text{detection}} = \frac{1}{N} \sum_{i=1}^{N} \frac{|H_i \cap \hat{H}_i|}{|H_i|},$$

$$A_{\text{classific}} = \frac{1}{N} \sum_{i=1}^{N} \frac{|C_i \cap \hat{C}_i|}{|C_i|},$$

$$A_{\text{macro}} = \frac{A_{\text{reaction}} + A_{\text{detection}} + A_{\text{classific}}}{3}.$$

Here, $r_i$ and $\hat{r}_i$ denote ground-truth and predicted reaction labels, $H_i$ and $\hat{H}_i$ the ground-truth and predicted hazardous objects, and $C_i$ and $$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$0 the ground-truth and predicted hazard classes or captions (Picek et al., 2024).

A related description of the benchmark emphasizes an over-prediction penalty in hazard identification. For each evaluation unit $$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$1, with ground-truth hazard set $$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$2, predicted set $$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$3, and correct matches $$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$4, the per-unit accuracy is

$$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$5

The same normalized form is used for hazard naming, with correctness determined by whether the ground-truth class label string appears in the predicted description (AlShami et al., 2024).

The captioning protocol is unusually restrictive. One challenge analysis reports that COOOL’s scoring uses only the first 35 characters of a caption and checks for their presence in ground-truth annotations. Another notes that public scoring relied on black-box evaluation using string subsequence matching to an undisclosed target set, reflecting an open-set language requirement but making offline validation difficult (Duong et al., 27 Jan 2025, Shriram et al., 18 Apr 2025).

The benchmark is also unusual in its split regime. No official training, validation, or test split was released to participants, and no ground-truth annotations were made available for development. One competition report states that the public leaderboard uses 8% of the data for online feedback and the private leaderboard the remaining 92%, while another notes that participants were limited to two final submissions on the private leaderboard (Duong et al., 27 Jan 2025, Picek et al., 2024).

4. Baselines, competition methods, and empirical behavior

Organizer baselines reported in solution papers are deliberately simple. For driver reaction detection, baseline variants infer reaction from bounding-box motion, such as object-center distance trends over time or a velocity-sign heuristic. For hazard identification, the baseline predicts the hazardous object as the bounding box closest to the image center. For captioning, the baseline crops the detected hazard and applies a CLIP-based captioner or CLIP-Interrogator, yielding generic phrases matched to the benchmark’s string-based metric (Picek et al., 2024, Duong et al., 27 Jan 2025).

A detailed 2024 challenge submission reports a three-part zero-shot pipeline: kernel-based change point detection on normalized total bounding-box area and optical flow for driver reaction detection; hazard identification via proximity heuristics plus pre-trained ViT classification and whitelist filtering; and hazard captioning via the MOLMO vision-LLM with category-focused prompts. The best reported subtask scores were 0.908 public / 0.829 private for driver reaction using mean-position ensembling, 0.799 public / 0.570 private for hazard identification with classification and trajectory-size filtering, and 0.239 public / 0.162 private for captioning with the MOLMO category prompt. The best final macro accuracy on the private leaderboard was 0.51772, corresponding to 2nd place out of 32 teams, with a reported 33% relative error reduction versus the organizers’ baseline (Picek et al., 2024).

Another competition study reports a label-free pipeline built from unsupervised reaction detection over speed and sound anomalies, heuristic weak classifiers for hazard localization, differentially private noisy weighting to mitigate overconfidence, and VLM-based caption aggregation. In that report, the organizers’ baseline scored 0.25560 on the private leaderboard and 0.25681 on the public leaderboard, while the full pipeline with all caption models ensembled reached 0.57261 private and 0.78453 public, and the authors report achieving first place overall on both public and private leaderboards (Duong et al., 27 Jan 2025).

These results illuminate several benchmark-specific regularities. First, driver reaction detection is often tractable with simple temporal signals derived from box dynamics, motion, or audio, whereas hazard captioning is disproportionately difficult because the official metric rewards concise string overlap under open-set uncertainty. Second, the nearest-to-center heuristic is weak on its own but remains a persistent baseline because many hazards are ego-relevant and near the camera axis. Third, category-oriented prompting can outperform fluent sentence generation when the evaluator emphasizes substring overlap rather than semantic richness (Picek et al., 2024, Duong et al., 27 Jan 2025).

A plausible implication is that COOOL simultaneously tests perception and metric alignment. Systems that reason well semantically may still underperform if they produce descriptions that are correct in meaning but mismatched to the benchmark’s constrained string-matching protocol.

5. COOOLER: benchmark extension with open-set ground truth

A major limitation of the original benchmark, as identified by later work, is that it does not supply open-set ground-truth hazard captions suitable for direct semantic evaluation. To address this, the paper “Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety” introduced COOOLER, an extension that denoises and extends COOOL with explicit natural-language hazard annotations (Shriram et al., 18 Apr 2025).

COOOLER preprocesses all 200 videos with NAFNet for denoising and deblurring and removes watermarks. It then adds two human annotations per video: a Hazardous Object Label such as “Cat” and a Hazard Description such as “A cat crossing the road.” The extension is intended for zero-shot evaluation, not closed-set training, and the paper does not define train/validation/test splits (Shriram et al., 18 Apr 2025).

The central methodological change is an embedding-based evaluation protocol. Predicted hazard descriptions are compared to ground truth using cosine similarity between text embeddings,

$$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$6

with a success threshold of 0.80. Two aggregate metrics are defined: BESM (Balanced Extremes Similarity Metric), which averages per-video maximum and minimum similarity to penalize inconsistency, and SAM (Similarity Average Metric), which averages all similarity scores to capture overall semantic alignment (Shriram et al., 18 Apr 2025).

COOOLER also formalizes a zero-shot localization workflow. A multi-agent pipeline uses VLMs for scene understanding, an LLM to consolidate and rank hazards into sets such as RHS, AES, COS, and AOS, and CLIP to match predicted hazard labels and descriptions to bounding-box snippets. For localization, snippets with width < 175 px, height < 175 px, or area $$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$7 px$$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$8 are ignored, and the top 10th percentile of snippet–label similarities per row in the CLIP similarity matrix are flagged as detections. The paper gives the standard IoU formula,

$$A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),$$9

but does not report IoU-based metrics in its results (Shriram et al., 18 Apr 2025).

Using OmniVLM, ViLA, GPT-4o-mini, and CLIP, the reported pipeline achieved BESM = 0.3922 and SAM = 0.3922 on COOOOLER. The paper juxtaposes this with the original challenge leaderboard, where the average among the top 15 was 0.386 and the best score 0.573 under a different scoring system, underscoring the difficulty of out-of-label hazard detection and captioning (Shriram et al., 18 Apr 2025).

COOOLER changes the benchmark’s epistemic character. Instead of black-box string validation, it enables explicit semantic evaluation and reproducible offline analysis. This suggests a shift from purely challenge-oriented scoring toward a more inspectable open-set benchmark for vision-language hazard reasoning.

6. 2COOOL: incident reports and hierarchical narrative evaluation

The benchmark family was extended further by 2COOOL, which reframes the task from hazard identification and naming to incident report generation. In 2COOOL, the required output is a human-interpretable incident report: a coherent, temporally ordered narrative that identifies whether a hazard or accident occurred and when it started, describes pre-incident context and post-incident outcomes, names and tracks involved entities, assesses severity and ego involvement, and explains causal factors consistent with the video evidence (Yokoi et al., 14 Oct 2025).

The 2COOOL dataset integrates COOOL, DADA, and Nexar, covering a broad spectrum of dashcam scenarios with emphasis on unusual and safety-critical events. Each clip is recorded at 30 fps and lasts from a few seconds to several tens of seconds. The annotation schema includes event type (hazard, accident, or no incident), crash severity, ego-vehicle involvement, counts of involved entities by type, time-to-hazard, detailed captions before and after the incident, and driver gaze information with gaze-based captions. These annotations are generated by vision-LLMs and verified by human validators (Yokoi et al., 14 Oct 2025).

The challenge defines a core task of incident report generation together with auxiliary subtasks: time-to-incident start estimation, incident detection, incident recognition, ego and other party involvement, crash severity assessment, caption-before-incident generation, and caption-after-incident generation. The official leaderboard ranks submissions by the average of CIDEr-D, METEOR, and SPICE computed on submitted reports (Yokoi et al., 14 Oct 2025).

A representative method, “Hierarchical Reasoning with Vision-LLMs for Incident Reports from Dashcam Videos”, decomposes the problem into frame-level captioning, incident frame detection, and fine-grained narrative generation. On the official 2COOOL open leaderboard, the method ranked 2nd among 29 teams and achieved the best CIDEr-D score. The paper reports the score for team “Turing Inc.” as SPICE 0.1822, METEOR 0.2605, CIDEr-D 0.0067, and Final Score 0.1498, while the top team “NotSoDeep” achieved Final 0.1518 with a lower CIDEr-D of 0.0040 (Yokoi et al., 14 Oct 2025).

2COOOL preserves the benchmark’s open-world orientation but shifts the evaluation target from sparse labels to narrative coherence, causal attribution, and temporally grounded explanation. A plausible implication is that it operationalizes a broader conception of hazard understanding: not only what is dangerous, but what happened, when, and why.

7. Positioning, limitations, and research directions

COOOL is typically positioned against autonomous-driving datasets such as KITTI, nuScenes, and Cityscapes, which are rich and well annotated for common categories but centered on predefined labels and typical scenarios. It is also compared with Lost and Found, which is strong for small road obstacles but lacks temporal reasoning and open-set language evaluation, and DHPR, which provides hazard descriptions and bounding boxes but uses single images rather than videos. The benchmark description additionally contrasts COOOL with LAVA, which is not designed for out-of-label hazard identification across open-ended categories (Shriram et al., 18 Apr 2025, AlShami et al., 2024).

Its distinctiveness lies in the conjunction of four properties. First, it targets out-of-label hazards rather than only known categories. Second, it uses video rather than isolated images, making temporal context available. Third, it couples hazard understanding to driver reaction. Fourth, later extensions evaluate open-set language outputs, first via constrained caption scoring and then via semantic embedding similarity or full incident narratives (AlShami et al., 2024, Shriram et al., 18 Apr 2025, Yokoi et al., 14 Oct 2025).

The published literature also identifies recurring limitations. The original benchmark is evaluation-only and does not define official train/validation/test splits, which complicates methodological comparison and creates dependence on external pretraining choices. Public documentation is incomplete with respect to certain schema details, scene distributions, and matching rules. The official caption metric is brittle because it emphasizes short strings and substring matching. Studies on COOOLER report that small hazards often fall below snippet-quality thresholds, harming recall, and that some videos yield no final predictions after VLM and LLM filtering. Work on 2COOOL reports persistent weaknesses in spatial grounding, such as left-right confusions, and errors in temporal ordering under subtle or occluded cues (AlShami et al., 2024, Shriram et al., 18 Apr 2025, Yokoi et al., 14 Oct 2025).

Future directions are already embedded in the benchmark family’s evolution. The original benchmark paper explicitly mentions anomaly detection, open-set recognition, open-vocabulary recognition, domain adaptation, low-resolution hazard recognition, forecasting hazards from image sequences, and avoidance maneuvers as compatible or future directions. COOOOLER points toward reproducible open-set semantic evaluation, while 2COOOL extends evaluation to causally structured incident narratives (AlShami et al., 2024, Shriram et al., 18 Apr 2025, Yokoi et al., 14 Oct 2025).

In that sense, COOOL is best understood not as a conventional dataset for closed-set object detection, but as a benchmark family for open-world hazard understanding in autonomous driving. Its technical significance comes from forcing systems to bridge object localization, temporal inference, zero-shot semantics, and safety-oriented explanation under long-tail conditions that standard autonomous-driving taxonomies leave unresolved.