---
title: COOOL Benchmark for Out-of-Label Hazards
url: https://www.emergentmind.com/topics/coool-benchmark
type: topic
---

# COOOL Benchmark for Out-of-Label Hazards

Searching arXiv for recent papers on the COOOL benchmark and related extensions.
COOOL, short for **Challenge Of Out-Of-Label**, is an evaluation-only benchmark for autonomous driving that targets the novelty problem: hazards that are critical for safe driving but are not included in a model’s closed set of training labels. In the autonomous-driving literature, “out-of-label” denotes rare, safety-critical objects or events such as exotic animals, debris, smoke, or other unusual hazards that fall outside standard taxonomies. COOOL uses dashcam video to evaluate whether a system can determine when the driver reacts, identify which objects are hazardous, and name or describe those hazards under zero-shot or open-set conditions rather than conventional closed-set detection assumptions [2412.05462][2412.19944].

## 1. Conceptual scope and problem setting

COOOL was introduced to probe a persistent weakness of autonomous-driving perception systems: strong performance on common, in-distribution categories does not guarantee robustness to rare, long-tail, or previously unseen hazards. The benchmark explicitly frames this as **out-of-label hazard understanding**, aligning it with open-set recognition and out-of-distribution reasoning while grounding the problem in traffic safety rather than generic novelty scoring [2412.05462][2412.19944].

A central distinction in COOOL is between **hazards** and **nuisance objects**. Hazards are objects or events that could require driver reaction, including pedestrians, vehicles, animals, and debris; nuisance objects are present in the scene but should not trigger a reaction. This makes the task more demanding than generic anomaly detection, because the system must not only recognize novelty but also decide which object is safety-relevant in context [2412.05462].

The benchmark is also defined by an evaluation-centric philosophy. The original release is described as serving **exclusively as an evaluation benchmark**, with no official train/validation/test split and no recommendation that participants train directly on benchmark labels. This design forces reliance on external pretraining, heuristics, or zero-shot reasoning rather than conventional supervised fitting to an in-domain training set [2412.05462][2412.19944].

Later work extends this core idea rather than replacing it. **COOOLER** adds explicit open-set natural-language hazard annotations and embedding-based evaluation, while **2COOOL** extends the task family from hazard identification and captioning to incident report generation, emphasizing temporally ordered, human-interpretable narratives about safety-critical events [2504.13399][2510.12190].

## 2. Dataset composition and annotation structure

The original benchmark is described as containing **over 200 collections of dashcam-oriented videos** curated to emphasize rare or novel hazards and corresponding driver reactions. The videos are high-definition and dashcam-perspective; no additional sensor modalities such as LiDAR or radar are specified in the benchmark description, so the benchmark is vision-focused [2412.05462].

Published analyses provide more granular statistics. One competition paper describes COOOL as comprising **over 200 dashcam videos collected worldwide**, with each frame annotated by bounding boxes and unique object IDs, enabling trajectory analysis. Another paper, in the context of the COOOOLER extension, reports **200 short video clips**, approximately **10 seconds per clip at 30 fps**, and states that the dataset includes annotated bounding boxes for all objects within each scene [2412.19944][2504.13399].

The annotation density is a defining property. The benchmark description reports **over 100,000 vehicle annotations**, **approximately 40,000 animal annotations**, and **over 10,000 hazard annotations smaller than $50 \times 50$ pixels**, stressing early detection of distant or low-resolution hazards. The majority of frames contain **1–5 potential hazards**, the maximum observed is **26 potential hazards in a single frame**, and at driver-reaction onset there are **1–18 potential hazards**, with the majority of reactions occurring when **3 potential hazards** are present [2412.05462].

The public challenge format exposes structured object annotations. One solution paper states that the organizers provide an `annotations_public` file with bounding boxes and tracking IDs for two categories: **challenge_object** and **traffic_scene**, each with unique tracking IDs. Another benchmark description states that every object is labeled in every frame and that “objects of interest” and “potential driving hazards” are explicitly annotated, together with a Tag system that adds information about vehicle movement and driver decisions [2501.16037][2412.05462].

Hazard semantics in the original benchmark are intentionally open-ended. Hazards are defined broadly as “an animal, person, debris or anything that is blocking the road,” and anomalous examples include debris, animals such as cats, and erratic pedestrian behaviors. This breadth is integral to the benchmark’s open-world character: the task is not merely to recover labels from a fixed ontology, but to recognize and describe unforeseen hazards in traffic scenes [2504.13399].

## 3. Official tasks and evaluation protocols

The current COOOL challenge is organized around three tasks. Systems are given the raw video together with a list of bounding boxes, and must **determine when the driver has started to react to a hazard**, **identify which object or objects in the scene are hazardous**, and **provide the name or description of the hazard** [2412.05462][2412.19944].

A common formalization in the competition literature defines three per-frame or per-sequence accuracies and a macro average:
\[
A_{\text{reaction}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(r_i = \hat{r}_i),
\]
\[
A_{\text{detection}} = \frac{1}{N} \sum_{i=1}^{N} \frac{|H_i \cap \hat{H}_i|}{|H_i|},
\]
\[
A_{\text{classific}} = \frac{1}{N} \sum_{i=1}^{N} \frac{|C_i \cap \hat{C}_i|}{|C_i|},
\]
\[
A_{\text{macro}} = \frac{A_{\text{reaction}} + A_{\text{detection}} + A_{\text{classific}}}{3}.
\]
Here, $r_i$ and $\hat{r}_i$ denote ground-truth and predicted reaction labels, $H_i$ and $\hat{H}_i$ the ground-truth and predicted hazardous objects, and $C_i$ and $\hat{C}_i$ the ground-truth and predicted hazard classes or captions [2412.19944].

A related description of the benchmark emphasizes an **over-prediction penalty** in hazard identification. For each evaluation unit $u$, with ground-truth hazard set $G_u$, predicted set $P_u$, and correct matches $C_u$, the per-unit accuracy is
\[
A_u = \frac{C_u}{\max(|G_u|, |P_u|)}.
\]
The same normalized form is used for hazard naming, with correctness determined by whether the ground-truth class label string appears in the predicted description [2412.05462].

The captioning protocol is unusually restrictive. One challenge analysis reports that COOOL’s scoring uses only the **first 35 characters** of a caption and checks for their presence in ground-truth annotations. Another notes that public scoring relied on **black-box evaluation using string subsequence matching to an undisclosed target set**, reflecting an open-set language requirement but making offline validation difficult [2501.16037][2504.13399].

The benchmark is also unusual in its split regime. No official training, validation, or test split was released to participants, and no ground-truth annotations were made available for development. One competition report states that the public leaderboard uses **8%** of the data for online feedback and the private leaderboard the remaining **92%**, while another notes that participants were limited to **two final submissions** on the private leaderboard [2501.16037][2412.19944].

## 4. Baselines, competition methods, and empirical behavior

Organizer baselines reported in solution papers are deliberately simple. For driver reaction detection, baseline variants infer reaction from bounding-box motion, such as object-center distance trends over time or a velocity-sign heuristic. For hazard identification, the baseline predicts the hazardous object as the bounding box closest to the image center. For captioning, the baseline crops the detected hazard and applies a CLIP-based captioner or **CLIP-Interrogator**, yielding generic phrases matched to the benchmark’s string-based metric [2412.19944][2501.16037].

A detailed 2024 challenge submission reports a three-part zero-shot pipeline: kernel-based change point detection on normalized total bounding-box area and optical flow for driver reaction detection; hazard identification via proximity heuristics plus pre-trained ViT classification and whitelist filtering; and hazard captioning via the **MOLMO** vision-language model with category-focused prompts. The best reported subtask scores were **0.908 public / 0.829 private** for driver reaction using mean-position ensembling, **0.799 public / 0.570 private** for hazard identification with classification and trajectory-size filtering, and **0.239 public / 0.162 private** for captioning with the MOLMO category prompt. The best final macro accuracy on the private leaderboard was **0.51772**, corresponding to **2nd place out of 32 teams**, with a reported **33% relative error reduction** versus the organizers’ baseline [2412.19944].

Another competition study reports a label-free pipeline built from unsupervised reaction detection over speed and sound anomalies, heuristic weak classifiers for hazard localization, differentially private noisy weighting to mitigate overconfidence, and VLM-based caption aggregation. In that report, the organizers’ baseline scored **0.25560** on the private leaderboard and **0.25681** on the public leaderboard, while the full pipeline with all caption models ensembled reached **0.57261 private** and **0.78453 public**, and the authors report achieving **first place overall** on both public and private leaderboards [2501.16037].

These results illuminate several benchmark-specific regularities. First, driver reaction detection is often tractable with simple temporal signals derived from box dynamics, motion, or audio, whereas hazard captioning is disproportionately difficult because the official metric rewards concise string overlap under open-set uncertainty. Second, the nearest-to-center heuristic is weak on its own but remains a persistent baseline because many hazards are ego-relevant and near the camera axis. Third, category-oriented prompting can outperform fluent sentence generation when the evaluator emphasizes substring overlap rather than semantic richness [2412.19944][2501.16037].

A plausible implication is that COOOL simultaneously tests perception and **metric alignment**. Systems that reason well semantically may still underperform if they produce descriptions that are correct in meaning but mismatched to the benchmark’s constrained string-matching protocol.

## 5. COOOLER: benchmark extension with open-set ground truth

A major limitation of the original benchmark, as identified by later work, is that it does not supply open-set ground-truth hazard captions suitable for direct semantic evaluation. To address this, the paper **“Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety”** introduced **COOOLER**, an extension that denoises and extends COOOL with explicit natural-language hazard annotations [2504.13399].

COOOLER preprocesses all **200 videos** with **NAFNet** for denoising and deblurring and removes watermarks. It then adds two human annotations per video: a **Hazardous Object Label** such as “Cat” and a **Hazard Description** such as “A cat crossing the road.” The extension is intended for **zero-shot evaluation, not closed-set training**, and the paper does not define train/validation/test splits [2504.13399].

The central methodological change is an embedding-based evaluation protocol. Predicted hazard descriptions are compared to ground truth using cosine similarity between text embeddings,
\[
\mathrm{cos}(\theta) = \frac{\mathbf{x} \cdot \mathbf{y}}{\|\mathbf{x}\|\,\|\mathbf{y}\|},
\]
with a success threshold of **0.80**. Two aggregate metrics are defined: **BESM (Balanced Extremes Similarity Metric)**, which averages per-video maximum and minimum similarity to penalize inconsistency, and **SAM (Similarity Average Metric)**, which averages all similarity scores to capture overall semantic alignment [2504.13399].

COOOLER also formalizes a zero-shot localization workflow. A multi-agent pipeline uses VLMs for scene understanding, an LLM to consolidate and rank hazards into sets such as **RHS**, **AES**, **COS**, and **AOS**, and **CLIP** to match predicted hazard labels and descriptions to bounding-box snippets. For localization, snippets with **width < 175 px**, **height < 175 px**, or **area $\le 35{,}000$ px$^2$** are ignored, and the top **10th percentile** of snippet–label similarities per row in the CLIP similarity matrix are flagged as detections. The paper gives the standard IoU formula,
\[
\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},
\]
but does not report IoU-based metrics in its results [2504.13399].

Using **OmniVLM**, **ViLA**, **GPT-4o-mini**, and **CLIP**, the reported pipeline achieved **BESM = 0.3922** and **SAM = 0.3922** on COOOOLER. The paper juxtaposes this with the original challenge leaderboard, where the average among the top 15 was **0.386** and the best score **0.573** under a different scoring system, underscoring the difficulty of out-of-label hazard detection and captioning [2504.13399].

COOOLER changes the benchmark’s epistemic character. Instead of black-box string validation, it enables explicit semantic evaluation and reproducible offline analysis. This suggests a shift from purely challenge-oriented scoring toward a more inspectable open-set benchmark for vision-language hazard reasoning.

## 6. 2COOOL: incident reports and hierarchical narrative evaluation

The benchmark family was extended further by **2COOOL**, which reframes the task from hazard identification and naming to **incident report generation**. In 2COOOL, the required output is a **human-interpretable incident report**: a coherent, temporally ordered narrative that identifies whether a hazard or accident occurred and when it started, describes pre-incident context and post-incident outcomes, names and tracks involved entities, assesses severity and ego involvement, and explains causal factors consistent with the video evidence [2510.12190].

The 2COOOL dataset integrates **COOOL, DADA, and Nexar**, covering a broad spectrum of dashcam scenarios with emphasis on unusual and safety-critical events. Each clip is recorded at **30 fps** and lasts from a few seconds to several tens of seconds. The annotation schema includes **event type** (hazard, accident, or no incident), **crash severity**, **ego-vehicle involvement**, counts of involved entities by type, **time-to-hazard**, detailed captions before and after the incident, and **driver gaze information** with gaze-based captions. These annotations are generated by vision-language models and verified by human validators [2510.12190].

The challenge defines a core task of incident report generation together with auxiliary subtasks: time-to-incident start estimation, incident detection, incident recognition, ego and other party involvement, crash severity assessment, caption-before-incident generation, and caption-after-incident generation. The official leaderboard ranks submissions by the average of **CIDEr-D**, **METEOR**, and **SPICE** computed on submitted reports [2510.12190].

A representative method, **“Hierarchical Reasoning with Vision-Language Models for Incident Reports from Dashcam Videos”**, decomposes the problem into frame-level captioning, incident frame detection, and fine-grained narrative generation. On the official 2COOOL open leaderboard, the method ranked **2nd among 29 teams** and achieved the **best CIDEr-D** score. The paper reports the score for team **“Turing Inc.”** as **SPICE 0.1822**, **METEOR 0.2605**, **CIDEr-D 0.0067**, and **Final Score 0.1498**, while the top team **“NotSoDeep”** achieved **Final 0.1518** with a lower **CIDEr-D of 0.0040** [2510.12190].

2COOOL preserves the benchmark’s open-world orientation but shifts the evaluation target from sparse labels to narrative coherence, causal attribution, and temporally grounded explanation. A plausible implication is that it operationalizes a broader conception of hazard understanding: not only *what* is dangerous, but *what happened*, *when*, and *why*.

## 7. Positioning, limitations, and research directions

COOOL is typically positioned against autonomous-driving datasets such as **KITTI**, **nuScenes**, and **Cityscapes**, which are rich and well annotated for common categories but centered on predefined labels and typical scenarios. It is also compared with **Lost and Found**, which is strong for small road obstacles but lacks temporal reasoning and open-set language evaluation, and **DHPR**, which provides hazard descriptions and bounding boxes but uses single images rather than videos. The benchmark description additionally contrasts COOOL with **LAVA**, which is not designed for out-of-label hazard identification across open-ended categories [2504.13399][2412.05462].

Its distinctiveness lies in the conjunction of four properties. First, it targets **out-of-label hazards** rather than only known categories. Second, it uses **video** rather than isolated images, making temporal context available. Third, it couples hazard understanding to **driver reaction**. Fourth, later extensions evaluate **open-set language outputs**, first via constrained caption scoring and then via semantic embedding similarity or full incident narratives [2412.05462][2504.13399][2510.12190].

The published literature also identifies recurring limitations. The original benchmark is evaluation-only and does not define official train/validation/test splits, which complicates methodological comparison and creates dependence on external pretraining choices. Public documentation is incomplete with respect to certain schema details, scene distributions, and matching rules. The official caption metric is brittle because it emphasizes short strings and substring matching. Studies on COOOLER report that small hazards often fall below snippet-quality thresholds, harming recall, and that some videos yield no final predictions after VLM and LLM filtering. Work on 2COOOL reports persistent weaknesses in spatial grounding, such as left-right confusions, and errors in temporal ordering under subtle or occluded cues [2412.05462][2504.13399][2510.12190].

Future directions are already embedded in the benchmark family’s evolution. The original benchmark paper explicitly mentions anomaly detection, open-set recognition, open-vocabulary recognition, domain adaptation, low-resolution hazard recognition, forecasting hazards from image sequences, and avoidance maneuvers as compatible or future directions. COOOOLER points toward reproducible open-set semantic evaluation, while 2COOOL extends evaluation to causally structured incident narratives [2412.05462][2504.13399][2510.12190].

In that sense, COOOL is best understood not as a conventional dataset for closed-set object detection, but as a benchmark family for **open-world hazard understanding in autonomous driving**. Its technical significance comes from forcing systems to bridge object localization, temporal inference, zero-shot semantics, and safety-oriented explanation under long-tail conditions that standard autonomous-driving taxonomies leave unresolved.

Source: https://www.emergentmind.com/topics/coool-benchmark