---
title: 'CompliVision: Hazard Compliance Benchmark'
url: https://www.emergentmind.com/topics/complivision-dataset
type: topic
---

# CompliVision: Hazard Compliance Benchmark

Searching arXiv for the CompliVision paper and closely related safety/compliance datasets to ground the article.
CompliVision is a general-purpose hazard dataset and benchmark for rule-based compliance assessment in vision–language systems, introduced in "General Hazard Detection" [2605.23304]. It reframes hazard detection from fixed hazard-category recognition to evaluation of whether a scene complies with or violates a natural-language safety rule grounded in real regulations and standards. In this formulation, a hazard is a rule violation in context, and the central task is: given an image \(I\) and a rule \(r_i\), predict a compliance label together with a natural-language explanation supported by visual evidence [2605.23304].

## 1. Rule-based formulation of hazard detection

CompliVision is motivated by three limitations of conventional hazard detection systems: noisy and sparse data, dynamic and context-dependent rules, and poor generalisation to novel hazards [2605.23304]. Existing systems are described as primarily detecting objects or coarse hazard categories and as assuming fixed, enumerated hazard types with many labelled examples per hazard. CompliVision instead treats hazard detection as rule-based compliance assessment.

The key design move is the decoupling of hazard concepts from image categories. Safety requirements are represented as language-based rules derived from regulations such as OSHA and Road Safety Road Rules 2017, and each image is evaluated against multiple textual rules. For every image–rule pair, the label is one of **Complied**, **Violated**, or **Not Applicable** [2605.23304]. Under this definition, a hazard is any \((I, r_i)\) for which the compliance label is:

\[
c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},
\]

with \(c_i = \text{Violated}\) implicitly corresponding to a hazard [2605.23304].

The formulation separates two conceptual subproblems. The first is **rule applicability**, namely whether a rule is relevant to the scene. The second is **compliance judgement**, namely whether the applicable rule is satisfied or violated. This structure matters because most possible rules are irrelevant to any particular image. The prevalence of the **Not Applicable** state is therefore not a defect of the dataset but a reflection of the problem structure [2605.23304].

A plausible implication is that CompliVision targets compositional reasoning rather than detector accumulation. Instead of adding a new classifier whenever a new safety requirement is introduced, the dataset encourages models to interpret arbitrary natural-language rules against scene evidence.

## 2. Dataset composition and domain coverage

CompliVision comprises **3,006 images** spanning **traffic**, **construction**, and **warehouse** environments [2605.23304]. These domains were selected because they are safety-critical, have authoritative regulations from which rules can be derived, and contain visually diverse scenes involving people, vehicles, equipment, infrastructure, and environmental conditions.

The dataset includes **15 domain-specific rule groups** and **54 total safety standards (subrules)**. For experiments, each image is paired with the **5 rule groups** for its domain, yielding **15,030 image–rule samples** [2605.23304].

| Domain | Images | Rule groups / subrules |
|---|---:|---|
| Traffic | 1,007 | 5 / 22 |
| Construction | 999 | 5 / 17 |
| Warehouse | 1,000 | 5 / 17 |

The traffic domain includes the rule groups **Driving Distraction**, **Pedestrian Crossing**, **Road Condition**, **Traffic Rules**, and **Vehicle Load**. Construction includes **Crane Use**, **Fire Risk**, **Ladder Use**, **Protective Equipment**, and **Scaffolding Risk**. Warehouse includes **Ergonomic Lifting**, **Forklift Use**, **Ladder Use**, **Protective Equipment**, and **Surface Condition** [2605.23304].

Across all image–rule pairs, the annotation counts are **3,567 Complied**, **2,393 Violated**, and **9,070 Not Applicable** [2605.23304]. The dominance of the last category follows from the multi-rule design: any single scene is ordinarily relevant to only a subset of possible rules.

Images were collected from open-source stock repositories including **Vecteezy**, **Unsplash**, and **Pexels**, using domain-specific keywords chosen so that the retrieved images naturally depict either compliance or violation. Preprocessing removes duplicates or near-duplicates using **perceptual hash**, resizes images to **336 × 336** pixels, and applies **center crop** to match VLM input, especially LLaVA [2605.23304].

The train/validation/test protocol is defined per domain. Each domain uses **100 images × 5 rules = 500 image–rule samples** for validation and the same amount for test, with the remaining images providing approximately **4,000 training samples per domain**. Aggregated across the three domains, training uses about **12k** image–rule pairs, while validation and test each contain about **1,500** samples [2605.23304].

## 3. Rule representation, annotations, and curation

For each image–rule pair \((I, r_i)\), CompliVision stores a compliance label and a natural-language explanation that identifies the supporting visual evidence [2605.23304]. Explanations mention concrete scene entities such as "helmet", "vest", "ladder", "forklift", and "pedestrian", as reflected in the paper’s word-cloud analysis. The dataset does **not** provide bounding boxes or segmentation masks; its focus is scene-level, rule-centric reasoning rather than local object detection [2605.23304].

Rules are concise natural-language statements derived from legislation and curated so that they remain visually checkable. The curation process selects relevant sections, preserves original wording as much as possible, removes clauses such as "except" and "unless" that create visually unresolved conditionals, and ensures that each rule expresses a single, clear compliance condition [2605.23304]. Example rule formulations include "The driver must not use a hand-held mobile phone while driving", "Workers must wear safety helmets in construction zones", and "Walking surfaces must be free from spills and obstacles that could cause slips or trips" [2605.23304].

Manual annotation assigns one of the three labels to every image–rule pair according to a predefined annotation schema specifying observable evidence, ambiguity handling, and criteria for **Not Applicable** [2605.23304]. These labels serve as gold labels for evaluation and for full-supervision baselines. Human-in-the-loop annotation later corrects weak samples and contributes in-context explanations used to refine prompts during active learning.

The dataset’s formal input–output definition is:

- input: image \(I\) and rules \(R = \{r_1, \dots, r_n\}\);
- output, for each pair \((I, r_i)\): a compliance label, confidence score, and optionally an explanation [2605.23304].

When explanation prompts are used, the model output is written as:

\[
(\hat{y}, \hat{E}) = M(x, \psi),
\]

where \(x = (I, r_i)\) and \(\psi\) is the prompt context, potentially augmented with human feedback [2605.23304].

A notable design property is the absence of hidden-state requirements. Rules are filtered to remain visually verifiable, excluding conditions that depend on unavailable metadata such as speed unless that condition is visually implied. This makes the dataset particularly aligned with static-image VLM evaluation, while also delimiting its scope.

## 4. Baseline framework and active learning pipeline

The paper presents a general hazard detection framework built around a **VLM backbone**, **prompt engineering**, **active learning**, **human-in-the-loop feedback**, and **LoRA-based fine-tuning** [2605.23304]. The primary instantiated backbone is **LLaVA-1.5 7B**, with comparisons against **LLaVA-Next**, **LLaMA-3.2 Vision**, and **LLaVA-CoT**.

At an abstract level, the architecture consists of a visual encoder \(f_v(I)\), a projector \(g(\cdot)\) that maps visual features into language space, a text encoder \(f_t(T)\), and an autoregressive LLM decoder \(f_{\mathrm{LLM}}(\cdot)\). The token-generation relation is given as:

\[
\text{Output tokens} = f_{\mathrm{LLM}\big( [f_t(T),\ g(f_v(I))] \big).
\]

Prompts are organised into three families. **Task-focused** prompts describe the compliance task. **Classification-focused** prompts constrain the response to fixed labels and may request applicability checks, confidence scores, and chain-of-thought reasoning. **Explanation-focused** prompts extend classification prompts by requiring a short rationale [2605.23304]. The structured-output format can be JSON-like, for example:

```json
{"compliance": "Complied"}
```

Active learning is introduced because annotating every image–rule pair is costly. The method uses two complementary prompts, **C1** and **C4**, to probe the same sample. A sample is treated as weak if predictions disagree or confidence is low; weak samples are sent to human annotators, while strong samples receive pseudo-labels [2605.23304]. Confidence for pseudo-labelled samples is calibrated using distances in embedding space to centroids computed from human-labelled samples:

\[
C = C_{\text{base} \times \alpha\Big(\hat{y}, \arg\min_i \text{dist}(\text{emb}_x, \text{emb}_{\text{centroid}_i})\Big).
\]

The feedback mechanism extends beyond labels. Human annotators can provide short explanations, which are stored in a feedback memory and retrieved for similar future samples. A stability check evaluates whether adding a candidate explanation to the global prompt context harms performance on previously seen samples; unstable feedback is discarded [2605.23304]. This mechanism functions as a prompt-level consistency control rather than a conventional retraining-only loop.

A plausible implication is that CompliVision is as much a benchmark for multimodal instruction following and explanation grounding as it is a dataset for classification. The explanations are operational: they are used to modify model behaviour, not merely to improve interpretability post hoc.

## 5. Evaluation protocol and empirical results

CompliVision evaluates models on per-rule compliance classification, explanation generation, and generalisation across traffic, construction, and warehouse domains [2605.23304]. The reported metrics are **Accuracy**, **Macro F1-score**, **Coverage**, **Inference time**, and **Annotation saved** \(P_s\), where:

\[
P_s = \frac{N_p}{N},
\]

with \(N_p\) the number of pseudo-labelled samples and \(N = N_p + N_h\) the total number of pseudo-labelled and human-labelled training samples [2605.23304].

For prompt selection with LLaVA, increasing prompt structure improves zero-shot performance. Among classification prompts, **C4** achieves **macro F1 \(\approx 0.5661\)**, **accuracy \(0.6653\)**, and full coverage. Among explanation prompts, **E1** reaches **macro F1 \(0.5655\)**, **accuracy \(0.6507\)**, and roughly **2.5 s/sample**, with richer outputs but higher inference time [2605.23304].

Across VLM architectures, **LLaVA-CoT + E1** gives the best zero-shot result, with **macro F1 \(\approx 0.6358\)** and **accuracy \(0.6793\)**, but at about **12 s/sample**. **LLaMA Vision** is reported as substantially weaker, with best macro F1 around **0.43** and incomplete coverage with some prompts [2605.23304]. The reported conclusion is that contemporary VLMs can perform non-trivial rule-based compliance reasoning zero-shot, but still struggle with fine-grained and context-dependent safety rules.

Fine-tuning and active learning produce much larger gains. On the **traffic** domain, full fine-tuning on **4,035 labelled samples** yields **macro F1 \(0.8368\)** and **accuracy \(0.8960\)**. After three active-learning rounds, using **1,054 manual labels** and **2,946 pseudo labels**, the model reaches **macro F1 \(0.8416\)** and **accuracy \(0.8920\)**, with **73.65% annotation saved** [2605.23304]. The traffic results also show that explanation feedback can improve early-round performance, with **macro F1 \(0.8428\)** and **accuracy \(0.9040\)** already at Round 0.

On the **construction** domain, full fine-tuning on **3,995 labels** achieves **F1 \(0.8375\)** and **accuracy \(0.9060\)**. The final active-learning result uses **1,362 manual** and **2,633 pseudo** labels, reaches **F1 \(0.8392\)** and **accuracy \(0.8980\)**, and saves **65.91%** of annotation [2605.23304].

On the **warehouse** domain, the full fine-tuning baseline with **4,000 labels** yields **F1 \(0.7503\)** and **accuracy \(0.8320\)**. The final active-learning configuration uses **1,246 manual** and **2,754 pseudo** labels, reaches **F1 \(0.8748\)** and **accuracy \(0.8960\)**, and saves **68.85%** annotation [2605.23304]. In this domain, active learning exceeds the full fine-tuning baseline.

Qualitative analyses reinforce the quantitative findings. t-SNE plots show progressively cleaner separation of **Complied**, **Violated**, and **Not Applicable** clusters across active-learning rounds, and example outputs demonstrate correct identification of distracted driving, improper pedestrian crossing, unsafe forklift or ladder use, and non-compliant PPE usage [2605.23304]. The paper also reports that explanations become more specific and less hallucinated when guided by human feedback.

## 6. Position within the dataset landscape, applications, and limitations

CompliVision is positioned as distinct from prior hazard and safety datasets because it is organised around explicit regulatory rules rather than object classes or coarse incident categories [2605.23304]. Construction datasets such as Construction-PPE, SH17, SODA, MOCS, and ConstructionSite10K are described as focusing on object detection or scene-level categories. Traffic datasets such as CLEVRER, DeepAccident, and WTS are described as emphasising event or trajectory reasoning, early accident prediction, or pedestrian behaviour, rather than textual traffic rules and per-rule compliance labels [2605.23304]. Industrial and fire datasets are similarly characterised as hazard-specific rather than generalised, rule-grounded safety datasets.

The paper identifies five aspects as making CompliVision unique: explicit natural-language rule grounding, multi-domain coverage, per-rule multi-label structure including **Not Applicable**, natural-language explanations, and integration with a complete VLM-plus-active-learning pipeline [2605.23304]. This suggests that CompliVision occupies a distinct position between dataset release and benchmarked methodology.

Its intended uses include training and benchmarking general hazard detection systems, research on rule applicability detection and explanation generation, development of semi-automated safety inspection assistants, and study of active learning, confidence calibration, and pseudo-labelling in multimodal settings [2605.23304]. The dataset and code are reported as available at **https://github.com/Haruharu-hub/hazard-project**.

The paper also states several limitations. CompliVision uses **static images only**, so rules requiring temporal information, such as speed or duration of distraction, can only be approximated visually. Its domain scope is restricted to **traffic**, **construction**, and **warehouse** settings. Because images come from stock repositories, there is likely selection bias toward staged and visually clear scenarios. Legal nuance is reduced by removing "except" and "unless" clauses from rules. Some regulatory aspects, such as training or certification, are not visually representable. Finally, ambiguity remains in the distinction between **Not Applicable**, **Complied**, and **Violated**, especially under occlusion, and no explicit inter-annotator agreement statistics are reported [2605.23304].

A useful comparison emerges with compliance datasets outside hazard detection. For example, DFIC addresses ICAO and ISO/IEC facial-image compliance using 26 photographic and pose requirements and balanced demographic partitions [2602.10985]. That comparison does not imply methodological equivalence, but it clarifies that CompliVision belongs to a broader class of datasets that reformulate recognition problems as compliance verification against external standards. In CompliVision, however, the standards concern scene safety rules across operational domains rather than portrait-image requirements.

CompliVision therefore marks a shift from perception-level hazard recognition to regulatory-aligned, rule-level hazard reasoning. Its central contribution is not merely that it adds images of unsafe scenes, but that it changes the unit of supervision from hazard category to image–rule compliance judgement, making natural-language rules the primary interface between regulations, annotations, and VLM reasoning [2605.23304].

Source: https://www.emergentmind.com/topics/complivision-dataset