CompliVision: Hazard Compliance Benchmark

• CompliVision is a hazard compliance benchmark that reframes detection as rule-based evaluation of visual scenes against natural-language safety rules.
• It comprises 3,006 images across traffic, construction, and warehouse domains with detailed labels of Complied, Violated, or Not Applicable.
• The dataset features natural-language explanations and an active learning pipeline to enhance fine-grained safety assessment in vision-language models.

Searching arXiv for the CompliVision paper and closely related safety/compliance datasets to ground the article. CompliVision is a general-purpose hazard dataset and benchmark for rule-based compliance assessment in vision–language systems, introduced in "General Hazard Detection" (Ng et al., 22 May 2026). It reframes hazard detection from fixed hazard-category recognition to evaluation of whether a scene complies with or violates a natural-language safety rule grounded in real regulations and standards. In this formulation, a hazard is a rule violation in context, and the central task is: given an image $I$ and a rule $r_i$, predict a compliance label together with a natural-language explanation supported by visual evidence (Ng et al., 22 May 2026).

1. Rule-based formulation of hazard detection

CompliVision is motivated by three limitations of conventional hazard detection systems: noisy and sparse data, dynamic and context-dependent rules, and poor generalisation to novel hazards (Ng et al., 22 May 2026). Existing systems are described as primarily detecting objects or coarse hazard categories and as assuming fixed, enumerated hazard types with many labelled examples per hazard. CompliVision instead treats hazard detection as rule-based compliance assessment.

The key design move is the decoupling of hazard concepts from image categories. Safety requirements are represented as language-based rules derived from regulations such as OSHA and Road Safety Road Rules 2017, and each image is evaluated against multiple textual rules. For every image–rule pair, the label is one of Complied, Violated, or Not Applicable (Ng et al., 22 May 2026). Under this definition, a hazard is any $(I, r_i)$ for which the compliance label is:

$$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$

with $c_i = \text{Violated}$ implicitly corresponding to a hazard (Ng et al., 22 May 2026).

The formulation separates two conceptual subproblems. The first is rule applicability, namely whether a rule is relevant to the scene. The second is compliance judgement, namely whether the applicable rule is satisfied or violated. This structure matters because most possible rules are irrelevant to any particular image. The prevalence of the Not Applicable state is therefore not a defect of the dataset but a reflection of the problem structure (Ng et al., 22 May 2026).

A plausible implication is that CompliVision targets compositional reasoning rather than detector accumulation. Instead of adding a new classifier whenever a new safety requirement is introduced, the dataset encourages models to interpret arbitrary natural-language rules against scene evidence.

2. Dataset composition and domain coverage

CompliVision comprises 3,006 images spanning traffic, construction, and warehouse environments (Ng et al., 22 May 2026). These domains were selected because they are safety-critical, have authoritative regulations from which rules can be derived, and contain visually diverse scenes involving people, vehicles, equipment, infrastructure, and environmental conditions.

The dataset includes 15 domain-specific rule groups and 54 total safety standards (subrules). For experiments, each image is paired with the 5 rule groups for its domain, yielding 15,030 image–rule samples (Ng et al., 22 May 2026).

Domain	Images	Rule groups / subrules
Traffic	1,007	5 / 22
Construction	999	5 / 17
Warehouse	1,000	5 / 17

The traffic domain includes the rule groups Driving Distraction, Pedestrian Crossing, Road Condition, Traffic Rules, and Vehicle Load. Construction includes Crane Use, Fire Risk, Ladder Use, Protective Equipment, and Scaffolding Risk. Warehouse includes Ergonomic Lifting, Forklift Use, Ladder Use, Protective Equipment, and Surface Condition (Ng et al., 22 May 2026).

Across all image–rule pairs, the annotation counts are 3,567 Complied, 2,393 Violated, and 9,070 Not Applicable (Ng et al., 22 May 2026). The dominance of the last category follows from the multi-rule design: any single scene is ordinarily relevant to only a subset of possible rules.

Images were collected from open-source stock repositories including Vecteezy, Unsplash, and Pexels, using domain-specific keywords chosen so that the retrieved images naturally depict either compliance or violation. Preprocessing removes duplicates or near-duplicates using perceptual hash, resizes images to 336 × 336 pixels, and applies center crop to match VLM input, especially LLaVA (Ng et al., 22 May 2026).

The train/validation/test protocol is defined per domain. Each domain uses 100 images × 5 rules = 500 image–rule samples for validation and the same amount for test, with the remaining images providing approximately 4,000 training samples per domain. Aggregated across the three domains, training uses about 12k image–rule pairs, while validation and test each contain about 1,500 samples (Ng et al., 22 May 2026).

3. Rule representation, annotations, and curation

For each image–rule pair $(I, r_i)$, CompliVision stores a compliance label and a natural-language explanation that identifies the supporting visual evidence (Ng et al., 22 May 2026). Explanations mention concrete scene entities such as "helmet", "vest", "ladder", "forklift", and "pedestrian", as reflected in the paper’s word-cloud analysis. The dataset does not provide bounding boxes or segmentation masks; its focus is scene-level, rule-centric reasoning rather than local object detection (Ng et al., 22 May 2026).

Rules are concise natural-language statements derived from legislation and curated so that they remain visually checkable. The curation process selects relevant sections, preserves original wording as much as possible, removes clauses such as "except" and "unless" that create visually unresolved conditionals, and ensures that each rule expresses a single, clear compliance condition (Ng et al., 22 May 2026). Example rule formulations include "The driver must not use a hand-held mobile phone while driving", "Workers must wear safety helmets in construction zones", and "Walking surfaces must be free from spills and obstacles that could cause slips or trips" (Ng et al., 22 May 2026).

Manual annotation assigns one of the three labels to every image–rule pair according to a predefined annotation schema specifying observable evidence, ambiguity handling, and criteria for Not Applicable (Ng et al., 22 May 2026). These labels serve as gold labels for evaluation and for full-supervision baselines. Human-in-the-loop annotation later corrects weak samples and contributes in-context explanations used to refine prompts during active learning.

The dataset’s formal input–output definition is:

• input: image $I$ and rules $R = \{r_1, \dots, r_n\}$;
• output, for each pair $(I, r_i)$: a compliance label, confidence score, and optionally an explanation (Ng et al., 22 May 2026).

When explanation prompts are used, the model output is written as:

$(\hat{y}, \hat{E}) = M(x, \psi),$

where $r_i$0 and $r_i$1 is the prompt context, potentially augmented with human feedback (Ng et al., 22 May 2026).

A notable design property is the absence of hidden-state requirements. Rules are filtered to remain visually verifiable, excluding conditions that depend on unavailable metadata such as speed unless that condition is visually implied. This makes the dataset particularly aligned with static-image VLM evaluation, while also delimiting its scope.

4. Baseline framework and active learning pipeline

The paper presents a general hazard detection framework built around a VLM backbone, prompt engineering, active learning, human-in-the-loop feedback, and LoRA-based fine-tuning (Ng et al., 22 May 2026). The primary instantiated backbone is LLaVA-1.5 7B, with comparisons against LLaVA-Next, LLaMA-3.2 Vision, and LLaVA-CoT.

At an abstract level, the architecture consists of a visual encoder $r_i$2, a projector $r_i$3 that maps visual features into language space, a text encoder $r_i$4, and an autoregressive LLM decoder $r_i$5. The token-generation relation is given as:

$r_i$6

Prompts are organised into three families. Task-focused prompts describe the compliance task. Classification-focused prompts constrain the response to fixed labels and may request applicability checks, confidence scores, and chain-of-thought reasoning. Explanation-focused prompts extend classification prompts by requiring a short rationale (Ng et al., 22 May 2026). The structured-output format can be JSON-like, for example:

$c_i = \text{Violated}$2

Active learning is introduced because annotating every image–rule pair is costly. The method uses two complementary prompts, C1 and C4, to probe the same sample. A sample is treated as weak if predictions disagree or confidence is low; weak samples are sent to human annotators, while strong samples receive pseudo-labels (Ng et al., 22 May 2026). Confidence for pseudo-labelled samples is calibrated using distances in embedding space to centroids computed from human-labelled samples:

$r_i$7

The feedback mechanism extends beyond labels. Human annotators can provide short explanations, which are stored in a feedback memory and retrieved for similar future samples. A stability check evaluates whether adding a candidate explanation to the global prompt context harms performance on previously seen samples; unstable feedback is discarded (Ng et al., 22 May 2026). This mechanism functions as a prompt-level consistency control rather than a conventional retraining-only loop.

A plausible implication is that CompliVision is as much a benchmark for multimodal instruction following and explanation grounding as it is a dataset for classification. The explanations are operational: they are used to modify model behaviour, not merely to improve interpretability post hoc.

5. Evaluation protocol and empirical results

CompliVision evaluates models on per-rule compliance classification, explanation generation, and generalisation across traffic, construction, and warehouse domains (Ng et al., 22 May 2026). The reported metrics are Accuracy, Macro F1-score, Coverage, Inference time, and Annotation saved $r_i$8, where:

$r_i$9

with $(I, r_i)$0 the number of pseudo-labelled samples and $(I, r_i)$1 the total number of pseudo-labelled and human-labelled training samples (Ng et al., 22 May 2026).

For prompt selection with LLaVA, increasing prompt structure improves zero-shot performance. Among classification prompts, C4 achieves macro F1 $(I, r_i)$2, accuracy $(I, r_i)$3, and full coverage. Among explanation prompts, E1 reaches macro F1 $(I, r_i)$4, accuracy $(I, r_i)$5, and roughly 2.5 s/sample, with richer outputs but higher inference time (Ng et al., 22 May 2026).

Across VLM architectures, LLaVA-CoT + E1 gives the best zero-shot result, with macro F1 $(I, r_i)$6 and accuracy $(I, r_i)$7, but at about 12 s/sample. LLaMA Vision is reported as substantially weaker, with best macro F1 around 0.43 and incomplete coverage with some prompts (Ng et al., 22 May 2026). The reported conclusion is that contemporary VLMs can perform non-trivial rule-based compliance reasoning zero-shot, but still struggle with fine-grained and context-dependent safety rules.

Fine-tuning and active learning produce much larger gains. On the traffic domain, full fine-tuning on 4,035 labelled samples yields macro F1 $(I, r_i)$8 and accuracy $(I, r_i)$9. After three active-learning rounds, using 1,054 manual labels and 2,946 pseudo labels, the model reaches macro F1 $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$0 and accuracy $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$1, with 73.65% annotation saved (Ng et al., 22 May 2026). The traffic results also show that explanation feedback can improve early-round performance, with macro F1 $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$2 and accuracy $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$3 already at Round 0.

On the construction domain, full fine-tuning on 3,995 labels achieves F1 $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$4 and accuracy $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$5. The final active-learning result uses 1,362 manual and 2,633 pseudo labels, reaches F1 $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$6 and accuracy $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$7, and saves 65.91% of annotation (Ng et al., 22 May 2026).

On the warehouse domain, the full fine-tuning baseline with 4,000 labels yields F1 $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$8 and accuracy $$c_i \in \{\text{Complied}, \text{Violated}, \text{Not Applicable}\},$$9. The final active-learning configuration uses 1,246 manual and 2,754 pseudo labels, reaches F1 $c_i = \text{Violated}$0 and accuracy $c_i = \text{Violated}$1, and saves 68.85% annotation (Ng et al., 22 May 2026). In this domain, active learning exceeds the full fine-tuning baseline.

Qualitative analyses reinforce the quantitative findings. t-SNE plots show progressively cleaner separation of Complied, Violated, and Not Applicable clusters across active-learning rounds, and example outputs demonstrate correct identification of distracted driving, improper pedestrian crossing, unsafe forklift or ladder use, and non-compliant PPE usage (Ng et al., 22 May 2026). The paper also reports that explanations become more specific and less hallucinated when guided by human feedback.

6. Position within the dataset landscape, applications, and limitations

CompliVision is positioned as distinct from prior hazard and safety datasets because it is organised around explicit regulatory rules rather than object classes or coarse incident categories (Ng et al., 22 May 2026). Construction datasets such as Construction-PPE, SH17, SODA, MOCS, and ConstructionSite10K are described as focusing on object detection or scene-level categories. Traffic datasets such as CLEVRER, DeepAccident, and WTS are described as emphasising event or trajectory reasoning, early accident prediction, or pedestrian behaviour, rather than textual traffic rules and per-rule compliance labels (Ng et al., 22 May 2026). Industrial and fire datasets are similarly characterised as hazard-specific rather than generalised, rule-grounded safety datasets.

The paper identifies five aspects as making CompliVision unique: explicit natural-language rule grounding, multi-domain coverage, per-rule multi-label structure including Not Applicable, natural-language explanations, and integration with a complete VLM-plus-active-learning pipeline (Ng et al., 22 May 2026). This suggests that CompliVision occupies a distinct position between dataset release and benchmarked methodology.

Its intended uses include training and benchmarking general hazard detection systems, research on rule applicability detection and explanation generation, development of semi-automated safety inspection assistants, and study of active learning, confidence calibration, and pseudo-labelling in multimodal settings (Ng et al., 22 May 2026). The dataset and code are reported as available at https://github.com/Haruharu-hub/hazard-project.

The paper also states several limitations. CompliVision uses static images only, so rules requiring temporal information, such as speed or duration of distraction, can only be approximated visually. Its domain scope is restricted to traffic, construction, and warehouse settings. Because images come from stock repositories, there is likely selection bias toward staged and visually clear scenarios. Legal nuance is reduced by removing "except" and "unless" clauses from rules. Some regulatory aspects, such as training or certification, are not visually representable. Finally, ambiguity remains in the distinction between Not Applicable, Complied, and Violated, especially under occlusion, and no explicit inter-annotator agreement statistics are reported (Ng et al., 22 May 2026).

A useful comparison emerges with compliance datasets outside hazard detection. For example, DFIC addresses ICAO and ISO/IEC facial-image compliance using 26 photographic and pose requirements and balanced demographic partitions (Gonçalves et al., 11 Feb 2026). That comparison does not imply methodological equivalence, but it clarifies that CompliVision belongs to a broader class of datasets that reformulate recognition problems as compliance verification against external standards. In CompliVision, however, the standards concern scene safety rules across operational domains rather than portrait-image requirements.

CompliVision therefore marks a shift from perception-level hazard recognition to regulatory-aligned, rule-level hazard reasoning. Its central contribution is not merely that it adds images of unsafe scenes, but that it changes the unit of supervision from hazard category to image–rule compliance judgement, making natural-language rules the primary interface between regulations, annotations, and VLM reasoning (Ng et al., 22 May 2026).