General Hazard Detection

Abstract: Hazard, as an abstract concept, is typically defined through cognitive-level logical reasoning rather than concrete examples. In contrast, existing hazard detection systems rely on predefined hazard categories and require intensive collection of labelled examples within detection or classification architectures. This approach faces three fundamental challenges when addressing abstract safety concepts: (1) noisy and sparse training data, (2) dynamically evolving definitions that change across contexts and time, and (3) limited generalisation to unseen or novel scenarios. To address these limitations, we present the CompliVision dataset, the first general-purpose hazard dataset designed for rule-based compliance assessment, along with a baseline framework for hazard evaluation. Our key innovation is decoupling the hazard concept from image-based examples by expressing safety requirements through language-based rules. We ground our approach in authoritative domain regulations and ISO standards to define diverse hazard concepts across multiple domains. The CompliVision dataset comprises 3,006 images spanning traffic, construction, and warehouse environments, with each image annotated for compliance against specific safety rules, accompanied by natural language explanations highlighting the supporting visual evidence. To achieve robust generalisation, we develop an active learning framework to more effectively guide and refine vision-LLMs in assessing hazard compliance. While state-of-the-art VLMs demonstrate strong capabilities, they struggle with the fine-grained, context-dependent interpretation required for accurate safety assessment. We proposed a general hazard detection framework to address this limitation which combines LLaVA-based visual reasoning with with human-in-the-loop feedback.

• The paper's main contribution is a novel rule-grounded approach that decouples hazard reasoning from object detection using language-based safety rules.
• It presents a comprehensive framework integrating prompt engineering, active learning, and human-in-the-loop feedback to boost interpretability and performance.
• Empirical results show high macro-F1 scores and significant annotation savings, validating scalable, context-sensitive compliance assessment across domains.

General Hazard Detection: Rule-Grounded Vision-Language Reasoning for Contextual Safety Compliance

Motivation and Challenges in Hazard Detection

Hazard detection has historically been constrained by object-level approaches and predefined hazard categories, primarily through supervised models relying on extensive labelled data. This paradigm fails to capture the abstraction, temporal variability, and contextual heterogeneity of hazards in dynamic environments. Existing computer vision models are robust at detecting safety equipment or environmental cues, but they exhibit limited performance in interpreting complex safety rules, evolving standards, and unseen scenarios. Addressing these fundamental limitations requires decoupling hazard reasoning from object-centric detection and grounding compliance assessment in rule-based language representations.

Figure 1: Overview of the proposed approach for general hazard detection, highlighting limitations of object detection models and the integration of VLMs, prompt engineering, active learning, human feedback, and contextual safety-compliance.

CompliVision Dataset: Design and Properties

The CompliVision dataset is introduced as the first multi-domain hazard benchmark for rule-grounded compliance assessment. It comprises 3,006 images spanning traffic, construction, and warehouse domains, with each image paired with explicit safety rules informed by authoritative regulations and ISO standards. The dataset facilitates multi-rule compliance classification through human-in-the-loop annotation cycles, delivering labels ("Complied", "Violated", "Not Applicable") and natural language explanations for supporting visual evidence. The rule sets were curated with high semantic precision, eliminating ambiguous modifier clauses for consistent interpretation by annotators and vision-LLMs.

Methodological Framework

The hazard detection problem is formulated as a per-image, per-rule compliance reasoning task. Given a scenario image $I$ and safety rule set $R$, the model determines rule applicability and assesses compliance based on visual cues. Central to the framework are parameter-frozen VLMs, advanced prompt engineering, active learning for annotation efficiency, human-in-the-loop feedback for explainability, and parameter-efficient fine-tuning (LoRA).

Figure 2: Detailed illustration of the proposed framework combining prompt engineering, VLM embeddings, feedback interactions, and fine-tuning.

Prompt Engineering

Prompt templates are classified into task-focused, classification-focused, and explanation-focused variants. Structured prompts with conditional phrasing and chain-of-thought reasoning (e.g., "think-before-answering") markedly improve macro-F1 and interpretability, highlighting the importance of task constraints and explicit guidance in VLMs.

Vision-LLM Architecture

Models integrate a vision encoder (ViT/CNN), multimodal projector, tokenizer, and transformer-based LLM. Frozen parameters preserve visual-linguistic alignment, while LoRA adapters enable efficient task adaptation. Four leading VLMs—including LLaVA, LLaVA-Next, Llama Vision, and LLaVA-CoT—are benchmarked.

Active Learning and Annotation Efficiency

Active learning pipelines minimize manual annotation by iteratively querying labels for informative (low-confidence, high-disagreement) samples and applying confidence-calibrated pseudo-labelling to strong samples. Dual prompt probes (baseline vs. reasoning) facilitate sample selection, optimizing human effort.

Figure 3: t-SNE visualization of embeddings for training, validation, and test data across active learning rounds, reflecting informed sample selection.

Human-in-the-loop Feedback

Two feedback modes—compliance labels and in-context explanations—drive model refinement. Explanations augment prompt context, retrieved via embedding similarity for similar cases; stability checks ensure reliable updates, preventing regressions.

Figure 4: Word clouds of generated explanations for training, validation, and test data, evidencing most salient visual cues referenced for feedback.

Fine-tuning with LoRA

LoRA adapters are injected into transformer and select vision projector layers, with only 0.3% of parameters trainable. This preserves pre-trained alignment while enabling task-specific adaptation using modest annotation budgets.

Experimental Results

Systematic benchmarking across prompts, VLM architectures, and training strategies reveals:

• Structured prompts (classification-focused, chain-of-thought) achieve highest macro-F1 and output consistency.
• LLaVA-CoT delivers strongest numerical results (macro-F1 = 0.6358, Accuracy = 0.6793) in zero-shot settings, albeit with elevated inference time.
• Active learning pipeline achieves equivalent performance to fully supervised fine-tuning (macro-F1 > 0.84, Accuracy > 0.89) with ≥65% reduction in manual labelling across domains.
• In-context explanation feedback compensates for smaller labelled datasets, enhancing interpretability and efficiency.
• t-SNE projections and feedback process visuals corroborate robust learning and stability across annotation rounds.

  Figure 5: t-SNE embeddings with rule-compliance regions contrasting AL Round 0 vs Round 3, illustrating efficient sample stratification and embedding separation.

  

  Figure 6: Representative hazard detection outputs across application domains, showing nuanced classification and interpretability.

  

  Figure 7: Human-in-the-loop feedback workflow, demonstrating iterative prompt refinement and stability validation.

Implications: Practical and Theoretical

The decoupling of hazard detection from object-category enumeration enables scalable, generalizable compliance reasoning across domains. The rule-grounded approach closes the semantic gap between vision and regulatory requirements, supporting adaptive monitoring in dynamic environments. Empirically, annotation cost is minimized without sacrificing performance, validating active learning with expert feedback for safety-critical domains.

Theoretically, the integration of language-based rules as context for VLMs advances multimodal commonsense reasoning, opening pathways for contextual compliance assessment, interpretable output generation, and regulation-aligned AI systems. Limitations remain in capturing temporal risk dynamics and modeling multi-agent interactions from static imagery. Future work includes temporal dataset expansion, complex scene understanding, and joint reasoning models for evolving safety standards.

Conclusion

The CompliVision framework demonstrates efficient hazard detection via rule-grounded vision-language reasoning. Active learning and human-in-the-loop strategies enable substantial annotation savings alongside high performance, providing a scalable architecture for compliance assessment across diverse environments. The work not only establishes new empirical baselines for rule-compliance reasoning but also informs the design of interpretable, context-sensitive safety monitoring systems, with implications for future AI methodologies in safety-critical applications.

Reference: "General Hazard Detection" (2605.23304)