TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

Abstract: Recent advances in vision-LLMs (VLMs) have accelerated their application to indoor safety hazards assessment. However, existing benchmarks suffer from three fundamental limitations: (1) heavy reliance on synthetic datasets constructed via simulation software, creating a significant domain gap with real-world environments; (2) oversimplified safety tasks with artificial constraints on hazard and scene types, thereby limiting model generalization; and (3) absence of rigorous evaluation protocols to thoroughly assess model capabilities in complex home safety scenarios. To address these challenges, we introduce TSHA (\textbf{T}rustworthy \textbf{S}afety \textbf{H}azards \textbf{A}ssessment), a comprehensive benchmark comprising 81,809 carefully curated training samples drawn from four complementary sources: existing indoor datasets, internet images, AIGC images, and newly captured images. This benchmark set also includes a highly challenging test set with 1707 samples, comprising not only a carefully selected subset from the training distribution but also newly added videos and panoramic images containing multiple safety hazards, used to evaluate the model's robustness in complex safety scenarios. Extensive experiments on 23 popular VLMs demonstrate that current VLMs lack robust capabilities for safety hazard assessment. Importantly, models trained on the TSHA training set not only achieve a significant performance improvement of up to +18.3 points on the TSHA test set but also exhibit enhanced generalizability across other benchmarks, underscoring the substantial contribution and importance of the TSHA benchmark.

• The paper introduces TSHA, a benchmark integrating multi-source real and synthetic data to evaluate hazard recognition in vision-language models.
• It employs a robust four-stage annotation pipeline combining chain-of-thought AI and expert reviews to ensure high-quality and contextually accurate hazard labels.
• Empirical results reveal an 18.3 point performance gain in key models and improved generalization across varied safety-critical benchmarks.

TSHA: A Comprehensive Benchmark for VLMs in Safety Hazard Assessment

Motivation and Problem Statement

Deployment of vision-LLMs (VLMs) in safety-critical inspection remains fundamentally bottlenecked by synthetic benchmarks, constrained hazard taxonomies, and unrealistic evaluation settings. Current datasets rely heavily on simulator-generated imagery with limited scene diversity, resulting in substantial model–environment domain gaps and poor generalization in real-world indoor hazard assessment. Rigorous, realistic, and broad-coverage resources are required to both advance the field and ensure reliable hazard recognition in unconstrained environments.

TSHA Benchmark Design

TSHA (Trustworthy Safety Hazard Assessment) is introduced as a large-scale, multi-source, and highly realistic benchmark specifically targeted at the safety hazard scenario. It comprises 81,809 training samples and a challenging 1,707-sample test set, all curated from four complementary sources: real-world indoor datasets (NYU v2, MIT Indoor Scenes), internet images and videos, AIGC-generated images, and newly captured real scenes—spanning typical bedrooms, kitchens, bathrooms, and office spaces.

Figure 1: Sample images, videos, and question types illustrating the coverage and complexity of TSHA data.

TSHA test data includes both traditional static images and advanced synthetic scenarios (time-evolving Sora2-generated videos and Hunyuan panoramic images with multiple concurrent hazards). This ensures the benchmark robustly probes models’ ability to generalize to sequence and world-scale, multi-risk contexts.

Data Curation and Quality Pipeline

A four-stage annotation pipeline is utilized for all samples: scene description, preliminary hazard identification, final hazard verification (with supporting evidence, confidence, and mis-ID analysis), and final human verification by expert annotators.

Figure 2: The four-stage TSHA annotation pipeline ensures high-quality, contextually verified hazard labels through consecutive machine and human refinement.

Generation of detailed, chain-of-thought-based QA pairs for each data point leverages GPT-4o with subsequent manual spot checks (10% sampling). All samples are validated for physical plausibility, realistic depiction of danger, and semantic consistency, especially to filter out low-fidelity or exaggerated AIGC content.

Dataset Statistics and Distribution

TSHA's train set consists of 80,102 image–QA pairs sourced from existing datasets (35.2%), internet images/videos (32.4%), AIGC (21.6%), and new captures (10.9%). The test set mixes these sources with significantly more complex video and panoramic cases. Hazard category coverage is comprehensive, including not only common fire and trip hazards but also complex scenarios such as obstructed egress and ambiguous situations involving at-risk individuals (e.g., children near dangerous objects).

Figure 3: Train data distribution across the four principal TSHA sources, revealing balanced and diverse domain sampling.

Bar charts (not displayed here) quantitatively confirm the inclusion and frequency of a broad spectrum of hazards required for robust model training and evaluation.

Task Formulation and Evaluation Framework

TSHA introduces two complementary task types:

• Choice Questions: Includes binary (true/false) and multi-class (four-option, including “no hazard”) questions for closed-format hazard identification.
• Open-ended QA: Demands unconstrained, natural language hazard enumeration and description for each sample.

For open-ended questions, scoring employs LLM-based evaluators (e.g., ChatGPT-4o) with explicit weighting on accuracy (0.7), conciseness (0.2), and coherence (0.1), ensuring that factual correctness is paramount. Multiple evaluators and human review further calibrate model assessment.

Experimental Analysis

Main Results

Systematic benchmarking of 23 VLMs exposes substantial failure modes in existing models:

• Closed-source models averaged 66.06 (max: 82.5, min: 47.5 across models such as Claude-3.7-Sonnet and Claude-4-Sonnet).
• Open-source VLMs averaged 63.52, with severe inconsistency by parameter count and architecture (InternVL2.5-26B at 82.8, Gemma3-4B at 34.5).

TSHA training yields robust improvements: Qwen2.5-VL-3B achieves an 18.3 point gain post-TSHA tuning, confirmed over a suite of open and closed-source models and across both question types. Gains are most pronounced for open-ended QA, suggesting prior datasets overfit closed-choice settings and that TSHA’s real-world realism is critical for genuine hazard reasoning.

Cross-Benchmark Generalization

VLMs trained on TSHA manifest consistent gains on standard benchmarks (BLINK, MMStar, AI2D, MUIRBench, SEEDBench2), notably in low-parameter regimes, thereby countering concerns about overfitting to safety-specific distributions. For example, TSHA-trained Qwen2.5-VL-3B enjoys a +1.6 average point improvement on general benchmarks, and the effect persists—albeit moderately—for models scaling up to 72B parameters.

Ablations and Data Synergy

Ablation studies confirm that diverse data sources (real, AIGC, internet, new captures) act synergistically: models trained on all domains outperform those trained on any individual subset, underscoring the necessity of comprehensive task distributional coverage for safety assessment.

Further, evaluations using multiple LLMs and human raters exhibit strong alignment, demonstrating that TSHA improvements are robust to evaluator idiosyncrasies and indicate genuine advances in semantic hazard understanding.

Implications and Future Directions

TSHA exposes critical limitations in current VLM real-world hazard reasoning—this applies regardless of model size, commercial status, or prior benchmark performance. The data curation and evaluation principles established in TSHA should substantially inform future dataset construction, especially for deployment in safety-critical and embodied AI domains, such as autonomous robotics and home-care assistants.

Practically, end-users should exercise significant caution in relying on off-the-shelf VLMs for unsupervised hazard monitoring. Theoretically, TSHA sets a new standard for the type, granularity, and coverage of data and evaluation required to close the domain gap between laboratory and real-world safety tasks. The synergy between human validation and chain-of-thought LLM pipeline annotation also provides a scalable pattern for future multi-modal safety resources.

Progress beyond TSHA likely depends on integrating temporal reasoning (from Sora2-esque videos), world-consistent holistic scene understanding (from Hunyuan-style panoramas), and the continued evolution of robust, open-source LLM-based evaluators.

Conclusion

"TSHA: A Benchmark for Visual LLMs in Trustworthy Safety Hazard Assessment Scenarios" (2603.29759) defines a rigorous, comprehensive, and domain-realistic standard for safety hazard assessment and exposes major practical gaps in current VLMs. TSHA’s data diversity, annotation quality, and evaluation rigor both highlight current model weaknesses and offer a blueprint for future research in trustworthy, safety-critical AI reasoning and deployment.