LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

Abstract: Artificial intelligence is increasingly catalyzing scientific automation, with multimodal LLM (MLLM) agents evolving from lab assistants into self-driving lab operators. This transition imposes stringent safety requirements on laboratory environments, where fragile glassware, hazardous substances, and high-precision laboratory equipment render planning errors or misinterpreted risks potentially irreversible. However, the safety awareness and decision-making reliability of embodied agents in such high-stakes settings remain insufficiently defined and evaluated. To bridge this gap, we introduce LABSHIELD, a realistic multi-view benchmark designed to assess MLLMs in hazard identification and safety-critical reasoning. Grounded in U.S. Occupational Safety and Health Administration (OSHA) standards and the Globally Harmonized System (GHS), LABSHIELD establishes a rigorous safety taxonomy spanning 164 operational tasks with diverse manipulation complexities and risk profiles. We evaluate 20 proprietary models, 9 open-source models, and 3 embodied models under a dual-track evaluation framework. Our results reveal a systematic gap between general-domain MCQ accuracy and Semi-open QA safety performance, with models exhibiting an average drop of 32.0% in professional laboratory scenarios, particularly in hazard interpretation and safety-aware planning. These findings underscore the urgent necessity for safety-centric reasoning frameworks to ensure reliable autonomous scientific experimentation in embodied laboratory contexts. The full dataset will be released soon.

• The paper introduces LABSHIELD to evaluate safety-critical reasoning in lab environments with a multimodal, PRP-based framework.
• It employs dual-track evaluations, using MCQ and semi-open QA metrics, to expose vulnerabilities in safety planning among SOTA MLLMs.
• It highlights challenges in hazard detection and planning stability, calling for enhanced visual grounding and risk-aligned training in embodied AI.

LABSHIELD: Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

Motivation and Background

LABSHIELD addresses a critical gap in embodied AI evaluation: safety-critical reasoning and planning in real scientific laboratory environments. As MLLM-driven embodied agents transition from passive assistants to autonomous operators, the intrinsic risk profile of laboratory settings—characterized by fragile apparatus, hazardous chemicals, and irreversible planning errors—demands rigorous verification of agent safety awareness and reliability. Traditional benchmarks either treat safety as a textual policy problem or reduce it to low-level motion planning, failing to capture the semantic-physical interplay and latent hazards endemic to laboratories. LABSHIELD is grounded in OSHA and GHS standards and introduces a unified, high-fidelity framework to diagnose safety-critical cognition and action in the context of experimental science (2603.11987).

Benchmark Design and Taxonomy

LABSHIELD comprises 164 operational tasks spanning three laboratory domains (Workbench, Sink, Fume Hood), each annotated with synchronized multi-view (head, torso, wrist) RGB-D data to capture complex spatial relationships and mitigate occlusion-induced perceptual bias. The taxonomy is structured along two orthogonal axes:

• Task Hierarchy: Four operation levels (atomic, short-horizon, long-horizon, mobile manipulation) encapsulate increasing complexity from single-step primitives to multi-zone workflows.
• Safety Levels: Four tiers define risk profiles, ranging from harmless (S0), low-risk (S1), moderate-risk requiring reactive alert (S2), to high-risk violations necessitating outright refusal (S3).

The evaluation protocol follows the Perception-Reasoning-Planning (PRP) paradigm, decoupling sensory anomaly detection, causal hazard reasoning, and constraint-sensitive planning for granular diagnostic analysis.

Evaluation Protocols and Metrics

LABSHIELD employs a dual-track evaluation scheme:

• MCQ-based Safety Evaluation: Hierarchical VQA probes object/symbol detection, spatial understanding, causal/counterfactual safety reasoning, and error recovery, scored via micro-averaged accuracy.
• Semi-open QA: Quantifies model performance across unsafe factor detection (Jaccard, precision, recall), hazard pattern attribution, judge-reviewed planning scores, and alignment metrics (pass rate versus ground truth). Special emphasis is placed on high-risk scenarios, evaluating models' ability to trigger stop/alert or refusal protocols.

A domain-trained human baseline establishes the upper bound for each metric, enabling rigorous benchmarking against automated agents.

Empirical Results and Error Analysis

MCQ-Safety Decoupling

Across 33 SOTA MLLMs, leading proprietary and open-source models cluster around 74–78% MCQ accuracy. However, this proficiency is non-transferable: average Safety Score drops by 32.0% in semi-open QA for laboratory scenarios, with catastrophic failures in grounded hazard reasoning and planning. Reasoning-intensive models (e.g., GPT-o3, Gemini-3-Pro) attain higher stability and hazard attribution scores, yet residual vulnerabilities persist, especially under S2/S3 risk tiers.

Perceptual Bottlenecks

Transparent glassware and subtle GHS pictograms consistently evade robust detection, resulting in missed hazards and degraded downstream planning performance. Attention visualizations confirm that opaque/high-contrast objects disproportionately dominate model focus, amplifying the "completion bias" observed in planning metrics.

Planning Instability

LLM judges are prone to over-optimism; feasibility scores (Sco.) can diverge by >45% from strict pass rates (Pas.), leading to "hallucinated success". Dual-metric regularization is necessary to anchor evaluation to expert standards and penalize subtle safety violations.

Systematic Safety Underestimation

In high-risk (S2/S3) tracks, underestimation rates exceed 30% for most models, with embodiment-specific variants (RoboBrain family) failing to yield substantial safety dividends versus general MLLMs. Error analysis across MCQ and semi-open QA confirms that perception errors propagate through the PRP pipeline, cascading into weak hazard reasoning and flawed planning.

Implications and Future Directions

LABSHIELD exposes the profound decoupling between abstract linguistic safety knowledge and grounded, multisensory reasoning required for physical agency. Embodiment alone is insufficient; improvements in high-fidelity visual grounding (especially for transparent objects) and explicit risk-context integration are essential. The results highlight the necessity for safety-aligned training, robust perceptual augmentation, and task refusal mechanisms to secure laboratory autonomy.

Practically, LABSHIELD catalyzes advances in:

• Dataset curation for safety-critical science environments,
• Diagnostic benchmarking for safety-aware embodied cognition,
• Design of multi-view, expert-validated evaluation suites,
• Transitioning from cognitive simulation to real-world deployment of laboratory agents.

Future research must focus on integrating these safety reasoning models with physical robotic platforms, curating real-world datasets with granular annotation, and deploying agents capable of safety-centric collaboration and autonomous discovery.

Conclusion

LABSHIELD establishes a rigorous, multimodal diagnostic suite for safety-aware reasoning and planning in scientific laboratories. Its dual-track, PRP-based evaluation elucidates fundamental deficiencies in current MLLMs, particularly in perception of visually ambiguous hazards and execution of safety-constrained plans. By systematically highlighting these bottlenecks, LABSHIELD provides actionable insights for bridging the digital-physical divide, guiding the development of robust, safety-aligned embodied AI for high-stakes scientific automation (2603.11987).