SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models

Abstract: Multimodal LLMs are increasingly adopted as autonomous agents in interactive environments, yet their ability to proactively address safety hazards remains insufficient. We introduce SafetyALFRED, built upon the embodied agent benchmark ALFRED, augmented with six categories of real-world kitchen hazards. While existing safety evaluations focus on hazard recognition through disembodied question answering (QA) settings, we evaluate eleven state-of-the-art models from the Qwen, Gemma, and Gemini families on not only hazard recognition, but also active risk mitigation through embodied planning. Our experimental results reveal a significant alignment gap: while models can accurately recognize hazards in QA settings, average mitigation success rates for these hazards are low in comparison. Our findings demonstrate that static evaluations through QA are insufficient for physical safety, thus we advocate for a paradigm shift toward benchmarks that prioritize corrective actions in embodied contexts. We open-source our code and dataset under https://github.com/sled-group/SafetyALFRED.git

• The paper introduces SafetyALFRED, a benchmark that extends ALFRED with six real-world kitchen hazard categories to test MLLMs’ ability to identify and mitigate risks.
• The evaluation reveals that while models excel in static QA-based hazard recognition, they struggle to integrate safety measures into dynamic embodied planning tasks.
• The multi-agent approach demonstrates that decoupling hazard recognition from action planning improves mitigation performance, yet a critical alignment gap remains.

SafetyALFRED: Evaluating Safety-Conscious Planning Capabilities of Multimodal LLMs

Benchmark Design and Problem Formulation

SafetyALFRED is introduced as an extension to the ALFRED embodied instruction-following benchmark, augmenting it with six distinct categories of real-world kitchen hazards: appliance misuse, spoilage, fall/trip hazards, fire hazards, property damage, and unsanitary conditions. The benchmark formulates a safety-constrained planning problem wherein Multimodal LLMs (MLLMs) must not only achieve task-specific goals but also proactively identify and mitigate critical hazards. The benchmark is implemented within the AI2Thor simulator, with environmental perturbations instantiating each hazard type for robust evaluation.

The planning challenge is formally encapsulated as a tuple $$\mathcal{P} = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{G}, \mathcal{H}, \mathcal{R}_{\text{safe}}\rangle$$, integrating environmental state, action space, transition dynamics, task goals, hazard predicates, and remediation functions. Crucially, the desired policy $\pi^*$ must always prioritize hazard remediation over task advancement, conditioning action selection on the presence of hazardous states.

Evaluation Protocols: Hazard Recognition vs. Mitigation

SafetyALFRED evaluates eleven state-of-the-art MLLMs from the Qwen, Gemma, and Gemini families in two distinct settings:

• QA-based Hazard Recognition: Agents act as external judges tasked with identifying safety hazards in static multimodal scenes. Input modalities include egocentric RGB images (vision-only) and optional textual metadata describing environmental state (metadata-augmented). Prompting protocols vary in complexity, sometimes including description of actions, subgoals, and demonstrations. Responses are processed via structured fields and assessed using a two-stage verification (structural criterion and NLI entailment).
• Embodied Hazard Mitigation: Agents must generate action plans that both achieve the assigned goal and remediate hazards encountered during planning. The evaluation traces frame-by-frame agent decisions in perturbed AI2Thor trajectories, requiring explicit reasoning and subgoal specification for each next action. Metrics include Mitigation Success Rate (MSR), task success, and safety alignment rate quantifying consistency between recognized hazards and effective mitigation.

Core Findings: Alignment Gap and Performance Analysis

Recognition Performance and Perceptual Bottlenecks

MLLMs exhibit strong performance in hazard detection within QA settings, with closed-weight models achieving up to 92.5% accuracy when augmented with metadata, and open-weight models up to 60.8%. Vision-only settings significantly degrade recognition rate, with persistent failures for certain categories such as appliance misuse and property damage. Metadata inclusion alleviates perceptual bottlenecks, especially for hazards involving less salient visual cues or complex interactions.

Model size correlates positively with recognition performance across both settings and families. Larger Qwen models consistently outperform smaller variants. However, with metadata-augmented input, the presence of extraneous embodied task descriptions (complex prompts) can introduce noise, reducing accuracy for hazard categories reliant on explicit object-state tracking.

Embodied Mitigation: Planning Deficits and Misalignment

Despite high QA recognition rates, models fail to translate abstract safety knowledge into effective hazard mitigation in embodied tasks. Average mitigation success rates seldom exceed 60%, even with access to ground truth state descriptions. Models reliably mitigate only fire hazards (due to simple stimulus-response mapping), while categories like appliance misuse and fall/trip hazards remain challenging.

MLLMs consistently prioritize task completion over hazard remediation. Incorrect actions during hazardous turns are dominated by goal-oriented behaviors, suggesting persistent misalignment between safety awareness and task execution. Action prediction accuracy on non-hazardous turns (often exceeding 80%) underscores that planning failures are not due to general task competence but to inadequate integration of safety constraints.

The relationship between QA recognition and embodied mitigation is weak: high hazard detection in static QA does not reliably predict mitigation success in dynamic embodied planning. Hallucinated hazards flagged in QA are typically ignored in embodied execution, leading to poor alignment rates. Notably, models display strong conservative bias, frequently hallucinating hazards in safe environments (false positives), particularly in vision-only settings.

Multi-Agent Architectural Insights

To diagnose cognitive interference stemming from task vs. safety reasoning, the paper implements a multi-agent system decoupling hazard recognition (safety judge) and mitigation (embodied actor). This architecture yields substantial improvements in certain hazard categories (e.g., mitigation accuracy for appliance misuse jumps from 0.7% to 71.1% for Qwen 3 VL 32B with metadata), yet the alignment gap persists: even when hazards are correctly flagged, mitigation often fails.

Error Taxonomy and Failure Mechanisms

Manual error analysis classifies failures into six dominant types, including hazard ignored (cross-task inconsistency), perception error (failure to localize hazards, exacerbated without metadata), hallucinated/misidentified hazards (over-conservative reporting), physical commonsense deficits (e.g., ignorance of the risks of microwaving metal), state tracking errors (temporal inconsistency), and output format errors. The distribution of failure types underlines the multifaceted nature of misalignment between safety knowledge, perception, and action synthesis.

Implications, Recommendations, and Limitations

Practical and Theoretical Implications

SafetyALFRED empirically demonstrates that QA-based hazard recognition is insufficient for embodied safety evaluation of MLLMs in agentic settings. The alignment gap revealed by dynamic mitigation tasks exposes a limitation in current MLLM architectures and training regimes: abstract knowledge is not reliably synthesized into context-aware, corrective behaviors. This has significant implications for robotics and autonomous agent deployment in environments with physical safety requirements.

The findings advocate for a methodological pivot in safety benchmarking: embodied, dynamic evaluations must supplement static QA regimes. Furthermore, the persistent planning deficits underscore the need for more embodied safety data, rigorous evaluation methodologies that jointly score safety awareness, corrective actions, and the trade-off with task performance, and architectural improvements that integrate safety reasoning more fluidly with goal-directed planning.

Deployability constraints further complicate the picture: while large models show superior hazard reasoning, their computational and hardware demands preclude use in embedded robotic platforms. Smaller models suitable for native deployment lag significantly in safety reasoning and mitigation capacity, raising concerns for real-world reliability.

Limitations

Evaluation is conducted in a simulated domain with controlled hazards; real-world environments present significantly greater complexity and unpredictability. Hazard evaluation relies on NLI-based verification, introducing potential bias and imperfect semantic discrimination. The benchmarking focuses on a limited subset of MLLM families; generalizability to other architectures remains to be established. Embodied tasks use pre-rendered trajectories rather than real-time agent interaction, potentially underestimating planning complexity and exposure to novel hazard states.

Conclusion

SafetyALFRED highlights a structural alignment deficiency in current MLLMs: hazard recognition in static QA settings does not translate into proactive safety mitigation within dynamic, embodied planning contexts. This necessitates a paradigm shift toward embodied safety benchmarks and evaluation methodologies. Future research must develop more sophisticated, context-sensitive planning architectures, integrate richer embodied safety datasets, and devise lightweight models capable of safe agentic operation. The benchmark provides a rigorous foundation for such developments, with implications spanning the deployment of autonomous agents in human environments and the advancement of safety-aligned foundation models.