FailSafe-VLM: Safety-Enhanced Multimodal Systems

• FailSafe-VLM is a safety-enhanced vision-language model architecture that embeds semantic reasoning inside a safety envelope to guide recovery actions.
• It decouples high-level semantic inference from low-level control using detectors, impedance regulators, and candidate fallback strategies.
• Empirical evaluations show improved failure recovery and risk detection, boosting success rates in robotic manipulation and content safety tasks.

Searching arXiv for the cited FailSafe-VLM-related papers and closely related safety-VLM/VLA work. arXiv search query: "FailSafe-VLM vision-language-action failure recovery safety alignment variable impedance". FailSafe-VLM denotes both a specific vision-LLM for robotic failure recovery and a broader architectural pattern for safety-enhanced multimodal systems. In the narrow sense, it is a LLaVA-OneVision-7B-based model fine-tuned to detect manipulation failures, identify failure types, and output executable 7-DoF recovery actions (Lin et al., 2 Oct 2025). In the broader sense used across recent work, it refers to layered systems that place a VLM or VLA inside an additional safety envelope—failure detectors, impedance regulators, visual safety-alignment modules, policy-conditioned safeguards, or candidate-constrained fallback selectors—so that semantic reasoning is coupled to bounded, auditable, and overridable mechanisms rather than trusted as an unconstrained end-to-end controller (Zhang et al., 21 Jan 2026, Gu et al., 11 Jun 2025, Liu et al., 2024, Liu et al., 14 Feb 2025, Christensen et al., 30 Dec 2025).

1. Terminological scope and representative systems

Across the literature, FailSafe-VLM is not a single standardized architecture. One line of work uses the name for a concrete failure-recovery model in robotic manipulation; several others use it as a conceptual label for systems in which a VLM contributes semantic understanding while a separate mechanism enforces safety, detects failure, or selects a constrained fallback. The common denominator is not a particular backbone, but a systems design principle: semantic multimodal inference is treated as advisory, supervisory, or recovery-oriented rather than as the sole arbiter of action (Lin et al., 2 Oct 2025).

System	Domain	Core fail-safe role
FailSafe-VLM	Manipulation	Detects failure and outputs executable 7-DoF recovery actions
SAFE	VLA monitoring	Predicts a scalar failure likelihood from VLA internal features
CompliantVLA-adaptor	Contact-rich manipulation	Wraps VLA actions with VLM-guided variable impedance control
SafeVLM / VLM-Guard / LlavaGuard	Multimodal content safety	Adds visual safety-alignment or policy-conditioned safeguarding
Semantic Lookout	Maritime autonomy	Selects one cautious fallback maneuver from constrained candidates
VILTA / VLM as Strategist	Driving safety evaluation	Generates adversarial or safety-critical scenarios for robustness testing

This dual usage matters because the concrete FailSafe-VLM model is only one member of a wider family. The wider family includes runtime monitors, representation-level safety interventions, modular wrappers around existing VLMs, and semantic fallback selectors for embodied autonomy. A plausible implication is that FailSafe-VLM is best understood as a research direction centered on safety envelopes for multimodal foundation models, rather than as a fixed algorithmic template.

2. Core architectural pattern: semantic reasoning inside a safety envelope

A recurrent pattern is strict separation of concerns. High-level modules perform semantic interpretation—task phase recognition, hazard recognition, multimodal safety classification, or failure likelihood estimation—while lower-level modules impose bounded action spaces, calibrated thresholds, or physically grounded control laws. In "CompliantVLA-adaptor" (Zhang et al., 21 Jan 2026), the VLA still decides where to move, but a VLM-informed variable impedance controller determines how stiff or compliant the motion should be under contact. In SAFE (Gu et al., 11 Jun 2025), the VLA remains frozen and a lightweight detector probes its internal embeddings to produce a scalar risk score,

$s_t = f(e_{0:t}),$

which is then thresholded by conformal prediction rather than by ad hoc confidence heuristics. In SafeVLM (Liu et al., 2024), a parallel safety path—safety projector, safety tokens, and safety head—is attached without altering the original vision encoder or projector. In VLM-Guard (Liu et al., 14 Feb 2025), the intervention is even more localized: internal hidden states are projected relative to a safety steering direction extracted from the underlying aligned LLM.

A second pattern is bounded or discrete intervention. Semantic Lookout in maritime autonomy does not allow arbitrary trajectory synthesis; it chooses a single candidate ID from a pre-vetted set of water-valid, world-anchored trajectories, with Station-keeping as ID 0 and as the default on tie, timeout, or invalid output (Christensen et al., 30 Dec 2025). CompliantVLA similarly constrains VLM outputs to impedance ranges and then applies a force-based scaling factor

$$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$

with $\alpha_{\text{force}} \in [0.2,1]$, so the safety layer can scale stiffness down but not beyond prescribed limits (Zhang et al., 21 Jan 2026). SafeVLM uses type and level control codes to select safety prompts and embeddings, turning safety policy into a structured inference-time control interface rather than an implicit emergent behavior (Liu et al., 2024).

A third pattern is explicit fallback behavior. SAFE raises alarms early enough that a robot can stop, backtrack, or ask for help (Gu et al., 11 Jun 2025). Semantic Lookout is designed specifically for the alert-to-takeover gap and enforces immediate human override with a short-horizon fallback maneuver (Christensen et al., 30 Dec 2025). CompliantVLA aborts trials after repeated force threshold violations (Zhang et al., 21 Jan 2026). These systems do not assume that semantic models are infallible; they assume the opposite and surround them with mechanisms that degrade toward conservative behavior.

3. Robotic manipulation: failure detection, recovery, and compliant execution

The model explicitly named FailSafe-VLM is introduced in "FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models" (Lin et al., 2 Oct 2025). It is built by full instruction fine-tuning of LLaVA-OneVision-7B, whose language backbone is Qwen2-7B-Instruct, whose vision encoder is SigLIP, and whose multimodal connector is a 2-layer GELU MLP. The model consumes 10 consecutive multi-view image observations and a task instruction, and outputs the current sub-task, a binary failure decision, the failure type, and a 7-DoF recovery action $\Delta A$. The training data are generated automatically by the FailSafe pipeline in ManiSkill using three generic failure modes—translation failures, rotation failures, and no-op failures—and validated by replaying the recovery action to ensure the failed task now succeeds. The resulting dataset contains approximately 131k failure-action pairs and approximately 56k ground-truth success trajectories (Lin et al., 2 Oct 2025).

Quantitatively, FailSafe-VLM substantially outperforms general-purpose VLM baselines on held-out rollout reasoning. On 1,712 entries from 20 held-out seeds, Qwen2.5-VL achieves Binary 0.2401, Accuracy 0.2401, and Cosine 0.0000; Gemini-2.5-flash achieves 0.6229, 0.1412, and -0.0121; GPT-4o achieves 0.7007, 0.1960, and 0.0117; FailSafe-VLM achieves 0.9094, 0.8368, and 0.6522 (Lin et al., 2 Oct 2025). When periodically invoked every 10 control steps as an external assistant, it raises the average success rate of OpenVLA from 14.7% to 37.3%, of OpenVLA-OFT from 90.7% to 98.7%, and of $\pi_0$-FAST from 78.7% to 82.7%, with improvement up to 22.6% on average across tasks (Lin et al., 2 Oct 2025).

SAFE addresses a different layer of the same problem: failure detection rather than recovery action synthesis. It is trained on rollout-level success/failure labels, not per-timestep annotations, and operates on frozen internal VLA features from OpenVLA, $\pi_0$, and $\pi_0$-FAST (Gu et al., 11 Jun 2025). Across four simulated settings, SAFE-MLP reaches 78.0% average ROC-AUC on unseen tasks and SAFE-LSTM reaches 77.0%. On the real Franka Panda setup with $\pi_0$-FAST, SAFE-MLP reaches 64.16 ROC-AUC on unseen tasks, above the best baseline at 60.27. Its conformal thresholding gives a time-varying alarm boundary with a direct trade-off between TPR, TNR, Balanced Accuracy, and detection time, and its runtime overhead is reported as negligible, with inference below 1 ms (Gu et al., 11 Jun 2025).

CompliantVLA-adaptor addresses physical safety under contact. It preserves pretrained VLA policies such as Pi0, RDT-1B, and OpenVLA-oft, but routes their position actions through a VLM-guided variable impedance controller and a force-based supervisory layer (Zhang et al., 21 Jan 2026). The VLM classifies the task phase as Free_motion, Approaching, Contact, or Retreat; proposes anisotropic stiffness and damping; and a 1 kHz force loop scales stiffness via $\alpha_{\text{force}}$ and terminates a trial if force exceeds 30 N three consecutive times. Across all tasks, overall success rises from 9.86% to 17.29%, a 1.74× improvement, while failure modes shift from large force violations and toppled objects toward slight misalignment, slip, or non-completion within time (Zhang et al., 21 Jan 2026). The system therefore exemplifies a physically grounded FailSafe-VLM: semantic reasoning configures a compliant control regime, but hard force thresholds and deterministic impedance design remain authoritative.

4. Visual safety alignment and safeguard wrappers for VLMs

In multimodal content safety, FailSafe-VLM often denotes a modular safeguard layered onto an existing VLM rather than a task-execution policy. SafeVLM is an add-on safety module for projector-based architectures such as LLaVA-v1.5-7B (Liu et al., 2024). It adds a safety projector, two sets of 64 trainable safety tokens of dimension 4096, and a safety head with 8-head cross-attention that predicts both safety type and safety level. Training proceeds in two stages: Stage I freezes the base VLM and trains the safety projector, tokens, and head on a visual safety alignment dataset of approximately 14,000 examples; Stage II freezes the safety modules and LoRA-tunes the LLM on approximately 100,000 mixed clean and unsafe examples. On the RTVLM benchmark, SafeVLM (+LoRA) reaches 8.26, exceeding GPT-4V at 7.92. On harmful politics, porn, and cyberbullying, its average is 7.80 versus 5.06 for baseline LLaVA-v1.5-7B, while classifier performance reaches level accuracy 0.96 and F1 0.97, and type accuracy 0.94 and F1 0.97 (Liu et al., 2024).

VLM-Guard addresses the safety alignment gap between a text-aligned LLM and a multimodal VLM (Liu et al., 14 Feb 2025). It extracts a Safety Steering Direction from the LLM using harmful/harmless prompt pairs, then intervenes at inference time by projecting VLM representations relative to that subspace and selectively steering harmful inputs. The core projection is

$$\hat{h}^l(q) = h^l(q) - h^l(q)V_{m,l}^\top V_{m,l}.$$

Empirically, this sharply reduces Attack Success Rate. For MaliciousInstruct in text-only mode, ASR drops from 15% for vanilla LLaVA to 4%; with a blank image, it drops from 34% to 6%. For jailbreak instructions with a blank image, ASR drops from 46% to 7%. On MM-Harmful Bench, ASR drops from 72% to 6% (Liu et al., 14 Feb 2025). The stated goal is not new safety supervision, but restoration of the LLM’s existing refusal geometry after multimodal perturbation.

LlavaGuard pursues a more openly policy-conditioned safeguard model (Helff et al., 2024). Fine-tuned from LLaVA backbones at 7B, 13B, and 34B, it generates strict JSON with a binary rating, one of nine safety categories O1–O9 or NA, and a rationale. The policy is encoded as text through "Should not" and "Can" sections, and category exceptions can be declared at inference time. Full fine-tuning substantially outperforms LoRA, and LlavaGuard-34B reaches Acc 90.74%, Recall 87.50%, Specificity 93.98%, and PER 84.29% on the held-out test set (Helff et al., 2024). In large-scale dataset auditing, LlavaGuard-13B flags 16,640 unsafe images in ImageNet’s 1.3M images. A likely implication is that FailSafe-VLM in content moderation is increasingly treated as a programmable wrapper with structured outputs and auditable rationales, not merely as a binary NSFW classifier.

5. Semantic fallback and safety-critical scenario generation beyond manipulation

The maritime system "Semantic Lookout" extends the FailSafe-VLM pattern to short-horizon fallback maneuver selection under the draft IMO MASS Code (Christensen et al., 30 Dec 2025). It combines a fast anomaly detector with a slow semantic selector that receives a forward camera frame overlaid with numbered candidate trajectories and returns a strict JSON containing see, implications, action, choice_id, and confidence. The action space is deliberately constrained to one cautious maneuver or Station-keeping. On 40 harbor scenes, gpt-4.1 reaches awareness $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$0 at $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$1 s, gpt-5-minimal reaches $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$2 at $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$3 s, and gpt-5-high reaches $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$4 at $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$5 s, supporting the claim that sub-10 s models retain most of the awareness of slower state-of-the-art models (Christensen et al., 30 Dec 2025). For action selection, FB-3 with gpt-5-low achieves Accept@1 = 0.68 [0.52, 0.80] and Best@1 = 0.48 [0.33, 0.63], outperforming Keep-station at 0.45 and 0.38 and Keep-starboard at 0.50 and 0.13. On fire scenes at 60 s, it yields mean $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$6 m $$\mathbf{K}_p^{\text{final}} = \mathbf{K}_p^{\text{VLM}} \cdot \alpha_{\text{force}},$$7, whereas Keep-course yields −4.3 m and Keep-starboard yields −2.9 m (Christensen et al., 30 Dec 2025). The field run verifies alert → fallback maneuver → operator handover.

In autonomous driving, the FailSafe-VLM role often shifts from runtime fallback to robustness engineering. VILTA inserts a VLM directly into the closed-loop training of an autonomous driving policy as a trajectory adversary (Chen et al., 19 Jan 2026). Rather than producing textual hazard descriptions for a downstream generator, the VLM directly edits future waypoints of a risky agent in a BEV scene, after which B-spline smoothing, sigmoid fusion with a CTRV-plus-map baseline, and LQR tracking enforce physical plausibility. In aggregate CARLA results, VILTA obtains lower crash rate than VLM-RL and CAT in both challenging and normal scenarios: 1.03 versus 1.44 and 1.70 in challenging settings, and 0.63 versus 0.76 and 0.86 in normal settings (Chen et al., 19 Jan 2026). The paper therefore uses a VLM not as a deployed policy, but as a scene-aware adversary that exposes long-tail safety weaknesses during training.

"VLM as Strategist" introduces a related three-layer architecture for safety-critical test generation (Wu et al., 2 Dec 2025). The strategic layer uses a VLM plus an accident knowledge database derived from NHTSA’s 37 pre-crash typologies to choose adversarial agents and instantiate guidance functions; the tactical layer formulates adaptive guidance; the operational layer uses guided diffusion to generate realistic multi-agent trajectories in closed-loop simulation. On generated scenarios, collision rates increase sharply for several autonomous driving systems—for example, PDM-Hybrid from 8.3% to 83.3% and PDM-Closed from 8.3% to 66.7%—while at-fault collision rate rises by about 4.2× on average (Wu et al., 2 Dec 2025). These results belong to safety validation rather than deployed fail-safe control, but they fit the broader FailSafe-VLM agenda: VLMs can act as semantic hazard strategists that systematically surface failure modes before deployment.

6. Limitations, misconceptions, and research directions

A common misconception is that FailSafe-VLM implies formal safety guarantees. The literature does not support that interpretation. SAFE provides high-probability alarm thresholding via conformal prediction, but explicitly notes that exchangeability is only approximate under multitask shift (Gu et al., 11 Jun 2025). VLM-Guard is an inference-only intervention and does not repair underlying training-time misalignment (Liu et al., 14 Feb 2025). SafeVLM improves safety but is more conservative on safe XSTest instructions, where the response rate falls from 91.20 for baseline LLaVA-7B to 78.09 for SafeVLM (+LoRA) (Liu et al., 2024). LlavaGuard depends on a comparatively small safety dataset and synthetic rationale supervision (Helff et al., 2024). Semantic Lookout validates fast anomaly detection only in small-n tests and reports a field run in a single hazard family (Christensen et al., 30 Dec 2025). FailSafe-VLM for manipulation is trained and evaluated in simulation, with only three ManiSkill tasks (Lin et al., 2 Oct 2025). CompliantVLA-adaptor explicitly notes VLM latency, API cost, and the fact that a fail-safe wrapper cannot compensate for all high-level VLA errors (Zhang et al., 21 Jan 2026).

A second misconception is that fail-safety can be achieved by prompting alone. The dominant trend is the opposite: prompt engineering is almost always paired with hard structure. SafeVLM uses explicit safety heads and control codes (Liu et al., 2024). VLM-Guard acts at the representation level (Liu et al., 14 Feb 2025). CompliantVLA enforces force thresholds and critical damping (Zhang et al., 21 Jan 2026). Semantic Lookout uses candidate-constrained motion primitives and continuous human authority (Christensen et al., 30 Dec 2025). This suggests that the field increasingly treats VLMs as semantic components inside a constrained supervisory architecture, not as standalone safety mechanisms.

The main research directions already present in the literature are broadening failure taxonomies, extending from manipulation to multi-agent and multimodal settings, improving sim-to-real transfer, reducing over-blocking, incorporating richer safety metrics directly into generation or control objectives, and combining semantic monitoring with hard low-level constraints. A plausible implication is that future FailSafe-VLM systems will be hybrid by design: semantic VLM modules for hazard interpretation, diagnosis, or recovery suggestion; frozen or partially frozen task models for capability; and deterministic envelopes—thresholds, candidate sets, impedance laws, world-anchored primitives, or calibrated monitors—for final authority over unsafe states and actions.