I-FailSense: Vision-Language Robotic Failure Detection

• I-FailSense is a framework for robotic failure detection that classifies visual trajectories as success or failure by identifying semantic misalignments between execution and instruction.
• It employs a two-stage method with LoRA-based post-training and lightweight FailSense blocks that leverage multi-layer representations for improved semantic reasoning.
• The system outperforms zero-shot baselines on benchmarks like SMF-CALVIN, demonstrating strong transfer to diverse failure types and real-world conditions.

I-FailSense is a vision-language-model framework for robotic failure detection in language-conditioned manipulation, with a deliberate emphasis on semantic misalignment failures: cases in which a robot performs a physically plausible manipulation that is inconsistent with the given instruction. In the formulation introduced in 2025, the task is to classify a visual trajectory as success or failure with respect to a language goal, rather than merely to detect low-level control breakdowns. The framework combines post-training of a base VLM with lightweight internal-layer classifiers, termed FailSense blocks, and aggregates their outputs through grounded arbitration. The resulting system is reported to outperform zero-shot VLM baselines on semantic misalignment detection, to transfer to broader robotic failure categories, and to retain utility under out-of-distribution simulation and real-world transfer settings (Grislain et al., 19 Sep 2025).

1. Problem formulation and failure semantics

I-FailSense studies language-conditioned manipulation as a goal-conditioned POMDP,

$$\mathcal{M} = (S, A, \mathcal{T}, \rho_0, \Omega, O, G),$$

in which the robot observes a partial visual state $O(s)$, acts in $A$, and receives a language goal $g \in G$. A trajectory is written as

$\tau = (o_0,\dots,o_T),$

and the detection problem is to classify $\tau$ as success or failure with respect to $g$ (Grislain et al., 19 Sep 2025).

The framework is centered on a distinction between control errors and semantic misalignment failures. The former include failed grasps, dropped objects, and wrong rotations; these are often visible directly in the scene. The latter are subtler. A robot may rotate the correct object in the wrong direction, lift the wrong block, or open a drawer when instructed to move a slider. Such behavior can be semantically coherent in isolation while still being wrong relative to the instruction. The paper therefore argues that a detector must jointly reason over the language instruction, the identity and relation of objects, and the temporal trajectory of robot motion (Grislain et al., 19 Sep 2025).

This framing is important because it rejects a common simplification in robotic monitoring: the assumption that failure is exhausted by obvious physical non-execution. In I-FailSense, a trajectory may be dynamically smooth and visually plausible yet still count as failure if instruction grounding is violated. The target of detection is therefore not merely mechanical breakdown but instruction-trajectory inconsistency.

2. Dataset construction for semantic misalignment

A major contribution is a procedure for constructing semantic misalignment detection datasets from existing expert-demonstration corpora. Starting from expert trajectories with correct language annotations $\{\tau, g^*\}$, the authors define a derived dataset of pairs $(\tau, g, y)$, where $y \in \{0,1\}$ is the success label. Positive examples are the original expert pairs $O(s)$0. Negative examples are built by pairing the same trajectory $O(s)$1 with a different instruction $O(s)$2, but crucially from the same task category, so that the mismatch remains semantically plausible rather than trivial (Grislain et al., 19 Sep 2025).

This yields two purpose-built resources. SMF-CALVIN is constructed from CALVIN task_D and spans 34 manipulation tasks grouped into six categories: lifting, rotating, pushing, opening/closing, placing, and lighting. SMF-DROID is constructed from real-world DROID demonstrations and contains 6K training and 276 test pairs, half positive and half negative. The work also evaluates on AHA, based on RLBench with 79 tasks and seven failure categories; because the original dataset was not publicly released, the paper uses a test set of 400 negative trajectory-instruction pairs (Grislain et al., 19 Sep 2025).

The negative-pair construction is methodologically significant because it forces the detector to distinguish between trajectories that remain task-like and trajectories that are actually instruction-consistent. This is a much harder setting than random mismatch generation. A plausible implication is that the dataset design itself functions as a curriculum for semantic grounding: the detector is trained not to reject implausible motion, but to reject plausible motion with the wrong referent or relation.

3. Architecture, post-training, and grounded arbitration

I-FailSense is built on PaliGemma2-mix-3B, chosen as a compact VLM backbone. The method has two stages. In the first stage, the base VLM is post-trained with LoRA. The input is a flattened trajectory image plus instruction, with multiple frames and viewpoints arranged into a single image,

$O(s)$3

and the instruction is inserted through a fixed prompt template $O(s)$4. Adaptation consists of fine-tuning the projection MLP between the frozen vision encoder and the LLM, together with LoRA modules in the key-query-value projection layers of the LLM attention blocks. Supervision is standard token-level cross-entropy against a target token such as <success> or <fail> (Grislain et al., 19 Sep 2025).

In the second stage, the VLM is frozen and lightweight binary classifiers called FailSense blocks (FS blocks) are attached to evenly spaced internal layers. For each selected layer $O(s)$5, a block receives the feature

$O(s)$6

Each FS block includes a hybrid attention pooling module, MLP layers with residual connections and batch normalization, and a final binary classifier producing a probability $O(s)$7. The blocks are trained independently with binary cross-entropy against the success/failure label (Grislain et al., 19 Sep 2025).

Inference combines these internal predictions with the base model’s own decoded output through grounded arbitration. Each FS block produces a binary decision, the VLM’s free-form output is converted to a binary prediction, and the final decision is made by weighted voting. In the default configuration, $O(s)$8, all FS blocks have weight 1, and the VLM output has weight 2, so that it acts as a tie-breaker (Grislain et al., 19 Sep 2025).

The architectural claim is not merely that auxiliary heads improve supervision, but that semantic failure detection benefits from explicit access to multiple representational depths. Different internal layers encode different levels of abstraction, and the detector is designed to exploit that stratification rather than to rely only on the final VLM output.

4. Empirical results and transfer behavior

On SMF-CALVIN, zero-shot VLMs are reported to be weak. GPT-4o reaches only about 0.58–0.63 accuracy, PaliGemma2-mix-3B is near chance, and Qwen2.5-VL-7B reaches about 0.68–0.69. I-FailSense improves substantially over these baselines, achieving 0.9064 accuracy, 0.9132 F1 with one exocentric view and 0.8818 accuracy, 0.8909 F1 with two views (Grislain et al., 19 Sep 2025).

Ablation results isolate the contribution of the two training stages. For one view, the base model is about 47% accuracy, LoRA only yields 85.7%, and the full model reaches 90.6%. For two views, the base model is about 52%, LoRA only yields 82.3%, and the full model reaches 88.2%. The paper interprets this as evidence that stage 1 teaches the model the failure-detection task, while stage 2 leverages intermediate representations more effectively than the final VLM output alone (Grislain et al., 19 Sep 2025).

Generalization is evaluated in three directions. First, although I-FailSense is trained only on semantic misalignment, it transfers strongly to AHA, which is mostly control-error focused, reaching 89% accuracy and outperforming AHA-trained baselines at 69–70%. Second, it generalizes from CALVIN to RLBench despite differences in objects, scenes, and camera settings. Third, on SMF-DROID, direct transfer from CALVIN helps but is not best; the strongest results come from fine-tuning FS blocks on DROID itself, reaching 0.7428 accuracy and 0.7300 F1 with two views (Grislain et al., 19 Sep 2025).

The real-world transfer results are technically notable because the best zero-shot accuracy is matched with roughly half the parameters, and because combining exocentric and egocentric views helps more in real-world data than in simulation. The paper attributes this to greater clutter and lower scene stability in real environments. At the same time, the reported drop from simulation to real-world performance makes clear that semantic misalignment detection remains sensitive to domain shift.

5. Position within the broader failure-detection literature

The term I-FailSense is attached most directly to the robotic VLM framework, but related papers outline a broader landscape of failure sensing in which unsafe latent states are inferred from externally observable evidence rather than privileged internal access. In autonomous driving, an intelligent intersection manager observes vehicle poses $O(s)$9 and estimates a latent unsafe state $A$0 from pose history, warning cross-traffic if the predicted failure probability exceeds a threshold. The corresponding recurrent classifier, FailureNet, is trained on control failures, upstream perception errors, speeding drivers, and manual reckless driving, and reports 84% overall accuracy on hardware in the MiniCity testbed (Buckman et al., 2023).

In Safety Instrumented Systems, Failure Mode Reasoning formalizes failure as a relation between the value reported by the program and the value that should have been reported, abstracts failure states into finite failure modes, and reasons backward from undesirable outputs to possible input failure modes. The approach is compositional over function blocks and is stated to be sound and complete relative to the abstraction used (Jahanian et al., 2020). In software debugging, failure indexing based on run-time values of program variables represents each failed execution as a dictionary of breakpoint-local variable values and clusters failures by a custom two-level distance metric, yielding reported improvements of 44.12% and 27.59% in faults number estimation and 47.30% and 26.93% in clustering effectiveness over MSeer in simulated and real-world environments, respectively (Song et al., 2023).

Other adjacent formulations emphasize weighted trust, observability engineering, or information-path traceability. The Impact Failure Detector assigns each monitored process an impact factor $A$1, computes

$A$2

and compares it with

$A$3

to determine whether a set remains trusted under crash failures (Rossetto et al., 2014). In segmented waveguide-enabled pinching-antenna systems, tagged pilots provide per-segment signatures that make otherwise passively combined uplink signals identifiable; the resulting framework uses per-segment ML detection for $A$4 and LASSO-based sparse recovery for $A$5 (Ouyang et al., 19 Feb 2026). In digital instrumentation and control safety analysis, Unsafe Information Flow (UIF) and intermediate processors extend RESHA-style fault-tree construction into feedback-heavy systems such as nuclear plant human-system interfaces, where the relevant hazard is often wrong, missing, mistimed, or misvalued information rather than an unsafe control action (Chen et al., 2022).

This suggests that I-FailSense belongs to a larger methodological family in which failure detection is framed as the inference of hidden unsafe conditions from trajectories, variable-state signatures, weighted trust summaries, tagged signals, or structured information flows. What changes across domains is the observable substrate and the failure ontology; what remains stable is the emphasis on externalized evidence and discriminative or compositional inference.

6. Limitations, interpretation, and significance

The robotic I-FailSense paper identifies several practical limits through its experiments. Performance drops when transferring directly from simulation to real-world data without real-world adaptation; real-world success improves with minimal post-training on DROID; and semantic misalignment remains subtle enough to produce precision-recall imbalance in harder settings (Grislain et al., 19 Sep 2025). These limitations are not incidental. They indicate that the hardest cases are those in which the visual scene is cluttered, the temporal trajectory is semantically plausible, and the mismatch resides in instruction grounding rather than in obvious control failure.

Within the wider literature, comparable constraints recur in different forms. Infrastructure-side trajectory monitoring is trained on a limited set of failure types and is evaluated in a 1/10th-scale physical city rather than full-scale roads (Buckman et al., 2023). Failure Mode Reasoning does not model non-termination or partial functions and focuses on input faults rather than systematic internal software bugs (Jahanian et al., 2020). UIF-based RESHA extensions are primarily qualitative and leave quantification and full root-cause analysis to future work (Chen et al., 2022). These parallels suggest that failure sensing is consistently bounded by observability assumptions: what can be inferred is constrained by what is made visible, whether as trajectory history, internal VLM features, program abstractions, variable values, or tagged measurements.

The significance of I-FailSense lies in treating semantic alignment itself as the central object of failure detection. Its empirical message is that training on semantic misalignment is not merely useful for detecting “wrong object” mistakes; it yields a broadly transferable success/failure critic that can detect broader robotic failure categories and move across simulation and real-world settings with zero-shot or minimal post-training (Grislain et al., 19 Sep 2025). In that sense, I-FailSense marks a shift from detecting whether motion was executed to detecting whether the executed motion remained faithful to the instruction.