Fail-RAG : A Retrieval Augmented Generation Informed Framework for Robot Failure Identification

Abstract: Industry automation is witnessing an evolution in robotics driven by both technological breakthroughs and societal changes: progress towards generalist robots, embodied and physical AI, and increasing labor shortage in manufacturing.An intelligent autonomous robot needs to not only act according to planned motions but also react to any unexpected events. In this study, we focus on such unexpected events in warehouses where robots are used for material handling. Specifically, we refer to any unexpected events as failures and develop methods to detect robot operations related failures. Rule-based detection methods may break since the form of failures could change due to the dynamic nature of both environments and tasks. We propose 'Fail-RAG', a Retrieval Augmented Generation (RAG)-based failure detection framework where failure images and context information are embedded and queried against a failure database by calculating their similarities. Vision-LLMs (VLMs) are further used to analyze failures and provide details by following our instruction template. We evaluated the performance of Fail-RAG by conducting both simulation and physical experiments using fixed robot arms and a mobile manipulator for multiple tasks that are common in warehouse automation. Fail-RAG achieved 25 percentage point higher failure detection accuracy on average across five types of robot operations compared to using off-the-shelf VLMs, indicating its effectiveness for real-world failure detection.

• The paper introduces Fail-RAG, a three-stage framework that retrieves similar failure cases with CLIP embeddings and prompts an off-the-shelf VLM to generate structured JSON diagnoses without model training.
• Fail-RAG improves robot failure detection accuracy by about 25 percentage points on average, with gains of up to 40 points across five simulated and physical warehouse-style operations.
• The paper finds that greater separation between failure-scenario embeddings generally predicts better accuracy, offering a practical method for curating RAG databases while highlighting limitations for navigation-heavy tasks and latency-sensitive deployments.

Fail-RAG addresses robot failure identification by combining retrieval augmented generation (RAG) with off-the-shelf vision-LLMs (VLMs), avoiding the fine-tuning pipelines that dominate contemporary approaches. The framework embeds time-sequenced camera frames and their ground-truth failure scenarios as CLIP vectors, retrieves the most similar failure cases at inference time, and prompts a VLM to produce structured JSON diagnoses. Evaluated on five warehouse-style operations spanning simulation and physical hardware, Fail-RAG improves failure detection accuracy by roughly 25 percentage points on average over vanilla VLM inference, with gains up to 40 points, while requiring no model training.

Motivation and positioning

The authors motivate the work from the observation that rule-based fault detection and specialized sensor installations scale poorly across general-purpose robots operating in dynamic environments. VLM-based monitoring offers a camera-only alternative, but most recent VLM/VLA failure-detection systems rely on fine-tuning on curated failure datasets — for example AHA (Duan et al., 2024), FailSafe (Lin et al., 2 Oct 2025), Guardian (Pacaud et al., 1 Dec 2025), ViFailback (Zeng et al., 2 Dec 2025), I-FailSense (Grislain et al., 19 Sep 2025), and SAFE (Gu et al., 11 Jun 2025). The paper identifies two gaps: (a) fine-tuning-free frameworks that can be dropped into existing operation workflows without capturing large vision-action failure corpora or sourcing heavy compute, and (b) systems that exploit VLM semantic reasoning not only to flag failures but to explain their likely causes, which matters for human-in-the-loop intervention. Fail-RAG is positioned against both fine-tuned detectors and lighter-weight alternatives such as mixture-of-experts anomaly detection [10.1109/IROS6(0139.2025).11245878] and behavior-tree-integrated VLM monitors (Ahmad et al., 2024), which the authors argue are limited to categorical outlier detection without semantic reasoning about failure causes.

Framework formulation

The pipeline has three stages. First, an inspection video stream is sampled at one of several frame rates and compiled into a single composite image containing time-ordered frames. Second, this image is encoded into a $\mathbb{R}^{512}$ vector using CLIP (ViT-B/32) (Radford et al., 2021) and queried against a pre-built RAG database of embedded failure scenarios using cosine similarity, dot product, or $\mathcal{L}_2$ distance; cosine similarity was ultimately selected because it performs equivalently for image and text embeddings. Third, the top-ranked retrieved failure cases are supplied as context to an off-the-shelf VLM (Qwen2.5-VL 32B, served via Ollama) together with a task-specific prompt and a structured JSON answer template.

The instruction template enforces four attributes per operation: failure types (e.g., robot_arm, pallet, flipping_station), a deterministic status constrained to normal, anomalous, or unknown, a sub-failure identifier per type, and an unconstrained natural-language reason field for operator debugging. The authors report that constraining the status vocabulary significantly improved absolute accuracy relative to free-form responses, though they do not quantify this ablation separately. The structured output is designed for direct API integration.

Experimental design

Five operations were evaluated: simulated palletizing (UR5), MoMa transport and traverse (Ridgeback base + Franka arm) in NVIDIA Isaac Sim, and physical depalletizing (Fanuc M-20iB) and assembly (UR5). The RAG dataset spans 2–6 failure variations per operation, captured at five sampling frequencies between 0.05 Hz and 2.5 Hz, with five distinct instances per (operation, scenario, frequency) tuple and three held-out validation images per tuple — roughly a 60–40 train/validation split. The sampling-rate variation was intended to probe the trade-off between computational load and information content of the query image.

Results

Retrieval-only evaluation shows that performance depends more on the interaction between distance metric and operation than on either factor alone. $\mathcal{L}_2$ distance and cosine similarity perform well across operations and frame rates, while MoMa–Transport degrades under all metrics and frequencies — a notable negative result, since that operation involves whole-scene appearance changes during navigation rather than localized manipulation errors. Dot product is less consistent.

In the full VLM-RAG comparison, Fail-RAG outperforms the vanilla VLM baseline for all operations except one MoMa–Transport scenario, with average accuracy gains around 25 percentage points and individual gains up to 40. Gains are larger on the physical experiments (depalletizing, assembly) than in simulation, which the authors attribute to shadows and lighting variation making retrieved reference cases more informative in real-world imagery. No consistent trend emerged between frame rate and accuracy — best performance occurred at different rates per operation (15 frames for palletizing, 5 for MoMa–Transport) — motivating the embedding-space analysis below.

Embedding-space analysis

The central analytical contribution is a hypothesis that RAG performance is governed by how distinct the failure-scenario embeddings are from one another within the database. Cosine-distance heatmaps over all pairwise embeddings for each operation–frame-rate combination show substantial variability: some databases have tightly clustered scenario embeddings (low mean cosine distance), others well-separated ones. Plotting Fail-RAG accuracy against mean intra-database cosine distance reveals a positive trend — greater separation between scenario embeddings yields higher detection accuracy — albeit with outliers the paper does not fully characterize. This finding has a practical implication: RAG dataset curation can be guided by measuring embedding separability before deployment, potentially reducing data-collection effort. However, the relationship remains empirical; the authors explicitly note that no mathematical formalization of it is provided.

Limitations and open questions

Several limitations are acknowledged or evident. The composited multi-frame query introduces detection latency whose acceptability is task-dependent; the authors propose latent-space processing or additional viewpoints as candidate remedies but do not evaluate them. The positive cosine-distance/accuracy trend lacks formal modeling, and its outliers are unexplained. MoMa–Transport's poor retrieval performance suggests the framework may not transfer cleanly from manipulation to navigation-centric tasks where scene-level appearance dominates the CLIP embedding. Evaluation uses a single VLM family (Qwen) and a single encoder (CLIP ViT-B/32), so cross-model generality is untested. Finally, the claimed benefit of constrained response vocabularies is asserted but not isolated experimentally.

Conclusion

Fail-RAG demonstrates that retrieval augmentation over a modest library of embedded failure scenarios substantially improves off-the-shelf VLM failure identification in industrial robot operations, without fine-tuning, and offers an actionable diagnostic — inter-scenario embedding distance — for predicting where the approach will succeed. The open questions left by the work concern formalizing the embedding-separability/accuracy relationship, extending the approach beyond manipulation to navigation tasks, and reducing detection latency.