Semantic Anomaly Detection with Large Language Models (2305.11307v2)

Abstract: As robots acquire increasingly sophisticated skills and see increasingly complex and varied environments, the threat of an edge case or anomalous failure is ever present. For example, Tesla cars have seen interesting failure modes ranging from autopilot disengagements due to inactive traffic lights carried by trucks to phantom braking caused by images of stop signs on roadside billboards. These system-level failures are not due to failures of any individual component of the autonomy stack but rather system-level deficiencies in semantic reasoning. Such edge cases, which we call semantic anomalies, are simple for a human to disentangle yet require insightful reasoning. To this end, we study the application of LLMs, endowed with broad contextual understanding and reasoning capabilities, to recognize such edge cases and introduce a monitoring framework for semantic anomaly detection in vision-based policies. Our experiments apply this framework to a finite state machine policy for autonomous driving and a learned policy for object manipulation. These experiments demonstrate that the LLM-based monitor can effectively identify semantic anomalies in a manner that shows agreement with human reasoning. Finally, we provide an extended discussion on the strengths and weaknesses of this approach and motivate a research outlook on how we can further use foundation models for semantic anomaly detection.

• The paper introduces a novel LLM-based method to detect semantic anomalies in autonomous robotic systems.
• It transforms vision inputs into textual descriptions using open vocabulary detectors and structured prompts.
• Experiments in driving and manipulation tasks demonstrate improved anomaly detection compared to traditional OOD methods.

Semantic Anomaly Detection with LLMs

The paper "Semantic Anomaly Detection with LLMs" addresses a pertinent challenge in the context of autonomous robotic systems—namely, the identification and handling of semantic anomalies within complex environments. As these systems become increasingly prevalent in various domains, such as autonomous driving and robotic manipulation, the imperative to safeguard against non-trivial failure modes becomes critical. This paper presents a novel approach to semantic anomaly detection utilizing LLMs to monitor and reason about potential discrepancies in visual input that could lead to erroneous or unsafe behaviors in autonomous systems.

Overview and Methodology

Robotic systems often rely on learned components that are susceptible to out-of-distribution (OOD) inputs, which undermine their ability to generalize beyond their training datasets. The authors propose leveraging the contextual reasoning capabilities inherent in LLMs to detect semantic anomalies—scenarios where individual elements may appear nominal, but their combination presents atypical or misleading patterns. The monitoring framework introduced in this paper converts vision-based observations into natural language descriptions, which are analyzed by an LLM through structured prompts designed to identify task-relevant anomalies.

The methodology emphasizes the transformation of these observations into textual scene descriptions using open vocabulary object detectors, integrating these descriptions within prompt templates tailored for the LLM. This enables the system to simulate human-like reasoning in identifying scenarios that might confound or disrupt autonomous decision-making processes. To validate this approach, the authors design and evaluate it across two fundamental systems: an autonomous driving system and a learned manipulation policy.

Experimental Results

In the autonomous driving domain, implemented within the CARLA simulator, the paper investigates how effectively the proposed LLM-based monitor can detect contextual anomalies—such as traffic lights mounted on moving trucks or images of stop signs on billboards—anomalies that real-world systems like Tesla's might encounter. The results indicate that the LLM correctly identifies most semantic anomalies while maintaining a reasonably low false positive rate in nominal scenarios. When object detection was flawed due to the simulation's visual fidelity, the LLM relied on the semantic content rather than mere visual features.

For learned manipulation policies, the paper explores whether LLMs can discern distractors in a tabletop manipulation task. Despite inherent policy randomness, the LLM was able to reason about visual distractors and align anomaly classifications with human intuition, better than common OOD detection baselines that predominantly depend on visual distinctiveness.

Comparative Analysis and Implications

The paper contrasts the LLM approach with traditional OOD detection methods like SCOD and Mahalanobis distance metrics, illustrating that these baseline methods fall short in scenarios where semantic context supersedes simple visual anomaly identification. Such traditional methods often fail to identify system-wide anomalies since they focus on model uncertainty or visual distinctiveness instead of semantic misalignments.

The implications of this research extend beyond immediate anomaly detection. It opens avenues for embedding semantic reasoning within robotic systems that mirror human-like insight, crucial for complex and safety-critical applications like autonomous driving. Moreover, as foundation models continue to evolve, integrating multimodal capabilities could enhance the fidelity and applicability of such frameworks across diverse robotics applications.

Future Directions

The authors identify several key areas for future exploration:

1. Multimodal Context: Incorporating visual inputs directly into LLM prompts to better preserve context may enhance detection fidelity.
2. System Grounding: Explicitly informing LLMs about specific system capabilities through fine-tuning can improve contextual grounding.
3. Complementary Techniques: Combining LLM-based methods with robust OOD detectors can provide a broader coverage for different failure modes.

In conclusion, this paper provides a compelling advancement in employing LLMs for semantic anomaly detection, proposing a flexible, insightful approach to address challenges in autonomous system reliability. The results suggest a promising trajectory for further development and integration of LLM-based semantic reasoning in real-world robotic applications.