Unsupervised Anomaly Detection Framework

• The unsupervised anomaly detection framework is a system that learns only from normal data to identify deviations without relying on labeled anomalies.
• It integrates techniques like continual prompting, structure-based contrastive learning, and memory modules that store task-specific keys and prototypes for efficient anomaly scoring.
• The framework achieves high detection accuracy (e.g., 0.930 AUROC on MVTec AD) while mitigating challenges such as domain shift and catastrophic forgetting.

Unsupervised Anomaly Detection Framework

Unsupervised anomaly detection frameworks aim to identify samples or regions in data that deviate from an unknown “normal” distribution, without relying on labeled anomaly data. These frameworks are central to domains where anomalies are rare, unpredictable, or expensive to annotate, such as industrial manufacturing, cybersecurity, and medical imaging. Fundamental challenges include catastrophic forgetting in continual scenarios, domain shift, high dimensionality, and the lack of ground-truth annotations for both methods and benchmarks.

1. Problem Definition and Core Objectives

Unsupervised anomaly detection frameworks are generally defined by the absence of anomaly labels during training. The primary objective is to distinguish between samples generated from a normal, but unknown, data distribution $P_{\mathrm{norm}}$, and samples drawn from an unseen, anomalous distribution $P_{\mathrm{anom}}$ that may only appear during testing. The central tasks include:

• Training a model $f(\cdot;\theta)$ solely on "normal" data, typically under a one-class assumption.
• Scoring unseen samples $x$ at test time, producing a scalar anomaly score $s(x)$, with higher values indicating a higher likelihood of abnormality.
• Localizing anomalies at the level of features, patches, image regions, or time intervals, as appropriate for the problem modality.
• For continual or lifelong scenarios, maintaining model performance on past tasks without catastrophic forgetting, ideally without storing raw training data.

In the continual setting as formalized in “Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt” (Liu et al., 2024), a sequence of object categories (or tasks) $\{1, ..., T\}$ is presented incrementally, with only task-specific normal data available per increment. The framework must learn to detect and localize anomalies in all categories after sequential training, while minimizing forgetting.

2. Key Methodological Components

Unsupervised anomaly detection frameworks encompass a range of architectures and innovations. The following methodological components are recurrent in state-of-the-art designs:

Continual Prompting and Memory Modules

The Unsupervised Continual Anomaly Detection (UCAD) framework (Liu et al., 2024) introduces a Continual Prompting Module (CPM), which stores for each task:

• A “key” vector representing task identity, sampled by farthest-point sampling from a feature layer of a frozen Vision Transformer (ViT).
• A set of task-specific “prompts” (learnable tokens), injected into the frozen ViT backbone at multiple layers.
• A coreset of normal patch embeddings representing “normal knowledge.”

During inference, similarity to stored keys is used for task identification, and corresponding prompts and normal coresets are used to obtain contextual, task-adaptive anomaly scores.

Structure-Based Contrastive Learning (SCL)

UCAD includes a structure-based contrastive loss that leverages segmentation masks from the frozen Segment Anything Model (SAM). The SCL objective drives patch-level features within the same SAM segment to cluster and those across segments to repel, enhancing within-class compactness and cross-task separability under the constraint that the base model remains frozen and only prompts are updated.

Joint Representation Learning, Likelihood Modeling, and Coresets

Many frameworks blend representation learning (autoencoders, contrastive learning, or deep metric learning) with a downstream density or nearest-neighbor estimation module. For pixel-wise localization in medical or industrial imaging, reconstruction losses or Gaussian Mixture Models (GMMs) are often employed to model per-pixel or patch embeddings (Liu et al., 2024, Kim et al., 2021), while coresets (prototypical patch sets) enable efficient nearest-neighbor retrieval for anomaly scoring and mitigate memory constraints.

Task-Agnostic Inference

A general principle is to avoid training and storing a separate model per task or data category; instead, a unified architecture leverages task-identity retrieval, conditional adaptation via prompts, or global embeddings.

3. Algorithmic Workflow and Optimization

A canonical unsupervised anomaly detection workflow proceeds as follows (specializing terminology for the continual task setting (Liu et al., 2024)):

1. Initialization: Start with an empty memory structure (e.g., key-prompt-knowledge bank).
2. Incremental Training (per task $t$):
   • Initialize prompts for the new task.
   • For each epoch on normal data $D_t^\mathrm{train}$:
     • Use SAM to segment each image, extract prompt-injected patch features, and compute the structure-based contrastive loss.
     • Update prompts via backpropagation of the SCL loss; the base encoder remains frozen.
     • Update normal coresets using coreset sampling on patch embeddings.
   • After task training, extract the task key from patch features.
   • Store the triplet (key, prompts, normal coreset) in memory.
3. Inference:
   • For a test sample $x$, extract features and find the most similar stored task key.
   • Retrieve and inject the corresponding prompts, extract adapted features, and compute patch-level anomaly scores (e.g., nearest-neighbor distance to the normal coreset).
4. No Replay, No Supervision: All training avoids raw sample replay and direct anomaly supervision.

An explicit pseudocode block from (Liu et al., 2024) (see paper for details) operationalizes these steps, ensuring efficient continual adaptation.

4. Evaluation Strategies, Datasets, and Metrics

Benchmarking unsupervised anomaly detection is complicated by the lack of anomaly annotations. Key strategies and metrics include:

• Datasets: MVTec AD (industrial, 15 classes, pixel-level masks), VisA (complex industrial surface defects), among others.
• Metrics:
  • Image-level Area Under the Receiver Operating Characteristic curve (AUROC)
  • Pixel-level Area Under the Precision–Recall curve (AUPR)
  • Forgetting Measure (FM), quantifying performance drop on previous tasks after training additional increments, with:

  $${\rm avg\ FM} = \frac{1}{T-1} \sum_{j=1}^{T-1} \max_{l<j} (\mathrm{AUC}_{l,j} - \mathrm{AUC}_{T,j})$$

• Ablations: Removing CPM and SCL in UCAD leads to substantial performance degradation (e.g., pixel AUPR drops from 0.456 to 0.183 on MVTec AD) (Liu et al., 2024).
• Comparison Baselines: DNE, UniAD (replay-based), PatchCore, and others.

Key results from (Liu et al., 2024):

Framework	MVTec AD Image AUROC	MVTec AD Pixel AUPR	MVTec AD avg FM	VisA Image AUROC	VisA Pixel AUPR	VisA avg FM
DNE	0.870	0.116	0.116	0.610	0.179	0.116
UniAD*	0.904	0.393	0.076	0.825	0.283	0.062
PatchCore*	0.669	0.190	0.318	0.633	0.181	0.349
UCAD	0.930	0.456	0.013	0.874	0.300	0.015

*Replay-based methods.

5. Design Principles, Innovations, and Theoretical Guarantees

Key innovations and principles validated by recent research include:

• Key-Prompt-Knowledge Memory: Abstracts the notion of “task identity” and “normal prototypes,” supports task-agnostic and memory-efficient continual anomaly detection without raw data replay (Liu et al., 2024).
• Contrastively-Learned Prompting: Drives adaptation at the prompt level, rather than full encoder retraining, aligning with continual learning constraints.
• Structure-based Contrastive Loss via SAM: Regularizes feature space to respect underlying image structure and semantic boundaries, directly improving anomaly localization and class separation.
• No Catastrophic Forgetting: The UCAD approach robustly preserves anomaly detection accuracy across increments, achieving low average forgetting metrics compared to replay and distribution-estimation baselines.
• Scalability and Efficiency: The total memory footprint scales linearly with the number of tasks (e.g., ∼23 MB for 15 tasks), facilitating practical lifelong deployment.

6. Limitations, Open Issues, and Future Directions

Despite their strengths, current unsupervised anomaly detection frameworks face several limitations:

• Domain Shift Sensitivity: Dependence on frozen pretrained backbones (e.g., ViT on ImageNet) and off-the-shelf segmentation models like SAM leads to performance degradation under strong distribution shift; retrieval and adaptation of prompts/keys rely on domain similarity.
• Memory Growth: Storing a distinct set of keys, prompts, and coresets per task means memory requirements grow linearly with the number of increments or categories. Strategies such as memory pruning or dynamic compression could alleviate this.
• Modality Generalization: While the prompting and coreset paradigm generalizes, extensions to videos or 3D point clouds require additional consideration (e.g., motion segmentation or geometric priors).
• Absence of Raw Image Storage: The constraint of not retaining previous raw images may complicate some scenarios where sample replay could benefit adaptation.
• Algorithmic Complexity: While the overall memory cost is modest, the computation for structure-based contrastive learning and coreset selection can be non-trivial, especially for large-scale or real-time scenarios.

Potential extensions identified include dynamic adaptation to new task modalities, online pruning of memory banks, and leveraging structure priors beyond static images (e.g., motion in surveillance videos) (Liu et al., 2024).

7. Context within the Broader Anomaly Detection Landscape

The UCAD framework represents a significant step in equipping unsupervised anomaly detection with truly continual, task-agnostic, and segmentation-capable capabilities, directly addressing the lack of supervision and catastrophic forgetting without reliance on task-specific tuning or replay. By integrating prompting, structure-driven contrastive learning, and compact memory organization, it demonstrates state-of-the-art results on challenging industrial benchmarks.

This work exemplifies a growing trend in unsupervised anomaly detection: leveraging advances in foundation models, prompt engineering, and structure-aware self-supervision to transcend the limitations of per-task modeling. It establishes new design patterns for both academic investigation and industrial deployment, setting a new methodological paradigm for lifelong anomaly detection and segmentation (Liu et al., 2024).