---
title: Zero-Shot Anomaly Classification & Segmentation
url: https://www.emergentmind.com/topics/zero-shot-anomaly-classification-and-segmentation-ac-as
type: topic
---

# Zero-Shot Anomaly Classification & Segmentation

Zero-shot Anomaly Classification and Segmentation (AC/AS) refers to the automatic identification and delineation of defects or anomalies in data (typically images, but also 3D volumes or multimodal inputs) without access to any labeled anomaly samples from the target distribution. In the zero-shot paradigm, models must generalize their notion of abnormality from either unlabeled test batches or from auxiliary datasets (often with prompt- or language-derived supervision), precluding any target-specific anomaly fine-tuning. This has become the dominant formulation for industrial inspection, open-world perception, and medical imaging, where anomaly distributions are vast, shifting, or costly to annotate.

## 1. Core Principles and Problem Formalization

In zero-shot AC/AS, the model is given either (a) only the unlabeled images and asked to predict per-image (classification) and per-pixel (segmentation) anomalies, or (b) auxiliary data from unrelated domains/classes to support prompt learning or feature alignment. No anomaly data from the test domain is available for training or threshold selection [2305.17382][2512.01427][2310.18961][2303.14814][2401.16753][2511.10047][2510.10456][2512.02520].

The standard mathematical formalization is as follows:

- Let $\mathcal{D} = \{I_1, \dots, I_N\}$ denote a batch of test images, each tokenizable into $M$ patches/volumetric tokens with features $\mathbf{z}_i^h \in \mathbb{R}^D$.
- The anomaly scoring function $a : \mathbb{R}^D \to \mathbb{R}$ is defined so that higher $a(\mathbf{z})$ indicates greater anomaly.
- Image-level classification: $A(I_i) = \max_h a(\mathbf{z}_i^h)$; decision threshold $\tau$ yields binary label [2512.02520].
- Anomaly segmentation: apply $a(\mathbf{z}_i^h)$ map over all $h$, producing a heatmap or binary mask after thresholding.

Two main paradigms exist:
- **Batch-based/representation-driven methods**: assign anomaly scores by mutual comparison of patch features within the batch, based on the empirical rarity of their representations [2401.16753][2512.02520][2511.10047].
- **Auxiliary-trained/prompt-driven methods**: align learned visual/textual representations via prompt engineering or prompt learning on auxiliary data, enabling anomaly/normality discrimination in joint embedding space [2305.17382][2310.18961][2303.14814][2506.23577][2508.03447][2504.06740].

The zero-shot regime is further complicated by the phenomenon of **consistent anomalies**: anomalous patterns that recur frequently, causing naive rarity-based scoring to fail and prompting the need for advanced graph, community-detection, or joint prompt-masked learning [2512.02520][2510.10456].

## 2. Representation-based and Mutual Scoring Approaches

Batch-based zero-shot AC/AS approaches assign anomaly scores entirely within the unlabeled test batch, eschewing any prompts or language structure:

- **Mutual Scoring Mechanism (MSM)**: For each patch, compute its minimum feature distance to patches in all other images. The anomaly score is typically the mean minimal distance to the $K$-nearest images ($K=30\%$ by default) [2401.16753]. Multi-scale feature aggregation (LNAMD/SNAMD) across different receptive fields and transformer layers improves detection of both small and large defects [2401.16753][2511.10047].
- **Constraint-based Re-scoring**: To reduce false positives, especially from noisy isolated patches in otherwise normal images, a graph-smoothing procedure over image-level similarity graphs is employed [2401.16753]. Nodes represent images, edges reflect dot-product similarity of their global (CLS/token) embeddings, and final scores are refined via local neighbor averaging.
- **Multimodal/3D Extensions**: Recent advances (MuSc-V2) extend all representation-based operations to work with both 2D (image) and 3D (point cloud or volumetric) modalities, using curvature-aware neighborhood pooling in 3D and similarity-weighted pooling across multiple scales in both 2D and 3D [2511.10047].

Batch-based approaches are fully training-free, require no auxiliary dataset or normal/defect labeling, and achieve SOTA metrics especially on out-of-distribution industrial datasets:
- MVTec AD: AUROC$_{\text{cls}}$ 97.8%, F1$_{\text{seg}}$ 62.6%, PRO 93.8% [2401.16753]; MVTec 3D-AD: AP-seg 54.7% (+23.7 pp over prior SOTA) [2511.10047].

## 3. Prompt-Driven, Language- and Vision-Language Model-Based Methods

Methods built on vision-language models (VLM) such as CLIP leverage semantic alignment between images and text to enable flexible anomaly reasoning:

- **Prompt Engineering and Learning**: Prompt sets (state words × templates) are crafted to represent generic “normal” and “abnormal” states (“flawless [object]”, “damaged [object]”) [2303.14814][2305.17382]. Compositional prompt ensembles (WinCLIP) and stacked/clustering-driven prompts (StackCLIP) offer improved generalizability and stability [2506.23577].
- **Object-Agnostic Prompt Learning**: AnomalyCLIP learns two short prompt vectors (“[object]”, “damaged [object]”) optimized on auxiliary domains, avoiding overfitting to specific seen classes and generalizing normal/abnormal concepts [2310.18961].
- **Conditional / Dynamic Prompt Synthesis**: CoPS synthesizes prompts conditioned on image/patch features and class semantics, injecting prototypes and sampled VAE tokens into prompt structure, enabling highly adaptive state modeling [2508.03447].
- **Multi-Type and Open-Vocabulary Segmentation**: MultiADS extends zero-shot AS to multi-type segmentation, generating specific masks (and class probabilities) for each defect type using defect-aware textual prompts and layered similarity heads [2504.06740]. In open-world tasks (Clipomaly), VLMs dynamically expand segmentation vocabularies at inference by discovering unknown regions and assigning interpretable names via CLIP-based region matching [2512.01427].
- **Few-Shot Memory Extensions**: APRIL-GAN and WinCLIP+ store features from a small set of normal reference images and compare against them at inference, significantly boosting performance on logical defects and hard-to-prompt anomalies [2305.17382][2303.14814].

In all cases, classification and segmentation scores are derived from similarity logits between learned/prompted text embeddings and spatial (patch) or global (CLS) image features, typically normalized via softmax. The best-performing approaches fuse features across several transformer depths/stages.

Representative results:
- AnomalyCLIP: Industrial dataset average Img-AUROC 91.5%, Pix-AUROC 91.1% [2310.18961].
- WinCLIP (ViT-B/16+): MVTec-AD (zero-shot) AC AUROC 91.8%, AS pixel-AUROC 85.1%; + few-shot (WinCLIP+) AUROC 95.2%, pixel-AUROC 96.2% [2303.14814].
- StackCLIP: MVTec-AD zero-shot AS AUPRO 86.4%, F1-max 47.6%, AUROC 91.7% [2506.23577].

## 4. Dealing with Consistent Anomalies: Graph-Based and Theoretical Advances

Classic representation methods fail when faced with **consistent anomalies**—recurrently appearing defects that are no longer rare in batch statistics. This scenario is theoretically and algorithmically formalized as follows:

- **Neighbor-Burnout Phenomenon**: Consistent anomalies exhibit abruptly increasing growth rates in sorted batchwise distances (log-gradients spike at the burnout index $i=H$, the number of repeated anomaly images) [2512.02520][2510.10456]. For normal patches, this growth rate $\tau_i$ decays smoothly as $i^{-\alpha}$ (power-law governed by EVT).
- **CoDeGraph Algorithm**: Constructs an image-level graph where nodes with shared suspicious low-distance patterns (weighted endurance ratio $\zeta'$) are joined, and community detection (Leiden+CPM) isolates densely co-matching clusters. Patches in detected anomaly communities are selectively filtered based on their dependency ratio (see $r(p)$), yielding a cleaned batch for robust mutual scoring [2512.02520][2510.10456].
- **Theoretical Foundation**: The statistical signature is grounded in extreme value theory (Fréchet/Beta/Exp distributions of sorted minimal distances), predicting (and separating) the scaling patterns of normal vs. consistent-anomaly patch neighbor relationships [2512.02520].
- **Batch-Text Bridging**: CoDeGraph-generated pseudo-masks can supervise prompt-driven models like APRIL-GAN or AnomalyCLIP, closing the domain adaptation gap and reducing reliance on labeled anomaly masks [2512.02520].

Empirically, CoDeGraph achieves robust zero-shot performance, notably excelling on consistent-anomaly-heavy datasets:
- MVTec-CA: AC AUROC 98.5%, AS F1 73.8%, AP 77.2% [2510.10456].

## 5. Practical Implementations, Robustness, and Evaluation

Zero-shot AC/AS methods are evaluated primarily on industrial (MVTec AD, VisA, MPDD, BTAD), medical (HeadCT, BrainMRI, ISIC, etc.), and open-world datasets (SMIYC, RoadAnomaly, Cityscapes anomaly tracks) [2401.16753][2310.18961][2506.23577][2512.01427].

Key implementation attributes:
- **Training vs. Training-Free**: Batch-based mutual scoring and Maskomaly are fully training-free; prompt-based models require auxiliary dataset fine-tuning or pseudo-mask bootstrapping.
- **Inference and Complexity**: Mutual scoring and graph-based models require $O(N^2)$ memory/computation; subset partitioning and attention-layer optimization mitigate costs.
- **Thresholding**: Optimal segmentation/classification thresholds are typically chosen per-dataset or via validation, although metrics such as Maximal Detection Margin (Maskomaly) gauge robustness to gradual threshold shifts [2305.16972].
- **Modalities**: MuSc-V2 and CoDeGraph now support 3D volumetric and multimodal data using ViT and point cloud features; volumetric tokenization achieves 99.96% AUROC on BraTS-METS MRI, Dice 75.4% [2512.02520][2511.10047].

Performance highlights (all zero-shot unless noted):

| Method             | Testbed        | AUROC (Cls) | F1-max (Seg) | Comments                                  |
|--------------------|---------------|-------------|--------------|--------------------------------------------|
| MuSc               | MVTec AD      | 97.8        | 62.6         | Training-free, batch-based                 |
| AnomalyCLIP        | Industrial    | 91.5        | 91.1         | Object-agnostic prompt learning            |
| CoDeGraph          | MVTec-CA      | 98.5        | 73.8         | Robust to consistent anomalies             |
| StackCLIP          | MVTec-AD/VisA | 91.7/84.7   | 47.6/—       | Stacked prompts, cluster-specific heads    |
| MultiADS           | VisA (Zero)   | 83.6        | 89.7         | Multi-type, per-defect class segmentation  |
| FiSeCLIP           | MVTec-AD      | 95.3        | 49.1         | Mutual filtering, batch and prompt fusion  |
| APRIL-GAN (few-shot)| VAND          | 0.8687 (F1) | ~0.43 (F1)   | 1st in K-shot classification, state of art |

## 6. Ablations, Challenges, and Limitations

Ablation studies across methods have shown:
- Multi-stage fusion is essential; combining shallow and deep features typically boosts segmentation performance by >10 points [2305.17382].
- Prompt design (state word × template ensembles, stacked category names, or VAE-generated class tokens) critically affects both stability and peak accuracy [2506.23577][2508.03447][2310.18961].
- Consistent anomaly filtering must be precise; naive top-k removal can degrade performance for classes where anomalies are not dominant [2510.10456].
- Mutual scoring and cross-modal fusion (as in MuSc-V2) add up to 20+ percentage points in AP over previous best for 3D-AD and Eyecandies [2511.10047].

Common limitations include:
- Failure to detect anomalies that are common in-batch (“consistency” bias); statistical methods can be circumvented if anomalies dominate numerically [2510.10456][2512.02520].
- Lack of fine-grained segmentation at the resolution limit of ViT backbones.
- Auxiliary dataset choice (for prompt learning) can introduce domain bias.
- Sensitivity to batch composition (for batch-based methods) if the batch is not diverse.
- Manual threshold tuning for robust application in deployment.
- Maskomaly’s region-level AC must be heuristically aggregated [2305.16972].

## 7. Emerging Trends and Future Directions

- **Hybridization of paradigms**: Bridging batch-based and VLM paradigms via pseudo-masks and co-training (CoDeGraph-supervised VLMs) [2512.02520].
- **3D and Multimodal Generalization**: Expansion to 3D industrial and medical datasets, with surface-aware patch grouping and synchronized patch-pooling/fusion [2511.10047][2512.02520].
- **Open-Vocabulary and Human-Interpretable Anomalies**: Labeling and segmenting unknown classes with structured language integration (Clipomaly), dynamic vocabulary expansion, and region-level tag assignment [2512.01427].
- **Multi-type and Open-World Anomaly Detection**: Simultaneous segmentation and precise defect-type discrimination for per-object, per-defect action [2504.06740].
- **Graph-theoretic and EVT-Grounded Filtering**: Community detection, endurance-ratio filtering, and statistical modeling peripheral to batchwise similarity scaling [2512.02520][2510.10456].
- **Efficiency and Robustness**: Algorithmic optimizations to reduce $O(N^2)$ time and memory, including batch partitioning, CLS-guided screening, and constrained neighbor re-ranking [2512.02520][2511.10047].
- **Integration with segmentation transformers**: Post-processing mask networks (Mask2Former) for anomaly segmentation with minimal engineering (Maskomaly) [2305.16972].

In the current landscape, zero-shot AC/AS methods have established both theoretical and practical foundations for anomaly detection in highly variable, open, and data-scarce environments. Their continued evolution toward robust, multiscale, multimodal, open-world, and explainable systems remains a central research focus.

Source: https://www.emergentmind.com/topics/zero-shot-anomaly-classification-and-segmentation-ac-as