FiSeCLIP: Zero-Shot Anomaly Detection

• FiSeCLIP is a training-free zero-shot anomaly detection framework that combines semantic filtering via CLIP text prompts with patch-based feature matching.
• It leverages a frozen CLIP ViT backbone, employing dual text prompts and iterative mutual-filtering to produce refined anomaly masks for both image and pixel-level tasks.
• Empirical evaluations on benchmarks like MVTec-AD and VisA show significant performance gains over previous state-of-the-art methods in industrial inspection.

FiSeCLIP is a training-free zero-shot anomaly detection (ZSAD) framework that leverages a frozen CLIP ViT backbone to achieve high-performance, batch-based anomaly classification and segmentation. It combines two core mechanisms: semantic filtering via cross-modal alignment informed by CLIP’s text encoder (SeCLIP-AD), and feature matching across batches using patch neighborhoods and mutual-filtering between images (FiCLIP-AD). This hybrid approach addresses the lack of labeled data and supports industrial-scale batch testing, providing both global and fine-grained anomaly measures by bridging visual and textual modalities (Bai et al., 15 Jul 2025).

1. Architecture and Methodology

FiSeCLIP consists of two principal submodules, both of which operate on top of a frozen CLIP ViT-L/14-336 backbone (24 layers):

• SeCLIP-AD (Semantic-filtering CLIP for Anomaly Detection): Local semantic correlation in CLIP is restored by replacing the last self-attention layer with an intermediate layer (e.g., layer 12) and removing its FFN and residuals. Using dual textual prompts—“a [detail] photo of normal [category]” and “a [detail] photo of abnormal [category]”—features are projected into a class-conditional textual space. This produces image-level and pixel-level anomaly probability maps. A binary filtering mask $M_{u,p}$ is generated by thresholding patch-level probabilities.
• FiCLIP-AD (Feature-matching CLIP for Anomaly Detection): Each image in a batch of $B$ unlabeled images is compared against others serving as reference memories. Reference features are filtered, masking out patches identified as anomalous by SeCLIP-AD. Multi-layer ($i\in\{6,12,18,24\}$) and multi-radius ($r\in\{1,3,5\}$) feature aggregation enables detection of both small and large anomalies. Patch-level distances are computed, and a second mask is generated based on these feature-based distances.

A mutual-filtering scheme iteratively refines the anomaly mask through majority voting (pixel-wise AND/aggregation) between the text-guided and feature-matching masks, typically converging in 1–2 rounds for practical efficacy.

2. Mutual-Filtering: Mechanism and Signal Refinement

The mutual-filtering process constitutes the core novelty in FiSeCLIP. It integrates:

• Text-Guided Filtering (SeCLIP-AD): For each patch $p$ in image $u$, the model computes the softmax score for “abnormal” and “normal” using $F_{s,u,p}$ and textual features $F_t$, defining $M_{u,p}=1$ if $$\text{SEG\_prob}_{u,p}(\text{abnormal}) > \lambda \, \text{SEG\_prob}_{u,p}(\text{normal}), \lambda=1.10$$.
• Feature-Matching Filtering (FiCLIP-AD): For each patch, increments are made over multi-scale ($B$0) pooled neighborhoods and reference images $B$1, with abnormal reference patches filtered out by $B$2. Minimum L2 patch-wise distances are calculated:

$B$3

Aggregated distances yield new binary masks thresholded at $B$4.

• Voting and Iteration: Text and feature-based masks are refined via pixel-wise majority, yielding mask $B$5 for subsequent FiCLIP-AD passes, producing a progressively denoised anomaly map.

This collaborative signal refinement between modalities mitigates ambiguities arising from unlabeled batch test images, reducing false positive “noise” and improving granularity of the segmentation.

3. Bridge Feature Matching and Cross-Modal Alignment

FiSeCLIP capitalizes on cross-modal alignment in two ligatures:

• Textual alignment initializes anomaly localization using class-conditional textual priors as prompt embeddings, facilitating coarse but robust anomaly heatmaps.
• Feature-matching recursively aligns local visual features across mutually referenced test images, after masking out unstable regions detected via text.

The final anomaly map $B$6 is computed as the pixel-wise maximum of normalized SeCLIP and FiCLIP anomaly scores, ensuring reinforcement between modalities.

Alignment Type	Source	Output
Textual (SeCLIP-AD)	Prompts+Vision	Initial mask $B$7, $B$8
Feature (FiCLIP-AD)	Pooled Patches	Mask $B$9, $i\in\{6,12,18,24\}$0
Fusion	Both	Final anomaly map $i\in\{6,12,18,24\}$1

This dual-branch mechanism enables fine-grained, semantic, and context-sensitive anomaly detection even in the absence of training labels.

4. Anomaly Detection: Classification and Segmentation Protocols

FiSeCLIP’s output is a patch-level anomaly score map $i\in\{6,12,18,24\}$2 for each image:

• Pixel-Level Segmentation: $i\in\{6,12,18,24\}$3 is upsampled to original resolution and thresholded at 0.5 to yield a binary anomaly mask.
• Image-Level Classification: $i\in\{6,12,18,24\}$4 or the mean of the top-$i\in\{6,12,18,24\}$5 scores is used as the image’s anomaly score, further thresholded for classification.

This enables zero-shot performance on industrial-scale benchmarks and subsumes both classification-based and segmentation-based anomaly evaluation schemes.

5. Empirical Performance on Standard Benchmarks

On the MVTec-AD benchmark, FiSeCLIP markedly outperforms previous zero-shot methods:

• Segmentation (AU-ROC / F₁-max): 93.3% / 49.1%
• Classification (AU-ROC / F₁-max): 95.3% / 95.4%
• Previous SOTA AdaCLIP: 88.7% / 43.4% (segmentation), 89.2% / 90.6% (classification)

Resulting in +4.6 pp (segmentation AU-ROC), +5.7 pp (F₁-max), +6.1 pp (classification AU-ROC), and +4.8 pp (F₁-max) improvements. On VisA, segmentation AU-ROC gains reach 97.4%. These results establish a new zero-shot baseline for large-scale industrial anomaly detection (Bai et al., 15 Jul 2025).

6. Implementation Characteristics and Design Parameters

FiSeCLIP uses the following configurations:

• Backbone: CLIP ViT-L/14-336, layers {6,12,18,24} used, backbone weights frozen throughout.
• Pooling Radii: $i\in\{6,12,18,24\}$6 for multi-scale anomaly detection.
• Batch Size: $i\in\{6,12,18,24\}$7 (practical B=2), larger batches provide greater robustness via richer mutual referencing.
• Thresholds: $i\in\{6,12,18,24\}$8 (text filter), $i\in\{6,12,18,24\}$9 (feature mask).
• Text Prompts: Dual prompts for normal and abnormal, possible detail tokens including “cropped”, “rotated”, etc.
• Computation: Runs on a single NVIDIA RTX 3090, images are resized to $r\in\{1,3,5\}$0.
• Mutual-Filtering Rounds: 1–2 iterations.

Inference is fully training-free, requiring only batch forwarding of unlabeled images. Code and model weights are available at the official repository: https://github.com/ybai111/FiSeCLIP.

7. Context, Significance, and Implications

FiSeCLIP demonstrates that CLIP’s zero-shot capabilities can be effectively extended to pixel-level anomaly localization and industrial inspection scenarios without additional training, via bridge feature matching and mutual cross-modal filtering. Its batch-based protocol aligns with real-world operational requirements where full-dataset inference is infeasible. The mutual-filtering between text and cross-image features directly addresses the ambiguity and noise arising from unlabeled references during testing.

A plausible implication is that similar mutual-filtering methodologies could further broaden zero-shot adaptation to other dense prediction tasks by exploiting cross-modal priors and inter-instance reference structures (Bai et al., 15 Jul 2025).