---
title: 'FiSeCLIP: Zero-Shot Anomaly Detection'
url: https://www.emergentmind.com/topics/fiseclip
type: topic
---

# FiSeCLIP: Zero-Shot Anomaly Detection

FiSeCLIP is a training-free zero-shot anomaly detection (ZSAD) framework that leverages a frozen CLIP ViT backbone to achieve high-performance, batch-based anomaly classification and segmentation. It combines two core mechanisms: semantic filtering via cross-modal alignment informed by CLIP’s text encoder (SeCLIP-AD), and feature matching across batches using patch neighborhoods and mutual-filtering between images (FiCLIP-AD). This hybrid approach addresses the lack of labeled data and supports industrial-scale batch testing, providing both global and fine-grained anomaly measures by bridging visual and textual modalities [2507.11003].

## 1. Architecture and Methodology

FiSeCLIP consists of two principal submodules, both of which operate on top of a frozen CLIP ViT-L/14-336 backbone (24 layers):

- **SeCLIP-AD (Semantic-filtering CLIP for Anomaly Detection):** Local semantic correlation in CLIP is restored by replacing the last self-attention layer with an intermediate layer (e.g., layer 12) and removing its FFN and residuals. Using dual textual prompts—“a [detail] photo of normal [category]” and “a [detail] photo of abnormal [category]”—features are projected into a class-conditional textual space. This produces image-level and pixel-level anomaly probability maps. A binary filtering mask $M_{u,p}$ is generated by thresholding patch-level probabilities.

- **FiCLIP-AD (Feature-matching CLIP for Anomaly Detection):** Each image in a batch of $B$ unlabeled images is compared against others serving as reference memories. Reference features are filtered, masking out patches identified as anomalous by SeCLIP-AD. Multi-layer ($i\in\{6,12,18,24\}$) and multi-radius ($r\in\{1,3,5\}$) feature aggregation enables detection of both small and large anomalies. Patch-level distances are computed, and a second mask is generated based on these feature-based distances.

A mutual-filtering scheme iteratively refines the anomaly mask through majority voting (pixel-wise AND/aggregation) between the text-guided and feature-matching masks, typically converging in 1–2 rounds for practical efficacy.

## 2. Mutual-Filtering: Mechanism and Signal Refinement

The mutual-filtering process constitutes the core novelty in FiSeCLIP. It integrates:

- **Text-Guided Filtering (SeCLIP-AD):** For each patch $p$ in image $u$, the model computes the softmax score for “abnormal” and “normal” using $F_{s,u,p}$ and textual features $F_t$, defining $M_{u,p}=1$ if $\text{SEG\_prob}_{u,p}(\text{abnormal}) > \lambda \, \text{SEG\_prob}_{u,p}(\text{normal}), \lambda=1.10$.

- **Feature-Matching Filtering (FiCLIP-AD):** For each patch, increments are made over multi-scale ($r$) pooled neighborhoods and reference images $v \neq u$, with abnormal reference patches filtered out by $M_v$. Minimum L2 patch-wise distances are calculated:
  $$
    \hat a_{u,v}^{i,r} = \min_{x \in F_u^{i,r}, y \in \hat F_v^{i,r}} \|x-y\|_2.\  
  $$
  Aggregated distances yield new binary masks thresholded at $\mu=0.57$.

- **Voting and Iteration:** Text and feature-based masks are refined via pixel-wise majority, yielding mask $M'$ for subsequent FiCLIP-AD passes, producing a progressively denoised anomaly map.

This collaborative signal refinement between modalities mitigates ambiguities arising from unlabeled batch test images, reducing false positive “noise” and improving granularity of the segmentation.

## 3. Bridge Feature Matching and Cross-Modal Alignment

FiSeCLIP capitalizes on cross-modal alignment in two ligatures:

- Textual alignment initializes anomaly localization using class-conditional textual priors as prompt embeddings, facilitating coarse but robust anomaly heatmaps.
- Feature-matching recursively aligns local visual features across mutually referenced test images, after masking out unstable regions detected via text.

The final anomaly map $S_u(p)$ is computed as the pixel-wise maximum of normalized SeCLIP and FiCLIP anomaly scores, ensuring reinforcement between modalities.

| Alignment Type         | Source       | Output              |
|-----------------------|--------------|---------------------|
| Textual (SeCLIP-AD)   | Prompts+Vision | Initial mask $M$, $A_{Se}$ |
| Feature (FiCLIP-AD)   | Pooled Patches | Mask $M^{inter}$, $A_{Fi}$ |
| Fusion                | Both         | Final anomaly map $S_u(p)$  |

This dual-branch mechanism enables fine-grained, semantic, and context-sensitive anomaly detection even in the absence of training labels.

## 4. Anomaly Detection: Classification and Segmentation Protocols

FiSeCLIP’s output is a patch-level anomaly score map $S_u(p)$ for each image:

- **Pixel-Level Segmentation:** $S_u(p)$ is upsampled to original resolution and thresholded at 0.5 to yield a binary anomaly mask.
- **Image-Level Classification:** $S^{img}_u = \max_p S_u(p)$ or the mean of the top-$k$ scores is used as the image’s anomaly score, further thresholded for classification.

This enables zero-shot performance on industrial-scale benchmarks and subsumes both classification-based and segmentation-based anomaly evaluation schemes.

## 5. Empirical Performance on Standard Benchmarks

On the MVTec-AD benchmark, FiSeCLIP markedly outperforms previous zero-shot methods:

- **Segmentation (AU-ROC / F₁-max):** 93.3% / 49.1%
- **Classification (AU-ROC / F₁-max):** 95.3% / 95.4%
- **Previous SOTA AdaCLIP:** 88.7% / 43.4% (segmentation), 89.2% / 90.6% (classification)

Resulting in +4.6 pp (segmentation AU-ROC), +5.7 pp (F₁-max), +6.1 pp (classification AU-ROC), and +4.8 pp (F₁-max) improvements. On VisA, segmentation AU-ROC gains reach 97.4%. These results establish a new zero-shot baseline for large-scale industrial anomaly detection [2507.11003].

## 6. Implementation Characteristics and Design Parameters

FiSeCLIP uses the following configurations:

- **Backbone:** CLIP ViT-L/14-336, layers {6,12,18,24} used, backbone weights frozen throughout.
- **Pooling Radii:** $r \in \{1,3,5\}$ for multi-scale anomaly detection.
- **Batch Size:** $B \geq 2$ (practical B=2), larger batches provide greater robustness via richer mutual referencing.
- **Thresholds:** $\lambda = 1.10$ (text filter), $\mu=0.57$ (feature mask).
- **Text Prompts:** Dual prompts for normal and abnormal, possible detail tokens including “cropped”, “rotated”, etc.
- **Computation:** Runs on a single NVIDIA RTX 3090, images are resized to $518 \times 518$.
- **Mutual-Filtering Rounds:** 1–2 iterations.

Inference is fully training-free, requiring only batch forwarding of unlabeled images. Code and model weights are available at the official repository: https://github.com/ybai111/FiSeCLIP.

## 7. Context, Significance, and Implications

FiSeCLIP demonstrates that CLIP’s zero-shot capabilities can be effectively extended to pixel-level anomaly localization and industrial inspection scenarios without additional training, via bridge feature matching and mutual cross-modal filtering. Its batch-based protocol aligns with real-world operational requirements where full-dataset inference is infeasible. The mutual-filtering between text and cross-image features directly addresses the ambiguity and noise arising from unlabeled references during testing.

A plausible implication is that similar mutual-filtering methodologies could further broaden zero-shot adaptation to other dense prediction tasks by exploiting cross-modal priors and inter-instance reference structures [2507.11003].

Source: https://www.emergentmind.com/topics/fiseclip