AnomalyVFM: Zero-Shot Anomaly Detection

• AnomalyVFM is a framework for zero-shot anomaly detection and localization that adapts pretrained vision models using synthetic images and efficient LoRA-style adjustments.
• It employs a three-stage synthetic data generation pipeline with anomaly-free synthesis, defect inpainting, and feature-based verification to robustly simulate real-world defects.
• Using a RADIO backbone and confidence-weighted loss, AnomalyVFM achieves superior performance (94.1% AUROC) across diverse industrial and medical benchmarks.

AnomalyVFM is a framework for zero-shot anomaly detection that turns any pretrained vision foundation model into a strong zero-shot anomaly detector. It addresses the setting in which an input image must be judged for anomalousness and abnormal regions must be localized without access to any in-domain training images. The method attributes the historical gap between pure vision foundation models and vision-LLMs to two practical issues—limited diversity in existing auxiliary anomaly detection datasets and overly shallow VFM adaptation strategies—and answers them with a three-stage synthetic dataset generation scheme, low-rank feature adapters, and a confidence-weighted pixel loss. With RADIO as a backbone, it achieves an average image-level AUROC of 94.1% across 9 diverse datasets, surpassing previous methods by significant 3.3 percentage points (Fučka et al., 28 Jan 2026).

1. Conceptual scope and problem setting

AnomalyVFM is defined for zero-shot anomaly detection and zero-shot anomaly localization. In this setting, no in-domain normal or anomalous images from target datasets are used during training; training data consist only of synthetic triplets generated by the method itself. The framework is designed to adapt pretrained VFMs such as RADIO, DINOv2, and DINOv3, while freezing the original backbone weights and learning only lightweight residual modules and small heads (Fučka et al., 28 Jan 2026).

The motivating claim is that purely visual foundation models should be well-suited to the fundamentally visual nature of anomaly detection, yet historically trail VLM-based approaches. The method identifies two reasons for this gap: limited diversity in auxiliary datasets and shallow VFM adaptation. Its central design therefore couples synthetic-data diversification with deeper, but still parameter-efficient, backbone adaptation. This design directly contrasts with prior VFM-based zero-shot anomaly detection systems that commonly attach shallow MLP heads and train with simple pixel-wise losses (Fučka et al., 28 Jan 2026).

A broader usage of the term can be inferred from adjacent literature. A recent survey on foundation-model-based anomaly detection classifies such systems into FM as Encoder, FM as Detector, and FM as Interpreter, and notes that vision foundation models and vision-LLMs are used for image-level detection, pixel-level localization, and explanation generation (Ren et al., 10 Feb 2025). This suggests that AnomalyVFM is both a specific framework and part of a broader research direction centered on anomaly detection with pretrained visual or multimodal foundation models.

2. Synthetic data generation pipeline

AnomalyVFM’s training set is produced by a robust three-stage synthetic dataset generation scheme. The pipeline begins from anomaly-free synthesis, injects defects by inpainting, and then verifies and extracts masks by feature comparison. The stages and their reported settings are summarized below (Fučka et al., 28 Jan 2026).

Stage	Operation	Key details
A	Anomaly-free image synthesis	FLUX; 1024×1024; 100 object categories; 50 textures
B	Defect inpainting	IS-Net foreground mask; rectangle with $w \sim U(50,350)$ and $h \sim U(50,350)$; 204 anomaly types; RePaint-style iterative generation
C	Verification and mask extraction	DINOv2 features; $M_d = 1 - \cos\_sim(f,f_a)$; accept if $D = \max(M_d) > T$ with $T = 0.3$

In Stage A, the generator $G$ is a high-resolution flow-matching model, FLUX. The normal-image prompt template is: “A close-up photo of [Object] for industrial visual inspection. Top-down view. Centered. [Texture] background.” The object list contains 100 object categories generated via GPT-4o, and the texture list contains 50 backgrounds such as concrete, brushed aluminum, denim, and parquet flooring. Sampling produces an anomaly-free image $I = G(p)$ at $1024 \times 1024$ (Fučka et al., 28 Jan 2026).

In Stage B, a soft foreground mask $M_{fg}$ is obtained from $I$ using IS-Net. A pixel $h \sim U(50,350)$0 is sampled within $h \sim U(50,350)$1, then a rectangle $h \sim U(50,350)$2 centered at $h \sim U(50,350)$3 is sampled with width $h \sim U(50,350)$4 and height $h \sim U(50,350)$5 on the $h \sim U(50,350)$6 canvas. The anomalous prompt is “A close-up photo of a [Anomaly] [Object] for industrial visual inspection. Top-down view. Centered. [Texture] background.” The anomaly descriptors are per-object and total 204 types. RePaint-style iterative generation then inpaints only within $h \sim U(50,350)$7, keeping the rest of $h \sim U(50,350)$8 fixed, yielding the anomalous image $h \sim U(50,350)$9 (Fučka et al., 28 Jan 2026).

In Stage C, dense features $M_d = 1 - \cos\_sim(f,f_a)$0 and $M_d = 1 - \cos\_sim(f,f_a)$1 are extracted from $M_d = 1 - \cos\_sim(f,f_a)$2 and $M_d = 1 - \cos\_sim(f,f_a)$3 using frozen DINOv2, and a cosine-distance map is computed as $M_d = 1 - \cos\_sim(f,f_a)$4 per pixel. The sample is accepted if $M_d = 1 - \cos\_sim(f,f_a)$5 exceeds the threshold $M_d = 1 - \cos\_sim(f,f_a)$6, and the binary anomaly mask $M_d = 1 - \cos\_sim(f,f_a)$7 is obtained by thresholding $M_d = 1 - \cos\_sim(f,f_a)$8 at the same $M_d = 1 - \cos\_sim(f,f_a)$9. By default, the resulting dataset contains 10,000 training triplets $D = \max(M_d) > T$0, 4,596 unique object–background pairs, and anomalous regions with mean area 2.52%, minimum 0.28%, and maximum 11.24% of the image (Fučka et al., 28 Jan 2026).

The reported ablations make the role of this pipeline explicit. Replacing FLUX with QWEN-Image or WAN causes at most small drops in pixel AUROC, whereas omitting filtering or foreground-aware sampling significantly degrades results. This directly ties the framework’s performance to dataset verification and object-aware defect placement rather than to synthetic generation alone (Fučka et al., 28 Jan 2026).

3. Backbone adaptation and learning objectives

The model pipeline takes an image $D = \max(M_d) > T$1, passes it through a pretrained VFM $D = \max(M_d) > T$2 augmented with low-rank adapters in every transformer block, reshapes the final token map to a spatial feature map, and decodes that map into a pixel-level anomaly score map $D = \max(M_d) > T$3 and a per-pixel confidence map $D = \max(M_d) > T$4. In parallel, a linear layer on the $D = \max(M_d) > T$5 token produces the image-level anomaly score $D = \max(M_d) > T$6 (Fučka et al., 28 Jan 2026).

The adaptation mechanism is LoRA-style. For a linear projection $D = \max(M_d) > T$7, AnomalyVFM parameterizes

$D = \max(M_d) > T$8

with $D = \max(M_d) > T$9, $T = 0.3$0, and default rank $T = 0.3$1. These adapters are inserted into the attention query, value, and output projection linear layers in all transformer blocks. The original backbone weights are frozen; only the LoRA parameters and the lightweight heads are trained (Fučka et al., 28 Jan 2026).

The decoder consists of two upsampling blocks, each implemented as Conv $T = 0.3$2 GroupNorm $T = 0.3$3 ReLU $T = 0.3$4 bilinear upsample. The final convolution head outputs anomaly logits and confidence logits. No explicit multi-scale fusion is used in the reported results; the method uses final-layer features from the VFM backbone, reshaped to a spatial map for the decoder (Fučka et al., 28 Jan 2026).

The training objective combines a confidence-weighted segmentation loss and an image-level focal loss. The base pixel loss is

$T = 0.3$5

The predicted confidence logits are converted into positive weights by

$T = 0.3$6

The pixel-level loss is then

$T = 0.3$7

The image-level loss is

$T = 0.3$8

and the total loss is

$T = 0.3$9

The paper states that the $G$0 term discourages trivially inflating confidence and acts as a regularizer akin to heteroscedastic uncertainty weighting (Fučka et al., 28 Jan 2026).

The reported optimization setup uses RADIO v2.5 ViT-L/16, input resolution $G$1 for training and evaluation, AdamW with learning rate $G$2, batch size 32, and 500 iterations. GroupNorm is used in the decoder, the backbone is frozen except for LoRA weights, and data augmentation is not required; optional light flips and crops are described as acceptable but not essential to the reported results (Fučka et al., 28 Jan 2026).

4. Inference protocol, calibration, and practical use

Inference in AnomalyVFM is direct. Given a test image, features flow through the adapted VFM backbone, the decoder outputs $G$3 and $G$4, and the image-level head outputs $G$5. The image-level anomaly score is $G$6 itself, while the pixel-level anomaly score is the per-pixel anomaly probability $G$7. Feature normalization or reconstruction losses are not used at inference time; the model directly predicts anomaly probabilities (Fučka et al., 28 Jan 2026).

No post-hoc calibration is required for AUROC evaluation. For deployment, the method allows light Gaussian smoothing on $G$8, morphological opening or closing, a fixed decision threshold such as $G$9, or an image-adaptive threshold such as Otsu on $I = G(p)$0. For evaluation with ground truth, max-F1 is computed over thresholds, while AUROC remains threshold-free (Fučka et al., 28 Jan 2026).

The zero-shot protocol is strict: no target-domain images enter training. Practical guidance in the paper recommends at least 100 object tags, approximately 50 textures, and at least 200 anomaly descriptors; removing any objects present in evaluation datasets is explicitly advised to prevent leakage. The default target synthetic dataset size is 10k images, though 1k–5k still works with reduced performance, and increasing the number of objects and images improves results (Fučka et al., 28 Jan 2026).

The framework also supports a reported few-shot extension. If a few normal samples are available, the trained AnomalyVFM can be fine-tuned for about 50 iterations; the paper notes that this can further raise image-level AUROC, giving MVTec AD 1–4 shot results of 97.2–98.2. This does not alter the zero-shot definition of the main model, but it indicates a straightforward path from synthetic-only adaptation to low-shot specialization (Fučka et al., 28 Jan 2026).

5. Benchmarks, ablations, and efficiency

AnomalyVFM is evaluated on industrial and medical benchmarks. The industrial set comprises MVTec AD, VisA, BTAD, MPDD, Real-IAD, KSDD, KSDD2, DAGM, and DTD-Synthetic. The medical set comprises HeadCT, BrainMRI, BR35H for image-level evaluation, and ISIC, ClinicDB, ColonDB, Kvasir, EndoTect, and TN3K for pixel-level evaluation. The reported metrics are image-level AUROC and max-F1, and pixel-level AUROC and max-F1 (Fučka et al., 28 Jan 2026).

With RADIO as backbone, the framework reaches average image-level AUROC 94.1% across 9 industrial datasets and average pixel AUROC 96.9%. The paper reports that this surpasses previous best methods by 3.3 percentage points in image-level AUROC and by 0.9 percentage points in pixel-level AUROC on average. On medical benchmarks, without any medical data used for training, the method is described as competitive to SOTA in image-level detection and as improving prior methods in pixel-level AUROC on average by 1.2 percentage points (Fučka et al., 28 Jan 2026).

The ablation studies isolate the major contributors. Removing feature-based verification causes a drop of 3.8 percentage points in image AUROC and 14.6 percentage points in pixel AUROC. Removing foreground selection for inpainting causes a drop of 1.4 percentage points in image AUROC and 5.8 percentage points in pixel AUROC. Turning off the confidence-weighted loss causes a drop of 0.6 percentage points in image AUROC and 2.0 percentage points in pixel AUROC. AdaLN and VPT remain strong alternatives, but trail LoRA by up to approximately 1 percentage point in image AUROC and approximately 2.9 percentage points in pixel AUROC. Rank 64 is reported as robust, and inserting adapters into query, value, and output projection performs best among tested layouts (Fučka et al., 28 Jan 2026).

The reported model-size and runtime characteristics are equally specific. The full system contains approximately 345.8M parameters, of which approximately 35.4M are trainable. Average inference time per image on an A100 GPU is 20.5 ms, faster than Bayes-PFL at 208.5 ms and AdaCLIP at 82.4 ms. Training takes approximately 2 hours on one A100 GPU for 500 steps with batch size 32, whereas generation of 10k synthetic samples takes approximately 1 day and is described as the dominant cost (Fučka et al., 28 Jan 2026).

The limitations identified in the same work are methodologically important. Synthetic generation is a bottleneck; prompt adherence failures remain possible even with feature-based filtering; off-the-shelf image generators may struggle to produce realistic medical images; and very fine, low-contrast anomalies or domain-specific physics cues may remain challenging if they are not well represented in synthetic data. These limitations are presented as properties of the current system rather than of zero-shot anomaly detection in general (Fučka et al., 28 Jan 2026).

6. Position within the broader AnomalyVFM literature

Subsequent and adjacent work places AnomalyVFM within a larger research space rather than treating it as an isolated architecture. "VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer" removes the text branch entirely and inserts two learnable global tokens, an anomaly token and a normal token, into a frozen Vision Transformer such as the CLIP image encoder or DINOv2. It combines these tokens with Spatial-Aware Cross-Attention and a Self-Alignment Function, achieves state-of-the-art performance on 13 zero-shot anomaly detection benchmarks spanning industrial and medical domains, and is reported to use approximately 99% fewer trainable parameters than a VLM baseline in the paper’s exploratory study (Hou et al., 9 Mar 2026). This suggests that one major fault line in the literature is not whether to use foundation models, but whether anomaly semantics must remain tied to a text encoder.

A complementary direction appears in few-shot anomaly detection. "H2VLR: Heterogeneous Hypergraph Vision-Language Reasoning for Few-Shot Anomaly Detection" reformulates FSAD as a high-order inference problem over a heterogeneous hypergraph connecting visual patch tokens and support-conditioned text prompt embeddings. Using a frozen OpenCLIP ViT-B/16+ backbone with approximately 2.3M additional learnable parameters, it reports industrial image-level AUROC averages of 84.86, 87.22, and 90.37 in 1-, 2-, and 4-shot settings, as well as industrial pixel-level AUROC averages of 93.80, 93.85, and 95.68 (Huang et al., 16 Apr 2026). In this sense, AnomalyVFM’s synthetic-data-driven zero-shot adaptation and H2VLR’s support-conditioned hypergraph reasoning address closely related goals under different supervision regimes.

The same broadening is visible in video anomaly research and evaluation. "Anomize: Better Open Vocabulary Video Anomaly Detection" addresses open-vocabulary video anomaly detection with frozen CLIP encoders, a text-augmented dual-stream design, and group-guided textual encodings, while "Vision-LLMs Assisted Unsupervised Video Anomaly Detection" uses high-level semantic features from a VLM together with a Sequence State Space Module trained on normal videos only (Li et al., 23 Mar 2025, Jiang et al., 2024). "FineVAU: A Novel Human-Aligned Benchmark for Fine-Grained Video Anomaly Understanding" then shifts attention from detection alone to anomaly understanding, decomposing evaluation into What, Who, and Where, and introducing FVScore as a human-aligned metric (Pereira et al., 24 Jan 2026). These works do not redefine AnomalyVFM, but they show that the topic increasingly spans image, video, zero-shot, few-shot, detector, and interpreter settings.

Survey literature formalizes this expansion. Foundation-model-based anomaly detection is explicitly categorized into FM as Encoder, FM as Detector, and FM as Interpreter, covering training-free CLIP-style methods, prompt-adapted and parameter-efficient systems, and models that generate anomaly explanations or rationales (Ren et al., 10 Feb 2025). A plausible implication is that AnomalyVFM now names both a particular synthetic-data-and-LoRA framework and a wider paradigm in which pretrained visual or multimodal foundation models are repurposed for anomaly scoring, localization, and reasoning.