PatchCore: Memory-Based Anomaly Detection

• PatchCore is a memory-bank based algorithm that detects and localizes visual anomalies by matching local CNN patch features to a compact coreset of normal image data.
• It achieves state-of-the-art performance on benchmarks like MVTec AD with image-level AUROC nearly 99.1% and efficient inference times even in few-shot scenarios.
• Extensions incorporate few-shot, continual, and transfer learning methods, enhancing robustness and enabling effective anomaly detection under data scarcity.

PatchCore is a memory-bank–based algorithm for visual anomaly detection and segmentation that operates by storing and matching local patch-level features from normal (defect-free) images. PatchCore achieves state-of-the-art performance on industrial inspection benchmarks, offering accurate detection and localization of defects even in low-data (“few-shot”) and high-resolution scenarios. Its core principle is to compare local CNN features of a test sample against a compact coreset, allowing anomaly scoring based on nearest-neighbor distances in feature space. PatchCore forms the foundation for numerous research advancements, including extensions for few-shot, continual, and transfer learning, and improved strategies for detection thresholding under data scarcity.

1. Core Methodology and Algorithmic Structure

PatchCore employs a frozen and pre-trained CNN backbone (e.g., WideResNet-50, ResNet-50), extracting intermediate feature maps from designated network layers. For each input image $x \in \mathbb{R}^{3 \times H \times W}$, spatial patch embeddings are generated by aggregating $p \times p$ neighborhoods via adaptive average pooling. Frequently, features from multiple layers are concatenated to harness both local detail and wider context. The resulting set of patch feature vectors from all nominal training images is pooled into a “memory bank” $M = \{ m_i \in \mathbb{R}^{d} \}$, where $d$ denotes feature dimension.

To achieve efficiency and compactness, a greedy coreset subsampling algorithm is used. This maximizes the representational coverage of the memory bank by iteratively selecting patch vectors that are maximally distant in the feature space from previously chosen ones, effectively approximating the K-center covering problem (Roth et al., 2021).

At test time, each patch $a_k$ from a novel image is scored by its Euclidean distance to its closest memory-bank vector: $s_k = \min_{m \in M} \| a_k - m \|_2$ The image-level anomaly score is

$S(x) = \max_k s_k$

A spatial anomaly heatmap is created by mapping $\{ s_k \}$ back to the original image grid and upsampling to input resolution.

2. Theoretical Foundations and Design Rationale

PatchCore's design is motivated by the need to maximize the coverage of normal data variation while reducing overfitting and computational cost inherent in global feature matching. The use of mid-level CNN features (typically from ResNet’s blocks 2 and 3) mitigates bias from ImageNet pre-training, capturing textures and shapes more relevant to industrial context rather than semantically abstract object-level features. Local neighborhood aggregation (e.g., pooling over 3×3 windows) enhances robustness to alignment errors and small local variations.

The coreset selection ensures that even rare modes in the normal data are preserved while enabling rapid inference with minimal memory. Empirically, retaining as little as 1–10% of all possible patch features leads to negligible loss in detection accuracy but significant reductions in memory and compute requirements.

3. Performance Benchmarks and Evaluation Protocols

PatchCore achieves leading results on datasets such as MVTec AD and VisA. On the MVTec AD benchmark, PatchCore_{25%} attains a mean image-level AUROC of 99.1%, halving the error rate compared to PaDiM and other competitors (Roth et al., 2021). At the pixel-level, PatchCore achieves 98.1% AUROC and 93.5% PRO segmentation score, with inference times as low as 0.17 s per 256×256 image using a 1% coreset.

PatchCore maintains strong performance even under severe data constraints. With only 1–5 nominal samples per class, the image AUROC remains high (e.g., 83.4% at one-shot, 90.8% at five-shot, both using 10% coreset). Performance degrades gracefully with reduced supervision and remains robust under memory reduction.

Evaluation is typically performed by reporting threshold-independent metrics (AUROC, AUPR, AUHPROC, HPROC) and segmentation accuracy. Threshold-dependent metrics (F1, precision, recall, FPR, FNR) are computed using strategies such as setting global thresholds by held-out data or self-score estimation (Santos et al., 2023, Arodi et al., 2024).

4. Extensions: Few-shot, Continual, and Transfer Learning

Few-Shot and Enhanced-PatchCore

PatchCore has been extensively investigated and optimized for low-data regimes. Recent work demonstrates that tuning hyperparameters—backbone architecture (e.g., using anti-aliased WRN50), input resolution (e.g., 448×448), and tailored data augmentation—can yield new state-of-the-art results in few-shot anomaly detection (Santos et al., 2023). Sequential PatchCore introduces a memory-bounded, on-the-fly coreset construction, lowering resource demands and enabling deployment on images up to 8MP using commodity GPUs (Mao et al., 16 Jan 2025).

Enhanced-PatchCore applies leave-one-out scoring on nominal data to estimate the anomaly threshold without needing held-out anomalies. Several thresholding strategies (max, boxplot whisker, empirical 95th percentile, beta-prime fitted quantile) are compared; fitting a beta-prime prior offers balanced sensitivity and robustness, especially in few-shot regimes (Arodi et al., 2024).

Anomaly Multi-Classification and Few-Shot Meta-Learning

Recent advancements combine PatchCore with relation-based meta-learning for anomaly “multi-classification.” For example, the residuals between test patches and their memory-bank nearest neighbors are encoded and input to a RelationNet head, which is further improved with proxy class generation (via overlaid DTD textures) and contrastive InfoNCE loss. This approach substantially outperforms direct fine-tuning of classification networks for few-shot multi-class anomaly tasks (Liu et al., 2024).

Continual Learning: PatchCoreCL

PatchCore has been adapted to continual learning scenarios (Barusco et al., 25 Aug 2025). PatchCoreCL maintains a set of sub-banks (one per task or time window), each built using coreset subsampling, and keeps a fixed total memory capacity. At inference, the closest-matching sub-bank determines both anomaly localization and implicit task identification. The continual learning formulation demonstrates less than 1% catastrophic forgetting on real-world medical imaging datasets.

Transfer Learning with Synthetic Data and Coreset Melding

The Sequential PatchCore framework enables explicit transfer learning. Pre-trained coresets built on synthetic data (with or without augmented impurities) are updated and merged (melded) with those from real data. Fine-tuning with as few as 10–50 real samples can restore segmentation F1 to levels above models trained solely on real data, even in the presence of challenging surface impurities (water stains, fingerprints) (Mao et al., 16 Jan 2025).

5. Thresholding Strategies, Practical Deployment, and Limitations

A critical operational aspect is anomaly threshold selection, especially in scenarios where anomalous samples are rare or unavailable. Enhanced-PatchCore’s leave-one-out and statistical fitting approaches allow threshold calibration exclusively from nominal data, increasing deployability in production settings (Arodi et al., 2024). However, PatchCore’s score distribution can be sensitive to outlier patches, especially when the number of normals exceeds ~50, occasionally degrading detection reliability.

PatchCore is sensitive to background clutter and irrelevant textures—cropping to object-of-interest regions substantially improves score consistency. Nonetheless, for subtle, fine-grade anomalies, recall remains suboptimal compared to coarse defects. Vision-LLMs (VLMs) show strong zero-shot performance for object-level anomaly detection but are less effective for precise localization or identification of minor faults.

6. Empirical Studies and Comparative Analysis

Summary of Key Benchmark Results

Method	Dataset	Few-shot (1/5/10) AUROC	Many-shot AUROC	Pixel-level AUPR	Segmentation PRO
PatchCore (vanilla)	MVTec AD	83–91% (1–5 shot)	99.1% (25%)	98.1%	93.5%
PatchCore (optimized)	VisA	80.0% → 86.4%*	—	—	—
Enhanced-PatchCore (β-prime-95)	CableInspect	F1=0.75, AUROC=0.78	—	AUPR=0.84	AUPRO=0.53
PatchCoreCL (continual)	Medical (BMAD)	<1% forgetting	Comparable to task-specific	—	—

*Optimized PatchCore improved few-shot AUROC by 6.4pp on VisA (Santos et al., 2023).

Limitations

• In high-shot regimes, PatchCore’s memory bank becomes vulnerable to inclusion of outlier patches, which can inflate anomaly score variance.
• Performance on certain fine-grained or ambiguous real-world anomalies remains limited; richer augmentations or metric learning may be required (Liu et al., 2024, Arodi et al., 2024).
• On highly cluttered or non-standard backgrounds, background-sensitive scoring can degrade both image-level and localization accuracy.

7. Future Directions

Key research challenges include developing more robust patch selection and augmentation techniques (potentially with learnable augmentation or rotation-equivariant backbones), improving synthetic defect generation for proxy training, and enhancing PatchCore’s reliability for subtle and composite anomalies. Continual learning and domain-adaptive thresholding represent robust directions for deployment in open-ended and evolving industrial or medical environments (Barusco et al., 25 Aug 2025, Mao et al., 16 Jan 2025). Integration with meta-learning and contrastive frameworks is a promising avenue for few-shot multi-type anomaly recognition (Liu et al., 2024).

PatchCore’s methodological influence now extends beyond industrial inspection, providing algorithmic templates for robust feature-bank anomaly detection in domains such as medical imaging, robotics, and large-scale surface inspection (Mao et al., 16 Jan 2025, Barusco et al., 25 Aug 2025, Arodi et al., 2024).