SPG: Sparse-Projected Guides with Sparse Autoencoders for Zero-Shot Anomaly Detection

Abstract: We study zero-shot anomaly detection and segmentation using frozen foundation model features, where all learnable parameters are trained only on a labeled auxiliary dataset and deployed to unseen target categories without any target-domain adaptation. Existing prompt-based approaches use handcrafted or learned prompt embeddings as reference vectors for normal/anomalous states. We propose Sparse-Projected Guides (SPG), a prompt-free framework that learns sparse guide coefficients in the Sparse Autoencoder (SAE) latent space, which generate normal/anomaly guide vectors via the SAE dictionary. SPG employs a two stage learning strategy on the labeled auxiliary dataset: (i) train an SAE on patch-token features, and (ii) optimize only guide coefficients using auxiliary pixel-level masks while freezing the backbone and SAE. On MVTec AD and VisA under cross-dataset zero-shot settings, SPG achieves competitive image-level detection and strong pixel-level segmentation; with DINOv3, SPG attains the highest pixellevel AUROC among the compared methods. We also report SPG instantiated with OpenCLIP (ViT-L/14@336px) to align the backbone with CLIP-based baselines. Moreover, the learned guide coefficients trace decisions back to a small set of dictionary atoms, revealing category-general and category-specific factors.

• The paper introduces a sparse autoencoder-based framework for zero-shot anomaly detection that eliminates the need for handcrafted prompts.
• It employs a two-stage pipeline using patch-token features and sparse-projected guides, achieving high AUROC scores on benchmark datasets.
• The method demonstrates superior cross-dataset transferability and interpretability, making it highly relevant for industrial visual inspection.

Sparse-Projected Guides with Sparse Autoencoders for Zero-Shot Anomaly Detection

Introduction and Problem Formulation

The paper presents Sparse-Projected Guides (SPG), a prompt-free, backbone-agnostic framework for zero-shot anomaly detection (ZSAD) and segmentation leveraging patch-token features from frozen foundation models. Unlike prevalent CLIP-based approaches which rely on handcrafted or learned prompts as reference embeddings for "normal" or "anomalous" states, SPG parameterizes guide vectors via sparse coefficients in a Sparse Autoencoder (SAE) latent space. This leads to a transparent and tunable anomaly criterion and enables compatibility with both vision-language and vision-only encoders.

ZSAD is particularly relevant in industrial visual inspection, where new defect or object categories often lack target-domain training data. SPG's cross-dataset transfer protocol ensures all learnable parameters are trained only on an auxiliary dataset with pixel-level anomaly masks and deployed to unseen target categories without any adaptation.

Methodology

SPG consists of a two-stage learning pipeline (Figure 1). Stage 1 trains an SAE on patch-token features from a frozen backbone $\phi$, learning a dictionary $D$ for sparse decomposition of patch representations. In Stage 2, both $\phi$ and the SAE are frozen, and only non-negative, sparsity-regularized guide coefficients for "normal" and "anomalous" guides are optimized. These guides are generated through the SAE dictionary, producing interpretable vectors anchored in a sparse set of dictionary atoms.

Patch-wise anomaly scoring is performed via cosine similarities between patch features and guide vectors, followed by a temperature-scaled two-class softmax to extract an anomaly probability map. Aggregation of this map into an image-level score is achieved through temperature-controlled log-sum-exp pooling, allowing a continuum from max-pooling (which preserves localized high-confidence anomalies) to mean-pooling (which aggregates uniformly).

Figure 1: SPG two-stage pipeline: SAE pretraining on frozen encoder features and subsequent optimization of sparse guide coefficients for normal/anomaly states.

Quantitative Results and Competitive Performance

SPG was evaluated on MVTec AD and VisA, two established anomaly detection datasets, using cross-dataset zero-shot transfer. DINOv3 (ViT-L) was used as the default encoder; OpenCLIP compatibility was also demonstrated. Metrics included image-level AUROC/AP and pixel-level AUROC/AUPRO.

SPG with DINOv3 achieves image-level AUROC of 91.4 on MVTec AD (VisA$\rightarrow$MVTec) and 80.2 on VisA (MVTec$\rightarrow$VisA), closely matching state-of-the-art CLIP prompt-based approaches. Critically, SPG attains the highest pixel-level AUROC across both datasets (92.3 for MVTec AD and 96.0 for VisA), establishing its superiority in segmentation tasks. AUPRO scores are competitive, although prompt-adaptation methods such as VCP-CLIP occasionally outperform SPG in certain category transfers.

SPG's backbone flexibility was rigorously analyzed. DINOv3 consistently exhibited the best performance across both image and pixel metrics (Figure 2), although other self-supervised ViTs and VLM backbones are compatible and yield competitive outputs.

Figure 2: Influence of backbone selection on both image-level and pixel-level anomaly metrics under cross-dataset transfer.

Ablations and Design Sensitivity

Extensive ablations reveal the sensitivity of SPG to SAE hyperparameters. The optimum SAE dictionary width ($C$) and TopK sparsity level ($k$) are dependent on both transfer direction and evaluation metric—no single configuration generalizes universally (Figure 3). This demonstrates the necessity for careful tuning in deployment, especially when cross-domain transferability is critical.

Figure 3: Performance sensitivity landscape with varying SAE dictionary width and TopK sparsity under VisA→MVTec AD and MVTec→VisA transfer.

Analysis of the pooling aggregation for image-level scoring indicates that max-like pooling (low $\tau$) reliably preserves high-confidence localized anomaly responses, outperforming mean-type aggregation (higher $\tau$). This empirically confirms that image-level detection benefits from highlighting spatially sparse anomaly responses (Figure 4).

Figure 4: Max-like aggregation (low $\tau$) maintains superior image-level anomaly detection.

Guide sparsification in Stage 2 was compared via explicit TopK and ReLU+$D$0 parameterizations. Increased sparsity budget leads to improved performance, but transfer direction can dictate the effective optimal sparsity (Figure 5), suggesting adaptive sparsity control may be preferable to fixed constraints.

Figure 5: Stage-2 guide learning performance as a function of explicit TopK guide sparsity.

Interpretability and Qualitative Analysis

SPG's sparse guides enable inspection of the decision anchor at the level of individual SAE atoms. Representative anomaly guide atoms, identified by largest learned coefficients, were visualized by retrieving top-activating patches from the auxiliary dataset and overlaying activation heatmaps onto images (Figure 6). Some atoms exhibit broad activation across categories (class-general, e.g., cracks, missing parts), while others are highly category-biased (class-specific structures). This dichotomy, enabled by the SAE's overcomplete dictionary and sparse latent encoding, enhances the transparency of SPG's anomaly criterion.

Figure 6: Qualitative visualization of SAE atoms emphasized by the anomaly guide; some exhibit class-general anomaly cues, others category-specific patterns.

Theoretical and Practical Implications

SPG bridges mechanistic interpretability and zero-shot deployment in anomaly detection. Its reliance on sparse SAE-based dictionary representations affords post-hoc analysis of contributing factors to detection decisions, enabling practical use in domains requiring explainability. The prompt-free, backbone-agnostic design circumvents issues in vision-language dependency and aligns with recent trends in leveraging self-supervised representation spaces. The ability to transfer guide vectors to unseen categories without adaptation underscores SPG's practical utility in industrial and open-world settings.

Theoretically, SPG highlights the value of sparse coding in overcomplete latent spaces for robust, interpretable anomaly criteria. The observed class-general and class-specific atoms suggest that foundation models encode both domain-independent and domain-tied anomalies, which can be harnessed through sparse projection.

Future Directions

SPG's prompt-free anomaly criterion suggests future work in:

• Adaptive sparsity regularization, potentially with learnable per-category constraints.
• Joint optimization of SAE dictionary and guide coefficients in multitask or domain adaptation settings.
• Integration with foundation models beyond vision (e.g., multimodal encoders), extending zero-shot anomaly detection to diverse modalities.
• Development of interpretable guide visualization protocols for deployment in high-stakes industrial inspection and medical imaging.

Conclusion

SPG demonstrates a competitive, interpretable, and domain-agnostic approach for zero-shot anomaly detection and segmentation founded on sparse autoencoder projections of frozen backbone features. Its robust performance across cross-dataset transfer, transparent decision-making via inspected SAE atoms, and prompt-free operation position it as a practical framework for open-world anomaly detection, with significant implications for theory and deployment in foundation model-powered computer vision.