Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models

Abstract: Large-scale vision-LLMs (VLMs) exhibit remarkable zero-shot capabilities, yet the internal mechanisms driving their anomaly detection (AD) performance remain poorly understood. Current methods predominantly treat VLMs as black-box feature extractors, assuming that anomaly-specific knowledge must be acquired through external adapters or memory banks. In this paper, we challenge this assumption by arguing that anomaly knowledge is intrinsically embedded within pre-trained models but remains latent and under-activated. We hypothesize that this knowledge is concentrated within a sparse subset of anomaly-sensitive neurons. To validate this, we propose latent anomaly knowledge excavation (LAKE), a training-free framework that identifies and elicits these critical neuronal signals using only a minimal set of normal samples. By isolating these sensitive neurons, LAKE constructs a highly compact normality representation that integrates visual structural deviations with cross-modal semantic activations. Extensive experiments on industrial AD benchmarks demonstrate that LAKE achieves state-of-the-art performance while providing intrinsic, neuron-level interpretability. Ultimately, our work advocates for a paradigm shift: redefining anomaly detection as the targeted activation of latent pre-trained knowledge rather than the acquisition of a downstream task.

• The paper introduces a training-free method (LAKE) that identifies sparse, high-variance neurons encoding latent anomaly knowledge in VLMs.
• It uses patch-level deviation probing and cross-modal semantic activation to accurately localize defects with state-of-the-art AUROC and PRO metrics.
• Findings demonstrate that activating anomaly-sensitive neurons enhances efficiency and interpretability in both industrial and medical imaging contexts.

Latent Anomaly Knowledge Excavation in Vision-LLMs

Introduction

The paper "Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-LLMs" (2604.07802) introduces LAKE, a framework that challenges the prevailing black-box paradigm in vision-LLM (VLM) based anomaly detection (AD). It hypothesizes and empirically validates that pre-trained VLMs, such as CLIP, intrinsically encode anomaly knowledge concentrated in a sparse set of neurons, which remains latent under standard inference. LAKE proposes a training-free procedure that identifies and elicits these anomaly-sensitive neurons using only a small set of normal (non-anomalous) samples. The framework reformulates AD as the targeted activation of latent knowledge, bypassing the common reliance on external adapters, prompt tuning, or memory banks.

Figure 1: Conceptual comparison of anomaly detection paradigms, highlighting LAKE's intrinsic activation of sparse anomaly-sensitive neurons instead of relying on external modules.

LAKE Framework

LAKE is founded on the premise that VLMs' discriminative capacities for anomaly detection are not distributed uniformly but are highly localized within a small subset of high-variance neurons. The LAKE pipeline comprises three principal stages:

1. Anomaly-Sensitive Neuron Detection: Neurons with highest variance across normal samples are identified, leveraging the theoretical equivalence between variance ranking (in highly disentangled VLMs) and truncated PCA.
2. Patch-Level Deviation Probing: Input image features are projected into this anomaly-sensitive subspace. For each patch token, the deviation from the normal reference manifold (gallery) is measured via nearest-neighbor search, using max-pooling to aggregate extremal deviations—essential for localizing sparse defects.
3. Cross-Modal Semantic Activation: Deeper visual features are aligned with textual embeddings of "normal" and "anomalous" prompts. Semantic anomaly scores are derived for each patch and fused with visual deviations via a tunable parameter $\alpha$.

   Figure 2: The LAKE framework integrates variance-based neuron selection, visual deviation, and cross-modal semantic probing with joint anomaly scoring.

Numerical Results

LAKE achieves state-of-the-art (SOTA) performance on standard industrial anomaly datasets (MVTec-AD, VisA, BTAD) and demonstrates robust generalization when transferred to the medical Brain-AD dataset. Key results include:

• Image-Level AUROC: 94.7% (MVTec-AD), 89.4% (VisA), and 96.2% (BTAD), outperforming recent prompt/adaptation-based methods by notable margins (e.g., $+70.5\%$ AUROC on BTAD over backbone).
• Pixel-Level PRO: 88.9% (MVTec-AD), 92.3% (VisA), 81.5% (BTAD); substantially higher than all baselines.

  Figure 3: Qualitative comparison of anomaly localization, showing that LAKE yields sharper and more accurate defect heatmaps, minimizing background false positives relative to SOTA competitors.

Ablations confirm that LAKE's data efficiency is high: with as few as 64 normal samples, both image-level and pixel-level metrics reach full-shot performance. The optimal subspace dimension is narrow ($K=100$), with performance degrading if the subspace is either too constrained or too dispersed, empirically validating the hypothesis of sparsity.

Figure 4: Image-level AUROC as a function of support set size illustrates rapid convergence (i.e., high data efficiency) in LAKE.

Figure 5: Pixel-level anomaly localization improves and stabilizes with increasing support set size, with little gain beyond 64-shots.

Figure 6: Ablations on subspace sparsity (top-$K$) and fusion weight ($\alpha$), confirming the importance of careful hyperparameter calibration and the dominance of visual signals in robust detection.

Mechanistic Interpretability

Through direct comparison, random channel selection versus LAKE's top-variance neurons results in a marked decrease in accuracy, reinforcing that anomaly sensitivity is not uniformly distributed. Qualitatively, only the top-variance-selected neurons align with true anomalies, while random selections yield noisy, non-discriminative heatmaps.

Figure 7: Heatmaps from random versus top-$K$ neuron subsets; only the latter provide precise and interpretable localization of anomalies.

A cross-category t-SNE analysis reveals that the same set of neurons underpins anomaly knowledge across a diverse array of industrial categories, supporting the universality and structural sharing of these sensitive units.

Figure 8: Cross-category neuron overlap analysis via t-SNE, confirming that the anomaly-sensitive neurons generalize and overlap in activation across disparate object types.

Theoretical and Practical Implications

LAKE demonstrates that diagnosing anomalies does not require further adaptation or complex, external modules. Its success with a narrow subspace provides a practical approach to memory and compute-efficient inference, reducing memory consumption and accelerating nearest-neighbor retrieval. From a theoretical standpoint, LAKE's use of direct variance-based neuron selection bounds its procedure in fundamental manifold learning theory, and the explicit fusion of geometric and semantic cues leverages the polysemantic nature of VLMs' representations.

Plug-and-play integration with existing methods (e.g., as an add-on for WinCLIP or ReMP-AD) consistently produces performance gains, substantiating the claim that latent anomaly knowledge can be efficiently activated in frozen models, regardless of the original model design.

Moreover, LAKE's competitive, domain-agnostic performance on medical imaging demonstrates that the activated anomaly-sensitive neurons encode general abnormality priors—suggesting potential for wide applicability across domains, given a compatible VLM backbone.

Limitations and Future Directions

The current formulation is restricted to 2D imagery and static analysis; extension to temporal or 3D VLMs remains to be addressed. The framework is also reliant on the presumed disentanglement property and structured variance profile of modern VLM activations—a consideration for future architectures with different representational properties.

Conclusion

LAKE provides a rigorous, interpretable, and training-free paradigm for anomaly detection in vision-LLMs, reframing the task as the targeted excavation and activation of natively encoded knowledge. Its empirically validated sparsity, data-efficiency, and cross-domain robustness mark it as a highly practical alternative to adaptation-heavy or black-box detection pipelines. The work advocates for a mechanistically rooted perspective in anomaly detection, with likely influence on both future interpretability research and efficient deployment strategies for industrial and medical vision systems.