Anomaly Feature Learning Overview

• Anomaly Feature Learning (AFL) is a technique that extracts discriminative residual features to distinguish anomalies from normal data using adaptive attention mechanisms.
• It employs a two-step process involving residual mapping from query images and abnormal residual proxy mining to effectively localize anomalies.
• AFL enhances anomaly detection accuracy and speed with a fusion of abnormal and normal guided scoring, reducing false positives in cross-domain contexts.

Anomaly Feature Learning (AFL) refers to representation learning modules and frameworks specifically designed to extract—adaptively and discriminatively—the priors, patterns, or residual components that set anomalous samples apart from normal background data. Unlike methods that operate purely on raw similarities with normal references or naive use of outlier exemplars, AFL as studied in the Normal-Abnormal Generalist Learning (NAGL) paradigm (Wang et al., 1 Oct 2025) systematically leverages both normal and abnormal reference data to achieve robust, transferable anomaly detection and localization, particularly in the context of generalist models intended for cross-domain deployment.

1. Purpose and Motivation

The principal objective of AFL within the NAGL framework is to deliver instance-aware, adaptive representations of abnormality in a query image. Standard reference-based anomaly detection approaches, which rely exclusively on the comparison to normal reference samples (e.g., via nearest-neighbor patch search in feature space), struggle to capture the diversity and local manifestations of real-world anomalies—especially when transferred across domains. Furthermore, naive inclusion of abnormal references as direct targets may trigger false activations due to background mismatches. AFL seeks to overcome these pitfalls through a residual-centric, proxy-guided, and attention-based mechanism that explicitly encodes the deviations characteristic of anomalies while retaining domain-relevant specificity.

2. Core Methodological Design

AFL operates on a multi-step pipeline:

a. Residual Mapping (Query Phase):

For each query image, deep patch-level feature representations are extracted using a frozen, pre-trained backbone. These patch descriptors are compared to feature sets obtained from normal references, generating a residual map Res($\mathcal{F}^q,\mathcal{F}^n$) for each query patch. This residual localizes the aspects by which the query diverges from exemplars deemed “normal.”

b. Mining of Abnormal Residual Proxies:

Leveraging abnormal references, the Residual Mining (RM) module computes residual patterns (difference between abnormal and corresponding normal features) and processes them through an attention mechanism to generate a set of abnormal “residual proxies” (%%%%1%%%%). These proxies encode the core signals of aberrant appearance learned from available outlier data, distilled across patch positions and reference samples.

c. Anomaly Feature Learning (AFL Attention):

A separate attention module is constructed where the learned abnormal residual proxies $\tilde{\mathcal{P}}$ are used as queries, and the query image’s own residual patch features serve as the attention keys. The attention output acts as anomaly proxies ($\hat{\mathcal{P}}$) adapted to the specific query instance. The core transformation is: $$Q_2 = W^{(Q_2)} \cdot \tilde{\mathcal{P}}, \quad K_2 = W^{(K_2)} \cdot \text{Res}(\mathcal{F}^q,\mathcal{F}^n), \quad V_2 = W^{(V_2)} \cdot \mathcal{F}^q$$ and

$$\hat{\mathcal{P}} = SA_2\left(\text{Softmax}\left(\frac{Q_2 K_2^T}{\sqrt{d}}\right)V_2\right)$$

where $SA_2$ denotes self-attention and $d$ is a scaling dimension.

d. Abnormal-Guided Scoring:

For each query patch $i$, its anomaly score $S_a^{(i)}$ is computed as the average $1-\cos$ similarity between its feature $f^{q}_i$ and each anomaly proxy $\hat{\mathcal{P}}_m$: $$S_a^{(i)} = \frac{1}{M} \sum_{m=1}^M \left[ 1 - d(f^{q}_i, \hat{\mathcal{P}}_m) \right]$$ where $d(\cdot,\cdot)$ is cosine distance, $M$ is the number of proxies.

e. Fusion with Normal-Guided Scoring:

The final anomaly score map is obtained by summing the abnormal-guided score $S_a$ and the normal-guided score $S_n$ (from a KNN distance to normal reference patches): $S = S_n + S_a$ yielding a pixel-level or patch-level anomaly localization.

3. Role within the Generalist NAGL Framework

AFL is inherently dependent on the RM stage, which generates the abnormal residual proxies. RM takes in abnormal reference samples, computes their residuals against normal reference features, and through attention distills a robust, learnable set of residual patterns (proxies) that generalize across domains and categories. AFL, in turn, applies these proxies as guidance to the query image’s own residuals, ensuring that only truly abnormal deviations (with strong similarity to real outlier patterns) are flagged as anomaly, whereas background differences or domain shift effects are suppressed.

This division of labor supports robust generalization: the RM proxies provide transferable abnormal patterns, while AFL adaptively reinterprets these in the instance domain via attention, producing finely discriminative anomaly signals.

4. Mathematical Formalization

The full sequence of operations in AFL is as follows:

1. RM residual proxy generation:

$$\tilde{\mathcal{P}} = SA_1(\text{Softmax}((Q_1 K_1^T)/\sqrt{d} + \mathcal{M}')V_1)$$

where $SA_1$ is self-attention, $(Q_1, K_1, V_1)$ are linear projections of residuals from abnormal–normal references, and $\mathcal{M}'$ is an attention mask.

1. AFL anomaly proxy adaptation:

$$\hat{\mathcal{P}} = SA_2(\text{Softmax}((W^{(Q_2)} \tilde{\mathcal{P}}) (W^{(K_2)} \text{Res}(\mathcal{F}^q,\mathcal{F}^n))^T/\sqrt{d}) W^{(V_2)} \mathcal{F}^q)$$

1. Score computation for localization:

$$S_a^{(i)} = \frac{1}{M} \sum_{m=1}^{M} (1 - d(f_i^q, \hat{\mathcal{P}}_m)), \quad S = S_n + S_a$$

where $M$ is the number of anomaly proxies.

5. Empirical Performance and Metrics

The effectiveness of AFL is reported as part of the overall NAGL pipeline. Standard metrics include image-level AUROC, average precision (AP), F1-max, pixel-level AUROC, and per-region overlap (PRO). Inclusion of AFL yields substantial improvements for both global anomaly classification and pixel-level segmentation, particularly across shifts from original to target domains, supporting its effectiveness for cross-domain anomaly detection scenarios. The architecture is demonstrably faster at inference than heavy generative models, and it reduces false activation rates compared to approaches that lack explicit residual-based attention mechanisms.

6. Comparative Advantages and Robustness

Compared to training-free, single-reference, or purely normal-based eg. KNN anomaly detectors, AFL’s explicit attention-driven transfer from abnormal proxies substantially enhances both accuracy and domain generalization. The residual mapping suppresses confounds arising from background/texture misalignment, while the two-stage attention structure allows for sharper, instance-aware anomaly focus. Unlike trivial combination strategies, this architecture prevents the propagation of false positives arising from mismatched abnormal backgrounds or superfluous abnormal pattern diffusion.

7. Potential Limitations and Extensions

While AFL provides significant improvements over standard GAD baselines, its reliance on the quality and variability of the abnormal reference set (and the expressivity of the learned residual proxies) represents a potential bottleneck: insufficient or non-representative abnormal reference patterns may constrain proxy coverage. Further, the transferability of learned proxies to completely unseen domains is fundamentally limited by the inherent proximity of available abnormal patterns to the possible anomalies encountered in the target. Extensions could include dynamic adaptation of the residual proxy set, proxy enrichment strategies, or more sophisticated fusion architectures between normal and abnormal cues. Nevertheless, the design currently represents the first principled approach for generalist anomaly detection using a mixture of normal and anomalous references as guidance for adaptive, residual-based anomaly feature learning (Wang et al., 1 Oct 2025).