Anomaly Detection Benchmark

• Anomaly detection benchmarks are standardized evaluation systems that aggregate diverse datasets, protocols, and metrics to assess methods across multiple modalities.
• They unify data formats and use rigorous metrics like AUROC, Average Precision, and F1 to expose scalability issues and domain-specific challenges.
• Advanced benchmarks incorporate multimodal annotations and mixture-of-experts models to enhance contextual reasoning and improve detection performance.

An anomaly detection benchmark is a standardized evaluation suite that aggregates datasets, protocols, and metrics for rigorously assessing anomaly detection methods across diverse domains and data modalities. Benchmarks of this kind have become a foundational tool for calibrating progress, exposing limitations in current algorithms, and guiding both methodological and application-driven research in anomaly detection.

1. Definition and Scope

Anomaly detection (AD) benchmarks provide curated datasets with ground-truth labels or masks for anomalies, standardized preprocessing, unified data structures, and precise evaluation protocols. They are stratified by modality—tabular, image, time series, graph, text, video, or multi-modal—and are often assembled to maximize domain diversity, sample complexity, and anomaly type variety. Benchmarks enforce reproducibility by mandating canonical train/test splits, reporting practices, and open-source implementations, and they support comparisons of classical unsupervised methods, semi-supervised detectors, supervised learners, and, increasingly, foundation model and multimodal approaches (Ling et al., 25 Nov 2025, Han et al., 2022).

2. Dataset Aggregation and Standardization

Benchmarks like ADNet unify the AD landscape by aggregating a large number of publicly available datasets—49 sources in ADNet's case—across five major domains (Electronics, Industry, Agrifood, Infrastructure, Medical). The resulting composite consists of 380 fine-grained object categories and 196,294 RGB images, all with rigorous, pixel-level MVTec-style anomaly masks and GPT-4-verified structured text annotations. These benchmarks enforce per-category caps to maintain data balance (e.g., 500 normal, 100 normal test, and 100 anomalies per defect type), harmonize formats (e.g., format conversions, tiling, or slicing to standard size), and standardize masking and attribute description protocols (Ling et al., 25 Nov 2025).

Standardization enables unified evaluation and extension. ADNet’s format conversion, pixel-mask schema, and multimodal annotation interface allow seamless ingestion and future expansion by the community.

3. Benchmark Protocols and Evaluation Metrics

Anomaly detection benchmarks specify strict evaluation paradigms that model distinct deployment constraints:

• Single-class (one-for-one): Trains one detector per category (or domain/task), measuring upper bounds of method-specific capacity and robustness.
• Multi-class (one-for-all): Trains a single model on all categories simultaneously, directly revealing scalability and inter-category generalization failures (“category catastrophe”).

Task protocols are divided between image-level (detecting if an image is anomalous) and pixel-level (localizing defective regions) detection. Metrics are rigorously defined:

• AUROC (Area Under ROC Curve): At both image and pixel level,

$$\mathrm{AUROC} = \int_0^1 \mathrm{TPR}(\mathrm{FPR}^{-1}(t)) \,dt$$

• Average Precision and F1: Reflecting class imbalance and precision–recall tradeoff.
• Pixel-level Average Precision: For localization, often never exceeding 30% in highly multi-class settings.

These protocols are complemented by auxiliary analyses (e.g., ablations, scalability breakdowns, context-awareness studies) to quantify domain shift, contextual semantics, and robustness (Ling et al., 25 Nov 2025, Han et al., 2022).

4. Baseline Methods, Scalability, and “Category Catastrophe”

Anomaly detection benchmarks include baselines representing state-of-the-art techniques in memory-based, reconstruction-based, and synthetic/synthesis paradigms, for example:

• PatchCore (memory-based)
• UniAD, RD, ViTAD, MambaAD, Dinomaly, LGC-AD (reconstruction-based)
• SimpleNet, DeSTSeg (synthesis-based)

On controlled-scale benchmarks (e.g., MVTec-AD), these approaches commonly exceed 95% image-level AUROC per category. However, ADNet reveals a striking scalability barrier: aggregate I-AUROC falls from 90.6% in single-class to 78.5% in unified multi-class settings (“category catastrophe”), mirrored by a sharp decline in foreground pixel precision. This phenomenon demonstrates that context-insensitive detectors fail to encode domain- or class-dependent semantics—visual cues like “cracks” or “spots” are ambiguous without categorical context (Ling et al., 25 Nov 2025). Comparable trends are observed in other modalities, such as NLP-ADBench for text (Li et al., 2024) and TAB for time series (Qiu et al., 22 Jun 2025).

5. Multimodal and Context-Adaptive Evaluation

Advanced benchmarks incorporate structured text, multimodal attributes, or language-based descriptions to support hybrid vision–language modeling, context reasoning, and cross-domain transfer. In ADNet, each anomaly’s spatial location and five key visual attributes (color, shape, area size, quantity, underlying reason) are described in standard fields, enabling joint vision–language analysis and supervised or self-supervised multimodal learning.

Addressing context-dependent failures, ADNet introduced Dinomalyᵐ, a context-guided Mixture-of-Experts Transformer decoder where a global feature ([CLS] token) routes information to specialized expert branches, providing context-conditional decoding capacity. This architecture yields substantial multi-class performance gains—raising I-AUROC from 78.5% to 83.2% and P-AUROC from 91.0% to 93.1% with negligible increase in inference cost over standard Transformers (Ling et al., 25 Nov 2025). Routing via the global context token is shown to outperform per-layer gating, and optimal expert number is empirically determined to be $K=8$.

6. Limitations, Extensibility, and Emerging Directions

Benchmarks reveal fundamental issues:

• Scalability: Unified detectors degrade beyond 100–200 categories, underscoring the need for more expressive, context-aware, or mixture-of-experts models.
• Contextual ambiguity: Identical visual anomalies may be normal or defective depending on application domain.
• Generalization: Category, domain, and modality transfer remain open challenges.

To foster extensibility, leading benchmarks such as ADNet provide:

• A modular ingestion and standardization pipeline for new datasets or modalities.
• Community extension scripts to support continuous benchmark evolution without disrupting core structure.
• Multimodal and language-grounded annotation to catalyze cross-disciplinary research.

Future research is driven toward large-scale, robust foundation models for AD, contextual mixture models, continual/incremental adaptation protocols, and hybrid human-AI severity scoring (as highlighted by MAD-Bench (Cao et al., 2024)).

---

References

• ADNet: A Large-Scale and Extensible Multi-Domain Benchmark for Anomaly Detection (Ling et al., 25 Nov 2025)
• ADBench: Anomaly Detection Benchmark (Han et al., 2022)
• NLP-ADBench: NLP Anomaly Detection Benchmark (Li et al., 2024)
• TAB: Unified Benchmarking of Time Series Anomaly Detection Methods (Qiu et al., 22 Jun 2025)
• MAD-Bench: Are Anomaly Scores Telling the Whole Story? A Benchmark for Multilevel Anomaly Detection (Cao et al., 2024)