Real-IAD Benchmark for Industrial Anomaly Detection

• Real-IAD Benchmark is a large-scale, multi-view industrial anomaly detection dataset offering over 150,000 high-resolution images across 30 diverse object classes with detailed pixel-level defect annotations.
• It features innovative evaluation protocols that use both single-view and aggregated multi-view metrics to simulate realistic inspection scenarios and account for noisy training data.
• The benchmark highlights the limitations of existing methods and encourages research into robust, multi-view fusion and noise-tolerant algorithms for practical industrial applications.

The Real-IAD benchmark is a large-scale, multi-view industrial anomaly detection dataset and evaluation protocol developed to address critical limitations of existing benchmarks in both scale and realism (Wang et al., 19 Mar 2024). It provides a challenging, high-resolution setting with over 150,000 images covering 30 diverse object classes, making it markedly more representative of real-world industrial inspection scenarios than previous datasets.

1. Dataset Composition and Collection Protocol

Real-IAD comprises approximately 150,000 images spanning 30 distinct object categories, including metal, plastic, wood, ceramics, and mixed materials. Each object is captured from five angles—a single top-down view and four side views at symmetric 45° increments—using a multi-camera setup. This multi-view acquisition ensures detection coverage for defects that may be partially occluded or angle-dependent, as encountered in automated inspection lines.

Defective samples include a broad spectrum of industrial anomalies: pit, deformation, abrasion, scratch, damage, missing parts, foreign objects, and contamination. Crucially, the range of defect-affected area proportions and aspect ratios is significantly larger than in existing datasets, with elongated, irregular, and very small defects present. Defect annotation is performed with pixel-level ground truth masks, supporting both detection and fine-grained segmentation.

2. Benchmark Protocol and Evaluation Metrics

Real-IAD introduces several benchmark protocols to mimic real production line scenarios:

• Standard Unsupervised IAD (UIAD): Training on defect-free (“normal”) images only, with evaluation by image-level AUROC (I-AUROC) and normalized per-region overlap (P-AUPRO).
• Multi-View Aggregation: In practice, inspections leverage multiple views per object. The sample-level metric (S-AUROC) aggregates anomaly predictions across all five views of each object sample, reflecting true line-level decision-making.
• Fully Unsupervised Industrial Anomaly Detection (FUIAD): Recognizing that in industrial contexts, training samples may accidentally include 10–40% defective images due to high yield rates. The “noisy ratio” (α) is explicitly controlled; test sets remain balanced. This setting is designed to benchmark model robustness against contaminated training data.

Relevant formulas include:

• $\text{I-AUROC} = \int_{FPR} TPR \, dFPR$ (over image-level predictions)
• $\text{S-AUROC}$ aggregates the AUROC of merged multi-view sample predictions.

This protocol is critical for benchmarking under realistic and production-relevant conditions.

3. Comparative Analysis and Difficulty

Real-IAD is an order of magnitude larger and more challenging than previous datasets such as MVTec AD (∼5,000 images, 15 classes, lower resolution). Images have a resolution of 2,000–5,000 pixels, supporting detection of finer defects. The use of multi-view imaging makes previously “easy” samples (e.g., detected in only one view) more difficult, and the broad range of defect morphologies exposes weaknesses in methods that rely on homogeneous assumptions about anomaly presentation.

Table summarizing scale comparison:

Dataset	Total Images	Classes	Resolution (px)	Multi-View
MVTec AD	~5,000	15	~700–1024	No
VisA	–	–	–	No
Real-IAD	150,000	30	2,000–5,000	Yes

This increased diversity and scale enable more discriminative evaluation: methods that nearly saturate (99%+ AUROC) on MVTec AD drop to 85–90% on Real-IAD single-view, and multi-view integration further highlights performance gaps.

4. Methodological Benchmarking and Reported Results

The benchmark includes evaluations of state-of-the-art unsupervised IAD algorithms, spanning:

• Feature embedding methods: PatchCore, PaDiM, CFlow.
• Data augmentation methods: SimpleNet, DeSTSeg.
• Reconstruction-based methods: RD, UniAD.

On Real-IAD:

• Single-view I-AUROC: Top methods achieve ~85–90%.
• Multi-view S-AUROC: Provides more nuanced model differentiation, with performance drops relative to simpler datasets.
• FUIAD setting: As α increases, most methods degrade in AUROC, but memory bank approaches (PatchCore, SoftPatch) exhibit greater robustness due to implicit filtering of contaminant anomaly features.

This suggests that Real-IAD captures the “industrial realism gap” missing from previous benchmarks.

5. Implications for Robustness, Generalization, and Practical Utility

By including multi-view images, a broader spectrum of defect morphologies, and a fully unsupervised noisy training regime, Real-IAD enables systematic paper of:

• Model robustness to non-purified training data (as in true factory settings, where labeling is expensive and yields are imperfect).
• The importance of model architectures that can aggregate or fuse multi-view information; single-view models are insufficient.
• The ability to separate genuine anomalies from natural object appearance variation in high-dimensional, heterogeneous datasets.

Real-IAD thus advances IAD research toward methods optimized for practical deployment, not merely laboratory benchmarks.

6. Prospective Research Directions

Real-IAD’s scale and protocols drive several methodological trajectories:

• Multi-view fusion: Model architectures for explicit feature fusion across camera views, including cross-attention mechanisms and latent feature aggregation.
• Noise tolerant learning: Development of robust learning algorithms leveraging importance re-weighting, self-supervision, or hybrid labeling to mitigate noisy anomaly contamination.
• Sample-level decision modeling: Algorithms operating on aggregate sample-level features to mimic real inspection workflows rather than naively treating each image in isolation.
• Scalable annotation and semi-supervised learning: Incentivized by the annotation challenges posed by scale, future IAD frameworks may increasingly use weak/soft label regimes and self-supervised pretraining.

7. Accessibility and Reproducibility

Dataset and code for Real-IAD are openly available to the community, supporting reproducible evaluation, extensibility to additional modalities, and comparison against emerging architectures. Its design and documented protocols facilitate plug-and-play benchmarking for new model classes and provide a standardized reference point for industrial computer vision.

---

Real-IAD thus constitutes a foundational resource for industrial anomaly detection, shaping evaluation, methodology, and deployment-readiness by simulating key facets of industrial practice: high-mix, multi-view, fine-grained, and label-imperfect settings. Its impact is manifest in the shift toward more robust, general, and practically useful IAD models.