InspectMM: Unified Multimodal Inspection

• InspectMM is a comprehensive, expert-annotated multimodal dataset that supports unified modeling for varied industrial inspection tasks across assets.
• It integrates image-level anomaly flagging, object detection, and keypoint localization using consistent expert protocols and rigorous data splits.
• The dataset enables robust evaluation of vision-language models in high-stakes industrial applications through detailed performance metrics and domain-specific challenges.

InspectMM is a large-scale, expert-annotated multimodal dataset designed to support unified modeling of industrial inspection tasks—including image-level anomaly flagging, object detection, and keypoint localization—across diverse asset classes such as wind turbines, solar farms, and building rooftops. Its design aims to enable systematic evaluation of vision-LLMs (VLMs) for precision-critical inspection scenarios, providing curated data, consistent annotation protocols, and rigorous split methodology for benchmarking multitask perception under realistic domain shift. InspectMM facilitates analysis of VLM robustness, grounding, and failure modes within a unified theoretical interface for industrial inspection research and model development (Wallace et al., 3 Aug 2025).

1. Motivation and Objectives

InspectMM addresses the operational limitations of using disjoint, task-specific models for industrial asset inspection pipelines. Traditional approaches require separate models for tasks such as anomaly flagging (classification), detection (bounding boxes), and inventory counting (keypoints), which results in redundancy, inefficiency, and elevated maintenance overhead. The principal objectives for creating InspectMM include:

• Providing a comprehensive, expert-validated benchmark for unified vision-LLM training and evaluation across inspection tasks.
• Enabling fine-tuning and stress-testing of VLMs on domain-specific aerial RGB imagery that is markedly out-of-distribution compared to web benchmarks.
• Systematically assessing model robustness, grounding, and critical failure modes in high-stakes industrial contexts (Wallace et al., 3 Aug 2025).

2. Dataset Composition and Structure

InspectMM comprises 292,341 aerial RGB frames collected via drone-based industrial inspections, annotated across multiple modalities by certified experts. The core annotation modalities are:

• Image-level class labels: "anomaly present" / "anomaly absent" for anomaly flagging (CLS).
• Bounding boxes: For defect (crack/damage) localization in anomaly detection (OD).
• 2D keypoints: Precisely localize and count components (solar panels, rooftop fixtures) for inventory counting (KD).

The primary inspection tasks and their characteristics (as mapped to standard computer vision paradigms) are:

Task Name	Data Modality	Annotation Quantity
CLS (Flagging)	Classification	145,000 images, 145,000 labels
OD (Detection)	Bounding box localization	50,000 images, 20,000 boxes
KD (Counting)	2D keypoints	97,000 images, 674,000 keypoints

Bounding boxes are encoded using normalized pixel coordinates scaled to 1,000 (Florence-2 "spatial token" convention), while keypoints are defined at the geometric center of each component. The dataset further distinguishes between "structured" (solar arrays) and "unstructured" (building rooftops) configurations for keypoint tasks, facilitating stratified evaluation.

3. Data Splits and Annotation Protocols

Data splits for each task are partitioned into disjoint training, validation, and test sets. These are stratified by asset type and geographic source to maximize representativity and minimize potential leakage. Keypoint validation datasets are further subdivided:

• Structured subset: Grid-aligned solar panel arrays characterized by high mean counts and low intra-image variance.
• Unstructured subset: Rooftop fixtures (ventilation, HVAC, etc.), typically few per image but with high spatial and geometric variance.

Annotation pipelines prioritize rigor and domain specificity:

• All labels and markups performed by certified industrial inspection experts adhering to consistent guidelines.
• Bounding boxes tightly circumscribe visible cracks or defects; keypoints are precisely positioned at component centers.
• Cross-review by a second expert and targeted spot-checks on label outliers are integral to annotation QA.
• Anomaly class labels are assigned only at a predefined severity threshold (e.g., crack width > 2 mm) to mitigate subjective judgment variance (Wallace et al., 3 Aug 2025).

4. Task Distribution and Metrics

InspectMM establishes standardized evaluation tasks with explicit class distributions, annotation densities, and metric protocols:

• Anomaly Flagging (CLS): 145,000 images, class-balanced overall (assets may vary in individual prevalence).
• Anomaly Detection (OD): 50,000 images, 20,000 bounding boxes, with a significant proportion of "negative" images (no annotated defects).
• Inventory Counting (KD): 97,000 images, 674,000 keypoints, mean of ~7 per image but with a long tail for large solar arrays.

Metrics used for evaluation—where directly stated—include:

• Learning rate for fine-tuning Florence-2 backbones: $\alpha = 10^{-6}$.
• Detection (OD): Intersection-over-Union (IoU) thresholds—IoU ≥ 0.5 for accurate detection; degenerate if IoU < 0.2 and area >30% of image.

$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$$

$$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

• Keypoint Localization (KD): A predicted keypoint is correct if within $\delta = 10$ or $20$ pixels Euclidean distance.
• mAP: Mean Average Precision is reported for zero-shot crack detection, though not explicitly defined in the data (Wallace et al., 3 Aug 2025).

5. Empirical Findings and Observed Challenges

Evaluation of VLMs trained on InspectMM, such as InspectVLM (Florence-2-based), yields key insights into both dataset utilization and task difficulty:

• Domain shift: VLMs pre-trained on general web imagery underperform without domain-specific fine-tuning, reflecting a challenging distributional gap.
• Fine-grained detection limitations: InspectVLM predicts overly large or spurious bounding boxes for cracks; small defects are frequently missed.
• Prompt overfitting: With low variability in anomaly-flagging prompts, models collapse to trivial "always-yes" answers by the fourth epoch.
• Spatial and architectural constraints: Tokenized encoders, unlike feature pyramid networks (FPNs), struggle with high-resolution or multiscale localization, notably in small object detection.
• Layout complexity: Keypoint accuracy on unstructured rooftops lags by over 20% (precision and recall) compared to structured solar array scenarios.
• Label imbalance and shortcut learning: Repetitive distribution in classification undermines robust visual grounding, leading models to learn spurious decision rules detrimental to critical safety applications (Wallace et al., 3 Aug 2025).

6. Significance and Limitations

InspectMM establishes a rigorous foundation for research into unified, multitask vision-language modeling for industrial inspection, reflecting real-world distributions, annotation complexities, and performance failure modes. However, several dataset and benchmarking limitations are apparent:

• Exact train/val/test partition counts are not published.
• The keypoint and detection modalities expose significant performance disparity between structured and unstructured settings.
• The dataset highlights the current inability of unified VLMs to simultaneously achieve robust grounding and fine-grained localization, especially for safety-critical industrial applications.

A plausible implication is that further advancements in architectural design, data augmentation, and active task prompting will be necessary to close the observed gap between the conceptual promise of unified perception and real-world industrial inspection demands (Wallace et al., 3 Aug 2025).