Boxes2Pixels: Learning Defect Segmentation from Noisy SAM Masks

Abstract: Accurate defect segmentation is critical for industrial inspection, yet dense pixel-level annotations are rarely available. A common workaround is to convert inexpensive bounding boxes into pseudo-masks using foundation segmentation models such as the Segment Anything Model (SAM). However, these pseudo-labels are systematically noisy on industrial surfaces, often hallucinating background structure while missing sparse defects. To address this limitation, a noise-robust box-to-pixel distillation framework, Boxes2Pixels, is proposed that treats SAM as a noisy teacher rather than a source of ground-truth supervision. Bounding boxes are converted into pseudo-masks offline by SAM, and a compact student is trained with (i) a hierarchical decoder over frozen DINOv2 features for semantic stability, (ii) an auxiliary binary localization head to decouple sparse foreground discovery from class prediction, and (iii) a one-sided online self-correction mechanism that relaxes background supervision when the student is confident, targeting teacher false negatives. On a manually annotated wind turbine inspection benchmark, the proposed Boxes2Pixels improves anomaly mIoU by +6.97 and binary IoU by +9.71 over the strongest baseline trained under identical weak supervision. Moreover, online self-correction increases the binary recall by +18.56, while the model employs 80\% fewer trainable parameters. Code is available at https://github.com/CLendering/Boxes2Pixels.

• The paper introduces a noise-robust distillation framework that refines SAM-generated pseudo-masks into accurate pixel-level defect segmentation.
• The method employs a dual-branch student network that fuses frozen DINOv2 features with a lightweight CNN to capture global semantics and fine local details.
• Experimental results demonstrate improved recall (+0.1856), higher IoU scores, and an 80% reduction in trainable parameters compared to baseline models.

Noise-Robust Defect Segmentation via Boxes2Pixels

Motivation and Problem Context

Accurate pixel-level defect segmentation underpins the automation of industrial infrastructure inspection, particularly in safety-critical applications such as wind turbine maintenance. Industrial datasets typically lack dense annotations due to cost and expertise requirements, with bounding boxes serving as the pragmatic annotation format. The emergence of Vision Foundation Models (VFMs), notably SAM, allows for the automatic generation of pseudo-masks from bounding box prompts, bridging localization and segmentation. Nevertheless, the systematic noise in such pseudo-masks—over-segmentation of backgrounds and omission of subtle or thin defects—limits their utility for direct supervision. Existing approaches, including state-of-the-art weakly or box-supervised segmentation, often fail in industrial scenarios due to the sparsity, variability, and low fill-rate of defects within bounding boxes, as well as the coarse spatial priors inherited from box-level annotation.

Boxes2Pixels Framework

Boxes2Pixels reframes the problem by considering foundation model pseudo-masks as a noisy, non-oracular teacher. The framework involves a hierarchical student network optimized for robustness to teacher errors, eschewing reliance on pixel-wise ground truth.

The workflow is as follows: bounding boxes from legacy datasets or detection models (e.g., YOLO) prompt SAM to generate pseudo-masks offline. These serve as supervision signals to train a compact student segmentation model. Unlike previous distillation efforts, SAM is not integrated into inference, decoupling deployment from the heavy computational burden of foundation models.

Figure 1: End-to-end Boxes2Pixels pipeline: annotations are provided as bounding boxes, SAM generates noisy pseudo-masks, and the student produces the final segmentation, with color codes indicating various detection scenarios.

Hierarchical Student Network

Boxes2Pixels employs a dual-branch student: a global semantic branch leveraging frozen DINOv2 ViT-S/14 features adapted via BitFit, and a parallel lightweight CNN detail branch for local spatial structures. BitFit adaptation enables parameter-efficient fine-tuning, thereby minimizing overfitting to noisy supervision. Intermediate DINOv2 features are projected, hierarchically fused via residual blocks, and upsampled. The detail branch preserves high-frequency content critical for thin or distributed defects.

The two branches are fused at a shared spatial resolution and processed by a feature mixer, followed by two independent heads: a binary defect localization head and a $(K+1)$-class semantic segmentation head. The structure is visualized below.

Figure 2: Student network architecture combining frozen global DINOv2 features and a local detail CNN for robust defect segmentation.

Noise-Aware Training Objectives

The training objective explicitly models annotation noise. The binary head is optimized with an Asymmetric Dice loss, up-weighting recall due to the operational cost of missed defects. The semantic head predicts classes with a class-weighted cross-entropy loss. Critically, an online one-sided self-correction mechanism is applied: background-class supervision is relaxed during training if the student’s prediction for any defect class exceeds a high confidence threshold. This asymmetric policy allows recovery of teacher/annotation-missed defects while minimizing the propagation of false positives and semantic drift.

Experimental Results

Quantitative evaluation is performed on a wind turbine blade defect dataset with only box annotations available at training, and manually annotated pixel-level masks for test evaluation. Boxes2Pixels is compared against U-Net, DeepLabV3, and SegFormer baselines, all trained in the same weakly supervised regime with identical SAM-derived pseudo-masks.

Boxes2Pixels demonstrates superior performance:

• Anomaly mIoU: 0.6523 (Boxes2Pixels) vs. 0.5881 (best baseline, SegFormer), a gain of +0.0642.
• Binary IoU: 0.6226 (Boxes2Pixels) vs. 0.5312 (best baseline), +0.0914.
• Binary Recall improvement via self-correction: 0.8051 vs. 0.6195, a +0.1856 absolute gain.

Self-correction yields notable improvements in missed-defect recovery, as confirmed by the anomaly-specific F1 and recall scores. Notably, these gains are achieved with 80% fewer trainable parameters due to the frozen backbone and BitFit adaptation, while real-time inference throughput exceeds 160 FPS on an NVIDIA H100.

Ablation studies confirm the necessity of the dual-branch structure, the explicit binary head, and the self-correction mechanism for robustness under teacher noise. Removing the binary head or self-correction degrades both recall and overall accuracy.

Figure 3: Qualitative results: comparison between bounding box annotations, SAM pseudo-masks, and Boxes2Pixels predictions, highlighting improved localization and recall of subtle defects.

Class- and Efficiency Analysis

The method achieves matched or better per-class performance than the strongest baseline, with significant improvement in the "Dirt" class without sacrificing "Damage" detection, corroborating that the approach generalizes across diverse defect types. Efficiency comparisons reveal a substantial reduction in trainable parameters and improved inference latency relative to baselines such as SegFormer.

Practical and Theoretical Implications

Practically, Boxes2Pixels facilitates pixel-level defect segmentation in industrial scenarios where only sparse box annotations and imperfect foundation model pseudo-masks are available. This decouples deployment from large foundation models and enables operational scale-up. The approach’s robustness to label noise generalizes to other dense prediction and scientific imaging tasks characterized by annotation incompleteness and foundation model limitations.

Theoretically, the work underscores the importance of noise-aware utilization of pseudo-labels, placing emphasis on supervision dynamics rather than attempting to perfect pseudo-label generation. The asymmetric self-correction mechanism provides a conservative pathway to recover missed annotations and to minimize systematic teacher errors without requiring iterative refinement or error modeling.

Future Directions

Further research directions include the application of uncertainty quantification within the self-correction mechanism, transfer to additional industrial and medical domains with different defect morphologies, integration with active learning and model calibration, and analysis of robustness under adversarial teacher noise or open-world distribution shift.

Conclusion

Boxes2Pixels establishes a noise-robust distillation framework for learning pixel-level defect segmentation from sparse, noisy pseudo-masks derived from bounding boxes via foundation models. By combining semantically-rich frozen representations, explicit structural supervision, and asymmetric self-correction, it achieves enhanced recall, localization, and parameter efficiency. The contributions emphasize the criticality of tailored label noise handling for industrial deployment and motivate broader adoption of such label-robust distillation pipelines.

(2604.11162)