Boundary-by-Mask: Few-Shot Instance Segmentation with Mask-Conditioned Boundary Learning for Texture-Poor Industrial Parts

Abstract: Recent advances in large pre-trained models have led to remarkable progress in instance segmentation on general images. However, industrial scenarios remain challenging. Instance definitions are often application-specific and inconsistent, and the domain gap from general imagery is substantial due to weak textures and limited contextual cues. Consequently, a direct application of existing models is unreliable. We propose Boundary-by-Mask, a few-shot instance segmentation framework that supervises boundaries instead of interior appearance. Given a few RGB images and corresponding instance masks, the method extracts rich visual features using a foundation-model encoder and trains a lightweight Signed Distance Function (SDF) head to predict boundary-aware distance maps. Segmentation masks are obtained through an SDF-to-mask reconstruction process. By explicitly estimating contours, the framework achieves reliable instance separation even on low-texture and color-uniform surfaces. The instance definition is conditioned by the instance mask. Replacing the mask specifies the segmentation target, such as the whole object or a sub-part. A pixel-wise shallow MLP head enables rapid training. Experiments on industrial parts and food items with ambiguous boundaries show strong few-shot generalization, robustness in feature-poor conditions, and precise control over mask-level targets.

• The paper presents a novel boundary-conditioned SDF framework that leverages reference masks to accurately segment texture-poor industrial parts.
• It employs a modular four-module pipeline using DINOv3 and SAM2 for appearance-invariant object localization and rapid adaptation via a shallow MLP head.
• Experimental validation shows the approach outperforms appearance-based FSIS baselines with mIoU ≈ 0.92 and AP75 = 0.66 using few references.

Boundary-by-Mask: Few-Shot Instance Segmentation with Mask-Conditioned Boundary Learning for Texture-Poor Industrial Parts

Introduction and Motivation

Industrial vision presents significant challenges for instance segmentation because texture-poor and color-uniform components defy appearance-centric approaches. Most current few-shot instance segmentation (FSIS) methods rely heavily on interior appearance features learned from large, natural image datasets, resulting in poor generalization when transferred to industrial contexts marked by ambiguous or highly task-dependent instance definitions. "Boundary-by-Mask" addresses this gap by constructing an instance segmentation pipeline grounded in boundary-based supervision—specifically, Signed Distance Function (SDF) regression—rather than interior texture, thereby achieving robust instance separation even under conditions with minimal appearance cues.

Figure 1: With a few RGB–mask references, the Boundary-by-Mask pipeline learns to predict an SDF and reconstruct instances in a task-adaptive and training-efficient manner.

Methodological Framework

The Boundary-by-Mask pipeline decomposes the segmentation process into four principal modules, organized along two axes: (1) Object vs. SDF, and (2) Offline (reference preparation) vs. Online (query inference). The pipeline exploits foundation models—specifically, DINOv3 as an appearance-invariant encoder for object localization and SAM2 for geometric context extraction. A shallow pixel-wise MLP head regresses per-pixel SDF values, conditioned directly on the reference mask, yielding a boundary-focused inductive bias.

Figure 2: End-to-end pipeline overview including reference collection, feature extraction, SDF head training, object localization, and mask reconstruction for arbitrary query images.

Ground-Truth SDF and Architectural Design

Reference masks are transformed into truncated, normalized ground-truth SDF maps, where the zero level set denotes the object boundary, with gradients clipped to emphasize the boundary zone. This formulation explicitly promotes boundary awareness rather than direct mask prediction, as evidenced by sharp improvements over edge- or mask-supervised heads. The SDF head—a shallow MLP—enables rapid adaptation and lightweight reconfiguration, critical for high-frequency production changeover in industrial settings.

Figure 3: SDF definition and interpretation—a continuous boundary encoding suitable for texture-poor objects; the zero level marks the boundary.

Instance Surface Reconstruction

During inference, the DINOv3-derived object feature vector is used to localize relevant regions in the query image. The SDF head then regresses a dense distance map, from which instance seeds are derived and an energy-based watershed refinement fuses SDF with image gradient cues, resulting in precise instance segmentation. The final mask is conditioned entirely on the chosen reference mask, so altering the reference definition dynamically re-specifies the segmentation granularity (object-level, part-level, etc.) without additional retraining beyond the shallow SDF head.

Figure 4: Qualitative inference examples, showcasing input images, SDF predictions, and the resulting output masks.

Experimental Validation

Comparative Analysis

The evaluation spans custom datasets with challenging industrial items (metal fasteners, tubes, bottles, and food) in both flat and pile layouts, incorporating substantial variation in shape, granularity, and task-dependent semantics. Strong baselines include appearance-driven few-shot approaches (YOLOv11 finetuned, PerSAM, PerSAM-F, No Time to Train!, SAM2) and zero-shot transfer methods.

Quantitative outcomes validate key claims:

• Boundary-by-Mask (BbM) consistently outperforms all appearance-driven FSIS baselines—including large-scale pretrained and fine-tuned models—on $\mathrm{AP}_{50}$, $\mathrm{AP}_{75}$, and mIoU at low reference counts ($K \leq 5$). For $\mathrm{AP}_{50:95}$, YOLOv11 surpasses BbM only at $K = 10$, reflecting a point at which scene-specific feature optimization becomes competitive.
• BbM achieves mIoU $\approx 0.92$ and $\mathrm{AP}_{75} = 0.66$ with just $K = 10$ references, corroborating both the data efficiency and boundary-centric advantage.
• SDF supervision delivers substantial gains over direct mask or edge regression under identical conditions, supporting the theoretical premise that boundary encoding yields better inductive bias for texture-poor part separation.

  Figure 5: Qualitative comparison (three instances) highlighting superior boundary localization and instance separation by BbM over SAM2, YOLOv11, PerSAM, and others.

Impact of Reference Granularity and Flexibility

Altering only the reference mask instantly changes the target instance definition, illustrated by seamless transitions between object- and part-level prediction with no need for backbone retraining.

Figure 6: Mask-conditioned flexibility—shifting from object-level to part-level segmentation by modifying the reference mask alone.

Reference-Shot Efficiency and Scaling

Accuracy increases rapidly with the number of references ($K$), saturating by $K\approx5$. This highlights the efficiency of boundary-based supervision in extracting structural cues from sparse, weakly-labeled data.

Figure 7: Accuracy curve vs. reference count $\mathrm{AP}_{75}$0 (measured by AP$\mathrm{AP}_{75}$1), showing rapid saturation and competitive performance throughout.

Training Efficiency

The offline phase (feature extraction, SDF head training, and threshold tuning) scales modestly with $\mathrm{AP}_{75}$2—annotation and training overhead is consistently lower compared to end-to-end finetuning approaches.

Figure 8: Offline reference-phase time as a function of $\mathrm{AP}_{75}$3, maintaining practical deployment suitability for real-world scenario adaptation.

Practical and Theoretical Implications

BbM demonstrates that boundary-focused, mask-conditioned SDF learning provides a robust and modular alternative to standard appearance-centric FSIS for industrial and texture-challenged domains. The capacity for rapid, reference-driven redefinition of instance targets is advantageous for industrial automation, reconfigurable manufacturing, and task-specific inspection, where the meaning of “instance” dynamically shifts. The method delivers annotation and retraining cost savings and enables efficient adaptation in operational environments with high variation and weak supervision.

Limitations remain, primarily in background complexity, extreme domain shift, and cluttered real-factory scenarios; these are prospective areas for future robustification and generalization. The approach’s modularity opens avenues for integration with uncertainty-aware reference selection, temporal or multi-view fusion, and further cross-domain adaptation leveraging boundary-centric priors.

Conclusion

Boundary-by-Mask establishes that conditioning on user-provided reference masks, combined with SDF-based boundary supervision over foundation-model features, is an effective strategy for few-shot, application-dependent instance segmentation in texture-poor scenarios. The modular architecture facilitates rapid redefinition, requiring only lightweight adaptation, and offers statistical and practical advantages over prevailing state-of-the-art FSIS pipelines. Further expansion to complex, dynamic industrial settings and exploration of uncertainty quantification are promising directions for subsequent work.