Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation

Abstract: Transforming foundation segmentation models from human-prompted tools into auto-promptable annotators is critical for scalable medical data annotation. Current methods commonly depend on external feature matchers or auxiliary networks to automate geometric prompting, but introducing architectural overhead and limiting performance scalability. Although SAM3 natively supports concept segmentation via reusable text prompts, its direct use in medical imaging is hindered by a lack of fine-grained clinical knowledge and the ambiguity of human-written descriptions. In this work, we propose Mask to Concept (M2C), an efficient framework that adapts SAM3 for medical few-shot annotation without external modules, parameter retraining, or manual text engineering. Using only a few labeled images, M2C enables SAM3 to automatically search for transferable visual concepts entirely within its frozen architecture: it initializes a learnable concept embedding, uses it to prompt segmentation, and updates the embedding by gradients of minimizing the concept segmentation error. We further introduce a Hybrid Uncertainty Estimation (HUE) module that calculates the prediction entropy and maps concept predictions back to the box prompts, measuring concept-geometry prompting inconsistency. Highly uncertain samples are flagged actively for human correction, and the corrected masks are then fed back to M2C to continuously search for more precise concept embeddings, forming a self-enhancing annotation loop with minimal expert effort. Experiments on medical segmentation benchmarks show that our method achieves SOTA few-shot segmentation performance and outstanding annotation efficiency, offering a practical and efficient pathway toward scalable medical image labeling. Codes are at https://github.com/Huster-Hq/M2C.

• The paper introduces M2C, a framework that embeds a learnable concept vector within SAM3 to optimize segmentation using few-shot annotations.
• It achieves state-of-the-art results on Kvasir-SEG and ISIC-2017 benchmarks, improving Dice scores by up to 11.8% over previous methods.
• The study leverages a hybrid uncertainty estimation method to efficiently direct human corrections, significantly reducing annotation efforts.

Mask to Concept (M2C): Efficient Few-Shot Medical Annotation with Auto-Promptable SAM3

Introduction

Semantic segmentation for medical imaging is critically limited by the expense and scarcity of high-quality, pixel-level annotations. Foundation models like Segment Anything Model (SAM) and its derivatives such as MedSAM and SAM3 have enabled prompt-driven segmentation. However, prior methods are hampered by the need for repetitive, manual geometric prompting on a per-image basis or depend on external networks and feature matchers that introduce significant architectural overhead and suboptimal scalability. Further, SAM3’s text prompt-based concept segmentation does not sufficiently capture the fine-grained clinical semantics required for annotation tasks across diverse, specialized medical imaging datasets.

Methodology

Mask to Concept (M2C): Test-Time Concept Embedding Search

The core technical advance is the M2C framework, which introduces a lightweight, optimizable concept embedding within SAM3’s architecture, obviating the need for retraining or auxiliary modules. Starting from a few human-annotated masks, M2C initializes a learnable vector $E$ that augments the standard textual concept embeddings output by SAM3’s text encoder. For every batch of support images, $E$ is optimized (with the SAM3 backbone frozen) using gradients from segmentation loss (combining IoU and cross-entropy terms), enabling the discovery of dataset-specific visual concepts directly at test time. This mechanism bypasses manual text prompt engineering and adapts to arbitrary medical domains. Unlike feature-matching or auxiliary network strategies, M2C maintains tight integration with SAM3’s native inference pipeline, enabling scalable cross-instance annotation within a compact search space.

Hybrid Uncertainty Estimation (HUE): Sample Selection for Human-in-the-Loop Annotation

Annotation efficiency and quality are further enhanced via HUE, which fuses two complementary, differentiable uncertainty estimators:

• Prediction Entropy ($U_\text{ENT}$): Local pixelwise entropy highlights ambiguous regions in the predicted segmentation.
• Concept-Geometry Prompting Inconsistency ($U_\text{CGPI}$): Measures divergence between concept-driven and geometry-driven segmentation masks (the latter generated by feeding the predicted mask’s bounding box as a prompt). These scores are combined into a single ranking metric, used to route high-uncertainty samples for human correction and allow confident (low-uncertainty) cases to be filtered out or finalized, enabling active sample selection and early exit strategies.

Results

State-of-the-Art Few-Shot Segmentation

On two challenging, zero-shot medical segmentation benchmarks—Kvasir-SEG (polyp) and ISIC-2017 (skin lesion)—M2C surpasses both training-based and training-free SOTA methods. In the one-shot scenario, M2C achieves Dice improvements of 4.2% and 11.8% over the second-best method (ProtoSAM) on Kvasir-SEG and ISIC-2017, respectively. This establishes the superiority of embedding-level domain adaptation over explicit feature matching, and demonstrates robust generalization with minimal annotation.

Efficiency and Annotation Cost Reduction

M2C, integrated with HUE's uncertainty-based selection, delivers improved annotation efficiency compared to methods such as Multiverseg and SPFS-SAM. The active learning loop iteratively refines the concept embedding with each cycle of human correction, driving down the number of samples requiring manual intervention for target segmentation quality. Ablation studies confirm that the hybrid uncertainty metric (combining entropy and geometric inconsistency) is necessary to minimize annotation overhead, with either sub-metric alone leading to higher human-in-the-loop costs.

Implications and Future Directions

This work signifies an inflection point in leveraging foundation models for practical, scalable medical annotation. By extracting medical concepts directly from annotated masks and optimizing over prompt embeddings, M2C obviates reliance on brittle textual semantics and external modules. This design can be extended beyond medical imaging to any domain where fine-grained, zero-shot concept adaptation is required. The active learning pipeline, combining concept discovery with uncertainty quantification, paves the way for efficient human-AI collaboration and minimal expert effort in dataset curation.

Future research may explore multi-concept prompt embedding, unsupervised discovery of subtypes within the learned embedding space, and extension to multi-modal tasks. Investigation into more nuanced uncertainty calibration and the integration of self-supervised refinement steps may further reduce human annotation cost and enhance model robustness in out-of-distribution settings.

Conclusion

Mask to Concept (M2C) unlocks the full semantic capacity of SAM3 via efficient test-time search for discriminative concept embeddings, establishing new benchmarks in few-shot medical annotation accuracy and human label efficiency. The proposed human-in-the-loop system, leveraging hybrid uncertainty routing and continual embedding refinement, presents a powerful and practical methodology for scalable dataset construction—readily adaptable to other foundation models and application domains.

---

Reference: "Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation" (2606.26711)