Annotation-Efficient Segmentation Approach

• The paper demonstrates that leveraging weak supervision, active learning, and transfer learning can drastically reduce annotation costs while maintaining high segmentation accuracy.
• It employs interactive tools and surrogate signals like LiDAR and foundation models to convert minimal labels into effective training data with near full-supervision performance.
• Empirical results reveal that methods such as active sampling and semi-supervised learning achieve cost-effective, high-accuracy segmentation in diverse domains.

An annotation-efficient segmentation approach seeks to maximize segmentation accuracy while minimizing the required amount or quality of manual annotations. This paradigm is especially critical in domains where dense, pixel-wise labels are costly, domain expertise is limited, or timely data curation is necessary. Annotation-efficient strategies span methodologies such as weak supervision (e.g., boxes, scribbles, points), semi-supervised and active learning, transfer/few-shot approaches, and the use of surrogate signals (e.g., physical markers, LiDAR, foundation models). These methods leverage advances in learning theory, model architectures, and data annotation processes to dramatically reduce labeling overhead without substantial sacrifice in task performance.

1. Definition and General Principles

Annotation-efficient segmentation comprises algorithmic and workflow innovations that enable the training of segmentation models with orders-of-magnitude fewer, weaker, or noisier annotations compared to classical fully-supervised protocols. The central motivation is the diminishing returns and prohibitive costs associated with hand-annotated, pixel-accurate masks in many application settings (medical imaging, remote sensing, industrial inspection, document analysis, and beyond).

Overarching principles of annotation efficiency include:

• Minimizing annotation complexity: Replacing polygon mask or dense pixel labeling with bounding boxes, rough contours, superpixels, points, or even image-level tags, often with the help of interactive or machine-assisted tools (Zhang et al., 2023, Acuna et al., 2018, Ge et al., 2024).
• Maximizing information gain per annotation: Curating which data points or regions to annotate through active learning, uncertainty sampling, or co-reset selection (Huang et al., 2024, Ge et al., 2024, Zhang et al., 2023).
• Leveraging unlabeled and auxiliary data: Incorporating self-supervision, domain adaptation, or multi-modal (e.g., LiDAR, calibration card) information to reduce the demand for direct annotations (Sharafutdinov et al., 2023, Zhang et al., 2023, Wang et al., 2020).
• Exploiting weak or proxy signals: Utilizing physical surrogates (calibration cards (Nicolaou et al., 2023)), large pretrained models (SAM, foundation models (Ward et al., 2024, Chen et al., 23 Sep 2025)), or prior datasets for transfer/few-shot learning (Ming et al., 2024).

2. Core Methodological Taxonomy

Annotation-efficient segmentation implementations are diverse. Notable classes include:

1. Weakly-supervised and noisy-label strategies: Replace exhaustive masks with weaker forms, such as bounding boxes, polygons, scribbles, or points, often combined with algorithmic conversion to trainable masks (e.g., GrabCut, CRF, SAM-prompted masks). Empirical benchmarking consistently shows that, under realistic annotation time constraints, these approaches match or exceed the cost-effectiveness of full supervision (Zhang et al., 2023). Notable quantitative insight is that a coarse contour or SAM-prompted mask achieves near top performance at a fraction (<30%) of the annotation budget required for dense masks.

2. Interactive and human-in-the-loop annotation: Interactive tools—often incorporating deep network prediction and iterative correction—enable annotators to generate high-quality masks with minimal intervention. Polygon-RNN++ reduces the median annotation interaction to ~5 clicks per object (out of 400–600 for manual polygon drawing), with mean IoU improvements and substantial generalization to new domains (Acuna et al., 2018). Entity–Superpixel Annotation (ESA) further reduces the cost by prioritizing annotator effort towards superpixels or mask proposals with the highest entropy, translating to a 98% reduction in click cost (Ge et al., 2024).

3. Active learning and selective annotation: These frameworks direct annotation to the most informative or uncertain samples. For instance, uncertainty-weighted clustering selects polyp images for manual labeling, thus minimizing redundancy and targeting feature-space underexplored cases, yielding state-of-the-art segmentation with the smallest label budget (Huang et al., 2024). K-center core-set selection in OCT targets geometric cover in latent feature space, again achieving U-Net performance with ~10% annotation (Zhang et al., 2023).

4. Surrogate- and multimodal-labeling: Physical surrogates (calibration cards, LiDAR) or surrogate sources (public datasets) supplement or replace direct masks: for example, "Efficient Annotation of Medieval Charters" uses bounding-box detection for segmentation and exploits calibration card detection and regression to estimate physical metrics (Nicolaou et al., 2023); similarly, LiDAR-projected annotations supervise semantic segmentation with sparse masked loss, achieving 95–96% mIoU with only LiDAR and no dense masks (Sharafutdinov et al., 2023).

5. Foundation and promptable model leveraging: The Segment Anything Model (SAM) and its derivatives enable zero-shot or prompt-based mask generation. Hybrid frameworks such as SAM-Mix synergistically couple classifier- and prompt-driven segmentation, attaining >5% Dice improvement in extreme few-shot CT scenarios with as little as 0.04% explicit ground-truth (Ward et al., 2024). Prompt-DAS adapts this for electron microscopy, using prompt-guided contrastive learning and sparse/zero-shot points for cross-domain annotation efficiency (Chen et al., 23 Sep 2025).

6. Semi-supervised, transfer, and meta-learning approaches: These exploit unlabeled or weakly labeled data by enforcing consistency, proxy supervision, pseudo-labeling, or multi-domain prototype sharing. The AIDE framework co-trains dual networks with cross-model correction, enabling near-oracle performance on medical image segmentation with 10% of the annotations (Wang et al., 2020). Generalized few-shot instance segmentation (SGFSIS) uses prototype fusion and marker-based structural guidance for multi-class nucleus segmentation with <5% dense annotation (Ming et al., 2024).

3. Key Algorithms and Pipelines

A selection of archetypal workflows and algorithmic modules includes:

Method/Class	Core Methodology	Annotation Reduction
Polygon-RNN++ (Acuna et al., 2018)	RL-trained polygon prediction + GGNN	~95% fewer clicks
ESA (Ge et al., 2024)	Mask proposals + superpixels + entropy	98% click reduction over trad.
Vessel-CAPTCHA (Dang et al., 2021)	Patch-wise tags + K-means pseudo-labels	~77% time reduction
LiDAR-masked loss (Sharafutdinov et al., 2023)	Sparse LiDAR projection + masked loss	Near-zero manual masks
AIDE (Wang et al., 2020)	Twin nets, global/local label correction	Full perf. at 10% annotation
SAM-Mix (Ward et al., 2024)	GradCAM prompts + LoRA adapter to SAM	0.04% labels, +5/25% Dice
Prompt-DAS (Chen et al., 23 Sep 2025)	Point prompt multitask w/ detection	15% points, SOTA EM Dice
K-center Greedy (Zhang et al., 2023)	k-center subset for fine-tuning	~10% labels, 3–4× faster

The annotation mechanism and pipeline dictate which aspects (clicks, pixel masks, full images, multimodal tags) are needed and optimally traded.

4. Quantitative Impact and Empirical Results

Empirical benchmarks across domains demonstrate substantial annotation-efficiency gains, often quantified as:

• Reduction in click/time cost: ESA cuts clicks to ~40 per image vs. 5000–9000 in region/pixel annotation, yet mIoU increases by 1–2% (Ge et al., 2024).
• Fraction of pixels needed: In SGFSIS, <5% dense annotation suffices to close the performance gap to full supervision in nucleus segmentation (Ming et al., 2024).
• Dice/IoU preservation: Vessel-CAPTCHA achieves 79.3% DSC on vessels using patch tags, exceeding full-supervision U-Net (77.7%) with 77% less annotation time (Dang et al., 2021). AIDE matches full supervision at 10% label cost across multiple clinical segmentation tasks (Wang et al., 2020).
• Cross-domain and cross-modality robustness: Prompt-DAS achieves 93% Dice in domain-adaptive EM segmentation with only 15% center-point annotation (Chen et al., 23 Sep 2025); SAM-Mix achieves +25% Dice generalization under cross-cohort shifts with minimal explicit mask labels (Ward et al., 2024).
• Metric saturation under annotation budgets: In large comparative studies (Zhang et al., 2023), polygonal or rough boundaries (IoU ≈ 0.84–0.90) consistently achieve near-maximal mIoU at <30% of the annotation time.

Notably, across studies, systematic ablations confirm optimality of annotation-efficient regimes within constrained budgets, and that full fine-pixel annotation is only superior at extremely high time costs.

5. Toolkits, Open Frameworks, and Best Practices

Reusable frameworks such as PyMIC (Wang et al., 2022) and FRAT (for document boxes (Nicolaou et al., 2023)) provide modular recipes to implement semi-/weak-/noise-robust segmentation. Key toolkit features include dataset samplers mixing labeled/unlabeled images, flexible loss combinations (e.g. partial-CE, noise-robust Dice, regularization), and batch aggregation engines supporting common co-training, mean-teacher, and pseudo-labeling paradigms.

Best practices that consistently emerge include:

• Initial benchmarking against a small, fully labeled baseline before transitioning to weak/semi/active regimes.
• Adopting region proposals and superpixels for human annotation interaction to minimize redundant effort.
• Carefully tuning unsupervised loss ramp-up schedules and network confidence thresholds to avoid overfitting to spurious signals.
• Employing postprocessing (connected components, test-time augmentation) to improve mask quality.
• Integrating self- and cross-model label correction to mitigate pseudo-label noise.

6. Domain-Specific Adaptations and Physical Surrogates

Application-tailored annotation-efficient approaches show marked utility in specialized domains:

• Medieval documents: Detection as segmentation (rectangles) with calibration card detection for scale estimation enables charter segmentation with an order-of-magnitude less annotation time and comparable downstream OCR utility (Nicolaou et al., 2023).
• Autonomous driving: Sparse LiDAR provides nearly all supervision required for road segmentation, with masked loss adapting to the variable coverage of LiDAR (especially in upper image regions) (Sharafutdinov et al., 2023). Mixing sparse/dense masks offers a tunable trade-off between effort and fine boundary accuracy.
• Histopathology and microscopy: Promptable frameworks like Prompt-DAS handle dense micro-instance segmentation where full annotation is impractical; labeled points and auxiliary contrastive objectives regularize feature space structure, enabling effective instance recovery with minimal points (Chen et al., 23 Sep 2025, Ming et al., 2024).
• COVID-19 and other emergent domains: Dual-encoder collaborative learning or usage of shared knowledge from related disease or anatomy datasets addresses acute data scarcity (Zhang et al., 2020).
• Physical measurement tasks: Box detection/classification provides sufficient segmentation granularity and facilitates physical measurement extraction (calibration length estimation) absent pixel masks (Nicolaou et al., 2023).

7. Limitations, Open Challenges, and Future Directions

While annotation-efficient segmentation has achieved extensive empirical success, key limitations and continuing challenges include:

• Reliance on weak annotations: Some object classes or domains (e.g., highly irregular boundaries in infectious lesions) may not be suitably captured by bounding boxes or rough polygons, necessitating hybrid or custom workflows (Zhang et al., 2020).
• Quality of surrogate labels: The transferability of pre-existing masks or physical markers (e.g., LiDAR, calibration cards) is subject to acquisition protocol differences and residual noise, often requiring post-hoc correction, self-training, or noise-robust objectives (Sharafutdinov et al., 2023, Wang et al., 2020).
• Automated proposal and prompt quality: Over-segmentation or under-segmentation by class-agnostic proposals (e.g., in ESA) can limit utility, especially where objects are thin, sparse, or poorly defined (Ge et al., 2024).
• Scalability to 3D, multi-modal, or highly imbalanced scenarios: Although many frameworks operate efficiently on 2D data, annotation and domain adaptation in 3D imaging, or across significant intensity/appearance shifts, may introduce new annotation labor or information-theoretic bottlenecks (Zhang et al., 2023, Li et al., 2020).
• Budget-aware selection and stopping criteria: Diminishing returns are observed beyond modest annotation budgets, suggesting the need for automated or adaptive stopping rules (Ge et al., 2024).
• Mixing of label styles or annotation sources: Real-world annotation projects may combine weak, dense, and surrogate labels. Formalized strategies to harmonize label types and annotations from annotators of variable expertise remain open.

Future research may focus on universal promptable models, active learning with diversity constraints, robust real-time feedback to annotators, broader domain coverage (multi-modal, multi-institutional), and generalization to novel diseases or tasks with no direct annotation at all.

---

In summary, annotation-efficient segmentation unifies weak supervision, active learning, transfer/meta-learning, foundation model prompting, and domain adaptation to achieve high-accuracy dense prediction at a small fraction of manual labeling cost (Zhang et al., 2023, Ward et al., 2024, Ge et al., 2024, Nicolaou et al., 2023, Wang et al., 2020). This paradigm is now foundational for rapid dataset development across scientific, clinical, remote sensing, and industrial domains, with broad empirical validation and ongoing methodological innovation.