ARTEMIS: Agent-guided Reliability-aware Temporal Mask Evolution for Imperfectly Supervised Video Polyp Segmentation

Abstract: Imperfectly supervised video polyp segmentation (VPS) aims to learn dense, temporally consistent masks from inexpensive supervision, including weak annotations (points, scribbles) and semi-supervision with few densely labeled frames. This setting is clinically valuable but challenging due to weak contrast, ambiguous boundaries, motion blur, and specular highlights, compounded by sparse pixel-level guidance. While SAM2 can generate dense masks from sparse inputs, direct pseudo-labeling often yields geometry-degraded masks with boundary leakage, underutilizes temporal consistency, and ignores reliability. To address these issues, we propose ARTEMIS, a unified framework for imperfectly supervised VPS driven by agent-guided reliability-aware temporal mask evolution. ARTEMIS initializes coarse masks from available supervision: SAM2 converts points/scribbles, while dense labels serve as reliable anchors. A debate-and-judge vision-language agent selects reliable temporal anchors under weak supervision, which are propagated bidirectionally with SAM2 to refine unreliable or unlabeled frames. Finally, ARTEMIS trains the segmenter using temporal reliability-aware robust learning, incorporating reliability-guided reference selection, a Reference Prototype Transport Module, and reliability-aware robust loss. These components assess mask reliability, evolve anchors over time, transport target identity across frames, and down-weight noisy supervision instead of discarding difficult samples. Experiments on SUN-SEG and CVC-ClinicDB-612 under scribble, point, and limited-label settings demonstrate that ARTEMIS achieves state-of-the-art performance. Code will be released at https://github.com/wangtong627/ARTEMIS.

• The paper proposes a two-stage pipeline that integrates agent-guided anchor selection with bidirectional mask evolution to enhance segmentation accuracy under weak supervision.
• The paper demonstrates significant improvements in Dice and MAE metrics on SUN-SEG and CVC-ClinicDB-612, outperforming existing state-of-the-art methods.
• The paper introduces a reliability-aware robust loss and a reference prototype transport module (RPTM) to mitigate temporal drift and handle noisy pseudo-labels.

ARTEMIS: Agent-Guided Reliability-Aware Temporal Mask Evolution for Imperfectly Supervised Video Polyp Segmentation

Motivation and Background

Video Polyp Segmentation (VPS) in colonoscopy is critical for computer-aided diagnosis, yet is challenged by weak polyp-background contrast, ambiguous boundaries, motion blur, scale variability, and partial occlusion, as illustrated in typical clinical contexts.

Figure 1: Challenges in VPS. (a) Low contrast between polyp and background; (b) motion blur; (c) scale variations; (d) partial occlusion.

Annotation scarcity further complicates practical deployment: dense frame-level masks are labor-intensive, motivating weak (points, scribbles) and semi-supervised regimes (few dense masks per video). Existing pipelines treat weak/semi-supervised settings separately, do not leverage cross-frame temporal consistency, and lack robust pseudo-label filtering—foundational masks from SAM/SAM2 are unreliable under challenging imaging artifacts.

ARTEMIS Framework Overview

ARTEMIS establishes a unified paradigm for imperfectly supervised VPS by integrating both weak and semi-supervised signals. The key strategy is a two-stage complete-then-learn pipeline: Stage 1 generates coarse masks from supervision and selects reliable anchors via a vision-language agent; Stage 2 employs bidirectional mask propagation and reliability-aware robust learning, including Reference Prototype Transport Module (RPTM) and sophisticated loss formulations.

Figure 2: ARTEMIS paradigm: evolving reliable anchors, reliability-guided reference selection, RPTM, and robust loss across mixed supervision regimes.

Dense Prompting and Temporal Propagation

Dense mask prompts, even when noisy, provide superior localization priors compared to sparse points for low-contrast polyps—a claim supported by strong numerical gains in Dice and MAE. Injecting dense masks into difficult frames improves subsequent frames through SAM2 temporal propagation, a property exploited via anchor-based bidirectional mask evolution.

Figure 3: Dense noisy mask prompts (green) outperform sparse point prompts (yellow) in polyp localization.

Figure 4: Injected dense mask prompt propagates to improve masks in following frames.

Stage 1: Agent-Guided Bidirectional Mask Evolution

ARTEMIS uses a vision-language agent (multi-role debate-and-judge, instantiated with Qwen2.5-VL-7B) to assess anchor reliability. Selection is based on agent scores with T-NMS filtering and fallback logic for robust coverage. Reliable anchors are propagated both forward and backward in time, densifying weak/sparse annotations and mitigating temporal drift.

Figure 5: Agent-guided anchor selection and bidirectional propagation with SAM2 for evolved pseudo mask generation.

Dense pseudo masks thus cover unlabeled/unreliable frames, increasing annotation density without manual overhead. For semi-supervised VPS, all labeled frames are adopted as anchors, bypassing agent selection.

Stage 2: Temporal Reliability-Aware Robust Learning

Stage 2 selectively samples reference frames based on frame-level quality scores (foreground confidence, mask agreement, area validity). RPTM is then employed: reference tokens are extracted, transported to target frames via attention-based pooling, temporally evolved using bidirectional Mamba, and injected using relation gates.

Reliability-aware robust loss down-weights unreliable pseudo supervision at both pixel and frame levels (probability-based reliability, bidirectional consistency, agent-judged frame weights), improving final segmentation robustness. Multi-scale deep supervision is performed, balancing BCE and weighted Dice losses.

Figure 6: Reliability-guided reference selection, RPTM with bidirectional identity evolution, and reliability-aware robust loss for robust segmenter training.

Experimental Results and Analysis

ARTEMIS achieves consistent top performance under scribble/point and limited-label settings on SUN-SEG and CVC-ClinicDB-612, outperforming all baselines in Dice, IoU, $F_\beta^w$, and MAE. Under scribble supervision, ARTEMIS increases Dice by over 4% absolute (Easy-Seen) and reduces MAE by more than 50% relative (Easy-Seen). In semi-supervised settings with only 1/16 labeled frames, ARTEMIS yields 5–6% absolute Dice improvements over MCF, ST-SAM, KnowSAM, SEE, and competitive teacher-student approaches.

Qualitative results demonstrate superior boundary precision, continuity, and resistance to temporal drift in challenging blurry and small polyp cases compared to competing weak/semi-supervised methods.

Figure 7: ARTEMIS retains accurate boundaries and continuous predictions under scribble supervision and small polyp targets.

Dice-threshold and $F_\beta$-threshold curves confirm performance superiority across threshold selections.

Figure 8: Dice-threshold curves for SUN-SEG-Hard-Seen under four settings, showing ARTEMIS dominance over baselines.

Ablation studies validate significant contributions of each component: bidirectional mask evolution yields largest pseudo-label quality gains; agent-guided anchor selection outperforms policy-based and naïve filtering; reliability-guided reference selection and robust loss further enhance segmentation accuracy. RPTM ablation and token-number analyses establish $N=4$ as optimal for reference identity transport, with relation-gated injection and bidirectional Mamba necessary for drift-resistance.

Figure 9: RPTM boosts polyp-region activation and suppresses distractors in ambiguous background.

Despite strong quantitative and qualitative results, ARTEMIS still encounters failure cases under severe distractors, rapid motion, and uneven illumination, suggesting avenues for future temporal refinement and boundary modeling.

Figure 10: Example failure cases: strong distractors and motion-induced boundary errors.

Implications and Future Directions

ARTEMIS validates the clinical viability of imperfect supervision for VPS, demonstrating that low-cost sparse annotations, when combined with agent-guided anchor selection and bidirectional mask evolution, deliver temporally consistent and accurate segmentations. Theoretical contributions include robust pseudo-label completion, reliability-aware loss, and reference-identity transport via RPTM.

Future development should address long-video scaling, memory-efficient reference management, uncertainty-driven anchor selection, and adaptive human-in-the-loop strategies. Strong implications exist for broader foundation-model integration in medical imaging and other temporally structured domains.

Conclusion

ARTEMIS sets a new benchmark for imperfectly supervised video polyp segmentation. By unifying weak/semi-supervised settings and exploiting agent-driven anchor selection, bidirectional mask evolution, and reliability-aware learning, ARTEMIS converts sparse supervision into dense, temporally consistent masks and suppresses noisy supervision. Empirically, ARTEMIS achieves state-of-the-art accuracy and robustness under challenging clinical artifacts and annotation scarcity, laying groundwork for scalable, practical VPS deployment.

(2606.20161)