Weakly Supervised Incremental Segmentation via Semantic Anchors and Spatial Arbitration

Abstract: Weakly Incremental Learning for Semantic Segmentation (WILSS) suffers from the continuous introduction of noisy supervision, which progressively corrupts class-level representations, leading to severe feature drift and semantic corruption, thereby causing newly learned classes to overwrite old ones. To address these issues, we propose a drift-resilient WILSS approach, named SASA, designed to stabilize semantic learning via Semantic Anchors and Spatial Arbitration. Specifically, at the representation level, we introduce semantic anchors of learnable tokens as rigid class-level references to preserve long-term semantic identity. Complementary to this, an elastic residual adaptation facilitates controlled, instance-specific refinement, ensuring a stable yet flexible learning trajectory. At the supervision level, we develop a Spatial Label Arbitration mechanism that performs geometry-aware decisions to directly filter unreliable signals and enforce a strict "one object, one class" constraint. By synergistically stabilizing representations and improving supervision reliability, SASA effectively mitigates feature drift under weak supervision. Extensive experiments on standard benchmarks demonstrate that our approach consistently outperforms existing state-of-the-art methods, particularly in challenging multi-step incremental settings. The code is available at https://github.com/ZhonggaiWang/SASA.

• The paper introduces SASA, which integrates Drift-Resilient Semantic Anchors (DSA) and Spatial Label Arbitration (SLA) to stabilize class representations during incremental segmentation.
• It leverages cosine similarity and auxiliary losses for token alignment, while using spatial denoising with object masks to mitigate noisy pseudo-label supervision.
• Experimental results on VOC and COCO benchmarks show significant mIoU improvements, confirming SASA's ability to counter feature drift and catastrophic forgetting.

Weakly Supervised Incremental Segmentation via Semantic Anchors and Spatial Arbitration

Introduction and Problem Analysis

Incremental semantic segmentation, particularly in weakly supervised settings, introduces a persistent challenge where noisy pseudo-labels accumulate across sequential learning steps. This process results in feature drift: the semantic representations of classes degrade as novel categories overwrite and displace older ones. Weak supervision based on image-level labels, compounded by the use of class activation maps (CAMs) and the predictions from frozen models for pseudo-label generation, injects contradictory supervision and inherent noise, especially in visually ambiguous regions. This problem is exacerbated in long incremental learning schedules, leading to progressive semantic corruption and catastrophic forgetting.

Figure 1: Illustration of Feature Drift (left) and the stabilization effect of SASA via Semantic Anchors and Spatial Label Arbitration (right).

SASA Framework: Architecture and Core Innovations

The proposed SASA framework directly addresses noise-induced drift in WILSS by introducing two synergistic modules: Drift-Resilient Semantic Anchors (DSA) and Spatial Label Arbitration (SLA).

Drift-Resilient Semantic Anchors (DSA)

Semantic Anchors are implemented as class-wise learnable tokens, serving as persistent, rigid references in the embedding space. Unlike ad hoc feature prototypes prone to drift, these anchors enforce long-term class identity by optimizing for inter-class separability and semantic stability. Elastic Residual Tokens (ERTs) are introduced to capture instance-specific deviations, dynamically computed via cross-attention between anchors and pixel feature maps. The final tokens are regularized by an $\ell_2$ penalty to restrict adaptation to small, controlled steps.

Supervision is performed by computing cosine similarity between pixel features and final tokens, enforcing class-token alignment via cross-entropy and auxiliary losses, including a separation constraint and anchor-based distillation. This architecture maintains near-orthogonal class representations, balancing plasticity for new concepts with retention for older classes.

Spatial Label Arbitration (SLA)

SLA explicitly denoises pseudo-labels at the spatial level. Class-agnostic object masks, e.g., from SAM or Maskformer, are used to arbitrate between signals from old and new models, enforcing the "One Object, One Class" constraint. Pixel-wise reliability within each mask is computed via a Gaussian function centered on the mask centroid. Activation density of novel-class signals is assessed, and majority voting determines the mask-level label, propagating spatially consistent supervision—reducing semantic fragmentation and label contradictions.

Experimental Validation

Extensive evaluation on Pascal VOC 2012 and MS COCO benchmarks demonstrate the efficacy of SASA under various incremental settings, including rigorous multi-step and cross-dataset scenarios. SASA consistently surpasses state-of-the-art weakly supervised approaches, including token-based and distillation-based paradigms, with robust mIoU gains for both old and new class sets.

Segmentation Performance

SASA achieves significant improvements, notably:

• VOC 10-10: >5% mIoU boost across both old and new classes
• VOC 10-2 (6-step): +22.4% (old) and +16.1% (new) mIoU compared to strong baselines
• COCO-to-VOC: +7 points above previous SOTA, approaching fully-supervised upper bounds

Component Analysis and Ablations

Ablation studies reveal:

• DSA alone yields a +4.7% mIoU gain by stabilizing semantic cores.
• SLA alone provides +3.5% mIoU by suppressing spatial noise.
• Combined, a +7.6% improvement is observed, confirming their complementary effects.

Loss sensitivity analysis further establishes that default weighting achieves optimal results, with the SLA threshold $\tau=0.6$ giving the best trade-off between noise suppression and label coverage.

Feature Space and Qualitative Results

t-SNE visualization demonstrates that SASA prevents catastrophic forgetting and feature drift; old-class clusters remain compact and separable even after multiple incremental steps.

Figure 2: t-SNE visualization—SASA preserves old-class cluster integrity and clear boundaries in cumulative class embeddings after incremental learning.

Qualitative segmentation results show SASA producing sharp boundaries and semantically coherent masks, resolving common confusion and fragmentation issues in baseline architectures.

Figure 3: SASA yields more accurate object boundaries and mitigates semantic drift compared to baseline segmentation models.

Implications and Future Directions

SASA’s approach signifies a paradigm shift in handling weakly supervised incremental segmentation: stabilizing semantic representations at the token level and spatially rectifying supervision at the mask level. The explicit control over class anchors and residual adaptation directly addresses compounding supervision noise—a limitation left implicit in prior work. Practically, this enables deployment of robust semantic segmentation models in scenarios with sparse or noisy labels, such as medical imaging and autonomous perception, where annotation scarcity is a key bottleneck.

Theoretically, the design establishes a framework for further exploration of disentangled representation stability and geometry-aware supervision in lifelong learning contexts. Future developments could investigate generalized anchor regularization, dynamic arbitration with uncertainty quantification, and integrating foundation models for richer pseudo-label generation.

Conclusion

The SASA framework introduces drift-resilient semantic anchors and spatial label arbitration, robustly mitigating feature drift in weakly supervised incremental segmentation. Extensive experimental results confirm its superiority over prior state-of-the-art methods, particularly in challenging multi-step regimes. SASA’s principled architecture lays the groundwork for stable, scalable class-incremental segmentation under realistic supervision constraints, with broad implications for continual learning and weak supervision paradigms.

(2606.04060)