Semantic and Spatial Rectification (SSR)

• SSR is a unified framework that combines semantic and spatial rectification to enhance CLIP-based weakly supervised segmentation.
• It employs Cross-Modal Prototype Alignment to align image and text features using prototype-based contrastive learning.
• Superpixel-Guided Correction refines class activation maps by integrating local superpixel priors and multi-head self-attention, boosting segmentation mIoU.

Semantic and Spatial Rectification (SSR) is a methodology specifically designed to address the dual issues of semantic misalignment and spatial noise in CLIP-based weakly supervised semantic segmentation (WSSS) with only image-level annotations. SSR introduces a dual-pronged rectification framework: Cross-Modal Prototype Alignment (CMPA) for semantic feature alignment and Superpixel-Guided Correction (SGC) for spatial noise suppression. By integrating prototype-based contrastive learning with local superpixel-derived priors during affinity-based mask refinement, SSR achieves state-of-the-art segmentation performance in a unified, single-stage, end-to-end trainable architecture (Bi et al., 1 Dec 2025).

1. Motivation and Problem Setting

Weakly supervised semantic segmentation (WSSS) utilizing CLIP models faces two core challenges: (a) modality gap-induced misactivation where visual features erroneously activate non-target foreground regions (semantic confusion), and (b) background over-activation arising from noisy affinity propagation (spatial confusion). SSR is devised to mitigate these effects by introducing dedicated semantic and spatial rectification modules: CMPA targets cross-modal feature misalignment, while SGC suppresses spurious non-target region propagation during the affinity refinement stage.

2. Framework and Pipeline Structure

SSR accepts an input image $I$ and corresponding image-level text prompt $T$ (covering $C$ foreground classes and $M$ background classes). The pipeline is implemented as follows:

• Feature extraction: CLIP ViT-B/16 provides the initial visual embedding $v \in \mathbb{R}^{d_1}$ and CLIP text encoder yields text features $t \in \mathbb{R}^{c_f \times d_1}$.
• Semantic rectification (CMPA): Projected features $v'$ and $t'$ are generated and leveraged for category-wise multimodal prototypes and prototype-level contrastive loss.
• Initial CAM generation: GradCAM is applied on CMPA-refined $v'$ to produce class activation maps (CAMs).
• Spatial rectification (SGC): Superpixel-derived spatial priors and a fusion of CLIP and DINO ViT-S/16 multi-head self-attention (MHSA) features yield a spatially-constrained affinity matrix for CAM refinement.
• Mask generation and training: Refined pseudo-masks guide end-to-end segmentation model training via cross-entropy. SSR updates all modules with a combined prototype contrastive and segmentation loss.

Notably, the vision-language and spatial encoders are mostly frozen; only the projection heads (ISA/TSA) are optimized for downstream learning. The prototype clusters are updated at fixed intervals (every 5,000 iterations).

3. Semantic Rectification: Cross-Modal Prototype Alignment (CMPA)

CMPA addresses semantic leakage between foreground categories by refining and aligning image and text features through the following process:

• Feature projection: Image features $v$ are passed through ISA, and text features $t$ through TSA, yielding $v'_i = \mathrm{ISA}(v_i)$ and $t'_i = \mathrm{TSA}(t_i)$, both projected to $\mathbb{R}^{d_2}$. ISA and TSA are independent MLPs (Linear–BN–ReLU).
• Prototype construction: For each present category $c$ in image $i$, CAM$_{ij}^c$ is computed via GradCAM, and the foreground-aware feature is obtained as $$f_{\text{image}} = \mathrm{MAP}(\text{CAM}^c \odot v'_i)$$. Textual category features $f_{\text{text}} = t'_i[c]$ are similarly indexed.
• Clustering: All $f_{\text{image}}$ and $f_{\text{text}}$ are clustered using K-means ($K=C$) to yield visual prototypes $P^I$ and textual prototypes $P^T$.
• Contrastive objective: Prototype contrastive loss $L_{\text{proto}}$ enforces alignment between image features and their matched text prototypes while maintaining inter-class separation and cross-modal consistency. For projected feature $v'_i$, scores over all prototypes are scaled by a learnable temperature $\tau_{\text{proto}}$:

$p_i = \frac{v'_i \cdot P^T}{\tau_{\text{proto}}}$

The contrastive loss is:

$$L_{\text{proto}} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(S^i_{\text{pos}})}{\exp(S^i_{\text{pos}}) + \sum_{j \ne \text{pos\_idx}(i)} \exp(S^i_{\text{neg}_j})}$$

This mechanism yields compact intra-class and separated inter-class clusters, as verified via t-SNE analysis.

4. Spatial Rectification: Superpixel-Guided Correction (SGC)

SGC is introduced to address over-activation in background areas due to global affinity propagation and the lack of strong spatial priors in global attention schemes:

• Superpixel region extraction: The SLIC algorithm segments image $I$ into $S$ superpixels, which are further clustered in color space via K-means into $K_{\mathrm{sp}}$ spatial regions.
• Target region identification: For each region $r$, the ratio $\text{ratio}_r$ of high-confidence CAM pixels is computed. If $\text{ratio}_r$ exceeds a threshold $\tau_{\text{region}}$, $r$ is flagged as a "target region".
• Spatial mask construction: A binary mask $\text{Mask} \in \{0,1\}^{L \times L}$ (where $L$ is the number of patch tokens) restricts affinity propagation to within target regions.
• Affinity matrix fusion: The affinity matrix $A \in \mathbb{R}^{L \times L}$ is built by concatenating (CLIP-MHSA, DINO-MHSA) features in a 0.4:0.6 ratio, followed by normalization:

$$A = \text{Normalize}(\text{ConCat}(\text{MHSA}_{\text{CLIP}}, \text{MHSA}_{\text{DINO}}))$$

• Affinity masking and CAM refinement: The masked affinity is $A^* = A \odot \text{Mask}$, and the CAM is refined as $$\text{CAM}_{\text{refine}}^c = A^* \otimes \text{CAM}^c$$.

SGC progressively improves CAM seed quality, as ablation studies demonstrate increasing mIoU from 63.3% (CMPA alone) to 78.7% with full SGC integration.

5. Training Protocol and Optimization

• The pipeline employs CLIP ViT-B/16 and DINO ViT-S/16 as fixed visual backbones (except for the trainable ISA/TSA projection heads).
• Projection heads contain 2–3 MLP layers with $$(\text{Linear} \rightarrow \text{BN} \rightarrow \text{ReLU})$$ structure, projecting from $d_1$ (backbone output) to $d_2=256$.
• AdamW optimizer, with learning rate $1 \times 10^{-5}$ and weight decay $2 \times 10^{-3}$, is used.
• Loss is a composite: $$L_{\text{SSR}} = L_{\text{proto}} + \gamma L_{\text{seg}}$$ with $\gamma=0.1$.
• CLI:DINO affinity fusion is set at 0.4:0.6.
• Prototypes are updated every 5,000 iterations. Batch sizes are 128 (VOC, 30k iters) and 256 (COCO, 80k iters).

6. Empirical Evaluation and Analysis

SSR achieves state-of-the-art single-stage WSSS performance:

• PASCAL VOC: mIoU of 79.5% (val), 79.6% (test), surpassing previous single-stage (and even certain multi-stage) approaches by 0.3% mIoU relative to ExCEL.
• MS COCO: 50.6% mIoU (val), outperforming ExCEL by 0.3%.
• CAM seed mIoU (VOC train): 78.7% for SSR, higher than ExCEL (78.0%) and VPL (77.8%).

Ablation results indicate significant incremental benefit from stepwise integration of SGC elements and cross-modal contrastive loss. Additional metrics show SSR improves precision, recall, and confusion ratios by approximately 3–4% over WeCLIP and MoRe.

Comparison to fully supervised DeepLabV2 indicates SSR attains 97.4% of ViT-B’s supervised performance on VOC val set. CMPA and SGC are verified to produce compact and well-separated feature clusters, and lightweight superpixel priors effectively guide spatial correction with low architectural overhead.

7. Limitations and Prospective Developments

SSR’s reliance on static superpixel configurations (e.g., SLIC parameters, activation thresholds) may necessitate dataset-specific tuning. Prototype alignment accuracy depends on initial CAM seed quality, with potential degradation in the presence of extreme class imbalance affecting K-means clustering. Future research directions include adaptive/learnable superpixel generation, prompt or generative augmentation of textual prototypes, and iterative prototype refinement leveraging pseudo-labels from CAM refinement steps.

SSR establishes a principled, modular approach to overcoming both semantic and spatial challenges in CLIP-based WSSS, achieving state-of-the-art performance without architectural complexity (Bi et al., 1 Dec 2025).