ModuSeg: Modular Weakly-Supervised Segmentation

• ModuSeg is a training-free weakly supervised semantic segmentation framework that decouples object discovery and semantic retrieval to enhance boundary precision.
• It employs a modular design with offline feature bank construction and online retrieval stages, enabling independent optimization of object proposals and label assignment.
• Empirical results demonstrate superior mIoU and runtime efficiency compared to traditional WSSS methods on benchmarks like VOC and COCO.

ModuSeg is a training-free weakly supervised semantic segmentation (WSSS) framework that operationalizes a clear decoupling between object discovery and semantic retrieval, leveraging foundation models and non-parametric retrieval. This approach directly addresses the entanglement of localization and classification that has traditionally led to suboptimal segmentation performance in WSSS, particularly boundary degradation and overreliance on discriminative object parts. The framework dispenses with both end-to-end and multi-stage retraining, instead utilizing precomputed feature banks and modular proposal-generation, resulting in state-of-the-art segmentation quality and significantly reduced computational demands (He et al., 8 Apr 2026).

1. Modular Architecture and Dataflow

ModuSeg consists of two strictly separated stages: offline feature bank construction and online inference via retrieval. This decoupling enables each component—object proposal, feature embedding, prototype formation, and semantic label assignment—to be optimized and analyzed independently.

Stage 1: Offline Feature Bank Construction

• Input: Training images $I_i$ with image-level labels $\mathcal{Y}_i$.
• A class-conditional mask generator $\mathcal{G}_{mask}$ (CorrCLIP) creates pseudo-masks conditioned on present classes (logits for $c\notin \mathcal{Y}_i$ are set to $-\infty$).
• Semantic Boundary Purification (SBP) applies $t$-step morphological erosion with kernel size $k$ ($t=20$, $k=3$ typically) to masks, reducing label noise near object boundaries.
• Soft-Masked Feature Aggregation (SMFA) pools ViT patch embeddings using area-interpolated, downsampled soft masks to generate instance vectors $v_c$.
• Outlier filtering discards the top $\mathcal{Y}_i$0 vectors farthest from class centroid $\mathcal{Y}_i$1 by $\mathcal{Y}_i$2 norm.
• The union of these class-specific vectors forms the feature bank $\mathcal{Y}_i$3.

Stage 2: Online Inference via Retrieval

• Input: Test image $\mathcal{Y}_i$4.
• A class-agnostic high-precision mask proposer $\mathcal{Y}_i$5 (e.g., EntitySeg) generates geometric region masks $\mathcal{Y}_i$6.
• For each mask proposal, SMFA computes a query embedding $\mathcal{Y}_i$7.
• Retrieval in $\mathcal{Y}_i$8 (using cosine similarity) finds top-$\mathcal{Y}_i$9 nearest neighbors; majority-class voting with a similarity-weighted tiebreak labels each proposal.
• Non-maximum suppression (NMS) and confidence-priority rasterization consolidate overlapping proposals into the final segmentation map $\mathcal{G}_{mask}$0.

Summary Table: ModuSeg Pipeline Stages

Stage	Major Components	Outputs
Offline Feature Bank	CorrCLIP, SBP, SMFA, filtering	$\mathcal{G}_{mask}$1 (feature bank)
Online Inference	EntitySeg, SMFA, retrieval, NMS	$\mathcal{G}_{mask}$2 (segmentation map)

2. Mask Proposal and Boundary Extraction

The mask proposal stage is key to ModuSeg's reliability in object delineation.

• Inference-time Proposal: EntitySeg, a class-agnostic segmenter, is chosen for its robustness in clutter and ability to delineate fine object boundaries across diverse semantic categories.
• Training-time Proposal: CorrCLIP generates class-conditional pseudo-masks, constrained by image-level labels to prevent hallucinated categories.
• Boundary Filtering: To reduce boundary ambiguity arising from coarse mask labels and patch-grid misalignment, SBP performs iterative morphological erosion:

$\mathcal{G}_{mask}$3

with $\mathcal{G}_{mask}$4 erosions, using a $\mathcal{G}_{mask}$5 structuring element.

• No Score Calibration: There is no explicit scoring of proposal quality beyond the pipeline's morphological/semantic cleansings and voting.

3. Feature Extraction, Bank Formation, and Retrieval

ModuSeg leverages a foundation model backbone (frozen C-RADIOv4-SO400M, patch size $\mathcal{G}_{mask}$6, $\mathcal{G}_{mask}$7 grid, $\mathcal{G}_{mask}$8) for embedding all proposal regions.

• Feature Processing: Following C-RADIOv4 preprocessing, feature maps $\mathcal{G}_{mask}$9 are computed for each input.
• Soft-Masked Feature Aggregation (SMFA): Masks $c\notin \mathcal{Y}_i$0 are area-interpolated to the $c\notin \mathcal{Y}_i$1 feature grid, yielding weights $c\notin \mathcal{Y}_i$2. Vectors are pooled as

$c\notin \mathcal{Y}_i$3

with normalization enforcing unit $c\notin \mathcal{Y}_i$4 norm and $c\notin \mathcal{Y}_i$5.

• Outlier-rejection: The global class centroid $c\notin \mathcal{Y}_i$6 is calculated and the top $c\notin \mathcal{Y}_i$7 vectors by distance $c\notin \mathcal{Y}_i$8 are discarded, forming the clean set $c\notin \mathcal{Y}_i$9.
• Feature Bank: All cleaned per-class vectors are indexed in a flat or IVF FAISS structure for non-parametric nearest-neighbor search.
• Retrieval and Label Assignment: At test time, for each query $-\infty$0, the $-\infty$1 nearest neighbors are retrieved. Label assignment follows a lexicographic ordering: maximal majority vote, followed by highest summed similarity if a tie occurs. Semantic confidence $-\infty$2 is averaged over similarities of the winning class neighbors.

4. Performance and Empirical Results

ModuSeg demonstrates substantial advances in both quantitative and qualitative segmentation metrics compared to contemporary WSSS methods.

• Datasets: PASCAL VOC 2012 (21 categories), MS COCO 2014 (81 categories).
• Metrics: Mean Intersection over Union (mIoU).

Reported mIoU:

Method	VOC Test (%)	COCO Val (%)
ExCEL	78.5	50.3
SSR	79.6	50.6
ModuSeg	86.6	56.7

• Qualitative Observations: ModuSeg segmentations feature more complete and contiguous object regions, with pronounced boundary crispness and reduced background activation artifacts.
• Runtime Efficiency: End-to-end pipeline (bank construction + entire dataset inference) requires only ~84 minutes for VOC, compared to 400–1,500 minutes for prior state-of-the-art approaches.

5. Ablations, Scalability, and Generalization

Ablation studies delineate the contributions of each architectural element.

• Impact of SBP and SMFA (VOC val mIoU):
  • Baseline (no SBP, no SMFA): 84.3%
  • +SMFA: 84.6% (+0.3)
  • +SBP: 85.2% (+0.9)
  • +SBP+SMFA: 86.3% (+2.0)
• Mask Generator Filtering: Class-logits filtering in CorrCLIP increases seed quality from 68.7 to 78.8 mIoU (VOC train).
• Proposal Source: Using SAM-2 for proposals yields 82.6% mIoU (VOC), while EntitySeg increases performance to 86.3%.
• Backbone Generality: Upgrading from DINOv2 to C-RADIOv4 boosts VOC mIoU from 82.9% to 86.3%.
• Plug-in Architecture: Integration with Mask-Adapter framework achieves 86.6% VOC mIoU.

6. Limitations and Prospective Directions

• Background Over-Partitioning: Retrieval effectiveness is upper-bounded by the quality of region proposals; current mask proposers tend to over-segment background, limiting prediction fidelity. Oracle proposal experiments suggest potential mIoU of ~95.7%.
• Static Feature Bank: Fixed, offline feature banks preclude adaptation to out-of-distribution domains or novel visual semantics during test-time inference.
• Research Pathways: Future enhancements include adaptive prototype updating, better class-agnostic segmenters, dynamic outlier thresholds, and the incorporation of multimodal (e.g., text plus image) feature fusion.

ModuSeg thus enables a modular, scalable, and interpretable WSSS system by decoupling localization and classification, employing non-parametric semantic retrieval over purified region prototypes, and achieving SOTA performance without parameter fine-tuning, establishing new upper bounds in training-free segmentation (He et al., 8 Apr 2026). Code and models are available at https://github.com/Autumnair007/ModuSeg.