MaskSAM: Automated Class-Aware Segmentation

• MaskSAM is a family of methods that extends SAM by integrating prompt-free mask generation with CAM-based pseudo-labeling to produce precise, class-aware segmentation masks.
• For natural images, it refines weakly supervised segmentation by merging high-resolution SAM outputs with coarse CAM labels using normalized overlap metrics.
• In 3D medical imaging, MaskSAM employs a Prompt Generator and 3D adapters to achieve fully automatic, multi-class volumetric segmentation with state-of-the-art performance.

MaskSAM designates a family of methods that extend or adapt the Segment Anything Model (SAM) to enhance semantic segmentation in both natural and medical images. MaskSAM techniques systematically address limitations of prompt-driven, class-agnostic SAM models by producing high-fidelity, class-aware segmentation masks without requiring human input at inference, and, in recent work, by enabling fully automatic, multi-class volumetric (3D) medical image segmentation. Two principal lines of research are established under this nomenclature: (1) MaskSAM/enhanced pseudo-labeling for weakly-supervised semantic segmentation (WSSS) that leverages prompt-free SAM outputs; and (2) "MaskSAM" as a prompt-free adaptation of SAM, with efficient 3D adapters and embedded classification, for volumetric medical image segmentation (Chen et al., 2023, Xie et al., 2024).

1. Rationale and Development

Conventional SAM-based segmentation depends on user prompts (points or boxes) and yields class-agnostic masks, which limits its direct applicability for tasks demanding pixel-precise, semantically-labeled outputs or full 3D volume analysis. Original WSSS pipelines based on CAMs supply only coarse, class-aware region proposals, lacking precise object boundary adherence. In parallel, applying vanilla SAM to volumetric medical data exposes key deficiencies: absence of semantic label prediction for each mask, critical reliance on user-driven prompting, and 2D inductive bias incompatible with 3D anatomical structures.

The MaskSAM approaches were therefore introduced to bridge these gaps. In the WSSS setting, prompt-free SAM masks are systematically integrated with CAM pseudo-labels to yield object- and class-aware pixel-wise annotations. In volumetric medical image segmentation, MaskSAM adapts the SAM architecture with a Prompt Generator, mask classification mechanism, and 3D-specific adapters, producing fully automatic segmentation and classification of 3D volumes (Chen et al., 2023, Xie et al., 2024).

2. Weakly Supervised Semantic Segmentation Enhancement

MaskSAM (under the identifier SEPL) augments standard CAM-based WSSS pipelines by applying prompt-free SAM mask generation and overlap-based mask selection/merging:

• CAMs $\{M_k\}$ (per class $k$) are processed into binary pseudo-labels $P_k$, furnishing coarse spatial guidance but with imprecise boundaries.
• SAM is invoked without prompts, yielding a set $\mathcal{S} = \{S_i\}$ of high-resolution, object-specific masks.
• For each SAM mask $S_i$, the class $k^*$ of maximal overlap with $P_k$ is assigned.
• Two normalization metrics are key: mask coverage $o_s(i, k) = \frac{|S_i \wedge P_k|}{|S_i|}$ and pseudo-label coverage $o_p(i, k) = \frac{|S_i \wedge P_k|}{|P_k|}$. These are used to filter $S_i$ by thresholds $k$0 (mask is at least 50% covered by $k$1) and $k$2 (mask covers at least 85% of $k$3).
• Final pseudo-labels are constructed by taking the union of all SAM masks passing these gates, producing refined class-aware, object-accurate segmentation masks suitable for downstream fully supervised training (Chen et al., 2023).

Empirical evaluation demonstrates consistent improvements over state-of-the-art WSSS methods, with gains on PASCAL VOC and MS-COCO segmentation tasks (Chen et al., 2023).

3. Prompt-Free 3D Mask Classification for Medical Segmentation

The second MaskSAM paradigm extends to 3D volumetric medical images (CT, MRI), addressing the full automation and semantic labeling of anatomical structures:

• Prompt Generator: Integrates multi-level ViT encoder features to generate $k$4 auxiliary binary mask prompts $k$5, corresponding bounding-box prompts $k$6, and auxiliary classifier tokens $k$7. These serve as automatic mask and prompt initializations, supplanting the need for user interaction.
• Global Classification Tokens: Adopts $k$8 global learnable classifier tokens $k$9 appended to the mask decoder input, where $P_k$0 is the number of semantic classes.
• For each predicted mask $P_k$1, the semantic label is computed by summing the auxiliary classifier token $P_k$2 with each $P_k$3, projecting via a classification head:

$P_k$4

Label assignment proceeds via the $P_k$5 operation.

• 3D Adapter Modules: Lightweight 3D Depth-Convolution Adapters (DConvAdapter) and 3D Depth-MLP Adapters (DMLPAdapter) are inserted in the transformer encoder and decoder, respectively. The DConvAdapter processes $P_k$6 with channel reduction, depthwise 3D convolution, activation, and up-projection, facilitating 3D context capture with minimal parameter cost. The DMLPAdapter similarly processes prompt tokens along the depth axis.

This configuration preserves compatibility with pretrained SAM weights while activating 3D awareness and enabling multi-class, prompt-free segmentation (Xie et al., 2024).

4. Training, Inference, and Evaluation

MaskSAM for 3D segmentation is trained end-to-end using cross-entropy or Dice losses on volumetric datasets—such as AMOS2022 (16-organ CT), Synapse (8-organ CT), and ACDC (cardiac MRI)—with heavy data augmentation (rotations, scaling, noise, contrast, gamma, mirroring).

• Volumes are cropped into overlapping patches, processed along three anatomical orthogonal axes for enhanced context, with multiple augmentations per axis. Predictions are averaged across patches and axes.
• At inference, the model is entirely prompt-free: automated prompt generation and mask classification pipeline permit full automation—no clinician interaction.

Results on standard benchmarks show MaskSAM surpasses strong baselines:

• AMOS2022: MaskSAM achieves 90.5% Dice (nnUNet: 87.8%, SwinUNETR: 88.0%, +2.7 absolute improvement).
• Synapse: 87.2% Dice (nnFormer: 86.6%, +0.6).
• ACDC: 93.4% Dice (nnFormer: 92.06%, +1.33).

Ablation studies indicate the best performance when the Prompt Generator combines mask and averaged bounding box cues, and that the full DMLPAdapter and DConvAdapter configuration yields incremental gains over simpler architectures (Xie et al., 2024).

5. Integration and Extensibility

In the WSSS domain, MaskSAM/SEPL is directly pluggable into any pipeline using CAM pseudo-labels. It requires no architectural modifications to downstream decoders; instead, it refines training supervision for fully supervised segmentation networks.

For 3D medical segmentation, MaskSAM requires the integration of the Prompt Generator, global classifier tokens, and lightweight adapters, but these additions do not modify the core prompt encoder or mask decoder of SAM and retain original pretrained transformer weights.

A plausible implication is that the MaskSAM architecture, due to its modular adapters and token-based interfaces, is extensible to other dense labeling tasks, even those involving multi-modal or highly anisotropic 3D data (Xie et al., 2024).

6. Limitations and Future Directions

Notable limitations of MaskSAM include a minor increase in runtime and memory footprint due to the Prompt Generator and adapter modules. Future research directions include reducing the number of prompt candidates $P_k$7 via learned selection, parameter sharing in adapters for lighter models, incorporation of anatomical priors or weak text cues, and expansion to multi-modal registration or diffusion MRI.

Both MaskSAM methods represent significant steps toward automated, class-aware, high-fidelity segmentation across image modalities. Their architectural decisions—particularly prompt automation, mask classification via token summation, and efficient 3D adaptation—form the basis for automated, high-performance, foundation-model-powered segmentation in natural and medical imaging domains (Chen et al., 2023, Xie et al., 2024).