ScSAM: Subcellular Semantic Segmentation

• ScSAM is a subcellular semantic segmentation framework that adapts the Segment Anything Model (SAM) to extract detailed organelle features from electron microscopy images.
• It fuses frozen SAM embeddings with MAE-derived cellular priors using a Feature Alignment and Fusion Module to counteract morphological variability and label imbalance.
• The model employs a cosine similarity-based class prompt encoder and a lightweight mask decoder, achieving improved Dice scores for nucleus, mitochondria, and granules.

ScSAM is a subcellular semantic segmentation framework for electron microscopy images that adapts the Segment Anything Model (SAM) to a biomedical setting characterized by pronounced morphology variability across organelles and severe distributional imbalance in the label space. It is designed for multi-class segmentation of subcellular structures—specifically nucleus, mitochondria, and granules in BetaSeg islet-cell EM data—and combines a frozen pre-trained SAM encoder with a masked autoencoder (MAE) trained on the target subcellular domain, a Feature Alignment and Fusion Module (FAFM), and a cosine-similarity-based class prompt encoder to produce class-specific masks $M^{(c)} = \mathrm{ScSAM}(I,k)$ (Fang et al., 23 Jul 2025).

1. Problem setting and rationale

ScSAM addresses a long-standing difficulty in learning-based organelle segmentation: the significant morphological and distributional variability among subcellular components. In the target setting, nucleus, mitochondria, and granules differ markedly in size, shape, texture, and spatial distribution, and small, irregular, sparsely distributed organelles such as granules are especially hard to learn. Under standard training, dominant classes and easier morphologies bias optimization, so the model overfits to spurious correlations or coarse global appearance and under-represents rare fine structures (Fang et al., 23 Jul 2025).

The method is motivated by three limitations identified in prior approaches. First, many methods rely on a single feature mapping, which lacks the diversity needed to represent heterogeneous organelles. Second, some alternatives require many trainable parameters and domain-specific supervision. Third, SAM-based variants may fuse features from backbones trained for essentially the same task, leading to redundant rather than complementary semantics. Vanilla SAM is also not directly suitable for this domain: it was pretrained on natural images, emphasizes global contextual understanding, and tends to miss subtle ultrastructural details and minority-class morphology in EMIs; zero-shot transfer therefore performs poorly (Fang et al., 23 Jul 2025).

The central design claim is that the relevant failure mode is not merely generic domain shift. In the formulation of ScSAM, the more specific issue is biased feature learning induced by gaps in the label space created by morphology variation and long-tailed class distributions. This motivates a cross-task fusion strategy in which SAM contributes structure-aware, region-level, edge- and shape-oriented features, while MAE contributes multi-scale cellular texture and morphology priors learned from reconstruction on EM data (Fang et al., 23 Jul 2025).

2. Architectural organization

Architecturally, ScSAM consists of two frozen encoders, a Feature Alignment and Fusion Module, a cosine-similarity-based class prompt encoder, and a mask decoder or segmentation head (Fang et al., 23 Jul 2025).

Component	Role	Key specification
Frozen SAM encoder	Structure-aware embeddings	Pre-trained, frozen
Frozen MAE branch	Cellular prior learning	Pretrained on BetaSeg
FAFM	Shared-space alignment and adaptive fusion	Projection to 256 channels
Class prompt encoder	Class activation via prototype similarity	Cosine-similarity matrix
Mask decoder	Final mask prediction	128 hidden units

The MAE branch is explicitly a prior-learning component rather than the main segmentation backbone. According to the paper and supplement, MAE is pretrained on BetaSeg with cropped image size $256\times 256$, patch size $16$, mask ratio $0.5$, for 1600 epochs. To generate embeddings, each original EMI is resized so its longest side is 1024, padded to $1024\times 1024$, split into four pieces, processed by the pretrained MAE, and stitched back into a $64\times 64\times 512$ representation that matches SAM’s embedding size. The authors emphasize that MAE alone performs very poorly for segmentation because reconstruction features lack sufficiently high-level semantics, but as a complementary prior branch they encode texture, local morphology, and distributional cues that SAM misses (Fang et al., 23 Jul 2025).

A practical consequence of this design is that both SAM and MAE embeddings are precomputed and both encoders remain frozen throughout training. The trainable portion is therefore comparatively lightweight, and the paper presents this as a mechanism for reducing computational burden while retaining complementary priors from two different pretraining tasks (Fang et al., 23 Jul 2025).

3. Feature alignment and adaptive fusion

The Feature Alignment and Fusion Module is the first main technical component. Its role is to reconcile the heterogeneous feature distributions of SAM and MAE—different statistical distributions, semantic granularity, and attention patterns—and then merge them into a dense representation suitable for segmentation (Fang et al., 23 Jul 2025).

Let $E_S$ and $E_M$ denote SAM and MAE embeddings. Each is projected to the same channel dimension, 256, by a two-layer MLP. For SAM, the paper gives

$\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$

and the MAE embedding is transformed analogously to produce $\hat{E}_M$, with

$256\times 256$0

The alignment is supervised by a cosine-similarity loss

$256\times 256$1

In this expression, $256\times 256$2 and $256\times 256$3 are the $256\times 256$4-th embedding vectors or channels after resizing and projection. Minimizing $256\times 256$5 aligns feature directions and projects the two embedding sets into a common submanifold while preserving each source’s distributional spread (Fang et al., 23 Jul 2025).

The paper reports that t-SNE visualization shows the original SAM and MAE embeddings are well separated, whereas aligned embeddings mix substantially. This is presented as empirical support for the claimed common-space alignment rather than mere concatenation of heterogeneous features (Fang et al., 23 Jul 2025).

After alignment, FAFM performs adaptive fusion. The aligned embeddings are concatenated along channels and passed through two $256\times 256$6 convolution layers plus GroupNorm: $256\times 256$7 where $256\times 256$8 appears to refer to the aligned auxiliary embedding, i.e., the MAE branch, and CAM denotes a channel attention module. GroupNorm is used because microscopy training employs small batch sizes, making BatchNorm unstable. CAM uses global average pooling and max pooling to gather channel statistics and produce learnable reweighting, amplifying channels useful for organelle discrimination. A lightweight auxiliary stream is added, yielding

$256\times 256$9

FAFM therefore outputs $16$0, intended to integrate structure-aware SAM semantics with MAE-derived cellular texture priors (Fang et al., 23 Jul 2025).

A notable experimental finding is that this explicit alignment-plus-attention fusion clearly outperforms both naive concatenation and symmetric cross-attention. On low-glucose data, direct concatenation yields mean Dice $16$1, cross-attention $16$2, and FAFM $16$3, which the paper uses to argue that the gain is not simply due to using two encoders, but to explicit feature-space alignment and attention-weighted fusion (Fang et al., 23 Jul 2025).

4. Class prompt encoding and training objective

ScSAM replaces manual prompting with a cosine similarity matrix-based class prompt encoder. Rather than using user-provided points or boxes, the model learns a prototype bank of class embeddings and uses prototype-feature similarity to activate class-specific regions (Fang et al., 23 Jul 2025).

Let

$16$4

where $16$5 is the number of classes and each prototype $16$6. After $16$7-normalizing the fused feature $16$8 and class prototype $16$9, the class-$0.5$0 cosine similarity score is

$0.5$1

The class-activated feature map is then formed by residual modulation: $0.5$2 In this construction, the fused feature is amplified in regions similar to the prototype of class $0.5$3, while the residual connection preserves the original feature (Fang et al., 23 Jul 2025).

The paper further states that an MLP-based residual structure adapts these activated features, and that one-hot-coded positive or negative class pairings are used to generate dense and sparse embeddings in the shape expected by the mask decoder. Conceptually, the sparse embedding plays the role of a category-aware point prompt, and the dense embedding acts like a category-aware mask prior supplying shape and texture cues. The exact decoder architecture is not described in detail, but the implementation section notes that the mask decoder uses 128 hidden units (Fang et al., 23 Jul 2025).

Prototype learning is regularized with NT-Xent loss: $0.5$4 Here $0.5$5 is a temperature, $0.5$6 is the class-$0.5$7 prototype, $0.5$8 is an embedding sample from class $0.5$9, $1024\times 1024$0 is batch size, and $1024\times 1024$1 is cosine similarity. The full training objective is

$1024\times 1024$2

where $1024\times 1024$3 supervises semantic segmentation. The authors set $1024\times 1024$4, because $1024\times 1024$5 begins near 1 and a larger weight impedes early convergence; supplementary experiments report that 0.2 outperforms 0.1 and 0.3, while weight 1 severely slows convergence (Fang et al., 23 Jul 2025).

This prompt mechanism is central to the automation claim of ScSAM. A plausible implication is that the method reframes SAM prompting from interactive point or box specification to learned class-conditioned activation, which is especially relevant in subcellular EM settings where manual prompting is poorly matched to dense, fine-grained organelle segmentation.

5. Experimental setting, results, and ablations

ScSAM is evaluated on BetaSeg, an islet-cell electron microscopy dataset with high-glucose and low-glucose subsets from OpenOrganelle. The original data include 3 annotated cells in the high-glucose subset and 4 in the low-glucose subset; the study selects the middle 350 slices from each cell because annotations are only available for target cells and these slices reduce bias from unlabeled neighboring cells. The segmentation targets are nucleus, mitochondria, and granules among seven available subcellular masks. To simulate limited-label learning, the last cell of each subset is used for training and the remaining cells for validation. EMIs are normalized, randomly cropped, and resized for augmentation. Optimization uses Adam with learning rate $1024\times 1024$6, batch size 32, and 100 epochs. The model is implemented in PyTorch 1.11.0 and trained on a single RTX 3090 24GB GPU (Fang et al., 23 Jul 2025).

On the high-glucose subset, ScSAM attains Challenge IoU $1024\times 1024$7, mIoU $1024\times 1024$8, AJI $1024\times 1024$9, and overall Dice $64\times 64\times 512$0, with per-class Dice $64\times 64\times 512$1 for nucleus, $64\times 64\times 512$2 for mitochondria, and $64\times 64\times 512$3 for granules. The strongest supervised baseline, nnFormer, reaches mIoU $64\times 64\times 512$4 and overall Dice $64\times 64\times 512$5; SurgicalSAM reaches mIoU $64\times 64\times 512$6 and overall Dice $64\times 64\times 512$7. Relative to nnFormer, ScSAM improves by $64\times 64\times 512$8 mIoU and $64\times 64\times 512$9 overall Dice; relative to SurgicalSAM, the gains are $E_S$0 mIoU and $E_S$1 overall Dice. Against Vanilla SAM, granule Dice increases from $E_S$2 to $E_S$3, which the paper highlights as evidence of improved learning on minority small structures (Fang et al., 23 Jul 2025).

On the low-glucose subset, ScSAM obtains Challenge IoU $E_S$4, mIoU $E_S$5, AJI $E_S$6, and overall Dice $E_S$7, with Dice $E_S$8 for nucleus, $E_S$9 for mitochondria, and $E_M$0 for granules. The best supervised baseline nnFormer has mIoU $E_M$1 and overall Dice $E_M$2; SurgicalSAM has mIoU $E_M$3 and overall Dice $E_M$4; SAM* has overall Dice $E_M$5. The reported improvements are $E_M$6 mIoU and $E_M$7 overall Dice over nnFormer, and $E_M$8 mIoU and $E_M$9 overall Dice over SurgicalSAM. Relative to Vanilla SAM2, granule Dice improves from $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$0 to $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$1. The authors emphasize that ScSAM is especially strong on mitochondria and granules, the classes most affected by morphology complexity and imbalance (Fang et al., 23 Jul 2025).

The ablation studies support each major component. Removing FAFM reduces performance: on high-glucose data, ScSAM without FAFM yields Challenge IoU $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$2, mIoU $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$3, AJI $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$4, and overall Dice $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$5. The component analysis indicates that both dense and sparse embeddings are important; sparse prompts have strong class semantics but, without dense embeddings, they lack geometric detail, while using alignment loss without dense embeddings collapses performance badly. The residual connection in the class prompt encoder is also beneficial: on low-glucose, removing it drops mean Dice from $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$6 to $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$7 (Fang et al., 23 Jul 2025).

ScSAM also exhibits better cross-dataset generalization. Training on high-glucose and testing on low-glucose, it improves mean Dice from $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$8 for SAM* to $\hat{E}_{S}= L_1(\mathrm{ReLU}(L_2(E_S))),$9, including granule Dice from $\hat{E}_M$0 to $\hat{E}_M$1. Training on low-glucose and testing on high-glucose, mean Dice improves from $\hat{E}_M$2 to $\hat{E}_M$3, with granule Dice from $\hat{E}_M$4 to $\hat{E}_M$5. The paper interprets this as evidence that cross-task fusion disentangles morphology from imaging contrast and captures more domain-invariant subcellular features (Fang et al., 23 Jul 2025).

6. Efficiency, limitations, and relation to similarly named SAM variants

Although ScSAM uses dual frozen encoders, only 27.6M parameters are updated. The paper reports that it reaches optimal performance within 50 epochs and peak Dice in 3.2 hours, compared with SurgicalSAM at 6.5 h, SAM* at 4.7 h, and MAE* at 4.6 h. Inference takes 0.457 s per image, slower than Vanilla SAM’s 0.321 s but described as acceptable for offline cell behavior analysis. The authors attribute the training efficiency to freezing both backbones and to the contrastive loss, which enlarges inter-class angular margins and stabilizes optimization (Fang et al., 23 Jul 2025).

The principal innovations are summarized in the source as: cross-task fusion of SAM and MAE rather than same-task feature fusion; explicit alignment of heterogeneous embeddings with a cosine loss before fusion; and a learned class-prompt mechanism that replaces manual prompts with prototype-driven dense and sparse prompts. In practical terms, the method is positioned for subcellular semantic segmentation in EMIs, especially in limited-label settings with strong morphology variability and poor suitability for manual prompting (Fang et al., 23 Jul 2025).

The main limitations, implied by the paper, are that evaluation is confined to BetaSeg and only three organelle classes, MAE alone is weak and only useful as a complement, and the method remains somewhat slower at inference than single-encoder alternatives (Fang et al., 23 Jul 2025). This suggests that the present evidence is strongest for the specific regime of islet-cell EM segmentation rather than for a general organelle segmentation benchmark spanning multiple datasets and annotation ontologies.

The name “ScSAM” should also be distinguished from several similarly named SAM-derived systems in adjacent domains. “CS3: Cascade SAM for Sperm Segmentation” is a SAM-based method for sperm instance segmentation, particularly overlapping sperm tails, but the provided text states that it does not use the term “ScSAM” anywhere (Shi et al., 2024). “CycleSAM” is a one-shot surgical scene segmentation framework that keeps SAM frozen and focuses on automatic prompt generation through cycle-consistent feature matching (Murali et al., 2024). “CellSAM” is a foundation model for cell segmentation that uses a detector, CellFinder, to generate bounding-box prompts for SAM and is explicitly not named “ScSAM” (Israel et al., 2023). These comparisons are nomenclatural rather than architectural equivalences, but they clarify that ScSAM refers specifically to the subcellular EM segmentation framework introduced in “ScSAM: Debiasing Morphology and Distributional Variability in Subcellular Semantic Segmentation” (Fang et al., 23 Jul 2025).