Molecular-Empowered All-in-SAM Model

• The paper introduces the Molecular-Empowered All-in-SAM Model, which integrates molecular guidance, SAM-based annotation, and sharpness-aware optimization to reduce annotation costs in bioimaging.
• It presents a detailed three-stage workflow that uses bounding-box annotations, prompt-driven mask generation, and MOCL segmentation refinement for enhanced precision.
• Empirical evaluations across cell segmentation, molecular graph learning, and materials science show robust performance with minimal data and improved generalization.

The Molecular-Empowered All-in-SAM Model is a general framework that unifies molecular guidance, sharpness-aware optimization, and prompt-based adaptation with the Segment Anything Model (SAM) family to enable highly efficient and effective annotation, segmentation, and prediction in bioimaging, molecular graph learning, and certain materials science domains. By leveraging molecular-specific information—such as immunofluorescence (IF) markers in histology or domain-distinguishing signals in molecular graphs—the model democratizes expert-level annotation and achieves robust, generalizable segmentation or prediction, typically with dramatically reduced annotation cost and data requirements.

1. Conceptual Foundation and Scope

The core notion of the Molecular-Empowered All-in-SAM Model is to impart molecular-specific knowledge into general-purpose segmentation or property-prediction architectures—most notably the Segment Anything Model (SAM), its adapters, and sharpness-aware optimization—to address the bottleneck of expert annotation or unreliable training in molecular imaging and analysis. The design leverages three pillars:

• Molecular Guidance: Use of domain-specific auxiliary channels, e.g., registered IF in pathology, to expose class-distinguishing signals even to lay annotators.
• SAM-Centric Annotation: Prompt-based or adapter-augmented inference by SAM for mask generation, with the ability to convert weak annotations (e.g., bounding boxes) to pixel-level masks.
• Robust Optimization: Training with sharpness-aware minimization, such as SAM or efficient variants (GraphSAM), to ensure generalization and stability under label noise and out-of-domain data.

Applications span tissue (cell or nuclei) segmentation (Li et al., 2023, Li et al., 21 Aug 2025), molecular property prediction with graph transformers (Wang et al., 2024), protein particle picking in cryo-EM (He et al., 2023), and interfacial engineering in perovskite photovoltaics via thick self-assembled molecular layers (Huang et al., 23 Mar 2026).

2. Model Architecture and Workflow

The canonical computational pathology instantiation proceeds in three sequential stages (Li et al., 21 Aug 2025):

1. Molecular-Empowered Annotation: Annotators draw only bounding boxes on PAS-stained images, while overlaying IF images specify molecular markers (e.g., WT-1 for podocytes, α-SMA for mesangial cells). This replaces the need for full pixel-level expert contouring.
2. SAM-Powered Mask Generation and Adaptation: Pre-trained SAM is prompted with the bounding boxes and returns pixel-level segmentation masks. Optionally, lightweight adapter modules are inserted into the final Transformer blocks of SAM to inject task-specific features or rectify domain mismatch, while the rest of SAM remains frozen. Alternatively, prompt-based learning (prefix tokens, U-Net heads, or transformer adapters) enables robust molecular adaptation with minimal additional parameters (He et al., 2023).
3. Segmentation Refinement with MOCL: The generated masks are used as pseudo-ground-truth to train a deep segmentation model, generally a U-Net with a ResNet-34 encoder. Molecular-Oriented Corrective Learning (MOCL) is used to re-weight per-pixel losses based on confidence and similarity of feature embeddings to high-confidence regions, reducing the deleterious effects of residual SAM mask noise.

A representative schematic is:

Stage	Inputs/Operations	Outputs
Molecular-Empowered Annotation	PAS, IF; bounding box drawing by annotators	Box coordinates
SAM Adapter or Prompt Inference	SAM with (optional) adapter/prompt tuning; box prompts	Pixel-level segmentation mask
MOCL Segmentation Training	U-Net/ResNet-34, pseudo-labels, MOCL weighting	Fine-grained output/segmentation

3. Mathematical Formulation

Prompt-Driven Mask Generation

Box prompts are perturbed to model user variation: $$x'_\text{min} = x_\text{min} + \Delta x_\text{min}, \quad \Delta x_\text{min} \sim \mathcal{U}(-\alpha w, \alpha w)$$ where $w$ is width.

SAM returns a pixel-probability map $P(x, y)$. Binarization and morph (threshold $\tau=0.5$): $M(x, y) = \bigl[ P(x, y) \geq \tau \bigr]$ and small-region filtering/hole-filling yields masks $M(x, y) \in \{0,1\}$.

Segmentation and MOCL Losses

Training uses a composite of cross-entropy and Dice losses: $$\mathcal{L} = \mathcal{L}_\mathrm{CE} + \lambda \mathcal{L}_\mathrm{Dice}$$ MOCL dynamically adjusts pixel weights: $w_i = \frac{1 + \cos(f_i, \bar{f}_\text{top})}{2}$ where $f_i$ are per-pixel feature embeddings and $\bar{f}_\text{top}$ is the mean of top-k confident embeddings. The MOCL loss is then: $w$0 with $w$1 the per-pixel CE+Dice loss.

Adapter and Prompt Learning

Adapter block in Transformer layer $w$2: $w$3 where $w$4 are handcrafted PAS features.

Prompt-based variants inject U-Net heads (image pre-processing), prefix tokens (between transformer layers), or encoder adapters (low-rank perturbations within transform blocks), with only miniature modules trained while core SAM parameters remain frozen (He et al., 2023).

4. Empirical Performance and Quantitative Evaluation

Across cell/nucleus segmentation tasks (Li et al., 2023, Li et al., 21 Aug 2025):

• Annotation time reduced from ≈9 min/cell (manual outline) to ≈10 s/cell (box).
• F1 (Dice) performance for podocyte/mesangial classes:
  • Lay PAS+IF manual F1: 0.8496/0.8473
  • SAM-L (tight boxes): 0.8370/0.8312
  • SAM-L (randomized boxes): 0.8577/0.8469
  • Downstream U-Net trained on SAM-L masks with MOCL: F1 ≈ 0.735–0.695, matching expert-annotated pipelines.
• Robustness to data scarcity and label noise: All-in-SAM sustains Dice > 0.78 and AUC > 0.94 even with <1% annotated data (Li et al., 21 Aug 2025).
• Public datasets (Monuseg): Dice/IoU/Adj. Rand at parity or outperforms nnUNet under full and weak-label splits.

In molecular graph transformers (Wang et al., 2024), integration of GraphSAM yields matched or slightly superior ROC-AUC on property prediction benchmarks compared to full SAM, with ~1.35× throughput.

Cryo-EM protein picking: Prompt-based adaptation achieves Dice of 0.72 (prefix) to 0.78 (encoder adapter) with only a small fraction of SAM parameters and GPU memory compared to conventional fine-tuning (He et al., 2023).

5. Variants, Applications, and Generalizations

• Vision-Centric Biomedical Segmentation: Molecular-Empowered All-in-SAM has been validated for fine-grained multi-class nuclei segmentation using vision foundation models with prompt guidance and adapter modules, achieving SOTA in settings requiring only PAS at inference (Li et al., 21 Aug 2025).
• Molecular Graph Learning: Efficient sharpness-aware minimization (GraphSAM) integrated into graph transformers (e.g., GROVER, CoMPT) improves generalization and throughput in molecular property prediction, with applications in classification/regression suites (BBBP, Tox21, ESOL, etc.) (Wang et al., 2024).
• Cryo-EM and Beyond: Parameter-efficient prompt-based learning for SAM enables universal protein particle picking with minimal labeled data, extensible to other high-noise biomedical imagery modalities (He et al., 2023).
• Materials Science: The “All-in-SAM” paradigm in perovskite photovoltaics refers to a single thick, thermally-evaporated SAM film forming a graded molecular orientation and energy barrier, which serves simultaneously as hole-transport, interface passivation, and wetting layer, delivering record power conversion efficiencies and industrial scalability (Huang et al., 23 Mar 2026).

6. Theoretical and Computational Considerations

• Sharpness-Aware Minimization (SAM): Both classical SAM and efficient variants flatten the loss landscape, improving out-of-distribution robustness and delaying time-to-failure in quantum molecular dynamics (Ibayashi et al., 2023). Empirical laws show systematic improvements in scaling, e.g., for Allegro-Legato, $w$5 vs. $w$6 baselines.
• MOCL Noise Mitigation: By steering learning toward consistent high-confidence regions between annotator-generated and model-generated masks, MOCL suppresses the impact of annotation ambiguity or prompt misplacement (Li et al., 2023, Li et al., 21 Aug 2025).
• Adapter Overhead: Adapter and prompt modules increase inference time and GPU memory only marginally compared to full model fine-tuning but decouple domain adaptation from foundational model stability.

7. Limitations and Future Prospects

Documented limitations include reliance on co-registered IF or auxiliary signals during annotation, mild computational/memory overhead for adapter blocks, and need for further validation on large-scale, multi-institution cohorts (Li et al., 21 Aug 2025). Prompt-based approaches are constrained by the capacity of tiny modules and may be suboptimal for domains with severe domain shift. The All-in-SAM concept in perovskite assembly requires precise control over evaporation conditions and material purity (Huang et al., 23 Mar 2026).

Planned or suggested extensions include:

• Incorporation of additional molecular markers and expansion to multi-organ and multi-modality datasets.
• Active learning and further reduction of annotation requirements.
• Real-time or on-device deployment via dynamic adapter pruning.
• Extension to further areas in quantum simulation and graph-based multi-task learning via universal foundation models and sharpness-aware optimization.

The Molecular-Empowered All-in-SAM Model thus constitutes a cross-domain, modular, and annotation-efficient paradigm for bringing molecular expertise and robust generalization to large-scale segmentation, prediction, and materials engineering pipelines.