SAM-Med3D-140K Dataset

• SAM-Med3D-140K is a large-scale, multimodal 3D medical segmentation dataset comprising 21,000 volumes and 131,000 masks across 247 semantic classes.
• It integrates public and proprietary imaging data, using rigorous preprocessing protocols such as spatial standardization, intensity normalization, and connected-component analysis.
• The dataset underpins a fully 3D prompt-driven segmentation model, enabling zero-shot evaluation and transfer learning across diverse anatomical and modality benchmarks.

The SAM-Med3D-140K dataset is a large-scale, multimodal, and multiclass 3D medical image segmentation corpus developed for training and evaluating general-purpose segmentation models in volumetric medical imaging. It comprises 21,000 volumetric medical scans and 131,000 associated 3D segmentation masks, covering a total of 247 semantic classes. The dataset is foundational to the SAM-Med3D model, which deploys a fully learnable 3D architecture for prompt-driven, zero-shot segmentation across diverse anatomical structures and imaging modalities (Wang et al., 2023).

1. Dataset Composition and Sources

SAM-Med3D-140K unifies data from both public medical segmentation benchmarks and proprietary clinical imaging collections. The dataset contains 21,000 distinct 3D image volumes and 131,000 volumetric masks, with an average of approximately 6.2 masks per volume. Segmentation is provided for 247 unique semantic classes, including major organs and a spectrum of lesion types.

The imaging data spans 27 distinct volumetric modalities: 1 CT sequence and 26 MRI sequences (e.g., T1, T2, FLAIR, DWI). The anatomical coverage is comprehensive, enumerating seven broad groups: abdominal and thoracic organs, bone, brain, cardiac tissues, glandular organs, muscle, and other soft tissues. Lesion categories in the training set notably include liver, kidney, and brain tumors (enhancing, edematous, etc.), with evaluation extended to several held-out lesion types (e.g., breast tumors in ultrasound).

Public sources incorporated in the dataset include:

• AMOS (Abdominal Multi-Organ Segmentation),
• TotalSegmentator (104-structure CT),
• BraTS21 (brain tumor MRI),
• KiTS21 (kidney and tumors CT),
• ATLAS (liver tumor MRI),
• TDSC-ABUS (3D breast ultrasound lesion challenge).

Unspecified private datasets provided additional volumes and masks, especially for CT and MRI.

2. Preprocessing and Annotation Workflow

SAM-Med3D-140K underwent a multi-stage preprocessing and annotation protocol, standardizing heterogeneous public and private sources for consistent model training. The steps include:

1. Meta-data filtering: Volumes with physical size below 1 cm³ or any dimension under 1.5 cm were excluded to ensure anatomical relevance of masks.
2. Mask cleaning and noise suppression: All multi-class masks are one-hot encoded. For each channel, the five largest connected components are computed. Masks are retained only if the background occupies less than 99% of the volume. Within remaining channels, only the five largest connected regions are kept to reduce spurious label islands.
3. Symmetry-based relabeling: For paired organs (e.g., kidneys), channels are automatically split into left and right based on spatial heuristics.
4. Spatial standardization: All image volumes are mapped to a uniform 128×128×128 voxel grid. Volumes smaller along any axis are padded, and larger volumes are downsampled via trilinear interpolation.
5. Intensity normalization and augmentation: Per-volume Z-normalization (zero mean, unit variance) is applied, with random axis flips for augmentation (no rotations or elastic deformations).
6. Annotation format and quality control: Original DICOM and NIfTI formats are converted to one-hot NIfTI volumes. Quality control is enforced via connected component and symmetry-based pipelines, though no further human manual annotation was conducted.

3. Dataset Statistics and Splitting

The global statistics of SAM-Med3D-140K are summarized as follows:

Usage	Volumes	Masks	Modalities	Classes
Training	21,000	131,000	27 (1 CT + 26 MRI)	247
Zero-Shot Eval	–	–	CT, MRI, US	153 targets in 15 datasets
Transfer	–	–	–	Pre-trained encoder for UNETR

Organs constitute the majority of samples, while lesions—particularly those with small spatial extent—are less prevalent and typically range from a few dozen to several hundred volumes per category. The dataset was used in its entirety for training; no internal held-out splits were retained. All formal evaluation was conducted in a cross-dataset, zero-shot setting using 15 external public benchmarks (Wang et al., 2023).

4. Model Integration and Training Protocols

SAM-Med3D was trained exclusively from scratch using the full 3D dataset. Key features of the training protocol include:

• Network architecture: A fully 3D adaptation of the Segment Anything Model (SAM), incorporating 3D convolutions, positional encodings, attention mechanisms, and upsampling. No pretrained 2D SAM weights were reused, as experiments indicated negligible benefit.
• Optimization schedule: The Adam optimizer was employed with an initial learning rate of $8 \times 10^{-4}$ for 800 epochs, partitioned into four cycles of 200 epochs. The learning rate was decayed by a factor of 10 at epochs 120, 160, and 190 within each cycle, then reset at cycle completion. Training utilized 8×A100 GPUs, with a batch size of 12 and gradient accumulation over 20 steps. Weight decay was set at 0.1.
• Loss function: The objective combined Dice and Cross-Entropy components (DiceCELoss), defined as:

$$\mathcal{L}_{Dice} = 1 - \frac{2\sum_i p_i g_i}{\sum_i p_i + \sum_i g_i}, \ \mathcal{L}_{CE} = -\frac{1}{N}\sum_{i} [ g_i \log(p_i) + (1-g_i)\log(1-p_i)], \ \mathcal{L} = \lambda \mathcal{L}_{Dice} + (1-\lambda)\mathcal{L}_{CE}$$

with $\lambda=0.5$.

5. Evaluation Protocols and Benchmarks

Model evaluation was prompt-driven and performed in a zero-shot cross-dataset setting. Key protocols include:

• Point-prompt simulation: Interactive segmentation prompts were simulated by sampling a random foreground voxel for the initial prompt, then additional points from error regions. Scenarios with 1, 3, 5, and 10 points per volume (or per 2D slice for 2D variants) were assessed.
• Metrics: Dice coefficient and Intersection-over-Union (IoU) were reported, with:

$\mathrm{IoU} = \frac{|P \cap G|}{|P \cup G|}$

where $P$ and $G$ denote predicted and ground-truth sets.

• External validation: Testing was exclusively performed on 15 independent public volumetric benchmarks, including CT, MRI, and US modalities. No fine-tuning was conducted. Anatomical groups evaluated comprise abdominal & thorax organs, brain structures, and lesion tasks (e.g., AMOS-Val, TotalSegmentator-Test, BraTS21-Val*, KiTS21-Val, TDSC-ABUS).
• Transfer learning: The 3D ViT backbone of SAM-Med3D was used as a frozen encoder to initialize UNETR models for AMOS and TotalSegmentator, resulting in a 4–5% Dice improvement versus training from scratch.

6. Dataset Limitations and Biases

Several limitations are acknowledged:

• Class and modality imbalance: The dataset exhibits substantial overrepresentation of common organs (e.g., liver, spleen), with scant examples for small lesions and rare MRI protocols (e.g., diffusion imaging). Ultrasound lesions are excluded from training and solely appear in TDSC-ABUS evaluation, where segmentation performance is diminished.
• Mitigation strategies: Exclusion of very small volumes (<1 cm³) limits overfitting to sparse targets. Connected-component analysis reduces label noise from small islands, and symmetry-based relabeling ensures paired organs are distinctly labeled.

A plausible implication is that, despite extensive standardization and careful filtering, clinical or etiological diversity may remain underrepresented for rare pathologies or modalities, potentially affecting zero-shot generalization to such cases.

7. Significance and Applications

SAM-Med3D-140K establishes a new scale for curated, standardized 3D medical segmentation corpora, enabling the design and validation of prompt-driven, general-purpose volumetric segmentation models. Its thorough integration into training, zero-shot evaluation, and transfer learning pipelines demonstrates applicability across anatomical targets, modalities, and clinical tasks. The dataset, along with trained models and code, is publicly available for research use via https://github.com/uni-medical/SAM-Med3D (Wang et al., 2023).