Auto3DSeg: Automated 3D Medical Segmentation
- Auto3DSeg is an automated medical image segmentation pipeline within MONAI that configures task-specific 3D segmentation workflows with minimal user setup.
- It facilitates advanced ensembling and hyperparameter tuning while allowing expert overrides for modality-specific adaptations.
- The framework supports diverse architectures like SegResNet and incorporates optimized techniques such as deep supervision and cross-validation to achieve robust clinical outcomes.
Auto3DSeg is an automated medical image segmentation pipeline included in MONAI and designed to facilitate and speed up the pipeline-design for 3D medical image segmentation. Across reported uses, it is presented as an automated, user-friendly framework that can auto-analyze data, generate suitable hyperparameters, execute training with GPU scalability, perform inference, and construct model ensembles, while still permitting expert override of hyperparameters and design choices when task-specific constraints require it (Siddiquee et al., 2022, Myronenko et al., 29 Oct 2025). Its published applications span CT, MRI, and CBCT, including intracranial hemorrhage, aorta, kidney, brain tumor, pancreatic tumor, and dental structure segmentation, with outcomes ranging from challenge-winning submissions to more modest performance on small, heterogeneous datasets (Myronenko et al., 2023, Myronenko et al., 2023, Jha et al., 28 Aug 2025).
1. Position within MONAI and configuration model
Auto3DSeg is consistently described as part of the MONAI ecosystem. In challenge reports and application papers, MONAI provides the surrounding infrastructure for preprocessing, training, inference, deployment, and reproducibility, while Auto3DSeg is the component that automates much of the segmentation pipeline design (Myronenko et al., 2023, Myronenko et al., 29 Oct 2025).
A recurrent design principle is minimal user setup. In the BraTS 2023 report, configuration is reduced to a short YAML file that specifies the modality, data root location, input list, and label mapping; for overlapping BraTS subregions, the configuration also sets sigmoid: true to request multi-label segmentation with sigmoid activation (Myronenko et al., 29 Oct 2025). In the intracranial hemorrhage study, Auto3DSeg is described as automating most parameter choices, including preprocessing, augmentation, model configuration, and neural architecture settings that fit the given data, allowing the investigators to focus on higher-level decisions such as whether to use a 2D or 3D formulation and how to ensemble models (Siddiquee et al., 2022).
This division of labor is central to how Auto3DSeg is used in practice. The framework handles routine optimization and code infrastructure, but the published reports show that investigators still intervene when imaging physics, label structure, or clinical targets demand departures from defaults. A plausible implication is that Auto3DSeg is best understood not as a single model, but as a self-configuring pipeline layer within MONAI for dataset-specific segmentation workflows.
2. Architectural patterns and optimization recipes
The dominant backbone in the reported literature is SegResNet. In BraTS 2023, Auto3DSeg was used with a 3D SegResNet encoder-decoder CNN based on a U-Net variant with residual blocks and batch normalization, with deep supervision and sigmoid output for overlapping subregions (Myronenko et al., 29 Oct 2025). In the aorta, dental CBCT, pancreatic MRI, and intracranial hemorrhage reports, SegResNet likewise appears as the principal architecture, although normalization details and dimensionality vary by task (Myronenko et al., 2023, LaBella et al., 18 Aug 2025, Jha et al., 28 Aug 2025, Siddiquee et al., 2022).
The reports also establish that Auto3DSeg is not limited to SegResNet. The BraTS 2023 paper states that Auto3DSeg supports DiNTS and SwinUNETR in addition to SegResNet, even though only SegResNet was used in that entry (Myronenko et al., 29 Oct 2025). The KiTS 2023 submission makes this broader search space concrete: Auto3DSeg automatically explored and trained SegResNet, DiNTS, and SwinUNETR, and the final ensemble retained SegResNet and DiNTS while excluding SwinUNETR because of inferior cross-validation performance (Myronenko et al., 2023).
Several challenge solutions built with Auto3DSeg converged on a recognizable optimization pattern: deep supervision, five-fold cross-validation, extensive augmentation, AdamW, an initial learning rate of , weight decay of , and cosine annealing to zero (Siddiquee et al., 2022, Myronenko et al., 2023, Myronenko et al., 29 Oct 2025). In multiple reports, loss terms are summed across supervision scales with weights of , for example
with downsampled targets matched to intermediate decoder outputs (Siddiquee et al., 2022, Myronenko et al., 2023). That pattern is not universal: the dental CBCT study used Dice + Cross-Entropy Loss with equal weighting, and KiTS used Dice + Focal loss for DiNTS (LaBella et al., 18 Aug 2025, Myronenko et al., 2023). The commonality lies less in a single fixed recipe than in Auto3DSeg’s systematic generation of strong task-specific training configurations.
3. Representative applications and reported outcomes
The empirical record of Auto3DSeg is most visible in challenge settings, where the framework has been used both as an almost out-of-the-box baseline and as the core of more customized pipelines.
| Application | Auto3DSeg-based configuration | Reported outcome |
|---|---|---|
| INSTANCE 2022 intracranial hemorrhage (Siddiquee et al., 2022) | 2D slice-wise SegResNet, 18-model ensemble | Dice 0.721; highest Dice; overall rank 2 |
| SEG.A 2023 aorta (Myronenko et al., 2023) | 3D SegResNet, 15-model ensemble | Dice 0.920; HD95 6.013; rank 1 |
| KiTS 2023 kidney/tumor/cyst (Myronenko et al., 2023) | Auto3DSeg search over SegResNet, DiNTS, SwinUNETR; 15-model final ensemble | Average Dice 0.835; surface Dice 0.723; rank 1 |
| BraTS 2023 brain tumors (Myronenko et al., 29 Oct 2025) | SegResNet used “out of the box” with minor customization | 1st place in Brain Metastasis, Brain Meningioma, BraTS-Africa; 2nd in Adult and Pediatric Glioma |
| BraTS-PEDs 2023 pediatric tumors (Kazerooni et al., 2024) | Team NVAUTO with Auto3DSeg (SegResNet) | FRS 10.25; final rank 1 |
| PANTHER 2025 pancreatic MRI (Jha et al., 28 Aug 2025) | SegResNet with 5-fold CV, ROI-focused two-phase training, STAPLE ensemble | Task 1 DSC 0.56; Task 2 DSC 0.33 |
| ToothFairy3 dental CBCT (LaBella et al., 18 Aug 2025) | Lightweight 3D SegResNet, 5-fold CV, Multi-Label STAPLE, two-phase inference | Average Dice 0.87 on out-of-sample validation |
These results show that Auto3DSeg has been used successfully across markedly different imaging regimes: anisotropic head CT, isotropic or resampled body CT, pre-aligned skull-stripped multi-parametric MRI, pancreatic MRI, and dental CBCT. The strongest results are in tasks where the framework was paired with careful ensembling and task-aware preprocessing, especially in SEG.A 2023, KiTS 2023, and multiple BraTS 2023 sub-challenges (Myronenko et al., 2023, Myronenko et al., 2023, Myronenko et al., 29 Oct 2025). Conversely, the pancreatic MRI results indicate that Auto3DSeg does not remove the intrinsic difficulty of limited-data, high-variability segmentation problems (Jha et al., 28 Aug 2025).
4. Task-specific adaptations beyond default automation
Although Auto3DSeg emphasizes automation, its published uses repeatedly involve targeted departures from default behavior. The intracranial hemorrhage solution is a clear example: because the 3D CTs had high in-plane resolution of about mm but a 5 mm inter-slice gap, and hemorrhages often appeared on only 2–3 slices per case, the investigators chose a 2D slice-wise SegResNet rather than 3D or 2.5D processing. They then ensembled three input-channel strategies—adjacent slices, multiple CT windows, and a combined 9-channel formulation—because validation performance was similar across them (Siddiquee et al., 2022).
The aorta challenge entry introduced an adaptive foreground normalization scheme to address multi-site CT intensity inconsistencies. After an initial segmentation pass, the method estimated 5th and 95th intensity percentiles within the aorta foreground and rescaled the entire CT image to with soft clipping; this adaptive normalization was used for 10 of the 15 ensembled models and was reported as bolstering robustness when some scans used non-standard positive-only scaling (Myronenko et al., 2023).
Auto3DSeg-based systems also frequently rely on anatomically focused cropping. In KiTS 2023, a preliminary network generated kidney bounding boxes so that the main models could train on rectangular crops around the kidneys, reducing compute demands and focusing learning (Myronenko et al., 2023). In pancreatic MRI, two-phase training first operated on a window-cropped image and then retrained on a tighter pancreas-centered bounding box derived from the predicted pancreas, with a 30 mm extension in all axes (Jha et al., 28 Aug 2025). The dental CBCT pipeline similarly performed a first-pass whole-volume segmentation, then used Multi-Label STAPLE on the fold predictions to define a tight mandible-centered crop for a second-phase model targeting smaller nerve structures (LaBella et al., 18 Aug 2025).
BraTS 2023 provides additional examples of task-aware customization inside the Auto3DSeg framework: label mappings were tailored to the nested and overlapping Whole Tumor, Tumor Core, and Enhancing Tumor regions; channel dropout was applied in the Metastasis challenge by randomly zeroing the T2 channel with probability 0.5; and the BraTS-Africa models were initialized from the adult glioma subchallenge in a low-data regime (Myronenko et al., 29 Oct 2025). These cases collectively show that Auto3DSeg is compatible with substantial expert intervention when modality, class structure, or dataset scale calls for it.
5. Comparative performance, failure modes, and evaluation without ground truth
Challenge rankings document strong performance, but the comparative papers also identify limits. In BraTS 2023, internal cross-validation scores could exceed hidden-test performance, and the authors explicitly attribute this gap to data domain variability (Myronenko et al., 29 Oct 2025). In the pancreatic MRI study, performance dropped from Task 1 to Task 2, and the discussion ties this to small dataset size, sequence variability between T1-weighted contrast-enhanced arterial MRI and T2-weighted MR-Linac MRI, and the intrinsic anatomical difficulty of pancreatic tumor delineation (Jha et al., 28 Aug 2025).
The BraTS-PEDs 2023 challenge results further caution against overinterpreting leaderboard margins. Team NVAUTO, which used Auto3DSeg, achieved a cumulative rank of 246 and an FRS of 10.25, yielding final rank 1, but pairwise permutation testing found that the differences between CNMCPMI2023 and NVAUTO () and between NVAUTO and SherlockZyb () were not statistically significant on the test set of 24 cases (Kazerooni et al., 2024). This indicates competitive performance, but not a decisive separation from other strong U-Net-based methods.
A distinct kind of evidence comes from the 2025 concordance study on anatomy segmentation without ground truth. There, an Auto3DSeg model trained on the TotalSegmentator dataset segmented 117 structures, and 24 chest-relevant structures were evaluated on 18 NLST chest CT series after harmonization to DICOM SEG with SNOMED-CT labeling. In that comparison, Auto3DSeg showed excellent agreement on lung lobes, with Dice consistently above 95% and volume overlap above 90%, and good agreement on the heart once anatomical convention differences were accounted for. By contrast, agreement was moderate to poor for ribs and poor for thoracic vertebrae, with frequent missed rib ends near the spine, leakage into neighboring vertebrae, fusion of multiple vertebrae, and mislabeling; the study therefore states that Auto3DSeg is not recommended for accurate segmentation of thoracic vertebrae or ribs on CT in that setting (Giebeler et al., 17 Dec 2025).
The important nuance is that these failures were observed for a specific Auto3DSeg model trained on the TotalSegmentator dataset, not for every Auto3DSeg pipeline. Even so, the study demonstrates that automated pipeline generation does not immunize downstream models against systematic annotation or training-data deficiencies.
6. Relation to foundation models and broader significance
Within MONAI, Auto3DSeg occupies a different role from newer segmentation foundation-model efforts. The VISTA3D paper describes Auto3DSeg as an automatic segmentation pipeline that self-configures deep learning models such as nnUNet and SegResNet for specific datasets and tasks, but also states that models produced by Auto3DSeg are task-specific and typically lack generalization to unseen classes and built-in interactive correction mechanisms (He et al., 2024). VISTA3D is positioned as complementary rather than substitutive: a single promptable model for 127 classes with interactive and zero-shot capabilities, whereas Auto3DSeg remains centered on task-specific automated model generation.
That distinction clarifies Auto3DSeg’s significance. Its major contribution is not zero-shot universality, but practical automation, reproducibility, and accessibility for high-performing supervised segmentation workflows. In BraTS 2023, this is explicitly framed as minimizing errors and manual intervention and facilitating reproducibility and access for non-experts; the same paper situates the approach within broader AutoML and auto-segmentation trends in medical imaging (Myronenko et al., 29 Oct 2025). The challenge reports on aorta, kidney, intracranial hemorrhage, and dental CBCT show that such automation can still be compatible with highly technical refinements, including adaptive normalization, ROI-focused multi-phase training, per-subregion checkpoint selection, and sophisticated ensembling (Myronenko et al., 2023, Myronenko et al., 2023, Siddiquee et al., 2022, LaBella et al., 18 Aug 2025).
A plausible implication is that Auto3DSeg has become an infrastructural layer for segmentation research rather than merely a convenience tool. It provides strong, reproducible baselines with little user intervention, yet it also serves as a scaffold for expert customization when default recipes are insufficient. The published record shows both sides of that role: challenge-winning performance in multiple domains and transparent exposure of failure modes when training data, anatomical conventions, or dataset size limit what automated configuration alone can achieve.