SegResNet: Residual Encoder–Decoder Segmentation

• SegResNet is a residual encoder–decoder segmentation architecture that employs multiscale skip connections and residual units to enhance training stability and feature preservation.
• It supports diverse 2D and 3D implementations across modalities such as MRI, CT, and PET/CT, and can be extended with deep supervision, attention, or prompt conditioning.
• The architecture demonstrates competitive Dice scores and efficiency in various medical imaging tasks while being adaptable for low-resource training and real-time deployment.

SegResNet is a residual encoder–decoder segmentation architecture that, in recent arXiv literature, is most often instantiated as a U-Net-like network in which encoder and decoder stages are built from residual units and linked by multiscale skip connections. Contemporary usage treats it both as a canonical MONAI backbone and as a flexible design family that can be deployed in 2D or 3D, trained on MRI, CT, PET/CT, CBCT, and IVOCT, and extended with deep supervision, attention, prompt conditioning, or low-bit deployment schemes (Pandya et al., 5 May 2026, Pajouh et al., 23 Oct 2025, Stenhede et al., 19 Sep 2025).

1. Architectural identity

Across the cited studies, SegResNet is consistently described as a residual encoder–decoder network for dense segmentation. In its recurrent 3D MONAI-style form, the encoder has four stages with residual-block counts $(1,2,2,4)$, the decoder has three stages with $(1,1,1)$, the initial number of filters is 16, dropout is 0.2, downsampling is performed by stride-2 convolutions, and the final head is a $1\times1\times1$ convolution mapping the last decoder features to class logits (Pandya et al., 5 May 2026, Pajouh et al., 23 Oct 2025, Bayer et al., 19 Mar 2026). Several papers explicitly frame this as a residual U-Net variant in which skip connections preserve fine spatial detail while residual units improve optimization stability in deeper volumetric models (Pandya et al., 5 May 2026, Dzikunu et al., 4 Feb 2025).

The internal definition of a residual unit is not completely uniform across implementations. One reported MONAI configuration uses 3D convolutions with $3\times3\times3$ kernels, InstanceNorm3d, and ReLU within residual units (Pandya et al., 5 May 2026). Other studies instead specify Group Normalization with ReLU after each 3D convolution, again within a residual encoder–decoder topology (Dzikunu et al., 4 Feb 2025, Bayer et al., 19 Mar 2026). In 2D IVOCT and 3D head-and-neck MRI, residual blocks are similarly described as stacks of normalization, ReLU, and convolution, followed by identity addition (Lee et al., 2023, Tie et al., 2024). This indicates that SegResNet is defined more by residual encoder–decoder structure than by a single immutable normalization or activation choice.

The architecture is also not confined to one depth. Five-stage encoders with convolutional-block counts $(1,2,2,4,4)$ were used for CBCT tooth segmentation and MRI-based pancreatic tumor segmentation (LaBella et al., 18 Aug 2025, Jha et al., 28 Aug 2025). A deeper six-level configuration with residual-block counts $(1,2,2,4,4,4)$ was used for longitudinal gross tumor volume segmentation in MRI-guided adaptive radiotherapy (Tie et al., 2024). At the other extreme, SegResNet has been used slice-wise in 2D for prostate CDI$^s$ segmentation and intracranial hemorrhage segmentation, where the volumetric formulation was replaced by a 2D residual encoder–decoder operating on resized slices or anisotropic axial CT slices without resampling (Dewbury et al., 15 Jan 2025, Siddiquee et al., 2022).

2. Variants and extensions

Recent work often uses SegResNet as a baseline that is then altered at skip connections, in the bottleneck, or in the supervision pathway. In meningioma segmentation, a baseline SegResNet uses additive skip fusion, whereas an attention-augmented variant inserts attention gates at every skip pathway and replaces summation by channel-wise concatenation. The gating formulation projects encoder features $x$ and decoder features $g$ into a shared space,

$$\tilde{x} = W_x * x + b_x, \qquad \tilde{g} = W_g * g + b_g,$$

then computes

$(1,1,1)$0

and reweights encoder features as

$(1,1,1)$1

before concatenating them with decoder features (Pajouh et al., 23 Oct 2025). The stated motivation is increased representational flexibility relative to element-wise skip addition.

Other extensions attach additional training heads or regularizers. In the BraTS 2023 setting, SegResNet was trained both with and without a VAE branch, and also with deep supervision applied at decoder stages with lower-resolution losses down-weighted by a factor of $(1,1,1)$2 per stage (Maani et al., 2024). Deep supervision also appears in slice-wise intracranial hemorrhage segmentation and in longitudinal head-and-neck MRI segmentation, where losses are applied at multiple output scales rather than only at the full-resolution head (Siddiquee et al., 2022, Tie et al., 2024).

A more structural alteration appears in longitudinal MRI-guided adaptive radiotherapy. There, a mask-aware SegResNet inserts attention modules at each encoder level so that deformably registered pre-radiotherapy GTV masks modulate mid-radiotherapy features. Channel attention is computed from pooled feature descriptors, while spatial attention concatenates pooled feature maps with downsampled prior masks; the refined tensor is fused residually as $(1,1,1)$3 (Tie et al., 2024).

SegResNet can also function as the encoder of a larger hybrid system. ENSAM uses a SegResNet-style encoder with four downsamplings, starting at 16 channels and doubling at each stride, then injects prompts at the bottleneck by latent cross-attention with 3D relative positional encoding and normalized-attention updates. In that study the full interactive model has 5.5 million parameters and 368 GFLOPs per forward pass at maximum patch volume (Stenhede et al., 19 Sep 2025). This suggests that, in current literature, SegResNet is often treated less as a closed architecture than as a reusable residual feature hierarchy.

3. Training regimes, losses, and inference practice

Training practice around SegResNet is heterogeneous and strongly task-dependent. In brain MRI, one study trained MONAI SegResNet on BraTS 2021 with four input channels (FLAIR, T1, T1Gd/T1wCE, T2), Dice loss, Adam, learning rate $(1,1,1)$4, L2 weight decay $(1,1,1)$5, automatic mixed precision for training and validation, batch size 1, and 10 epochs, after reorienting images to RAS, applying modality-specific normalization, and cropping to $(1,1,1)$6 (Pandya et al., 5 May 2026). In BraTS-MEN 2025, SegResNet variants used four co-registered MRI modalities, center crops of $(1,1,1)$7, subject-wise z-score normalization, AdamW with AMSGrad, batch size 1, 20 epochs, and a hybrid loss with Dice and Focal terms weighted by $(1,1,1)$8 and $(1,1,1)$9 (Pajouh et al., 23 Oct 2025).

Other domains use different losses entirely. Prostate CDI$1\times1\times1$0 segmentation employed Binary Cross Entropy with Logits, Adam, StepLR, batch size 16, and 200 epochs in a 2D slice-wise setup (Dewbury et al., 15 Jan 2025). Whole-body lymphoma PET/CT segmentation used binary Dice loss with AdamW, cosine annealing from $1\times1\times1$1 to 0 over 500 epochs, 3D patches of size 192, and lesion-centric crop sampling with probabilities $1\times1\times1$2 and $1\times1\times1$3 (Ahamed et al., 2023). In PSMA PET/CT, SegResNet was paired with Dice Loss, Dice Cross Entropy, Dice Focal Loss, and L1-weighted Dice Focal Loss, with patch size $1\times1\times1$4, AdamW, cosine annealing from $1\times1\times1$5 to 0 over 1000 epochs, and 80% foreground-centered cropping (Dzikunu et al., 22 Apr 2025, Dzikunu et al., 4 Feb 2025).

A common metric family is Dice-based. The set-form coefficient repeatedly used in these studies is

$1\times1\times1$6

with soft or multi-class generalizations used in optimization (Pajouh et al., 23 Oct 2025, Pandya et al., 5 May 2026). However, the literature does not converge on a single “SegResNet loss”: Dice-only, Dice+Cross-Entropy, Dice+Focal, BCE-with-logits, and task-specific weighted losses all appear.

Inference is likewise varied. Sliding-window inference is common in volumetric tasks, including BraTS, lymphoma PET/CT, and head-and-neck MRgART (Maani et al., 2024, Ahamed et al., 2023, Tie et al., 2024). Ensemble strategies include majority voting over discrete outputs, mean probability averaging across folds or models, and STAPLE-based fusion across cross-validation models (Pajouh et al., 23 Oct 2025, Maani et al., 2024, LaBella et al., 18 Aug 2025). Postprocessing ranges from none, to connected-component filtering, to lesion-size and mean-probability gating tuned to challenge metrics (Maani et al., 2024, Hresko et al., 2022).

4. Application breadth and reported behavior

In neuro-oncology MRI, SegResNet has been repeatedly competitive. On BraTS 2021, one SegResNet study reported validation mean Dice 0.84 with class-wise Dice 0.84 for tumor core, 0.90 for whole tumor, and 0.79 for enhancing tumor; a separate test report in the same paper gave mean Dice 0.86 with class-wise values 0.86, 0.92, and 0.81 for tumor core, whole tumor, and enhancing tumor, respectively (Pandya et al., 5 May 2026). In meningioma segmentation on BraTS-MEN 2025, standalone validation lesion-wise Dice reached ET 73.9%, TC 72.4%, WT 63.29% for baseline SegResNet and ET 73.3%, TC 75.7%, WT 67.8% for the attention-augmented variant, while the three-model ensemble containing both SegResNet variants achieved test lesion-wise Dice of 77.30% for ET, 76.37% for TC, and 73.92% for WT (Pajouh et al., 23 Oct 2025). In BraTS 2023 adult glioma, a single SegResNet without deep supervision achieved validation mean Dice 0.8643 and mean HD95 18.10; in pediatric tumors, ET-only specialization improved the validation mean Dice of the SegResNet ensemble to 0.7204 and the hidden-test mean Dice to 0.6992 (Maani et al., 2024).

Outside brain MRI, SegResNet has shown strong results on heterogeneous abdominal CT. In the RATIC benchmark, the CNN-based SegResNet achieved spleen 0.919, right kidney 0.954, left kidney 0.960, liver 0.970, bowel 0.923, and an average DSC of 0.945, outperforming UNETR, SwinUNETR, and UNETR++ under identical preprocessing and training conditions (Bayer et al., 19 Mar 2026). In prostate CDI$1\times1\times1$7 segmentation, a 2D MONAI SegResNet achieved DSC $1\times1\times1$8, the best among the compared models in that study (Dewbury et al., 15 Jan 2025).

In PET/CT lesion segmentation, performance depends not only on overlap but also on downstream quantification. For lymphoma PET/CT, SegResNet achieved internal-test mean DSC $1\times1\times1$9 and median DSC $3\times3\times3$0, with external-test mean DSC $3\times3\times3$1 and median DSC $3\times3\times3$2; it also had the lowest false-positive volume among the compared networks on both internal and external evaluation (Ahamed et al., 2023). In PSMA PET/CT, one study reported for SegResNet with L1DFL a mean Dice of $3\times3\times3$3, median Dice $3\times3\times3$4, false positives $3\times3\times3$5, and F1 $3\times3\times3$6, while a related study on the same clinical setting reported median DSC 0.68 and F1 0.66 for SegResNet with L1DFL, with statistically significant improvement over Dice Focal Loss for the DSC distribution (Dzikunu et al., 22 Apr 2025, Dzikunu et al., 4 Feb 2025).

The architecture has also been used far beyond standard radiology segmentation. In CBCT tooth segmentation, a lightweight Auto3DSeg/SegResNet pipeline with five-fold STAPLE fusion achieved an average Dice of 0.87 on the ToothFairy3 out-of-sample validation set (LaBella et al., 18 Aug 2025). In longitudinal head-and-neck MRI, an ensemble of 10 SegResNet-based models achieved average DSCagg 0.794 for pre-radiotherapy segmentation and 0.733 for mid-radiotherapy segmentation on the challenge test set (Tie et al., 2024). In intravascular OCT fibrous-cap segmentation, a modified 2D SegResNet reached five-fold Dice $3\times3\times3$7, held-out test Dice 0.816, and inference time 0.02 s per frame (Lee et al., 2023).

5. Efficiency, deployment, and systems considerations

A recurrent reason for choosing SegResNet is computational practicality. In the BraTS-MEN study, the baseline model was described as “quite light,” each component model was trained for only 20 epochs on a single GTX1080 with 8 GB VRAM and batch size 1, and the full training was said to complete in “one to two days” (Pajouh et al., 23 Oct 2025). In BraTS 2021, automatic mixed precision enabled training on $3\times3\times3$8 patches with batch size 1 on a single Tesla P100, with the paper attributing improved efficiency and reduced memory usage to AMP (Pandya et al., 5 May 2026).

Low-resource efficiency is also visible in architecture-level redesigns. ENSAM keeps a SegResNet-style CNN encoder specifically because it can be trained from scratch on under 5,000 multimodal 3D volumes on a single 32 GB GPU in approximately 6 hours; the resulting promptable model remains lightweight relative to many foundation-style alternatives (Stenhede et al., 19 Sep 2025). Slice-wise 2D usage is another efficiency pathway: for intracranial hemorrhage segmentation on anisotropic CT, a 2D SegResNet ensemble trained through Auto3DSeg achieved the top challenge Dice of 0.721 while avoiding resampling and exploiting large 2D batch sizes (Siddiquee et al., 2022).

SegResNet has also been examined from a deployment-engine perspective rather than only a training perspective. In a TensorRT post-training quantization study on TotalSegmentator V2, FP32 SegResNet achieved mean Dice 0.882, whereas the real INT8 engine achieved 0.879; engine size decreased from 42.40 MB to 17.48 MB, and inference latency decreased from 5.14 ms to 2.06 ms, corresponding to an approximately 2.49× speedup (Qu et al., 28 Jan 2025). These results indicate that SegResNet is not only trainable under constrained hardware, but also amenable to real low-bit deployment with minimal reported accuracy loss.

6. Limitations, misconceptions, and open directions

A common misconception is to treat SegResNet as a single fixed blueprint. The recent literature instead uses 2D slice-wise, 3D volumetric, deep-supervised, VAE-regularized, attention-gated, prompt-conditioned, and mask-aware variants, with differing stage counts, skip-fusion rules, normalization layers, and losses (Siddiquee et al., 2022, Maani et al., 2024, Stenhede et al., 19 Sep 2025). This suggests that “SegResNet” now functions as an architectural family centered on residual encoder–decoder segmentation rather than as one invariant graph.

Another recurring limitation is that favorable overlap scores do not automatically imply clinically faithful quantification. In PSMA PET/CT, SegResNet-based segmentations could show acceptable Dice while tumor volume and lesion spread remained variable, and high concordance in SUV-related metrics did not extend uniformly to TMTV or $3\times3\times3$9 (Dzikunu et al., 22 Apr 2025). In lymphoma PET/CT, SegResNet reproduced SUVmean and lesion count more reliably than $(1,2,2,4,4)$0, and performance deteriorated on small, faint, or highly disseminated disease (Ahamed et al., 2023).

Failure modes are strongly domain specific but structurally similar across studies. Small or subtle enhancing tumor regions remain difficult in brain MRI; pediatric ET segmentation, very small lesions, and domain shift are repeatedly reported as challenging (Pajouh et al., 23 Oct 2025, Maani et al., 2024). Small tubular nerves are poorly captured at coarse CBCT resolution and required a second native-resolution phase (LaBella et al., 18 Aug 2025). MRI-based pancreatic tumor segmentation remained modest, with DSC 0.56 on diagnostic T1-weighted arterial contrast-enhanced MRI and 0.33 on T2-weighted MR-Linac data, illustrating sensitivity to sequence-dependent domain shift and limited sample size (Jha et al., 28 Aug 2025). In slice-wise hemorrhage segmentation, the final challenge submission achieved the top Dice but had HD = inf, indicating outlier boundary or false-positive behavior under surface-distance metrics (Siddiquee et al., 2022).

The proposed future directions are correspondingly practical rather than doctrinal. The cited studies call for better handling of class imbalance and small lesions, richer or more tailored augmentations, diversity-aware ensembling and calibration, analysis of normalization and activation choices under batch size 1, stronger postprocessing for lesion-wise metrics, domain adaptation across scanners and sequences, and incorporation of prior masks or interactive prompts when longitudinal or user-guided information is available (Pajouh et al., 23 Oct 2025, Maani et al., 2024, Tie et al., 2024, Jha et al., 28 Aug 2025). Within this landscape, SegResNet remains important less because it is uniquely standardized than because it is repeatedly adaptable, computationally tractable, and empirically competitive across a wide range of medical segmentation regimes.