3D U-NetR: Residual 3D Architectures

• 3D U-NetR is a volumetric network architecture with an encoder–decoder structure enhanced by residual and recurrent connections, boosting gradient flow and enabling deeper models.
• It leverages skip connections and 3D convolutions to capture full spatial context, proving effective in medical segmentation, CT/MRI reconstruction, and robotic affordance learning.
• Multiple instantiations of 3D U-NetR have achieved state-of-the-art metrics by improving thin-structure recall and enhancing consistency in sequential and volumetric data analyses.

3D U-NetR refers to a class of three-dimensional U-Net architectures augmented with residual (and in some variants, recurrent) connections. This architecture generalizes the standard U-Net framework to full volumetric (3D) inputs and is most commonly employed in medical image analysis, volumetric segmentation, CT/MRI reconstruction, semantic scene completion, and robot affordance learning. Distinct 3D U-NetR instantiations have been proposed across several domains; these share key neuroanatomical encoder–decoder backbones with skip connections, but modify their convolutional modules to incorporate residual learning, recurrent modules, or both, boosting gradient flow and enabling deeper networks.

1. Architectural Foundation

3D U-NetR architectures implement a fully volumetric encoder–decoder with symmetric skip concatenations and residual modules at each resolution. The baseline 3D U-Net encoder pathway alternates two (or more) 3D convolutions (typically 3×3×3) with downsampling (via strided convolution or 2×2×2 max-pooling), doubling the feature map count at each level. The decoder up-samples via transposed 3×3×3 or trilinear interpolation, concatenates the corresponding encoder features, then applies convolution(s). Residual variants replace plain conv-norm-activation blocks with “residual blocks”: convolutional layers with batch/instance/group normalization and nonlinearity, with shortcut (identity or projection) summations. Pre-activation and post-activation block orderings are both in use across published works (Isensee et al., 2019, Li et al., 2020, Rassadin, 2020).

The defining architectural innovations are as follows:

• Volumetric Processing: Consistently applies 3D convolutions, capturing context in all spatial axes.
• Residual Learning: Encoder (and for some variants decoder) blocks sum their output with an identity or projected shortcut. This modification, verified across medical image segmentation and grasp affordance prediction, improves gradient propagation, model convergence, and enables deeper (5+ layer) encoders without training instabilities.
• Skip Connections: Classic U-Net long-range skip connections concatenate encoder outputs to decoder stages at matching spatial scales, promoting fine structure recovery.
• Recurrent Extensions: Some models embed recurrent residual units, in which convolutional feature maps are processed iteratively via 3D convolutions and summed over recurrence depth, further expanding the receptive field per block and enforcing spatio-temporal consistency for sequential tasks (Kadia et al., 2021, Cao et al., 2024).

2. Mathematical Formalism

Let input volume $$I \in \mathbb{R}^{X \times Y \times Z \times C_{in}}$$. The network comprises a stack of encoding functions $E_l$ and decoding functions $D_l$, possibly recurrent and/or residual. A typical encoder level is:

$$E_l(x) = \mathrm{ResBlock}_l\left(\mathrm{Down}(x)\right)$$

where $\mathrm{Down}$ is either max-pooling or strided 3D convolution.

The residual block at level $l$, assuming post-activation, is implemented as: $h_1 = \phi(\mathrm{Norm}_1(\mathrm{Conv}_1(x)))$

$h_2 = \mathrm{Norm}_2(\mathrm{Conv}_2(h_1))$

$y = \phi\left(x + h_2\right)$

where $\phi$ is a (Leaky)ReLU or ELU, and $\mathrm{Norm}$ is BatchNorm, GroupNorm, or InstanceNorm depending on the implementation (Isensee et al., 2019, Rassadin, 2020).

For recurrent residual variants (e.g., R2U3D (Kadia et al., 2021)), the recurrent residual unit at each level operates as: $$H_{t} = \phi \left( W_{x}\ast X + W_{h}\ast H_{t-1} + b\right),\quad H_{0}=0, t=1,\dots,T$$

$Y = X + H_T$

Decoders mirror the encoder but use upsampling (transposed conv or trilinear), channel halving, and re-application of residual/convolutional blocks, plus concatenation of the encoder skip activations. Output layers often consist of $1\times1\times1$ convolution and sigmoid (segmentation, affordance mapping) or softmax (semantic completion) heads.

3. Task-Specific Instantiations and Loss Functions

3.1 Segmentation and Super-Resolution

In root MRI segmentation (Zhao et al., 2020), 3D U-NetR is optimized with (weighted) binary cross-entropy, with plant root voxels upweighted (e.g. $w_r=10$) to enhance thin structure recall. Masked loss variants exclude root-soil borders, modestly improving F1 at the expense of anatomically implausible thickening. Super-resolution arises natively: the decoder upsamples input volumes and segmentations by a factor of 2 in each axis, supervised via high-resolution ground truth in a one-shot regime.

3.2 Joint Segmentation and Classification

Lung nodule analysis (Rassadin, 2020) implements a joint multi-task objective: $L = L_{dice} + \lambda\,L_{CE}$ with Dice loss for segmentation and weighted categorical cross-entropy for texture classes, $\lambda=0.2$. Segmentation and classification heads branch at the bottleneck, with global average pooling and MLP for texture prediction.

3.3 Reconstruction

For low-dose CT reconstruction (Gunduzalp et al., 2021), the 3D U-NetR architecture is tasked to map FBP reconstructions $X$ to denoised targets $Y$ with mean absolute error loss

$$\mathcal{L}_{L_1}(\hat{Y}, Y) = \frac{1}{N} \sum_{i=1}^N |\hat{Y}_i - Y_i|$$

Optionally, MSE and SSIM are jointly optimized to enforce local structural recovery.

3.4 Sequential Scene Completion (Recurrent U-NetR)

Sequence modeling is tackled in SLCF-Net (Cao et al., 2024) via a 3D U-NetR module injected with an aligned hidden-state volume. At each timestep: $(Y_t, H_t) = \mathrm{U\text{-}NetR}(X_t, H_{t-1})$ where $X_t$ is the frame-specific feature grid and $H_{t-1}$ is propagated via pose alignment. Standard U-NetR encoder–decoder processes the concatenated input. The loss comprises per-voxel cross-entropy, geometric regularizers, and an inter-frame pseudo-label consistency term.

4. Training Protocols and Augmentation

Protocols are domain-dependent but share key features:

• Patch-based training: Volumetric crops (e.g., $60^3$, $80\times160\times160$, $128^3$) are sampled to fit GPU memory.
• Augmentation: Includes axis-aligned flips, 90°/180°/270° rotations, elastic deformations, and intensity rescaling (for MR/CT). Some works (e.g., (Zhao et al., 2020, Rassadin, 2020)) report augmentation boosts of IoU/Dice by 0.05–0.06.
• Optimization: Adam or SGD optimizers predominate. LR schedules and early-stopping tactics are often reported. Deep supervision and composite loss functions (Dice + cross-entropy) are common in segmentation.
• Batch size: Typically small (1–4) due to volumetric memory demands.

5. Quantitative Performance Across Domains

Application	Test Metric	Baseline	3D U-NetR Result	Paper
Plant root MRI segmentation	F1(3 voxel tolerance)	0.94	0.964 (w/ data/loss)	(Zhao et al., 2020)
Lung nodule segmentation/challenge	Composite Dice	91.19% (plain UNet)	91.54% (residual)	(Isensee et al., 2019)
Graspable part detection	Val IoU	71.4% (plain)	77.6% (Res-U-Net)	(Li et al., 2020)
Low-dose CT reconstruction	SSIM %, PSNR dB (real)	74.58, 31.57	76.31, 32.29	(Gunduzalp et al., 2021)
3D lung segmentation	Soft-DSC (LUNA16)	—	0.9859 (R2U3D-Dyn.)	(Kadia et al., 2021)

Performance consistently shows residual 3D U-Net delivers modest or state-of-the-art gains over classical U-Net for both segmentation and volumetric regression tasks, with greatest impact observed in thin-structure recall, graspable part detection, and sequential semantic volumetric tasks.

6. Extensions: Recurrent, Pre-activation, and SE Modules

Several lines of work build upon the base 3D U-NetR:

• Recurrent Residual Units: R2U3D (Kadia et al., 2021) and 3D U-NetR in SLCF-Net (Cao et al., 2024) employ block-level or network-level recurrence, enhancing temporal and incremental 3D feature aggregation. R2U3D demonstrates further strength in minimal-data regimes (Soft-DSC up to 0.992 on VESSEL12 with 100 scans only).
• Pre-activation Residuals: Pre-activation layouts, positioning normalization and nonlinearity before convolution, provide minor empirical gains but have not yielded large benchmark improvements (Isensee et al., 2019).
• Squeeze-and-Excitation (SE): Channel-wise attention (SENet blocks) incorporated in R2U3D-Dynamic selectively recalibrates filter responses, further sharpening performance for 3D lung segmentation (Kadia et al., 2021).
• Global Residuals: Materialized in (Gunduzalp et al., 2021) as a skip sum of the input volume to the output, enhancing reconstruction fidelity in regression tasks.

7. Limitations, Observed Trends, and Application Landscape

While 3D U-NetR variants consistently provide improvements in challenging volumetric settings—especially for super-resolved, low-SNR, or thin-structure preservation—reported performance margins over plain 3D U-Net are dataset- and task-dependent. In some medical datasets with high structure and clean annotations, residual and pre-activation blocks yield only ~0.3–0.6% composite Dice gains (Isensee et al., 2019). Greatest benefits are reported when capturing faint, slender, or ambiguous features (e.g., roots, fine vessels, grasp handles), or when bridging inter-frame contextual gaps in 4D data (sequential scene completion).

A plausible implication is that 3D U-NetR benefits scale with data complexity and richness of context, with diminishing returns in highly redundant, well-structured domains.

• (Zhao et al., 2020) 3D U-Net for Segmentation of Plant Root MRI Images in Super-Resolution
• (Li et al., 2020) Learning to Grasp 3D Objects using Deep Residual U-Nets
• (Isensee et al., 2019) An attempt at beating the 3D U-Net
• (Rassadin, 2020) Deep Residual 3D U-Net for Joint Segmentation and Texture Classification of Nodules in Lung
• (Kadia et al., 2021) R2U3D: Recurrent Residual 3D U-Net for Lung Segmentation
• (Gunduzalp et al., 2021) 3D U-NetR: Low Dose Computed Tomography Reconstruction via Deep Learning and 3 Dimensional Convolutions
• (Cao et al., 2024) SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net