Perceptual-Drifting Hybrid Loss in 3D Imaging

• The paper introduces a cyclic 2.5D perceptual loss that sequentially applies voxelwise and perceptual metrics across axial, coronal, and sagittal planes, yielding improved PSNR and SSIM.
• The methodology combines 2D VGG-16 based feature extraction with MSE and SSIM losses, balancing fine-grained voxel accuracy with high-level semantic fidelity.
• The design leverages a decaying cyclic schedule and standardized preprocessing to robustly capture anatomical features, enhancing performance across diverse 3D medical synthesis models.

Perceptual-drifting hybrid loss is a loss function designed for cross-modal 3D medical image synthesis tasks, where accurate preservation of high-level semantic features across all anatomical planes is essential. It is characterized by the sequential application of a 2.5D perceptual loss, combined with MSE and SSIM voxelwise losses, using a cyclical schedule that alternates between axial, coronal, and sagittal planes with decreasing interval durations. This approach addresses challenges in balancing perceptual loss optimization across planes and leverages pre-trained 2D feature extractors, yielding improvements in both quantitative image similarity metrics and visual fidelity in diverse medical image synthesis models (Moon et al., 2024).

1. Mathematical Foundations of the Cyclic 2.5D Perceptual Loss

The cyclic 2.5D perceptual loss is defined for a pair of 3D volumes: prediction $\hat{y} \in \mathbb{R}^{H \times W \times D}$ and ground truth $y$. Let $\phi_j(\cdot)$ denote the feature map output at layer $j$ (specifically, $j=23$, conv4_3) of a 2D VGG-16 model pre-trained on ImageNet. The 2D perceptual loss for a set of $S$ slices along plane $p$ (axial, coronal, or sagittal) is:

$$L^{p}_{\text{perc}}(\hat{y}, y) = \frac{1}{S} \sum_{s=1}^S \|\phi_j(y_{p,s}) - \phi_j(\hat{y}_{p,s})\|^2_2$$

where $y_{p,s}$ and $\hat{y}_{p,s}$ are the single-channel ground truth and predicted slices for slice $y$0 in plane $y$1, repeated across three channels to match the required VGG input.

The key procedural innovation is the cyclic schedule for loss-plane selection:

• At each training epoch $y$2, only one orthogonal plane is used for the perceptual loss (axial, coronal, or sagittal).
• The schedule starts with an interval $y$3 per plane in cycle 1 and decays with factor $y$4 each cycle, down to a minimum $y$5. Within a cycle $y$6, plane selection is organized as:
  • Epochs $y$7: axial\ $y$8: coronal\ $y$9: sagittal

This schedule enacts a non-uniform, drifting focus, ensuring balanced feature learning across all planes while avoiding overfitting to a specific view. The full cyclic loss at epoch $\phi_j(\cdot)$0 is:

$\phi_j(\cdot)$1

2. Combined Perceptual-Drifting Hybrid Loss Function

The perceptual-drifting hybrid loss $\phi_j(\cdot)$2 blends voxelwise fidelity with perceptual similarity, combining MSE, SSIM, and the cyclic 2.5D perceptual term:

$\phi_j(\cdot)$3

• $\phi_j(\cdot)$4
• $\phi_j(\cdot)$5, as per Wang et al. (2004)

Empirically effective hyperparameters for the 2D/VGG16-based setting are $\phi_j(\cdot)$6, $\phi_j(\cdot)$7, $\phi_j(\cdot)$8. For the 3D MedicalNet variant: $\phi_j(\cdot)$9, $j$0, $j$1.

3. Training Algorithm and Drifting Schedule

Plane alternation is implemented by precomputing a per-epoch plane‑schedule:

$j=23$5

In each epoch, only the slices along the designated plane are used for the perceptual term. Slices are min-max normalized to [0,1], replicated to three channels, and processed through the truncated VGG-16 for feature map extraction. Early stopping is initialized only after three complete cycles to avoid premature convergence during single-plane transitions.

4. VGG-16 Feature Extractor and Data Handling

The perceptual term utilizes a 2D VGG-16 network (pre-trained on ImageNet), truncated after its 23rd layer (end of conv4_3), encompassing:

• conv1_1, conv1_2, pool1
• conv2_1, conv2_2, pool2
• conv3_1, conv3_2, conv3_3, pool3
• conv4_1, conv4_2, conv4_3

Input 2D slices are pre-processed by min-max normalization to [0,1] and channel replication. Feature maps $j$2 have $j$3, $j$4 the original spatial size, and pairwise Euclidean (ℓ₂) feature distances are averaged slice-wise. This design supports standardized feature comparison across medical modalities lacking large annotated 3D models.

5. Implementation Protocols and Hyperparameters

Preprocessing for T1w MRI employs N3 bias correction, Freesurfer intensity normalization, skull-stripping via SynthStrip, cropping/resampling to $j$5, and min-max scaling to $j$6. PET images undergo equivalent geometric processing, with by-manufacturer standardization (per-scanner mean-zero, unit variance) to enhance pathology “hot-spot” contrast and mitigate device variability. Data augmentation comprises 3D elastic deformation, affine transformations (rotation $j$7, scale $j$810%), random flipping, and MRI-only Gaussian noise.

Representative hyperparameters (2D perceptual/VGG16 scenario):

• Epoch interval $j$9, decay $j=23$0, $j=23$1
• Generator: 3D U-Net ($j=23$2 channels, instance norm, dropout 0.2 in bottleneck)
• Optimizer: Adam, learning rate $j=23$3 (U-Net) or $j=23$4 (GANs), cosine annealing for U-Net with period matching plane interval
• Batch size: 1 (full 3D volume)
• Early stopping: patience equals current plane interval, starts after three full triaxial cycles

The method is compatible with diverse models, including U-Net, UNETR, SwinUNETR, CycleGAN, and Pix2Pix.

6. Quantitative and Qualitative Performance

Evaluation on 516 paired MRI–PET (ADNI) samples demonstrates consistent improvement in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM):

Model	Baseline Loss	Baseline SSIM₃D	Baseline PSNR	+Cyclic2.5D SSIM₃D	+Cyclic2.5D PSNR	ΔSSIM	ΔPSNR
3D U-Net	MSE+SSIM+2.5D	0.897±0.037	28.18±2.83	0.900±0.036	28.73±2.63	+0.3%	+0.55
UNETR	MSE+SSIM+2.5D	—	—	+0.2–0.5%	+0.2–0.4
SwinUNETR	MSE+SSIM+2.5D	—	—	+0.2–0.5%	+0.2–0.4
Pix2Pix	MSE+SSIM+2.5D	0.861	—	0.886	+1.02	+2.9%	+1.02
CycleGAN	MSE+SSIM+2.5D	0.843	—	0.861	+0.98	+2.1%	+0.98

Qualitatively, the drifting schedule facilitates the learning of anatomical details in all three planes, reduces overfitting to individual views, and enhances high-contrast (“hot-spot”) tau uptake regions. Notable spikes in validation loss emerge at plane-switch boundaries but subside with successive cycles, paralleling the exploratory and fine-tuning behavior of cyclical learning-rate schedules.

7. Practical Summary and Significance

The perceptual-drifting hybrid loss enables robust, multi-planar semantic fidelity for 3D cross-modal image translation by:

1. Slicing volumes along three orthogonal planes;
2. Scheduling plane alternation with a fixed decaying-interval schedule;
3. Employing a frozen 2D perceptual backbone (VGG16, through conv4_3);
4. Combining MSE and SSIM voxel losses;
5. Utilizing standard preprocessing and augmentation routines.

This protocol is effective for a range of volume-to-volume synthesis architectures and yields reproducible gains in quantitative and qualitative outcomes for medical image translation, particularly where the preservation of high-level semantic features outweighs strict voxelwise alignment (Moon et al., 2024).