Unpaired 3D Brain Tumor Synthesis

• The paper introduces unpaired 3D brain tumor synthesis that leverages unpaired healthy MRIs and limited annotated tumor scans to generate realistic synthetic volumes.
• Two-stage frameworks, TF and TSGM, combine coarse augmentation with GAN and diffusion-based refinements to ensure anatomical accuracy and label consistency.
• Empirical evaluations show improved segmentation performance with higher Dice scores, highlighting the method's scalability and potential in clinical imaging.

Unpaired 3D brain tumor synthesis denotes the class of frameworks that generate synthetic three-dimensional MRI brain volumes containing tumors, without the need for paired healthy–tumor subject scans. These approaches aim to mitigate data scarcity in annotated tumor imaging by leveraging abundant unlabelled healthy MRI volumes and only a limited set of real annotated tumor scans. The principal methodologies involve two-stage frameworks combining augmentation, generative adversarial networks (GANs), and diffusion models. Notable systems are Tumor Fabrication (TF) (Dong et al., 23 Nov 2025) and Two-Stage Generative Model (TSGM) (Wang et al., 2023).

1. Motivation and Conceptual Foundations

Manual synthesis of brain tumors within MRI requires extensive expert input and fails to capture the statistical diversity and anatomical plausibility required for downstream deep learning applications. Deep generative models present a principled data-driven alternative; however, classical approaches such as pix2pix demand large paired datasets, which are infeasible under standard clinical constraints. Unpaired synthesis methods address this by decoupling anatomical background from pathological content, enabling scalable enrichment of supervised segmentation pipelines. The foundational principle is to generate realistic, anatomically-correct, and label-consistent tumor-bearing MRIs using only unpaired healthy images and a small set of labeled tumor cases (Dong et al., 23 Nov 2025, Wang et al., 2023).

2. Two-Stage Frameworks for Unpaired 3D Synthesis

Both TF and TSGM employ two-stage designs to disentangle generation and refinement or anomaly localization. The process can be summarized as follows:

Framework	Stage I	Stage II
TF	Coarse mask-based tumor synthesis	Refinement using a GAN with multiple loss terms
TSGM	CycleGAN-based healthy↔tumor translation	Diffusion-based conditional image reconstruction

TF: Coarse Synthesis and Refinement

• Stage I (TF-Aug): Synthetic tumor masks $m_s$ are generated from real annotated masks via random spatial augmentations (scale, rotation, translation, and combinatorial overlays). These masks are applied to healthy scans $\mathbf{x}_h$ using region-of-interest (ROI)-guided augmentation—Gaussian blurring inside masks and per-class learnable linear intensity transforms—producing coarse synthetic images $\mathbf{x}_{s'}$.
• Stage II (TF-GAN): The coarse image–mask pair $(\mathbf{x}_{s'}, m_s)$ is refined by a 3D U-Net GAN. The generator receives the concatenated pair and outputs a realistic tumor image $\mathbf{x}_s$. Losses include adversarial, class-wise perceptual (using a frozen segmentation encoder), and a hinge reconstruction term ensuring non-ROI anatomical consistency.

TSGM: Adversarial Translation and Diffusion-Driven Reconstruction

• Stage I (CycleGAN): A CycleGAN is trained for unpaired translation between healthy and pathological domains, mapping healthy MRIs to synthetic tumor-bearing MRIs without requiring correspondence. Losses comprise adversarial, cycle-consistency, and identity mapping constraints.
• Stage II (VE-JP): A joint-probability score-based diffusion model is trained to reconstruct healthy images from their synthetic tumorized versions, based on a variance-exploding stochastic differential equation framework. Only pathological regions are altered, with the reconstructed–input residual providing precise anomaly segmentation.

3. Technical Workflows and Model Architectures

TF Architecture and Training

• Generator: 3D U-Net with six resolution scales; upsampling via nearest-neighbor interpolation plus 3×3×3 convolution to avoid checkerboard artifacts.
• Discriminator: Five-stage 3D convolutional encoder with both PatchGAN and global discrimination heads.
• Losses:
  • Hinge reconstruction on non-tumor regions to enforce anatomical preservation outside synthetic tumors,
  • Class-wise ROI-guided perceptual loss via frozen U-Net features,
  • Patch and global adversarial losses,
  • Summed as

  $$\mathcal{L}_{\text{total}} = \lambda_a\,\mathcal{L}_{\text{adv}}^{\text{patch}} + \lambda_b\,\mathcal{L}_{\text{adv}}^{\text{global}} + \lambda_c\,\mathcal{L}_{\text{hinge}} + \lambda_d\,\mathcal{L}_{\text{percep}}$$

• Training: Stage I—200 epochs with SGD; Stage II—200 epochs, Adam optimizer, patch size $128^3$, learning rate scheduling; all data skull-stripped, registered, and intensity-scaled.

TSGM Architecture and Training

• Stage I (CycleGAN): 3D ResNet generator (9 blocks), 3D PatchGAN discriminator, instance normalization.
• Stage II (VE-JP): 3D U-Net-style score network with “BigGAN residual” blocks, group normalization, and FiLM conditioning on noise level. Joint probability conditioning achieved by concatenating the fixed “tumor” image to the input at all times.
• Training Losses: CycleGAN adversarial, cycle-consistency, and identity; diffusion denoising score matching as

$$\mathcal{L}_{\text{Stage2}}(\theta) = \frac{1}{2}\,\mathbb{E}_{i,X_0,z} \| \eta_i\, s_\theta(X_{i+1}, \sigma_i) + z\|_2^2$$

• Preprocessing: Brain extraction (HD-BET), registration to SRI24 template, intensity clipping and normalization, fixed 3D size.

4. Synthetic Data Generation, Post-processing, and Label Transfer

• TF workflow: Each synthetic volume $\mathbf{x}_s$ is paired with its mask $m_s$ from TF-Aug as the segmentation ground truth. Post-processing includes spatial smoothing and learned intensity alignment. Volumes can be generated at arbitrary scale (e.g., 50-300 pairs per experiment).
• TSGM workflow: Synthetic abnormal images resulting from CycleGAN are used as guides for joint-probability diffusion, and the difference between input and reconstructed healthy volumes localizes the synthetic tumor region.

5. Evaluation Protocols and Empirical Performance

Downstream Segmentation

• TF evaluation: nnU-Net (6-stage, 128³ input, 100 epochs) trained on either 100 real or real+synthetic pairs; performance assessed by Dice scores for enhancing tumor (ET), tumor core (TC), and whole tumor (WT).
• TF results: With 100 synthetic pairs, TF outperforms CarveMix and pix2pix: mean Dice for TF is $67.77 \pm 0.08$ vs. baseline $66.57 \pm 0.52$. Mean improvement is significant ($+1.20$, $p=0.02$).
• Ablation (TF): More synthetic data yields incrementally increased segmentation performance (e.g., 300 pairs lead to a mean Dice of $68.25 \pm 0.46$).
• TSGM evaluation: On BraTS2020, TSGM achieves DSC $0.8590$, outperforming CycleGAN-alone ($0.8458$); similar gains are noted on ICTS and in-house datasets.

Qualitative Results

• TF: GAN-refined volumes show improved texture, edge sharpness, and blending relative to coarse augmentation. Some limitations remain in edema realism and mass effect.
• TSGM: Stage 1 synthetic tumors are visually plausible; Stage 2 reconstructions maintain anatomical details outside pathological regions. The residual localization is anatomically precise.

6. Strengths, Limitations, and Future Directions

• TF advantages: Fully automated, scalable to large healthy cohorts, minimal expert input, and label-consistent; facilitates generation for rare and diverse tumor types. Absence of shape priors increases diversity through augmentation.
• TF limitations: Synthetic edema can be anatomically ambiguous, and lack of biomechanical modeling omits realistic mass-effect deformations.
• TSGM advantages: Modularity, use of joint-probability score-based diffusion, robust anomaly localization via residuals, and easy extension to other 3D modalities or organs.
• Common directions: Integration of biomechanical deformation (for mass-effect), multi-modality synthesis, richer attribute control, and improved modeling of diffuse/edematous regions (Dong et al., 23 Nov 2025, Wang et al., 2023).

7. Broader Implications and Clinical Applicability

Unpaired 3D tumor synthesis frameworks contribute substantially to medical imaging AI pipelines under data scarcity. The capacity to generate large, diverse, and realistic paired data from abundant healthy scans and minimal annotation can enhance model robustness, support domain adaptation for rare pathologies, and improve performance in low-data regimes, as shown by significant segmentation metric gains (Dong et al., 23 Nov 2025). These techniques are broadly applicable to other anatomical regions and pathologies given their reliance only on unpaired data and flexible, modular pipeline designs (Dong et al., 23 Nov 2025, Wang et al., 2023).