Make-A-Volume: 3D MRI Synthesis Framework

• Make-A-Volume is a diffusion-based generative framework for cross-modality 3D brain MRI synthesis that combines 2D-latent diffusion with sparse volumetric convolutions to ensure global anatomical coherence.
• The method overcomes the slice inconsistency of 2D GANs and the high resource demands of full 3D models by using a two-stage process with volumetric fine-tuning.
• Empirical evaluations show improved performance metrics (lower MAE, higher SSIM/PSNR) and practical benefits for clinical applications such as MRI post-processing and data augmentation.

Make-A-Volume denotes a diffusion-based generative framework for cross-modality 3D brain MRI synthesis that leverages a 2D-latent backbone with sparse volumetric convolutions, enabling computationally efficient, volumetrically consistent translation between contrasts such as T1↔T2 or SWI↔MRA in medical imaging. The method addresses the longstanding challenge of fusing slice-level generative capacity with global 3D coherence, while circumventing prohibitive memory requirements and instability issues endemic to adversarial and fully volumetric neural architectures (Zhu et al., 2023).

1. Problem Setting and Limitations of Prior Approaches

Cross-modality 3D MRI synthesis seeks to generate unavailable or corrupted MRI contrasts to provide comprehensive multi-contrast neuroimaging data. This is essential for downstream tasks in automated pipelines, radiation therapy planning, and algorithmic data augmentation. Conventional generative methods predominantly fall into two orthogonal regimes: slice-wise 2D GANs (e.g., Pix2pix, CycleGAN) and volumetric 3D GANs. While 2D methods operate on individual slices—resulting in high anatomical inconsistency across the orthogonal planes—3D models can enforce global spatial fidelity but at a severe computational and optimization overhead, including large GPU memory footprints, unstable training, and prominent risk of mode collapse. The Make-A-Volume approach was developed to combine the computational and statistical benefits of diffusion models with volumetric regularization, without resorting to full 3D convolutions or prohibitively expensive resources (Zhu et al., 2023).

2. Framework Architecture: Latent Diffusion and Volumetric Fine-Tuning

Make-A-Volume is architected as a two-stage generative pipeline:

Stage 1: Slice-wise Latent Diffusion

• An autoencoder $E(x), D(z)$ compresses 2D slices into a low-dimensional latent $z = E(x)$, reducing per-slice memory up to $16\times$ while maintaining semantic richness.
• A U-Net backbone is trained as a latent diffusion model to translate between modality latents, specifically by learning the conditional noise-prediction mapping

$$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$

where $z_t$ is the noisy target slice latent at noise step $t$ and $z_{ctr}$ is the clean latent for the source modality slice.

• The network operates on slices independently, with reconstruction of the target space performed after diffusion denoising.

Stage 2: Volumetric Layer Fine-Tuning

• To achieve 3D consistency, a subset of 1D convolutional layers ("volumetric layers") are inserted in the U-Net at points after encoder and before decoder blocks.
• Feature maps $$f \in \mathbb{R}^{(b_v \cdot n) \times c \times h \times w}$$ are reshaped such that the 1D convolution acts across the slice axis (of length $n$), then returned to the original tensor semantics.
• These volumetric layers are initialized as identities and fine-tuned end-to-end on paired 3D datasets.

This progression from 2D-latent translation to pseudo-3D consistency yields a memory-efficient generative process with robust intra-volume anatomical coherence.

3. Mathematical Formulation of Latent Diffusion and Training Objective

Forward (Noising) Process in Latent Space:

Let $z_0 \sim q(z)$ be a clean latent code for a target slice. The fixed variance schedule $z = E(x)$0 induces

$z = E(x)$1

and in closed form,

$z = E(x)$2

where $z = E(x)$3 with $z = E(x)$4.

Reverse (Denoising) Process:

The U-Net is trained to predict noise $z = E(x)$5 given $z = E(x)$6:

$z = E(x)$7

Training Loss:

The objective is a mean-squared error between the actual noise and the predicted noise:

$z = E(x)$8

This aligns (up to constants) with maximizing the variational lower bound for the diffusion process. Notably, all diffusion is performed in frozen latent space, avoiding the memory and compute intensities of pixel-space diffusion.

4. Volumetric Consistency and Coherence Enhancement

Volumetric layers implement 1D convolutions across the slice axis at selected U-Net locations. Their explicit updates are:

1. Reshape features $z = E(x)$9 from $16\times$0 to $16\times$1.
2. Apply a kernel of width $16\times$2 along slices.
3. Reshape back for subsequent 2D processing.

During fine-tuning, all slices in a volume share the same noise step $16\times$3, sampled with emphasis on later timesteps, to enhance inter-slice coherence, especially at low-noise (higher-fidelity) stages. No additional volumetric regularization loss is used; pseudo-3D convolutions suffice for implicit consistency.

5. Datasets, Training Regime, and Computational Considerations

The main datasets are:

• SWI$16\times$4MRA (in-house): 111 paired volumes ($16\times$5 mm SWI, $16\times$6 mm MRA), each resampled to $16\times$7 voxels.
• T1$16\times$8T2 (RIRE public): 17 volumes, resampled to $16\times$9.

Core hyperparameters:

• Autoencoder downsampling $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$0.
• Diffusion steps $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$1 (linear $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$2 schedule).
• Adam, learning rate $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$3.
• Batch size: 16 slices (slice-wise training), 4 volumes (volume-wise fine-tuning, 25–35 slices per volume).

Resource usage:

• Training on an NVIDIA A100 80GB GPU: slice-wise stage converges in $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$448h, volumetric fine-tuning in $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$54h.
• Peak memory is $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$6 of a naive 3D diffusion implementation.

6. Quantitative and Qualitative Evaluation

Key metrics are Mean Absolute Error (MAE, $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$7), Structural Similarity Index (SSIM, $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$8), and Peak Signal-to-Noise Ratio (PSNR, $$\epsilon_\theta : (z_t, t, z_{ctr}) \rightarrow \mathbb{R}^{c\times h \times w}$$9). Below is a summary table for the two main cross-modal translation tasks:

Method	MAE (S/M)	SSIM (S/M)	PSNR (S/M)
Pix2pix 2D	8.17/16.81	0.739/0.538	25.7/20.1
Pix2pix 3D	6.23/11.37	0.765/0.650	28.4/22.9
Ours (200 steps)	5.24/10.79	0.788/0.676	29.45/24.33
Ours (1000 steps)	4.80/10.62	0.801/0.684	30.14/25.46

S: SWI$z_t$0MRA, M: T1$z_t$1T2

Qualitatively, conventional 2D GANs produce jagged, spatially inconsistent outputs in coronal/sagittal reconstructions, while Make-A-Volume yields vessels and anatomy that are smooth and plausible in all planes. Ablation studies highlight that without volumetric layers, inter-slice flicker and metric drops are observed (MAE: 5.13/10.93, SSIM: 0.792/0.667, PSNR: 29.89/24.62 versus improved metrics with volumetric fine-tuning).

7. Strengths, Limitations, and Application Domains

Make-A-Volume uniquely fuses the computational efficiency and stability of diffusion models with explicitly encoded volumetric regularization, achieving state-of-the-art MRI synthesis without relying on full 3D convolutions or adversarial training dynamics. Its memory footprint is orders-of-magnitude smaller than volumetric GAN/diffusion approaches, yet it sidesteps slice-inconsistency inherent to purely 2D pipelines. The framework, however, requires paired 3D data for volumetric fine-tuning, and sampling speed (1–2 s/volume) may not be optimal for time-constrained scenarios. Downstream applications include synthesis of missing modalities in clinical post-processing pipelines, MRI-only radiotherapy, and augmentation for 3D deep learning in multi-modal neuroimaging (Zhu et al., 2023).