Direct Volumetric Patch Diffusion

• Direct volumetric patch diffusion is a family of methods for scalable 3D generative modeling that operates on local patches to bypass full-volume memory constraints.
• It leverages positional encoding and global context aggregation via patch blending to ensure anatomical coherence and minimize cross-patch artifacts.
• Applications in CT, PET, and 3D synthesis demonstrate significant efficiency gains, improved PSNR, and reduced runtime compared to traditional full-volume modeling.

Direct volumetric patch diffusion is a family of methodologies for generative modeling and inverse problem solving in high-dimensional volumetric data domains (notably 3D medical imaging), in which diffusion models are trained and/or deployed over small spatiotemporal patches of the data rather than over entire volumes. This approach is motivated by the prohibitive computational and memory requirements of direct full-volume diffusion modeling in modalities such as computed tomography (CT), positron emission tomography (PET), and 3D synthesis, where volumes often reach $256 \times 256 \times 500$ or larger. By learning 3D priors or score functions on local patches—augmented with positional and/or global context encoding—and systematically merging their predictions, direct volumetric patch diffusion enables scalable, consistent, and data-efficient generation, denoising, or reconstruction of large-scale volumetric data while maintaining high accuracy and anatomical coherence across all axes (Song et al., 2024, Yang et al., 20 Dec 2025, Hu et al., 2024).

1. Mathematical Foundations and Patchwise Formulation

In direct volumetric patch diffusion, the data domain is a high-dimensional tensor $x \in \mathbb{R}^{H \times W \times D}$, typically representing a volumetric medical image with $D$ slices. The key mathematical structure is a collection of patch extraction operators $P_i$, which map $x$ to overlapping or shifted sub-volumes $x_{i} = P_i x \in \mathbb{R}^{h \times w \times k}$, where $k \ll D$ is the local patch depth. The forward (noising) process follows the DDPM kernel,

$$q(x_t|x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$

where the global schedule $\alpha_t$ is inherited to the patch level. Training operates at the patch scale, where a score network $S_\theta(P_i x_t, t, p)$—often a U-Net—predicts $x \in \mathbb{R}^{H \times W \times D}$0, with $x \in \mathbb{R}^{H \times W \times D}$1 a positional embedding that encodes spatial location, slice index, or other coordinate information (Song et al., 2024, Bieder et al., 2023, Yoon et al., 2024).

Global inference aggregates the patch-level scores into a full-volume prediction by lifting each patch score back to the global domain via $x \in \mathbb{R}^{H \times W \times D}$2 and blending according to a position-aware weighting $x \in \mathbb{R}^{H \times W \times D}$3:

$x \in \mathbb{R}^{H \times W \times D}$4

In practice, blending can employ uniform weights or smooth window functions (e.g., Gaussian, Hann) in the slice direction, and Monte-Carlo sampling over different patch partitions for stochastic regularity (Song et al., 2024).

2. Position-Awareness, Global Context, and Patch Aggregation

Pure patchwise diffusion is inherently local and risks losing global coherence or introducing cross-patch artifacts. Position-awareness is addressed by explicit coordinate encodings—using normalized position channels, sinusoidal embeddings, or spatial indices concatenated to each patch input (Yang et al., 20 Dec 2025, Hu et al., 2024, Bieder et al., 2023, Yoon et al., 2024). Further, context coupling is realized by:

• Concatenating a downsampled global volume as an input channel to each patch denoiser, providing low-resolution global context for local patch predictions (Yang et al., 20 Dec 2025).
• Merging prediction scores across multiple partitions or offsets during inference, sometimes with recurrent denoise-re-noise cycles to reduce stitching or seam artifacts (Yang et al., 20 Dec 2025).
• Transformer tokenization and attention: In non-U-Net settings, as in VolDiT, volumetric patch tokens are processed via global attention, with positional signals and long-range context captured intrinsically (Seyfarth et al., 26 Mar 2026).
• Blending mechanisms: Patchwise scores are fused using sum, average, or windowed functions, often randomized per iteration, to promote isotropy and reduce boundary artifacts (Song et al., 2024).

3. Inverse Problem Solving and Posterior Sampling

Direct volumetric patch diffusion achieves practical inverse problem solutions for ill-posed settings (e.g., sparse-view CT, limited-angle CT, PET denoising) by integrating data-consistency steps into the generative sampling loop. The typical workflow involves:

1. Initialization of the entire 3D volume as Gaussian noise at the maximum diffusion timestep;
2. Iterative update: At each reverse diffusion iteration $x \in \mathbb{R}^{H \times W \times D}$5,
   • Random partition selection and patch extraction,
   • Patchwise score computation with position/context encoding,
   • Score blending and aggregation into full-volume gradients,
   • Estimation of the denoised image via Tweedie’s formula,
   • Data-consistency enforcement using CG or gradient steps w.r.t. forward operator $x \in \mathbb{R}^{H \times W \times D}$6 (e.g., Radon transform for CT),
   • Sampling of the next timestep using the denoised estimate and global score.

Algorithm pseudocode for DiffusionBlend++ and comparable approaches can be found in (Song et al., 2024) and (Yang et al., 20 Dec 2025). These workflows support unrolled optimization over volumes with $x \in \mathbb{R}^{H \times W \times D}$7 or higher spatial footprints.

4. Network Architectures and Training Regimes

The core denoising modules for patchwise diffusion are adapted for efficient memory usage and effective context modeling:

• 3D U-Net, PatchDDM, and UNet variants: Feature averaging is used rather than concatenation in skip connections to control feature variance and avoid memory explosion (Bieder et al., 2023, Song et al., 2024). Channel capacity is often increased (by $x \in \mathbb{R}^{H \times W \times D}$81.6x) due to the diminished activation blowup.
• Attention: Omitted in many 3D U-Nets due to cost, but included at coarse resolution in larger models (Yang et al., 20 Dec 2025), or implemented globally in transformers (Seyfarth et al., 26 Mar 2026).
• Transformer-based architectures: VolDiT replaces U-Nets entirely with transformer stacks over patch tokens, with time-step gated control adapters for spatial conditioning (Seyfarth et al., 26 Mar 2026).
• Coordinate encoding: Linear ramp, sine/cosine, and MLP-generated embeddings for slice/patch indices or coordinates (Bieder et al., 2023, Yang et al., 20 Dec 2025).
• Conditional input stacking: For conditional modalities (e.g., PET-MR), noisy input, modality-specific priors, and coordinate maps are stacked along the channel dimension (Yoon et al., 2024).

Training is conducted exclusively on local patches and their context. Memory savings are substantial: per-inference memory demand can be reduced by over two orders of magnitude (e.g., $x \in \mathbb{R}^{H \times W \times D}$9 for $D$0-slice patches) (Song et al., 2024).

5. Computational Efficiency and Empirical Performance

Direct volumetric patch diffusion excels in scalability compared to full-volume or 2D-slice priors:

• Memory/Runtime: For CT volumes of $D$1, inference on an A40 GPU is reduced from over 20 hours (DiffusionMBIR, 2000 NFE) to $D$230 minutes for DiffusionBlend++ (200 NFE) (Song et al., 2024). Training leverages patchwise stochasticity, enabling large batch sizes on limited hardware (Bieder et al., 2023, Yoon et al., 2024).
• Data Efficiency: Patchwise models achieve high fidelity with orders of magnitude less data. For PaDIS, 2,304 CT slices suffice for strong 3D reconstruction priors, compared to tens/hundreds of thousands for whole-image models (Hu et al., 2024).
• Accuracy and Consistency Metrics: Substantial quantitative gains in PSNR and SSIM over filtered backprojection, 2D slice models, or non-patch 3D diffusion are reported, especially in underdetermined reconstruction settings. DiffusionBlend++ on sparse-view CT with 8 views improves axial PSNR, e.g., FBP ≈14.7 dB, DDS≈33.2 dB, DiffusionBlend≈34.3 dB, DiffusionBlend++=35.7 dB (Song et al., 2024). Patch-based methods preserve z-axis (cross-slice) consistency, as quantified by total variation in $D$3, closely mirroring ground truth (Song et al., 2024).
• Generative Synthesis: VolDiT achieves improved FID (0.004 vs. 0.031) and Dice (0.94 vs. 0.89) on 3D medical image synthesis, demonstrating superior global coherence compared to U-Net LDMs (Seyfarth et al., 26 Mar 2026).

Model	Memory Efficiency	Inference Time	State-of-the-Art Results
DiffusionBlend++	$D$4	30 min (256×256×500)	PSNR 35.7 dB (SVCT, 8 views)
PatchDDM	7$D$5 training reduction	1.02s/step (A100)	Dice up to 0.899 (ensemble)
Global Context Patch	$D$632GB $D$7 Patch+Downsample	20 min (512²×256)	PSNR $D$833 dB (8-view CT)
VolDiT (transformer)	Tokenized memory	--	FID 0.004 (LUNA16)

6. Extensions, Limitations, and Model Variants

Several extensions of direct volumetric patch diffusion have been demonstrated:

• Hybrid Local-Global Patch Models: Coupling of patchwise and global context priors (e.g., downsampled volume) improves coherence and performance; removing the global context reduces fidelity (Yang et al., 20 Dec 2025).
• Residual Diffusion: Some models perform diffusion not on the raw volume but on the residual between low-dose and ground truth (e.g., in PET), enabling highly efficient denoising (Yoon et al., 2024).
• Primitive-based Patch Diffusion: In human generation, PrimDiffusion operates on a set of volumetric primitives (color, density, scale, pose) packed as a tensor, enabling real-time, decoder-free rendering at high resolution (Chen et al., 2023).
• Low-data Regime: Patchwise priors exhibit graceful degradation under data scarcity. For PaDIS, a reduction from 2,304 to 144 images led to only a modest PSNR drop (33.57 dB$D$932.28 dB), compared to dramatic deterioration for whole-image models (Hu et al., 2024).

Identified limitations include boundary artifacts for excessively small patch sizes, a dependence on coordinate/position encoding for global consistency, and an absence (in some frameworks) of true end-to-end joint training over the full volume (Yang et al., 20 Dec 2025, Bieder et al., 2023, Yoon et al., 2024, Chen et al., 2023). Further ablation studies confirm that removal of patch-based context or downsampled global views leads to performance degradation in both reconstruction and generative tasks (Yang et al., 20 Dec 2025).

7. Representative Applications and Modalities

Direct volumetric patch diffusion frameworks have achieved state-of-the-art results and practical deployment in:

• Sparse-view and limited-angle 3D CT reconstruction, with high-dimensional clinical datasets (e.g., AAPM, LIDC), preserving 3D consistency and structure (Song et al., 2024, Yang et al., 20 Dec 2025, Hu et al., 2024).
• Volumetric PET denoising, including tight PET-MR anatomical conditioning (Yoon et al., 2024).
• Medical 3D image segmentation and tumor detection (PatchDDM on BraTS2020) (Bieder et al., 2023).
• Controllable volumetric synthesis for medical imaging using global attention architectures (Seyfarth et al., 26 Mar 2026).
• 3D human shape and appearance synthesis using compact volumetric primitives (Chen et al., 2023).

The direct volumetric patch diffusion paradigm is extendable to other imaging and non-imaging 3D inverse problems that admit a convolutional forward model or global conditioning, including MRI, SPECT, and electron tomography (Yang et al., 20 Dec 2025). This approach continues to drive progress in scalable, anatomically consistent, and practical generative modeling across large-scale volumetric domains.