---
title: 3D Wavelet Latent Diffusion Model
url: https://www.emergentmind.com/topics/3d-wavelet-latent-diffusion-model-3d-wldm
type: topic
---

# 3D Wavelet Latent Diffusion Model

3D Wavelet Latent Diffusion Model (3D-WLDM) denotes a family of 3D generative models that move diffusion away from raw voxel or image space and into a wavelet-structured representation. In strict usage, the term refers to models that perform diffusion in a learned latent space while explicitly incorporating wavelet-aware encoding or residual wavelet structure, as in whole-body MR-to-CT translation and WaLa; in broader usage, it also encompasses 3D wavelet-domain diffusion systems that denoise fixed discrete wavelet coefficients directly, even when no learned autoencoder latent is present [2507.11557], [2411.08017], [2402.19043].

## 1. Terminology and conceptual scope

A central distinction in the literature is between **wavelet-domain diffusion** and **wavelet latent diffusion**. WDM, for example, is explicitly described as a diffusion model operating on a fixed wavelet-domain representation of a 3D medical image, not a latent diffusion model in the strict learned-latent sense: no encoder is learned, no decoder is learned, the representation is invertible via exact IDWT, and the compression comes from deterministic Haar DWT rather than a pretrained autoencoder [2402.19043]. POWDR inherits the same conceptual boundary: it is highly relevant to a 3D-WLDM agenda, but it remains a 3D conditional wavelet diffusion model rather than a canonical latent diffusion model because it applies diffusion directly to a fixed wavelet-domain representation of the full 3D volume [2601.09044].

By contrast, the whole-body MR-to-CT model explicitly named **3D Wavelet Latent Diffusion Model** performs modality translation in a learned latent space and augments the encoder-decoder with a Wavelet Residual Module, while WaLa first converts a high-resolution 3D shape into a wavelet representation, then compresses that representation into a very small latent grid using a VQ-VAE, and finally trains diffusion in that compact latent space [2507.11557], [2411.08017]. These systems satisfy the stronger sense of “wavelet latent diffusion,” because diffusion operates in a learned latent rather than directly on analytic coefficients.

A common misconception is therefore that any diffusion over wavelet coefficients is automatically a latent diffusion model. The literature repeatedly rejects that equivalence. The 3D shape papers on neural wavelet-domain diffusion, for instance, diffuse coarse wavelet coefficients and use auxiliary latent mechanisms for inversion or manipulation, but diffusion itself does not run in an encoder latent; similarly, UDiFF is described as a close variant of 3D-WLDM only if “latent” is interpreted broadly as a compact hidden representation rather than a VAE-style latent space [2302.00190], [2404.06851]. In the opposite direction, canonical 3D latent diffusion can be entirely non-wavelet: 3D-LDM diffuses a 256-dimensional auto-decoder latent vector for neural implicit SDFs, with no wavelet decomposition at all [2212.00842].

## 2. Representation spaces and multiresolution structure

The simplest 3D wavelet representation used in this literature is the single-level 3D Haar discrete wavelet transform. For a 3D volume \(y_0 \in \mathbb{R}^{H \times W \times D}\), POWDR writes
\[
x_0 = DWT(y_0) = \{LLL, LLH, LHL, LHH, HLL, HLH, HHL, HHH\},
\]
with
\[
x_0 \in \mathbb{R}^{8 \times \frac{H}{2}\times \frac{W}{2}\times \frac{D}{2}}.
\]
WDM uses the same basic construction: one low-frequency approximation sub-band and seven high-frequency detail sub-bands, concatenated along the channel dimension, so diffusion operates on an 8-channel tensor at half spatial resolution in every axis [2601.09044], [2402.19043]. This construction is attractive because halving each spatial dimension reduces the 3D spatial volume by a factor of \(2^3=8\), while retaining explicit detail channels.

Wavelet latent models introduce an additional learned compression stage on top of multiresolution structure. In the whole-body MR-to-CT 3D-WLDM, the encoder maps MR and CT volumes to learned latent tensors \(Z_{MR}, Z_{CT} \in \mathbb{R}^{c \times d \times h \times w}\), and each latent is partitioned as
\[
Z = [S, M],
\]
where the first half of the channels encode structural information and the second half encode modality-specific information [2507.11557]. WaLa adopts a different route: a \(256^3\) signed distance field is first converted to a diffusible wavelet tree \(W_n \in \mathbb{R}^{46^3 \times 64}\), then encoded by a convolutional VQ-VAE to a latent grid
\[
Z \in \mathbb{R}^{12^3 \times 4},
\]
yielding 6912 latent variables and a reported \(2427\times\) compression ratio relative to the original \(256^3\) field [2411.08017].

A third regime appears in wavelet-coefficient 3D shape generation. The 2022 and 2023 neural wavelet-domain diffusion papers represent a \(256^3\) TSDF by recursive multiscale coefficients \(C_i^j, D_i^j\), choose \(J=3\), and keep only \(\{C_i^3, D_i^3\}\) as a compact coarse/detail pair [2209.08725], [2302.00190]. UDiFF similarly decomposes a \(256^3\) UDF into coarse and fine coefficients,
\[
\{C_i, F_i\} = \phi^D_\theta(U_i),
\qquad
\hat U_i = \phi^I_\delta(C_i, F_i),
\]
but learns the decomposition and inversion filters themselves, describing them as a learnable biorthogonal wavelet transform adapted to UDF statistics [2404.06851].

This variety of representations shows that “3D-WLDM” does not refer to a single tensor layout. In existing work it may mean analytic DWT coefficients, a learned latent built from wavelet-processed features, or a learned latent over a wavelet tree. The unifying property is not a fixed architecture but the use of wavelet-structured multiresolution geometry to make full 3D diffusion tractable.

## 3. Diffusion objectives, denoisers, and conditioning

The diffusion objective is not uniform across the literature. WDM performs diffusion entirely in wavelet space and predicts denoised coefficients \(\tilde{x}_0\) rather than noise:
\[
\mathcal{L}_{MSE} = \|\tilde{x}_0 - x_0\|_2^2.
\]
The paper states that predicting the noise to remove produced checkerboard artifacts, so direct \(x_0\)-prediction was retained [2402.19043]. POWDR uses the same wavelet-space reconstruction objective,
\[
L = MSE_{wavelet} = \|x_0 - \widetilde{x}_0\|^2,
\]
and is therefore most naturally interpreted as an \(x_0\)-prediction conditional DDPM in wavelet space [2601.09044].

By contrast, the named 3D-WLDM for MR-to-CT synthesis adopts latent noise prediction:
\[
\mathcal{L}_{WLDM}
=
\mathbb{E}_{\epsilon_t \sim \mathcal N(0,I)}
\left[
\left\|
\epsilon_t - \epsilon_\theta(Z_{CT}^{t}, t, Z_{MR})
\right\|_2^2
\right],
\]
with latent forward noising
\[
Z_{CT}^{t} = \sqrt{\bar{\alpha}_t}\, Z_{CT}^{0} + \sqrt{1-\bar{\alpha}_t}\,\epsilon.
\]
Here the noisy CT latent is denoised under MR latent conditioning, and the structural component of the MR latent is meant to anchor anatomy during reverse diffusion [2507.11557]. WaLa also departs from standard \(\epsilon\)-prediction: its denoiser is trained to recover the clean latent directly,
\[
\mathcal{L} = \mathbb{E}_t \left[ \| f_\theta(Z_n^t, t, \Theta_n) - Z_n^0 \|^2 \right],
\]
and it uses a cosine noise schedule [2411.08017].

Other systems remain closer to standard DDPM practice. UDiFF minimizes
\[
\min_{\sigma} \mathbb{E}_{\mathcal{C}_0, t, {\epsilon}\sim\mathcal{N}(0,1)}
\left[
\left\|{\epsilon}-{\epsilon}_{\sigma}\left({\mathcal{C}_t, t\right)\right\|^{2}
\right]
\]
for coarse wavelet coefficients, with CLIP-based cross-attention for text or image conditioning [2404.06851]. HVDM similarly applies standard DDPM \(\epsilon\)-prediction in a fused latent volume
\[
z \in \mathbb{R}^{4 \times 16 \times 32 \times 32},
\]
using a 3D U-Net denoiser after a hybrid autoencoder combines a 2D projected latent branch with a 3D wavelet-volume branch [2402.13729].

Conditioning mechanisms are likewise heterogeneous. POWDR concatenates noisy target wavelet coefficients with DWT-transformed conditioning coefficients from a masked lesion-bearing image, producing a 16-channel input to a 3D ResUNet with attention [2601.09044]. The whole-body MR-to-CT 3D-WLDM conditions on MR latent features and inserts **Dual Skip Connection Attention** into the diffusion U-Net; its Structure Emphasis Module attends from shallow diffusion features to \(S_{MR}\), while its Modality Filtering Module subtracts modality-specific interference associated with \(M_{MR}\) [2507.11557]. SC-Diff, though non-wavelet, provides a relevant comparison: it combines CLIP cross-attention for optional RGB conditioning with a ControlNet-style 3D conditioning branch for partial TSDF observations [2403.12470].

## 4. Medical imaging formulations

Medical imaging is the domain in which 3D wavelet diffusion has been developed most explicitly as a response to cubic memory growth. WDM targets unconditional 3D medical volume generation for BraTS T1-weighted MRI and LIDC-IDRI lung CT at \(128^3\) and \(256^3\) resolution, and frames wavelet decomposition as a simple alternative to patch-wise, slice-wise, cascaded, or learned-latent 3D generation. It reports that WDM can be trained on a single 40 GB NVIDIA A100, requires 2.55 GB inference memory at \(128^3\) and 7.27 GB at \(256^3\), and is the only diffusion-based method in its comparison that could be trained at \(256^3\) on a single 40 GB GPU [2402.19043].

POWDR moves from unconditional synthesis to **pathology-preserving outpainting**. Rather than generating lesions from scratch, it retains a real pathology-bearing region and synthesizes only the surrounding anatomy. The condition is formed as \(c = m \odot y_0\), where \(m\) is a pathology mask or a random connected mask, then transformed to wavelet space and concatenated with the noisy sample at every denoising step [2601.09044]. Its most distinctive contribution is the random connected mask strategy, introduced because training only on pathology masks produced near-deterministic completions: repeated sampling with tumor-mask conditioning gave pairwise cosine similarity \(0.9947 \pm 0.00003\) and KL divergence \(0.00026\), while adding random connected masks reduced cosine similarity to \(0.9580\) and increased KL divergence to \(0.01494\) [2601.09044]. The method also reports downstream gains: using nnU-Net for whole-tumor segmentation, the baseline Dice \(0.6992 \pm 0.3009\) increases to \(0.7137 \pm 0.2868\) when 50 synthetic cases are added [2601.09044].

The explicitly named **3D Wavelet Latent Diffusion Model for Whole-Body MR-to-CT Modality Translation** adopts a stricter 3D-WLDM formulation. Its encoder-decoder backbone is VAE-style, wavelet processing is inserted through the **Wavelet Residual Module**, and the latent is disentangled into structural and modality-specific components. During diffusion, the noisy CT latent and MR latent are concatenated and passed through a U-Net with **Dual Skip Connection Attention**, whose SEM branch emphasizes structure from \(S_{MR}\) and whose MFM branch suppresses MRI-specific interference from \(M_{MR}\) [2507.11557]. Quantitatively, the model reports **PSNR \(25.33 \pm 2.60\)**, **SSIM \(0.81 \pm 0.07\)**, **MAE \(47.28 \pm 16.48\)**, and **NCC \(0.97 \pm 0.02\)** against baselines including DDPM, DDIM, StableDM-2D, ControlNet, StableDM-3D, and CycleGAN, and its full ablation stack improves PSNR by **2.31 dB**, SSIM by **0.23**, and MAE by **28.08** over the vanilla latent diffusion baseline [2507.11557]. The same study reports improved downstream TotalSegmentator Dice, with full-model values of **81.4 ± 9.6** for spleen, **75.9 ± 21.8** for kidney, **89.7 ± 5.4** for liver, **71.1 ± 10.5** for stomach, and **80.1 ± 17.0** for vertebrae [2507.11557].

Across these medical systems, the practical motivation is consistent: wavelet structure is used to preserve fine anatomy, reduce memory pressure, and avoid the artifacts or downstream restrictions attributed to patch-wise or slice-wise generation. The divergence lies in whether the wavelet representation is analytic and invertible, as in WDM and POWDR, or embedded inside a learned latent autoencoder, as in the whole-body 3D-WLDM.

## 5. Shape, geometry, and video analogues

In 3D shape generation, the earliest wavelet-diffusion line models geometry as a truncated signed distance field and performs diffusion on compact multiscale wavelet coefficients rather than raw voxels. “Neural Wavelet-domain Diffusion for 3D Shape Generation” normalizes each shape, samples a \(256^3\) TSDF, truncates to \([-0.1,+0.1]\), and uses a compact scale-\(J=3\) pair \((C^3, D^3)\); on ShapeNet chairs, reconstructing from only \(C_i^3\) and \(D_i^3\) changes TSDF magnitudes by only **2.8%** while removing **97% of the coefficients** [2209.08725]. Diffusion is applied only to the coarse coefficients, and a deterministic detail predictor restores fine structure; generation takes about **6 seconds per shape** on one RTX 3090 [2209.08725]. The 2023 extension adds an encoder \(z = Enc_\phi(C_0)\) for inversion and manipulation, but diffusion still operates on coarse wavelet coefficients rather than on the encoder latent, so it remains wavelet-domain diffusion with an auxiliary latent conditioner rather than a standard latent diffusion model [2302.00190].

UDiFF generalizes wavelet-domain diffusion from SDFs to **unsigned distance fields** so that open surfaces can be modeled. It learns both the decomposition filter \(\phi^D_\theta\) and inversion filter \(\phi^I_\delta\), diffuses only over coarse coefficients, predicts fine coefficients with a second U-Net, and reconstructs UDFs before meshing with DCUDF rather than marching cubes [2404.06851]. Its strongest evidence for learned wavelet optimization is the reconstruction ablation on DeepFashion3D: mesh recovery from Haar gives **264.8**, Biorthogonal3-3 gives **46.04**, Biorthogonal6-8 gives **42.92**, learnable decomposition only gives **36.12**, learnable inversion only gives **32.15**, and learning both gives **28.51** in L2 Chamfer Distance scaled by \(10^5\) [2404.06851].

HVDM shows how wavelet ideas migrate into latent video diffusion. It uses a deterministic hybrid autoencoder that decomposes video into a 2D projected latent branch and a 3D wavelet-volume branch, with the latter applying a true spatio-temporal 3D DWT over time, height, and width and separately encoding \(x_{lll}\) and the concatenated high-frequency sub-bands [2402.13729]. Diffusion is then trained on the fused latent rather than on raw wavelet coefficients. In UCF-101 reconstruction, the wavelet frequency-matching loss improves **R-FVD** from **8.08** to **5.35**, **LPIPS** from **0.041** to **0.038**, **PSNR** from **33.67** to **34.00**, and **SSIM** from **0.892** to **0.915** [2402.13729]. This is not a pure wavelet latent diffusion model, but it demonstrates that explicit low/high-frequency handling remains useful even when diffusion acts on a hybrid latent.

WaLa is the clearest large-scale 3D-WLDM instantiation in geometry generation. It starts from a \(256^3\) SDF, converts it to a wavelet tree \(W_n \in \mathbb{R}^{46^3 \times 64}\), compresses that representation to \(Z \in \mathbb{R}^{12^3 \times 4}\), and trains conditional or unconditional diffusion with models of roughly one billion parameters [2411.08017]. The method reports a **2427x compression ratio**, conditioned inference in **2 to 4 seconds**, and strong performance across point-cloud, voxel, single-view, and multi-view settings [2411.08017]. On GSO, for example, its representation comparison reports IoU **0.9780** with only **6912** variables, compared with **0.9956** for the full wavelet tree at **1,129,528** variables and **1.0** for the original \(256^3\) SDF at **16,777,216** variables [2411.08017]. WaLa is therefore not merely wavelet-domain diffusion; it is diffusion over a learned latent that is itself built on a wavelet representation.

## 6. Relation to non-wavelet 3D latent diffusion and recurrent limitations

Wavelet-based 3D latent diffusion is best understood against neighboring 3D latent diffusion systems that do not use wavelets. 3D-LDM trains diffusion over a **256-dimensional** DeepSDF-style latent vector and conditions on images or text via CLIP embeddings, but the latent has no explicit spatial locality or frequency separation [2212.00842]. SC-Diff compresses \(64^3\) TSDFs to a VQ-VAE latent
\[
\mathbf{z}\in \mathbb{R}^{3\times 16\times 16\times 16},
\]
then performs conditional latent diffusion for shape completion, showing that latent-space diffusion uses **30.72% less GPU memory** than a non-latent alternative while enabling \(64^3\) completion [2403.12470]. LaGeM builds a hierarchical latent set representation for native 3D geometry and performs cascaded diffusion over latent levels \(\mathbf{Z}_3,\mathbf{Z}_2,\mathbf{Z}_1\), but explicitly does not use wavelets; its hierarchy is multiresolution only in a learned latent-set sense [2410.01295]. “AutoDecoding Latent 3D Diffusion Models” similarly diffuses a dense volumetric latent \(8^3 \times 256\) learned from 2D-supervised autodecoding, again with no wavelet structure [2307.05445].

Relative to those systems, wavelet-based models repeatedly claim four advantages. First, they provide **explicit multiscale separation** between coarse and high-frequency detail rather than relying on a monolithic latent to preserve sharp geometry. Second, in fixed-transform variants such as WDM and POWDR, they provide **exact invertibility** without requiring a difficult 3D autoencoder pretraining stage [2402.19043], [2601.09044]. Third, in learned latent variants such as WaLa and the whole-body 3D-WLDM, they offer a route to combine latent compression with wavelet-aware preservation of fine structure [2411.08017], [2507.11557]. Fourth, they often make full 3D generation feasible without patch-wise or slice-wise decomposition, which is a recurring concern in medical imaging [2402.19043].

The limitations are equally recurrent. Terminology remains unstable: several important “3D-WLDM-like” systems are not latent diffusion models in the strict modern sense, because diffusion acts on analytic wavelet coefficients rather than on a learned latent [2402.19043], [2601.09044]. Learned wavelet formulations can also be underspecified: UDiFF does not fully document the exact neural parameterization of \(\phi^D_\theta\) and \(\phi^I_\delta\), including filter structure and constraints [2404.06851]. Some conditional settings collapse without additional training strategies, as POWDR explicitly shows [2601.09044]. Diffusion speed remains a bottleneck in unconditional or long-chain settings, and geometry pipelines still often depend on marching cubes or DCUDF for surface extraction [2411.08017], [2404.06851]. The whole-body MR-to-CT 3D-WLDM also depends on accurate MR-CT alignment during training, a limitation the authors identify directly [2507.11557].

A plausible implication is that future 3D-WLDMs will continue to hybridize these two traditions: the exact multiresolution structure of wavelet-domain modeling and the scalability of learned latent diffusion. The current literature already spans fixed Haar DWT, learned biorthogonal transforms, wavelet-aware residual autoencoders, hybrid projected-plus-wavelet video latents, and VQ-VAE compression of wavelet trees. The common thread is not a single canonical design, but the use of wavelet structure as a principled mechanism for making high-resolution 3D diffusion both computationally feasible and detail-aware.

Source: https://www.emergentmind.com/topics/3d-wavelet-latent-diffusion-model-3d-wldm