---
title: Multi-Level VQGAN Architectures
url: https://www.emergentmind.com/topics/multi-level-vector-quantized-generative-adversarial-network-vqgan
type: topic
---

# Multi-Level VQGAN Architectures

Multi-Level Vector-Quantized Generative Adversarial Network (VQGAN) denotes a class of VQGAN-derived models in which representational capacity is distributed across more than one “level,” but the literature uses that label in more than one way. In the narrowest sense, it refers to hierarchical quantization with multiple latent maps or multiple residual quantization stages; in broader usage, it can also denote multi-scale feature extraction, multi-grained attention, or semantic low/high-level feature partitions built around a single codebook. All such models retain the canonical VQGAN pattern—an encoder, a discrete codebook with nearest-neighbor quantization, a decoder, and adversarial and perceptual training—but they differ in where hierarchy is introduced: inside the quantizer, around the tokenizer, or in the prior over tokens [2508.04734] [2512.15947] [2409.06105].

## 1. Terminological scope and conceptual variants

Standard VQGAN maps an input \(x\) to a continuous latent \(z_e(x)\), replaces each latent vector with its nearest codebook entry, and reconstructs the signal with a decoder trained using reconstruction, perceptual, and adversarial losses. MCR-VQGAN states this explicitly as an encoder–quantizer–decoder pipeline with a learned codebook and GAN supervision, and also distinguishes VQGAN from VQ-VAE by noting that VQGAN augments discrete latents with adversarial and perceptual objectives to sharpen texture and improve realism [2512.15947].

Across recent papers, “multi-level VQGAN” is used for at least three non-identical design patterns. First, it can mean **hierarchical quantization**, where different codebooks or quantization stages encode coarse and fine information. The colorectal virtual H&E model is the clearest instance: it uses two latent feature maps at different spatial resolutions, with separate codebooks for coarse and fine structure [2508.04734]. DelightfulTTS 2 uses a different hierarchical form—multi-stage residual vector quantization at the bottleneck of a speech codec—where successive stages quantize residual error rather than separate spatial resolutions [2207.04646].

Second, it can mean **multi-scale or multi-level feature extraction around a single codebook**. MCR-VQGAN is explicit that “multi-level” refers to multi-resolution processing and multi-scale convolutions, not multiple codebooks or stacked quantizers [2512.15947]. SGC-VQGAN likewise keeps a single latent grid and a single codebook, but enriches it with pyramid feature learning and semantically guided low/high-level partitions [2409.06105].

Third, hierarchy may exist **outside the VQGAN tokenizer itself**. TATS states explicitly that its 3D VQGAN uses a single-level codebook and that hierarchy is introduced at the transformer level through sparse-frame autoregression plus an interpolation transformer [2204.03638]. Efficient-VQGAN similarly keeps a single-level codebook while introducing “multi-grained” attention in the generator prior through global block tokens and local windows [2310.05400].

| Usage of “multi-level” | Representative models | Location of hierarchy |
|---|---|---|
| Hierarchical quantization | virtual H&E VQGAN [2508.04734]; DelightfulTTS 2 [2207.04646] | multiple codebooks or residual quantization stages |
| Single-codebook multi-scale design | MCR-VQGAN [2512.15947]; SGC-VQGAN [2409.06105] | encoder/decoder features or semantic partitions |
| Hierarchy in the prior | TATS [2204.03638]; Efficient-VQGAN [2310.05400] | transformer or attention granularity |

A common misconception is therefore that any “multi-level VQGAN” necessarily uses hierarchical codebooks. Several papers explicitly contradict that reading. MCR-VQGAN has one codebook of size \(K=1024\) with embedding dimension \(D=512\) [2512.15947]; Efficient-VQGAN uses a single-level codebook with \(K=1024\) [2310.05400]; TATS uses a single codebook with \(K=16384\) and embedding dimension \(c=256\) [2204.03638].

## 2. Canonical formulation and multi-level generalizations

The core quantization rule is shared across the literature. For an encoder output \(z_e(x)\) and codebook entries \(\{e_k\}\), the quantized representation is obtained by nearest-neighbor assignment:
$$
k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.
$$
This formulation appears in standard VQGAN recaps in MCR-VQGAN, Efficient-VQGAN, SGC-VQGAN, and related papers [2512.15947] [2310.05400] [2409.06105].

The standard VQ objective combines codebook and commitment terms with stop-gradient:
$$
L_{vq} = \| \mathrm{sg}[z_e(x)] - z_q \|_2^2 + \beta \| z_e(x) - \mathrm{sg}[z_q] \|_2^2.
$$
These are then combined with reconstruction, perceptual, and adversarial losses. MCR-VQGAN presents a total generator loss
$$
L_G = \lambda_{adv} L_{adv} + \lambda_{rec} L_{rec} + \lambda_{perc} L_{perc} + \lambda_{VQ} L_{VQ},
$$
with \(\lambda_{adv}=1\), \(\lambda_{rec}=10\), \(\lambda_{perc}=10\), and \(\lambda_{VQ}=5\), and uses a PatchGAN discriminator with logistic GAN loss plus \(R1\) regularization with \(\gamma=10\) [2512.15947]. TATS uses a different adversarial configuration with spatial and temporal discriminators, feature matching, and a minimax objective whose typical weights are \(\lambda_{rec}=\lambda_{match}=4.0\), \(\lambda_{adv}=1.0\), and \(\beta=0.25\) [2204.03638]. Efficient-VQGAN uses a Projected GAN discriminator with hinge losses and a stage-1 total objective \(L_{total}=L_{perc}+L_2+L_{vq}+\lambda\cdot L_{adv}\) [2310.05400].

Multi-level formulations extend this template in two principal ways. In **hierarchical spatial quantization**, separate latent maps are quantized at different resolutions, each with its own codebook and commitment term:
$$
L_{VQ} = \sum_{l=1}^{2} L_{VQ}^{(l)}.
$$
This is the formulation used in the two-level virtual H&E VQGAN, where the coarse level captures tissue-scale structure and the fine level captures cellular detail [2508.04734].

In **residual multi-stage quantization**, later levels quantize the residual left by earlier levels:
$$
r_0 = z_e(x),\qquad r_l(t)=r_{l-1}(t)-q^{(l)}(t),\qquad z_q(t)=\sum_{l=1}^{L} q^{(l)}(t).
$$
DelightfulTTS 2 uses this stage-wise residual VQ at frame rate \(80\) frames/sec, combined with multi-resolution spectrogram loss, adversarial losses from multi-scale and multi-period discriminators, and feature matching [2207.04646].

These formulations share the same discrete-latent logic but differ in what each level represents. A plausible implication is that “multi-level” is best understood structurally rather than terminologically: the decisive question is whether levels are implemented as multiple codebooks, residual quantizers, feature partitions, or prior-level abstractions.

## 3. Hierarchical quantization in the strict sense

The colorectal virtual H&E model provides the most explicit image-domain instance of a multi-level VQGAN in the strict sense. The encoder produces two latent feature maps at different spatial resolutions, and each level is quantized by its own codebook of size \(1024\). The coarse level models high-level tissue structures, the fine level models cellular details, and the decoder integrates both levels to reconstruct the final H&E image [2508.04734]. The single-level baseline in the same study follows “Taming Transformers,” uses one codebook of size \(1024\), and differs from the multi-level model precisely in the absence of this hierarchical latent decomposition [2508.04734].

Its training objective combines \(L_{rec}\), \(L_{perc}\), \(L_{style}\), \(L_{stain}\), \(L_{VQ}\), and a small-weight adversarial term \(L_{adv}\), with \(\lambda_{adv}=0.02\). The commitment cost is \(0.2\) for the single-level model and \([0.2, 0.1]\) for the two-level model [2508.04734]. The adversarial term is introduced after a warm-up—after \(6000\) iterations on Orion and after \(300\) iterations on CODEX—to stabilize the codebooks [2508.04734].

On the Orion test set, the two-level VQGAN improved over both cGAN and the one-level VQGAN in image similarity, with \(L1=0.1491\), \(L2=0.0391\), \(SSIM=0.5221\), and \(PSNR=20.38\), compared with \(0.1545\), \(0.0435\), \(0.5199\), and \(19.92\) for the one-level VQGAN and \(0.1670\), \(0.0495\), \(0.4749\), and \(19.38\) for cGAN [2508.04734]. On the CODEX test set, the two-level model also gave the best reconstruction metrics among the three, with \(L1=0.1432\), \(L2=0.0417\), \(SSIM=0.6816\), and \(PSNR=20.89\) [2508.04734]. For downstream tissue classification, semantic preservation on Orion rose from \(0.6293\) for cGAN to \(0.6742\) for the one-level VQGAN and \(0.6994\) for the two-level model [2508.04734]. Nuclei segmentation with a watershed algorithm likewise favored the two-level model on Orion, with mean IoU \(0.6800\) [2508.04734].

The same paper also identifies an important limitation: increasing the number of quantization levels beyond two led to codebook collapse risks, and the two-level hierarchy offered the best trade-off between performance and stability [2508.04734]. This directly contradicts the assumption that deeper quantization hierarchies are monotonically beneficial.

DelightfulTTS 2 shows that strict multi-level VQGAN is not restricted to image pyramids. Its VQ-GAN codec uses hierarchical multi-stage residual vector quantization, reconstructs waveforms with a HiFi-GAN-style generator augmented with a bidirectional Long Expressive Memory layer, and applies multi-scale and multi-period discriminators with discrete wavelet transform downsampling [2207.04646]. The system uses a constant bitrate of \(12.8\) kbps and a \(12.5\) ms frame hop, and reports MOS \(4.26 \pm 0.09\) for DelightfulTTS 2 versus \(4.16 \pm 0.09\) for DelightfulTTS and \(4.08 \pm 0.09\) for FastSpeech 2, with CMOS gains of \(+0.14\) over DelightfulTTS and \(+0.13\) over FastSpeech 2 [2207.04646]. In this setting, multi-level structure corresponds to bitrate-efficient residual refinement rather than coarse/fine image scales.

## 4. Single-codebook models with multi-scale or multi-grained structure

A second lineage keeps the quantizer single-level but redistributes modeling power around it. MCR-VQGAN is explicit on this point: it is “a 2D VQGAN for MRI→tau PET synthesis with a single discrete latent codebook, but a multi-level/multi-scale feature extractor in the encoder and residual refinement blocks, plus CBAM attention throughout” [2512.15947]. Its encoder uses an initial \(7\times7\) convolution, three downsampling stages with channels \([64,128,256,512]\), multi-scale convolutional branches \(3\times3\), \(5\times5\), and \(7\times7\), six encoding ResNet blocks, and CBAM. The codebook size is \(K=1024\), the embedding dimension is \(D=512\), and the codebook is updated by EMA with decay \(0.99\) [2512.15947].

The decoder is symmetric but uses standard \(3\times3\) convolutions in the decoding ResNet blocks for efficiency, followed by three upsampling stages and a final \(7\times7\) convolution with Tanh output. The discriminator is a PatchGAN with three \(4\times4\) stride-2 convolution layers, and training includes one-sided label smoothing, annealed Gaussian noise on generated images, and \(R1\) gradient penalty [2512.15947]. On the ADNI-3 test set, MCR-VQGAN achieved \(MSE \approx 0.0056 \pm 0.0061\), \(PSNR \approx 24.39 \pm 4.49\) dB, and \(SSIM \approx 0.9000 \pm 0.0453\), outperforming cGAN, WGAN-GP, CycleGAN, and standard VQGAN [2512.15947]. Its downstream classifier achieved comparable accuracy on real and synthetic tau PET, \(63.64\%\) versus \(65.91\%\) [2512.15947]. The paper therefore uses “multi-level” to denote multi-resolution and multi-scale processing, not hierarchical codebooks.

Efficient-VQGAN makes an analogous distinction. Its stage-1 tokenizer uses a Swin Transformer-based encoder and decoder with local window attention, a single-level codebook with \(K=1024\), and a Projected GAN discriminator [2310.05400]. Hierarchy appears in stage 2 as **multi-grained attention**, where a latent token matrix is split into non-overlapping blocks, each block is summarized by one global token, and local generation attends to both global block tokens and an extended local neighborhood [2310.05400]. The paper states explicitly that “Multi-Level” refers to multi-grained attention rather than hierarchical quantization [2310.05400]. On ImageNet \(256\times256\), Efficient-VQGAN achieved reconstruction FID \(0.95\) at latent size \(32\times32\), and class-conditioned synthesis FID \(6.81\) with IS \(135.84\) under \(0.5\) acceptance-rate rejection sampling [2310.05400].

MoVQ also remains outside strict hierarchical quantization. It introduces multichannel quantization by splitting encoder channels into \(c\) chunks, quantizing each chunk independently with a shared codebook, and decoding through spatially conditional normalization so identical discrete indices can be modulated differently across locations [2209.09002]. For \(256\times256\) images, it uses a \(16\times16\times4\) latent and a \(1024\times64\) codebook, with recombination capacity scaling as \(K^c\) while sequence length remains \(16\times16\) [2209.09002]. The paper explicitly states that this design is not hierarchical across scales or residual stages [2209.09002]. On ImageNet, MoVQ-mask achieved FID \(7.22\) and IS \(130.1\) in \(12\) steps, while MoVQ-auto achieved FID \(7.13\) and IS \(138.3\) in \(1024\) steps [2209.09002].

These models are important because they show that a substantial fraction of “multi-level VQGAN” practice is actually **single-codebook VQGAN with structured feature allocation**. The improvement comes not from adding more quantizers, but from redesigning receptive fields, feature pathways, or attention topology around the same quantizer.

## 5. Semantic organization, codebook usage, and tokenizer objectives

A third line of work treats “levels” as semantic strata within the tokenizer. SGC-VQGAN keeps a single latent grid but splits features into low-level and high-level parts, introduces Semantic Online Clustering, and applies a semantic consistency objective based on CosFace [2409.06105]. Its quantization distance is explicitly multi-level:
$$
d(i,k)=\|\tilde{e}_i^t-\tilde{c}_k^t\|_2+\beta\|\hat{e}_i^t-\hat{c}_k^t\|_2.
$$
The codebook carries paired low/high-level features, and high-level entries are updated toward both encoded anchors and semantic class prototypes \(W_y\) obtained from segmentation-model inference [2409.06105]. Pyramid Feature Learning aggregates detail and semantic features without additional trainable modules in the core autoencoder path [2409.06105].

Empirically, SGC-VQGAN reports \(100\%\) active tokens and substantially improved semantic uniqueness relative to VQGAN and CVQ-VAE, with ratios \(0.8978\) at threshold \(\tau=0.7\) and \(0.7511\) at \(\tau=0.9\) for the reported configuration [2409.06105]. On ImageNet with latent \(32\times32\) and \(K=16384\), it reports PSNR \(23.26\), SSIM \(0.6903\), LPIPS \(0.1745\), and unconditional FID \(19.92\); on NuScenes it reports FVD \(825.95\), outperforming the listed VQGAN and CVQ-VAE baselines [2409.06105]. The model is therefore “multi-level” in feature semantics and codebook organization, not in the number of latent scales.

SeQ-GAN provides a closely related but explicitly conceptual contribution. It argues that VQ tokenizers face two competing objectives: **semantic compression** and **details preservation**, and that improving reconstruction fidelity does not necessarily improve downstream generation [2212.03185]. Its tokenizer is single-level, with latent grid \(16\times16\), codebook size \(K=1024\), and embedding dimension \(D=256\), but it is trained in two phases: a first phase with semantic-enhanced perceptual loss emphasizing deep features and classifier logits, and a second phase that freezes encoder and codebook while finetuning an enhanced decoder for local detail recovery [2212.03185]. Codebook usage is improved to \(100\%\) via entropy regularization [2212.03185]. On \(256\times256\) ImageNet, SeQ-GAN + AR-L reports FID \(6.25\) and IS \(140.9\), while SeQ-GAN + NAR-L reports FID \(4.55\) and IS \(200.4\) in \(12\) steps [2212.03185].

This paper is especially relevant to multi-level VQGAN because it proposes how the two-phase objective could be transferred to hierarchical tokenizers: coarse levels should emphasize semantic compression with high \(\alpha\), whereas fine levels should emphasize details preservation with low \(\alpha\) [2212.03185]. That recommendation does not instantiate a multi-level VQGAN by itself, but it provides an explicit recipe for level-specific objectives.

ViT-VQGAN reinforces a further corrective point: a carefully engineered single-level tokenizer can outperform hierarchical baselines [2110.04627]. It uses a single codebook of size \(K=8192\) on a \(32\times32\) latent grid, factorized code lookup in a low-dimensional space, and \(\ell_2\)-normalized latent and code vectors, reaching codebook usage of approximately \(95\%\)–\(96\%\) [2110.04627]. On ImageNet reconstruction, it reports FID \(1.28\), outperforming vanilla VQGAN variants including a hierarchical codebook baseline at FID \(1.45\) [2110.04627]. This does not invalidate hierarchical quantization, but it shows that “more levels” is not the only route to better tokenization.

## 6. Prior hierarchy, failure modes, and open directions

Multi-level behavior is often introduced after tokenization. TATS is explicit that its VQGAN is single-level: it uses a 3D VQGAN with one codebook, no hierarchical quantization inside the VQGAN, and hierarchy only in the transformer prior through sparse anchor-frame autoregression and an interpolation transformer [2204.03638]. The encoder compresses video with temporal compression rate \(d_t=4\) and spatial compression rate \(d_s=8\), uses codebook size \(K=16384\) and embedding dimension \(c=256\), and adopts replicate padding along the temporal axis so tokens become more shift-equivariant in time [2204.03638]. Long-range video structure is then modeled by a hierarchical transformer with \(24\) layers, \(16\) heads, and embedding size \(1024\), allowing generation up to \(1024\) frames [2204.03638]. On short-video benchmarks, TATS-base reports FVD \(132.6\pm2.6\) on Sky Time-lapse and \(94.6\pm2.7\) on Taichi-HD, while long-horizon coherence metrics on UCF-101 favor TATS-hierarchical over the listed baselines [2204.03638]. The article-level implication is that token hierarchy and prior hierarchy are separable design decisions.

The literature also converges on several recurrent limitations. First, **more levels can destabilize codebooks**: the virtual H&E study reports codebook collapse risks beyond two quantization levels [2508.04734]. Second, **GAN training remains fragile**: TATS reports instability when axial attention was used in the 3D VQ encoder/decoder and relies on pure convolutions, stronger decoders, Synced BatchNorm, feature matching, delayed GAN start, and gradient clipping to avoid discriminator collapse [2204.03638]; MCR-VQGAN similarly uses label smoothing, annealed noise, EMA codebook updates, and \(R1\) penalty for stability [2512.15947]. Third, **distributional mismatch persists in downstream use**: the virtual H&E study found that Cellpose and StarDist were sensitive to subtle domain shifts and that watershed was more robust [2508.04734], while MCR-VQGAN notes the need for external validation across scanners, vendors, and sites [2512.15947]. Fourth, **long-horizon autoregression accumulates error**: TATS reports drift, artifacts, and degeneration to repeated tokens over long horizons, especially in regions with large motion [2204.03638].

Several future directions recur across papers. MCR-VQGAN explicitly identifies hierarchical codebooks, pyramidal quantization, or hybrid diffusion refinement as possible extensions beyond its current single-codebook design [2512.15947]. SGC-VQGAN identifies automatic discovery of semantic prototypes without external segmentation and integration with hierarchical or residual quantization as future work [2409.06105]. Efficient-VQGAN suggests combining multi-grained attention with further memory optimizations or hierarchical latents for ultra-high resolution [2310.05400]. SeQ-GAN proposes level-wise semantic compression and decoder finetuning for multi-level tokenizers [2212.03185].

Taken together, these works establish that Multi-Level VQGAN is not a single architecture but a design space. In its strict form, it uses multiple codebooks or residual quantization stages to distribute coarse and fine information. In broader usage, it denotes single-codebook VQGANs whose effective hierarchy lies in feature extraction, semantic organization, or the prior over tokens. The technical consequence is that any description of a “multi-level VQGAN” must specify where the levels actually reside: in latent topology, in codebook structure, in feature semantics, or in the generative prior.

Source: https://www.emergentmind.com/topics/multi-level-vector-quantized-generative-adversarial-network-vqgan