Multi-Level VQGAN Architectures

• The paper demonstrates that multi-level VQGAN models distribute representations over multiple latent stages or feature partitions to enhance reconstruction quality and semantic consistency.
• It compares hierarchical quantization, multi-scale feature extraction, and prior-level designs, highlighting that the integration point of hierarchy is crucial for achieving stability and improved performance.
• Empirical results in image reconstruction, tissue classification, and audio synthesis indicate that carefully balancing levels—often optimally at two—avoids codebook collapse while refining both coarse and fine details.

Multi-Level Vector-Quantized Generative Adversarial Network (VQGAN) denotes a class of VQGAN-derived models in which representational capacity is distributed across more than one “level,” but the literature uses that label in more than one way. In the narrowest sense, it refers to hierarchical quantization with multiple latent maps or multiple residual quantization stages; in broader usage, it can also denote multi-scale feature extraction, multi-grained attention, or semantic low/high-level feature partitions built around a single codebook. All such models retain the canonical VQGAN pattern—an encoder, a discrete codebook with nearest-neighbor quantization, a decoder, and adversarial and perceptual training—but they differ in where hierarchy is introduced: inside the quantizer, around the tokenizer, or in the prior over tokens (Saurav et al., 5 Aug 2025, Kim et al., 17 Dec 2025, Ding et al., 2024).

1. Terminological scope and conceptual variants

Standard VQGAN maps an input $x$ to a continuous latent $z_e(x)$, replaces each latent vector with its nearest codebook entry, and reconstructs the signal with a decoder trained using reconstruction, perceptual, and adversarial losses. MCR-VQGAN states this explicitly as an encoder–quantizer–decoder pipeline with a learned codebook and GAN supervision, and also distinguishes VQGAN from VQ-VAE by noting that VQGAN augments discrete latents with adversarial and perceptual objectives to sharpen texture and improve realism (Kim et al., 17 Dec 2025).

Across papers, “multi-level VQGAN” is used for at least three non-identical design patterns. First, it can mean hierarchical quantization, where different codebooks or quantization stages encode coarse and fine information. The colorectal virtual H&E model is the clearest instance: it uses two latent feature maps at different spatial resolutions, with separate codebooks for coarse and fine structure (Saurav et al., 5 Aug 2025). DelightfulTTS 2 uses a different hierarchical form—multi-stage residual vector quantization at the bottleneck of a speech codec—where successive stages quantize residual error rather than separate spatial resolutions (Liu et al., 2022).

Second, it can mean multi-scale or multi-level feature extraction around a single codebook. MCR-VQGAN is explicit that “multi-level” refers to multi-resolution processing and multi-scale convolutions, not multiple codebooks or stacked quantizers (Kim et al., 17 Dec 2025). SGC-VQGAN likewise keeps a single latent grid and a single codebook, but enriches it with pyramid feature learning and semantically guided low/high-level partitions (Ding et al., 2024).

Third, hierarchy may exist outside the VQGAN tokenizer itself. TATS states explicitly that its 3D VQGAN uses a single-level codebook and that hierarchy is introduced at the transformer level through sparse-frame autoregression plus an interpolation transformer (Ge et al., 2022). Efficient-VQGAN similarly keeps a single-level codebook while introducing “multi-grained” attention in the generator prior through global block tokens and local windows (Cao et al., 2023).

Usage of “multi-level”	Representative models	Location of hierarchy
Hierarchical quantization	virtual H&E VQGAN (Saurav et al., 5 Aug 2025); DelightfulTTS 2 (Liu et al., 2022)	multiple codebooks or residual quantization stages
Single-codebook multi-scale design	MCR-VQGAN (Kim et al., 17 Dec 2025); SGC-VQGAN (Ding et al., 2024)	encoder/decoder features or semantic partitions
Hierarchy in the prior	TATS (Ge et al., 2022); Efficient-VQGAN (Cao et al., 2023)	transformer or attention granularity

A common misconception is therefore that any “multi-level VQGAN” necessarily uses hierarchical codebooks. Several papers explicitly contradict that reading. MCR-VQGAN has one codebook of size $K=1024$ with embedding dimension $D=512$ (Kim et al., 17 Dec 2025); Efficient-VQGAN uses a single-level codebook with $K=1024$ (Cao et al., 2023); TATS uses a single codebook with $K=16384$ and embedding dimension $c=256$ (Ge et al., 2022).

2. Canonical formulation and multi-level generalizations

The core quantization rule is shared across the literature. For an encoder output $z_e(x)$ and codebook entries $\{e_k\}$, the quantized representation is obtained by nearest-neighbor assignment:

$$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$

This formulation appears in standard VQGAN recaps in MCR-VQGAN, Efficient-VQGAN, SGC-VQGAN, and related papers (Kim et al., 17 Dec 2025, Cao et al., 2023, Ding et al., 2024).

The standard VQ objective combines codebook and commitment terms with stop-gradient:

$z_e(x)$0

These are then combined with reconstruction, perceptual, and adversarial losses. MCR-VQGAN presents a total generator loss

$z_e(x)$1

with $z_e(x)$2, $z_e(x)$3, $z_e(x)$4, and $z_e(x)$5, and uses a PatchGAN discriminator with logistic GAN loss plus $z_e(x)$6 regularization with $z_e(x)$7 (Kim et al., 17 Dec 2025). TATS uses a different adversarial configuration with spatial and temporal discriminators, feature matching, and a minimax objective whose typical weights are $z_e(x)$8, $z_e(x)$9, and $K=1024$0 (Ge et al., 2022). Efficient-VQGAN uses a Projected GAN discriminator with hinge losses and a stage-1 total objective $K=1024$1 (Cao et al., 2023).

Multi-level formulations extend this template in two principal ways. In hierarchical spatial quantization, separate latent maps are quantized at different resolutions, each with its own codebook and commitment term:

$K=1024$2

This is the formulation used in the two-level virtual H&E VQGAN, where the coarse level captures tissue-scale structure and the fine level captures cellular detail (Saurav et al., 5 Aug 2025).

In residual multi-stage quantization, later levels quantize the residual left by earlier levels:

$K=1024$3

DelightfulTTS 2 uses this stage-wise residual VQ at frame rate $K=1024$4 frames/sec, combined with multi-resolution spectrogram loss, adversarial losses from multi-scale and multi-period discriminators, and feature matching (Liu et al., 2022).

These formulations share the same discrete-latent logic but differ in what each level represents. A plausible implication is that “multi-level” is best understood structurally rather than terminologically: the decisive question is whether levels are implemented as multiple codebooks, residual quantizers, feature partitions, or prior-level abstractions.

3. Hierarchical quantization in the strict sense

The colorectal virtual H&E model provides the most explicit image-domain instance of a multi-level VQGAN in the strict sense. The encoder produces two latent feature maps at different spatial resolutions, and each level is quantized by its own codebook of size $K=1024$5. The coarse level models high-level tissue structures, the fine level models cellular details, and the decoder integrates both levels to reconstruct the final H&E image (Saurav et al., 5 Aug 2025). The single-level baseline in the same study follows “Taming Transformers,” uses one codebook of size $K=1024$6, and differs from the multi-level model precisely in the absence of this hierarchical latent decomposition (Saurav et al., 5 Aug 2025).

Its training objective combines $K=1024$7, $K=1024$8, $K=1024$9, $D=512$0, $D=512$1, and a small-weight adversarial term $D=512$2, with $D=512$3. The commitment cost is $D=512$4 for the single-level model and $D=512$5 for the two-level model (Saurav et al., 5 Aug 2025). The adversarial term is introduced after a warm-up—after $D=512$6 iterations on Orion and after $D=512$7 iterations on CODEX—to stabilize the codebooks (Saurav et al., 5 Aug 2025).

On the Orion test set, the two-level VQGAN improved over both cGAN and the one-level VQGAN in image similarity, with $D=512$8, $D=512$9, $K=1024$0, and $K=1024$1, compared with $K=1024$2, $K=1024$3, $K=1024$4, and $K=1024$5 for the one-level VQGAN and $K=1024$6, $K=1024$7, $K=1024$8, and $K=1024$9 for cGAN (Saurav et al., 5 Aug 2025). On the CODEX test set, the two-level model also gave the best reconstruction metrics among the three, with $K=16384$0, $K=16384$1, $K=16384$2, and $K=16384$3 (Saurav et al., 5 Aug 2025). For downstream tissue classification, semantic preservation on Orion rose from $K=16384$4 for cGAN to $K=16384$5 for the one-level VQGAN and $K=16384$6 for the two-level model (Saurav et al., 5 Aug 2025). Nuclei segmentation with a watershed algorithm likewise favored the two-level model on Orion, with mean IoU $K=16384$7 (Saurav et al., 5 Aug 2025).

The same paper also identifies an important limitation: increasing the number of quantization levels beyond two led to codebook collapse risks, and the two-level hierarchy offered the best trade-off between performance and stability (Saurav et al., 5 Aug 2025). This directly contradicts the assumption that deeper quantization hierarchies are monotonically beneficial.

DelightfulTTS 2 shows that strict multi-level VQGAN is not restricted to image pyramids. Its VQ-GAN codec uses hierarchical multi-stage residual vector quantization, reconstructs waveforms with a HiFi-GAN-style generator augmented with a bidirectional Long Expressive Memory layer, and applies multi-scale and multi-period discriminators with discrete wavelet transform downsampling (Liu et al., 2022). The system uses a constant bitrate of $K=16384$8 kbps and a $K=16384$9 ms frame hop, and reports MOS $c=256$0 for DelightfulTTS 2 versus $c=256$1 for DelightfulTTS and $c=256$2 for FastSpeech 2, with CMOS gains of $c=256$3 over DelightfulTTS and $c=256$4 over FastSpeech 2 (Liu et al., 2022). In this setting, multi-level structure corresponds to bitrate-efficient residual refinement rather than coarse/fine image scales.

4. Single-codebook models with multi-scale or multi-grained structure

A second lineage keeps the quantizer single-level but redistributes modeling power around it. MCR-VQGAN is explicit on this point: it is “a 2D VQGAN for MRI→tau PET synthesis with a single discrete latent codebook, but a multi-level/multi-scale feature extractor in the encoder and residual refinement blocks, plus CBAM attention throughout” (Kim et al., 17 Dec 2025). Its encoder uses an initial $c=256$5 convolution, three downsampling stages with channels $c=256$6, multi-scale convolutional branches $c=256$7, $c=256$8, and $c=256$9, six encoding ResNet blocks, and CBAM. The codebook size is $z_e(x)$0, the embedding dimension is $z_e(x)$1, and the codebook is updated by EMA with decay $z_e(x)$2 (Kim et al., 17 Dec 2025).

The decoder is symmetric but uses standard $z_e(x)$3 convolutions in the decoding ResNet blocks for efficiency, followed by three upsampling stages and a final $z_e(x)$4 convolution with Tanh output. The discriminator is a PatchGAN with three $z_e(x)$5 stride-2 convolution layers, and training includes one-sided label smoothing, annealed Gaussian noise on generated images, and $z_e(x)$6 gradient penalty (Kim et al., 17 Dec 2025). On the ADNI-3 test set, MCR-VQGAN achieved $z_e(x)$7, $z_e(x)$8 dB, and $z_e(x)$9, outperforming cGAN, WGAN-GP, CycleGAN, and standard VQGAN (Kim et al., 17 Dec 2025). Its downstream classifier achieved comparable accuracy on real and synthetic tau PET, $\{e_k\}$0 versus $\{e_k\}$1 (Kim et al., 17 Dec 2025). The paper therefore uses “multi-level” to denote multi-resolution and multi-scale processing, not hierarchical codebooks.

Efficient-VQGAN makes an analogous distinction. Its stage-1 tokenizer uses a Swin Transformer-based encoder and decoder with local window attention, a single-level codebook with $\{e_k\}$2, and a Projected GAN discriminator (Cao et al., 2023). Hierarchy appears in stage 2 as multi-grained attention, where a latent token matrix is split into non-overlapping blocks, each block is summarized by one global token, and local generation attends to both global block tokens and an extended local neighborhood (Cao et al., 2023). The paper states explicitly that “Multi-Level” refers to multi-grained attention rather than hierarchical quantization (Cao et al., 2023). On ImageNet $\{e_k\}$3, Efficient-VQGAN achieved reconstruction FID $\{e_k\}$4 at latent size $\{e_k\}$5, and class-conditioned synthesis FID $\{e_k\}$6 with IS $\{e_k\}$7 under $\{e_k\}$8 acceptance-rate rejection sampling (Cao et al., 2023).

MoVQ also remains outside strict hierarchical quantization. It introduces multichannel quantization by splitting encoder channels into $\{e_k\}$9 chunks, quantizing each chunk independently with a shared codebook, and decoding through spatially conditional normalization so identical discrete indices can be modulated differently across locations (Zheng et al., 2022). For $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$0 images, it uses a $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$1 latent and a $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$2 codebook, with recombination capacity scaling as $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$3 while sequence length remains $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$4 (Zheng et al., 2022). The paper explicitly states that this design is not hierarchical across scales or residual stages (Zheng et al., 2022). On ImageNet, MoVQ-mask achieved FID $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$5 and IS $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$6 in $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$7 steps, while MoVQ-auto achieved FID $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$8 and IS $$k^* = \arg\min_k \|z_e(x) - e_k\|_2,\qquad z_q = e_{k^*}.$$9 in $z_e(x)$00 steps (Zheng et al., 2022).

These models are important because they show that a substantial fraction of “multi-level VQGAN” practice is actually single-codebook VQGAN with structured feature allocation. The improvement comes not from adding more quantizers, but from redesigning receptive fields, feature pathways, or attention topology around the same quantizer.

5. Semantic organization, codebook usage, and tokenizer objectives

A third line of work treats “levels” as semantic strata within the tokenizer. SGC-VQGAN keeps a single latent grid but splits features into low-level and high-level parts, introduces Semantic Online Clustering, and applies a semantic consistency objective based on CosFace (Ding et al., 2024). Its quantization distance is explicitly multi-level:

$z_e(x)$01

The codebook carries paired low/high-level features, and high-level entries are updated toward both encoded anchors and semantic class prototypes $z_e(x)$02 obtained from segmentation-model inference (Ding et al., 2024). Pyramid Feature Learning aggregates detail and semantic features without additional trainable modules in the core autoencoder path (Ding et al., 2024).

Empirically, SGC-VQGAN reports $z_e(x)$03 active tokens and substantially improved semantic uniqueness relative to VQGAN and CVQ-VAE, with ratios $z_e(x)$04 at threshold $z_e(x)$05 and $z_e(x)$06 at $z_e(x)$07 for the reported configuration (Ding et al., 2024). On ImageNet with latent $z_e(x)$08 and $z_e(x)$09, it reports PSNR $z_e(x)$10, SSIM $z_e(x)$11, LPIPS $z_e(x)$12, and unconditional FID $z_e(x)$13; on NuScenes it reports FVD $z_e(x)$14, outperforming the listed VQGAN and CVQ-VAE baselines (Ding et al., 2024). The model is therefore “multi-level” in feature semantics and codebook organization, not in the number of latent scales.

SeQ-GAN provides a closely related but explicitly conceptual contribution. It argues that VQ tokenizers face two competing objectives: semantic compression and details preservation, and that improving reconstruction fidelity does not necessarily improve downstream generation (Gu et al., 2022). Its tokenizer is single-level, with latent grid $z_e(x)$15, codebook size $z_e(x)$16, and embedding dimension $z_e(x)$17, but it is trained in two phases: a first phase with semantic-enhanced perceptual loss emphasizing deep features and classifier logits, and a second phase that freezes encoder and codebook while finetuning an enhanced decoder for local detail recovery (Gu et al., 2022). Codebook usage is improved to $z_e(x)$18 via entropy regularization (Gu et al., 2022). On $z_e(x)$19 ImageNet, SeQ-GAN + AR-L reports FID $z_e(x)$20 and IS $z_e(x)$21, while SeQ-GAN + NAR-L reports FID $z_e(x)$22 and IS $z_e(x)$23 in $z_e(x)$24 steps (Gu et al., 2022).

This paper is especially relevant to multi-level VQGAN because it proposes how the two-phase objective could be transferred to hierarchical tokenizers: coarse levels should emphasize semantic compression with high $z_e(x)$25, whereas fine levels should emphasize details preservation with low $z_e(x)$26 (Gu et al., 2022). That recommendation does not instantiate a multi-level VQGAN by itself, but it provides an explicit recipe for level-specific objectives.

ViT-VQGAN reinforces a further corrective point: a carefully engineered single-level tokenizer can outperform hierarchical baselines (Yu et al., 2021). It uses a single codebook of size $z_e(x)$27 on a $z_e(x)$28 latent grid, factorized code lookup in a low-dimensional space, and $z_e(x)$29-normalized latent and code vectors, reaching codebook usage of approximately $z_e(x)$30–$z_e(x)$31 (Yu et al., 2021). On ImageNet reconstruction, it reports FID $z_e(x)$32, outperforming vanilla VQGAN variants including a hierarchical codebook baseline at FID $z_e(x)$33 (Yu et al., 2021). This does not invalidate hierarchical quantization, but it shows that “more levels” is not the only route to better tokenization.

6. Prior hierarchy, failure modes, and open directions

Multi-level behavior is often introduced after tokenization. TATS is explicit that its VQGAN is single-level: it uses a 3D VQGAN with one codebook, no hierarchical quantization inside the VQGAN, and hierarchy only in the transformer prior through sparse anchor-frame autoregression and an interpolation transformer (Ge et al., 2022). The encoder compresses video with temporal compression rate $z_e(x)$34 and spatial compression rate $z_e(x)$35, uses codebook size $z_e(x)$36 and embedding dimension $z_e(x)$37, and adopts replicate padding along the temporal axis so tokens become more shift-equivariant in time (Ge et al., 2022). Long-range video structure is then modeled by a hierarchical transformer with $z_e(x)$38 layers, $z_e(x)$39 heads, and embedding size $z_e(x)$40, allowing generation up to $z_e(x)$41 frames (Ge et al., 2022). On short-video benchmarks, TATS-base reports FVD $z_e(x)$42 on Sky Time-lapse and $z_e(x)$43 on Taichi-HD, while long-horizon coherence metrics on UCF-101 favor TATS-hierarchical over the listed baselines (Ge et al., 2022). The article-level implication is that token hierarchy and prior hierarchy are separable design decisions.

The literature also converges on several recurrent limitations. First, more levels can destabilize codebooks: the virtual H&E study reports codebook collapse risks beyond two quantization levels (Saurav et al., 5 Aug 2025). Second, GAN training remains fragile: TATS reports instability when axial attention was used in the 3D VQ encoder/decoder and relies on pure convolutions, stronger decoders, Synced BatchNorm, feature matching, delayed GAN start, and gradient clipping to avoid discriminator collapse (Ge et al., 2022); MCR-VQGAN similarly uses label smoothing, annealed noise, EMA codebook updates, and $z_e(x)$44 penalty for stability (Kim et al., 17 Dec 2025). Third, distributional mismatch persists in downstream use: the virtual H&E study found that Cellpose and StarDist were sensitive to subtle domain shifts and that watershed was more robust (Saurav et al., 5 Aug 2025), while MCR-VQGAN notes the need for external validation across scanners, vendors, and sites (Kim et al., 17 Dec 2025). Fourth, long-horizon autoregression accumulates error: TATS reports drift, artifacts, and degeneration to repeated tokens over long horizons, especially in regions with large motion (Ge et al., 2022).

Several future directions recur across papers. MCR-VQGAN explicitly identifies hierarchical codebooks, pyramidal quantization, or hybrid diffusion refinement as possible extensions beyond its current single-codebook design (Kim et al., 17 Dec 2025). SGC-VQGAN identifies automatic discovery of semantic prototypes without external segmentation and integration with hierarchical or residual quantization as future work (Ding et al., 2024). Efficient-VQGAN suggests combining multi-grained attention with further memory optimizations or hierarchical latents for ultra-high resolution (Cao et al., 2023). SeQ-GAN proposes level-wise semantic compression and decoder finetuning for multi-level tokenizers (Gu et al., 2022).

Taken together, these works establish that Multi-Level VQGAN is not a single architecture but a design space. In its strict form, it uses multiple codebooks or residual quantization stages to distribute coarse and fine information. In broader usage, it denotes single-codebook VQGANs whose effective hierarchy lies in feature extraction, semantic organization, or the prior over tokens. The technical consequence is that any description of a “multi-level VQGAN” must specify where the levels actually reside: in latent topology, in codebook structure, in feature semantics, or in the generative prior.