VQBridge: Efficient Codebook Optimization

• VQBridge is a projector that addresses quantization challenges by aligning codebook and encoder distributions using a compress-process-recover pipeline.
• It achieves 100% codebook usage by integrating learning annealing with a global attention mechanism, significantly enhancing reconstruction and generation quality.
• Empirical results demonstrate that VQBridge scales effectively, outperforming conventional linear and MLP projectors even with massive codebooks and diverse configurations.

VQBridge is a robust, scalable, and efficient projector based on the map function method for training vector quantized networks (VQN), introduced to maintain high codebook usage during learning annealing and codebook size expansion (Chang et al., 12 Sep 2025). It optimizes code vectors through a compress-process-recover pipeline, enabling stable and effective codebook training; when combined with learning annealing, the resulting VQN achieves full (100%) codebook usage across diverse codebook configurations, a regime termed FVQ (FullVQ) (Chang et al., 12 Sep 2025). In the surrounding literature, the label has also appeared in looser senses: ReVQ is described as a practical visual tokenizer, or “VQBridge,” for multimodal models, and an earlier diffusion-based VQ-VAE prior is “sometimes referred to as ‘VQBridge’,” reflecting a broader pattern in which vector quantization is used as a bridge between continuous representations and discrete tokens (Zhang et al., 14 Jul 2025, Cohen et al., 2022).

1. Problem setting and motivation

Vector Quantized Networks are central to converting images into discrete tokens for autoregressive and diffusion-based image generation models, but scaling VQNs to large codebook sizes and high vector channel dimensions has been challenging due to straight-through estimation bias, one-step-behind updates, and sparse codebook gradients (Chang et al., 12 Sep 2025). The paper identifies three fundamental pathologies. First, the straight-through estimator introduces estimation error because the quantized latent and the encoder output do not coincide. Second, codebook and latent representations are updated out of sync, destabilizing training through a one-step-behind update. Third, only a small subset of code vectors receive gradients, which leads to codebook collapse and low utilization.

The underlying quantization setup follows the standard VQ formulation. The encoder outputs a latent $\mathbf{z}_e$, the quantizer assigns it to the nearest codebook entry,

$x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$

and the quantized latent is

$\mathbf{z}_q = \mathbf{c}_x.$

Because $\arg\min$ is non-differentiable, the straight-through estimator is used for backpropagation,

$$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$

together with a commitment loss

$$L_{cmt}(\mathbf{z}_e, \mathbf{z}_q) = d(\mathbf{z}_e, \operatorname{sg}[\mathbf{z}_q]) + \beta d(\mathbf{z}_q, \operatorname{sg}[\mathbf{z}_e]).$$

The VQBridge proposal is motivated by the claim that linear projectors, MLPs, and related prior approaches insufficiently address these three issues jointly, especially at scale.

2. Map-function formulation

The key insight is that addressing straight-through estimation bias, one-step-behind updates, and sparse gradients jointly needs a strong map function $f(\cdot)$ to align codebook and encoder output distributions, together with learning annealing (Chang et al., 12 Sep 2025). VQBridge therefore acts as a mapping function on the codebook,

$\hat{\mathcal{C}} = f(\mathcal{C}),$

with the objective of reducing the distance between encoder outputs and the transformed codebook:

$$D(\mathcal{Z}_e, \hat{\mathcal{C}}) = \frac{1}{|\mathcal{Z}_e|} \sum_{\mathbf{z}_i \in \mathcal{Z}_e} \min_{\mathbf{c}_i \in \hat{\mathcal{C}}} d(\mathbf{z}_i, \mathbf{c}_i).$$

A smaller $D(\cdot,\cdot)$ indicates better distribution matching.

Within this formulation, VQBridge is not a replacement for vector quantization itself; it is a projector that jointly updates and regularizes the codebook. This suggests a specific interpretation of the method’s name: the “bridge” is the learned alignment between the codebook geometry and the encoder’s latent distribution. The paper’s emphasis is that global alignment, rather than only local winner-take-all updates, is necessary for scalable codebook training.

3. Compress-process-recover projector

VQBridge replaces the fragile linear or shallow MLP mapping with a compress-process-recover pipeline using global attention via ViT blocks (Chang et al., 12 Sep 2025). In the compress stage, the codebook $x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$0 is divided into $x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$1 groups, and each group is projected and normalized:

$x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$2

In the process stage, the grouped representation is passed through $x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$3 Vision Transformer blocks,

$x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$4

In the recover stage, the representation is expanded back to the original codebook size:

$x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$5

followed by reshaping to obtain the updated codebook $x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$6.

This architecture is designed to operate on the codebook rather than on per-sample latents. The details identify this as important operationally: VQBridge runs only during training and is not used at inference, so it introduces zero codebook or runtime overhead after training. The use of global attention is presented as the mechanism by which all codebook entries receive updates, rather than only the winning entries selected by nearest-neighbor quantization.

4. Learning annealing, FullVQ, and optimization behavior

The method is coupled to learning annealing, and the combination is presented as necessary for stable large-scale training (Chang et al., 12 Sep 2025). The paper argues that a smaller learning rate helps reduce the one-step-behind effect and improves overall alignment, and the appendix-level analysis summarized in the data states that learning annealing renders the delayed update error negligible, proportional to $x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$7.

The term FVQ, or FullVQ, denotes the regime obtained by combining VQBridge with learning annealing to achieve full (100%) codebook usage across diverse codebook configurations. The reported behavior is stronger than simple anti-collapse stabilization: codebook usage remains at 100% even with a 262k-codebook, and the model consistently improves with larger codebooks, higher vector channels, or longer training. Ablation results summarized in the data further state that linear and MLP projectors lose codebook usage, especially at scale, whereas VQBridge maintains 100% usage in all tested settings. The best ViT depth is reported as $x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$8, and patch-size scaling is described as a co-scaling rule, with the example $x = \arg\min_{i} d(\mathbf{z}_e, \mathbf{c}_i),$9k codebook $\mathbf{z}_q = \mathbf{c}_x.$0 $\mathbf{z}_q = \mathbf{c}_x.$1 patch size.

A plausible implication is that VQBridge should be understood not merely as an anti-collapse heuristic but as a training-time codebook optimizer whose effectiveness depends on the interaction between map-function capacity and annealed optimization.

5. Empirical performance and scalability

On ImageNet $\mathbf{z}_q = \mathbf{c}_x.$2 reconstruction, the reported results combine full codebook usage with state-of-the-art reconstruction quality (Chang et al., 12 Sep 2025).

Configuration	rFID $\mathbf{z}_q = \mathbf{c}_x.$3	Codebook Usage $\mathbf{z}_q = \mathbf{c}_x.$4
FVQ, 16,384 codebook, 256 channel, 40 epochs	1.30	100%
FVQ, 16,384 codebook, 256 channel, 120 epochs	1.17	100%
FVQ, 262,144 codebook, 256 channel	0.95	100%
FVQ, 262,144 codebook, 256 channel, 120 epochs	0.88	100%

The same summary compares FVQ with earlier VQ methods at $\mathbf{z}_q = \mathbf{c}_x.$5 codebook size and 256 vector channels: VQGAN has rFID $\mathbf{z}_q = \mathbf{c}_x.$6 with $\mathbf{z}_q = \mathbf{c}_x.$7 codebook usage, LlamaGen has rFID $\mathbf{z}_q = \mathbf{c}_x.$8 with $\mathbf{z}_q = \mathbf{c}_x.$9 usage, and IBQ has rFID $\arg\min$0 with $\arg\min$1 usage. The paper’s scaling claim is that larger codebooks improve reconstruction linearly and collapse is never observed in the reported experiments.

The downstream consequence is evaluated by integrating FVQ with LlamaGen for class-conditional ImageNet $\arg\min$2 generation. The FVQ-enabled variant, LlamaGen-XL + FVQ, reaches FID $\arg\min$3, IS $\arg\min$4, Precision $\arg\min$5, and Recall $\arg\min$6, compared with LlamaGen-XL at FID $\arg\min$7, IS $\arg\min$8, Precision $\arg\min$9, and Recall $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$0. The same comparison places FVQ-XL ahead of VAR-d20 at FID $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$1 and DiT-XL/2 at FID $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$2. The data also states that FVQ generalizes to multi-code VQ designs such as RQ-VAE and VAR, and that with only 10 epochs FVQ surpasses RQ-VAE at 50 epochs.

6. Terminological scope and related bridge-based VQ methods

The name “VQBridge” is not used uniformly across the literature (Zhang et al., 14 Jul 2025, Cohen et al., 2022, Liu et al., 2024). In the 2025 FVQ paper, VQBridge is the specific projector described above. In “Quantize-then-Rectify: Efficient VQ-VAE Training,” ReVQ is presented as a framework that transforms a pre-trained VAE into a VQ-VAE tokenizer by freezing the VAE encoder and decoder and training only a quantizer and a lightweight rectifier; the summary explicitly describes it as serving as a practical visual tokenizer, or “VQBridge,” for multimodal models. ReVQ compresses ImageNet images into at most 512 tokens with rFID $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$3, and full training finishes on a single NVIDIA 4090 in approximately 22 hours, in contrast to comparable methods requiring 4.5 days on 32 A100 GPUs.

A different usage appears in “Diffusion bridges vector quantized Variational AutoEncoders,” where the framework is “sometimes referred to as ‘VQBridge’.” There, the bridge is a learned diffusion bridge between a continuous coded vector and a non-informative prior distribution, replacing the usual autoregressive prior in VQ-VAE and enabling end-to-end training. The data reports latent-space NLL improvements over PixelCNN priors on mini-Imagenet and CIFAR, and sampling speedups of roughly $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$4–$$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$5.

Neighboring work also develops related bridge or alignment ideas without using the same name. AlignedVQ for edge-cloud collaborative VLMs uses quantization after normalization layers, Dual Linear Projection, and joint task-aware finetuning, achieving approximately $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$6 compression of intermediate features while remaining within $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$7 to $$\hat{\mathbf{z}} = \mathbf{z}_q + \operatorname{sg}[\mathbf{z}_e - \mathbf{z}_q],$$8 of original model accuracy across eight VQA datasets (Liu et al., 2024). This suggests that, across several subfields, “bridge” language recurs whenever vector quantization mediates between two representational regimes: encoder and codebook distributions, continuous latents and discrete states, or edge-side and cloud-side computation.

7. Significance within vector-quantized modeling

Within the specific 2025 formulation, VQBridge’s significance is that it turns high codebook utilization into a first-class optimization target rather than an incidental diagnostic (Chang et al., 12 Sep 2025). The method is presented as robust, scalable, and generalizable, with no inference cost increase, and the reported evidence ties full utilization directly to improved reconstruction and stronger downstream autoregressive image generation. In that sense, FVQ reframes codebook usage from a health metric into an enabling condition for scaling VQ tokenizers.

Across the broader literature, the recurring bridge motif indicates a shared systems-level role for vector quantization. ReVQ uses quantization and rectification to bridge pre-trained VAEs and multimodal tokenization; diffusion bridges connect continuous latent priors with discrete VQ-VAE codes; AlignedVQ bridges partitioned execution in edge-cloud VLMs. This suggests that VQBridge, in its narrow and broad senses, belongs to a family of methods that treat discrete token formation as a structured interface rather than only a compression primitive.