---
title: 'ReVQ: Quantize-then-Rectify for Efficient VQ-VAE'
url: https://www.emergentmind.com/topics/quantize-then-rectify-revq
type: topic
---

# ReVQ: Quantize-then-Rectify for Efficient VQ-VAE

Searching arXiv for the cited papers to ground the article.
{"query":"id:2507.10547 OR id:2504.19874 OR id:2602.02001 OR id:1911.07816 OR id:2509.19186 OR id:2403.17236","max_results":10}
{"query":"\"Quantize-then-Rectify: Efficient VQ-VAE Training\"","max_results":5}
Quantize-then-Rectify (ReVQ) denotes a class of two-stage quantization procedures in which a continuous object is first mapped to a discrete representation and is then corrected by a secondary mechanism that targets the dominant quantization-induced error. In the formulation introduced for efficient VQ-VAE training, ReVQ converts a pre-trained continuous VAE into a discrete VQ-VAE by quantizing the frozen VAE latent and learning a lightweight post rectifier that pushes the quantized latent back into the decoder’s acceptable manifold [2507.10547]. In the broader literature represented here, closely related quantize-then-rectify designs appear in vector quantization for inner-product preservation, post-training quantization for large language models, latent-space image compression, residual vector quantization for neural audio codecs, and quantized compressed sensing [2504.19874] [2602.02001] [2403.17236] [2509.19186] [1911.07816].

## 1. Conceptual structure and terminology

Across these works, the defining structure is consistent: the quantization stage enforces a bitrate, token, or numerical precision constraint, while the rectification stage compensates for the specific failure mode introduced by that constraint. The rectifier may be a learned residual predictor, a low-rank correction, a 1-bit residual estimator, a beam-search refinement, or a convex decoder based on rectified linear penalties. What is rectified is not uniform across domains: in some cases it is latent reconstruction error, in others inner-product bias, weight perturbation, or suboptimal discrete code selection.

| Setting | Quantized object | Rectification mechanism |
|---|---|---|
| Efficient VQ-VAE training [2507.10547] | Frozen VAE latents | Lightweight post rectifier \(g\) in latent space |
| TurboQuant [2504.19874] | Rotated vector coordinates | 1-bit QJL correction on the residual |
| LLM PTQ / QER baseline [2602.02001] | Weight matrix \(W\) | Low-rank correction \(LR\) after quantization |
| Neural image compression [2403.17236] | Latent \(q=Q(y)\) | Quantization Rectifier \(R(q)\) before synthesis |
| RVQ neural codecs [2509.19186] | Level-wise RVQ indices | Beam-search refinement of code combinations |
| Quantized compressed sensing [1911.07816] | One-bit or multi-bit measurements | Convex ReLU-based reconstruction program |

This common structure makes ReVQ less a single algorithm than a design pattern. The quantizer establishes the discrete bottleneck; the rectifier is then tailored to the quantity one wishes to preserve under that bottleneck.

## 2. ReVQ for efficient VQ-VAE training

In "Quantize-then-Rectify: Efficient VQ-VAE Training" [2507.10547], ReVQ is a fast, tolerance-aware method for turning a pre-trained continuous VAE into a discrete VQ-VAE. The central observation is that modern VAEs such as DC-AE exhibit strong noise tolerance: when Gaussian noise is injected into DC-AE’s 2048-D latent, reconstructions remain visually high-quality for noise variance \(\le 0.3\) and degrade beyond this, while the appendix uses latent MSE \(\le 0.1\) as a practical acceptability criterion. This motivates a training strategy that does not retrain the heavy VAE, but instead keeps quantization error within the decoder’s tolerance and learns a small corrector for the residual error.

The pipeline is explicit. A pre-trained DC-AE encoder maps an input image \(x\) to a latent \(z \in \mathbb{R}^{2048}\), normalized by global dataset mean and variance. The VAE encoder and decoder are frozen. The normalized latent is discretized by channel multi-group quantization, and the quantized latent is then processed by a lightweight post rectifier \(g\), implemented as an EfficientViT block with matched input and output dimensionality and no up- or down-sampling. The rectified latent is finally passed through the frozen decoder. Formally, if \(z_q = q(z)\) and \(z_r = g(q(z))\), the training objective is
\[
L_{\mathrm{ReVQ}} = \|z - g(q(z))\|_2^2,
\]
and the optimization problem is
\[
\min_{\theta_g,\mathcal{C}} \|z - g(q(z))\|_2^2.
\]
No perceptual, adversarial, or commitment losses are used; nearest-neighbor assignments are handled with standard straight-through behavior, while the VAE itself is treated as a black box [2507.10547].

This formulation shifts the optimization burden from full-resolution image space to low-resolution latent space. A plausible implication is that ReVQ’s efficiency is not merely an implementation detail but follows directly from the decision to freeze the compute-heavy encoder and decoder and to optimize only the latent quantizer and rectifier.

## 3. Quantization geometry, code capacity, and stabilization

The ReVQ quantizer in [2507.10547] partitions the latent \(z \in \mathbb{R}^{C \times H \times W}\) into \(G\) channel groups, with per-group dimensionalities \(d_g\) such that \(\sum_{g=1}^{G} d_g = C\). Each group has its own codebook \(E^{(g)} \in \mathbb{R}^{K_g \times d_g}\), and each group slice \(z_g(h,w)\) is quantized by nearest-neighbor search:
\[
k_g^*(h,w) = \arg\min_k \|z_g(h,w) - e_k^{(g)}\|_2^2.
\]
The full quantized representation is the concatenation
\[
q(z)(h,w) = \operatorname{concat}\!\left(e_{k_1^*}^{(1)}, \ldots, e_{k_G^*}^{(G)}\right).
\]

For DC-AE on ImageNet, \(H=W=1\) and \(C=2048\), so the token count \(B\) is taken to equal the number of channel groups \(G\). Each group therefore corresponds to one token index. The reported configurations are \(B=512\) and \(B=256\). With \(B=512\), each token covers \(d=2048/512=4\) channels and each group uses \(K=16{,}384\) codes; with \(B=256\), each token covers \(d=8\) channels and each group uses either \(K=65{,}536\) or \(K=262{,}144\) codes. If all groups share \(K\) entries, the number of code combinations per position is \(K^G\). The paper gives the capacity intuition that for \(B=256\) and a single shared codebook size \(N=1024\), the combinations scale as \(M=N^B \approx 2^{2560} \gg\) ImageNet’s 1.28M images, while arguing that conventional single-codebook spatial quantizers underutilize this capacity because of symmetry bias [2507.10547].

Two implementation choices are particularly important. First, channel splitting consistently outperforms spatial splitting: for \(B=512, K=16{,}384\), rFID is \(1.06\) with channel splitting versus \(1.11\) with spatial splitting, and for \(B=256, K=65{,}536\), rFID is \(2.57\) versus \(2.91\). Second, ReVQ uses a Non-Activation Reset at the end of each epoch to prevent index collapse. Without reset, codebook utilization falls to \(65.3\%\) as codebook size grows; with reset, utilization stays above \(97\%\). In a 2D synthetic setting, the same mechanism reduces quantization error from \(2.8\) to \(0.4\). The rectifier also matters materially: for \(B=512, K=16{,}384\), a ViT-based rectifier gives rFID \(1.06\), compared with \(1.08\) for a CNN and \(1.09\) for an MLP; for \(B=256, K=65{,}536\), the corresponding values are \(2.57\), \(3.00\), and \(4.58\). The paper further reports that adding another encoder before the quantizer is harmful, with rFID jumping to approximately \(189\) for \(B=512\), which is why the design remains rectifier-only [2507.10547].

## 4. Reconstruction quality and compute profile

ReVQ is reported to compress ImageNet images into at most \(512\) tokens while sustaining competitive reconstruction quality, with ReVQ\(_{512T}\) reaching rFID \(=1.06\) [2507.10547]. The reported ImageNet validation metrics and training times are as follows.

| Variant | Token / codebook setting | Reported outcome |
|---|---|---|
| ReVQ\(_{512T}\) | \(B=512,\ K=16{,}384\) | SSIM 0.690, PSNR 23.700, LPIPS 0.092, rFID 1.06, \(\sim 22\) hours on a single NVIDIA RTX 4090 |
| ReVQ\(_{256T}\) | \(B=256,\ K=65{,}536\) | SSIM 0.620, PSNR 21.690, LPIPS 0.129, rFID 2.57, \(\sim 26\) hours |
| ReVQ\(_{256T}\) | \(B=256,\ K=262{,}144\) | SSIM 0.640, PSNR 21.960, LPIPS 0.121, rFID 2.05 |
| ReVQ\(_{512T}\) variant | \(K=262{,}144\) | \(\sim 40\) hours |

The efficiency claim is central. ReVQ completes full training on a single NVIDIA 4090 in approximately \(22\) hours, whereas the cited comparison point for MaskBit is \(3456\) GPU hours, described as \(4.5\) days on \(32\times\)A100 for \(1.35\)M iterations. The reported explanation is architectural: compute-heavy shallow layers of image-space encoders and decoders dominate FLOPs in standard VQ-VAEs, whereas ReVQ freezes the pre-trained VAE and trains only a low-resolution latent quantizer and rectifier. The omission of GAN and perceptual branches, and the absence of encoder and decoder gradients, further reduce memory and wall-clock time [2507.10547].

Against other tokenizers, the paper reports representative rFID values of \(1.28\) for ViT-VQGAN with \(1024\) tokens, \(1.12\) for Mo-VQGAN with \(1024\) tokens, \(1.61\) for MaskBit with \(256\) tokens, \(1.71\) for TiTok-S-128 with \(128\) tokens, \(1.34\) for CODA with \(2560\) tokens, and \(1.11\) for TokenBridge with \(4096\) tokens. Within that comparison set, ReVQ\(_{512T}\) is presented as offering short sequences, competitive reconstruction, and single-GPU training [2507.10547].

## 5. Domain-specific variants of the quantize-then-rectify pattern

The same quantize-then-rectify logic appears in several adjacent areas, but the object being preserved differs from case to case.

**TurboQuant**: TurboQuant first applies a random orthogonal transform, then performs per-coordinate \(b\)-bit Lloyd-Max scalar quantization to minimize MSE, and finally rectifies the systematic inner-product bias of the MSE quantizer by adding an unbiased estimate of the residual inner product obtained from a 1-bit Quantized Johnson–Lindenstrauss transform [2504.19874]. The paper states that the MSE distortion satisfies
\[
D_{\mathrm{mse}}(Q_{\mathrm{mse}}) \le \frac{\sqrt{3}\pi}{2}\frac{1}{4^b},
\]
while any randomized vector quantizer obeys the lower bound
\[
D_{\mathrm{mse}}(Q) \ge \frac{1}{4^b},
\]
so TurboQuant is within the universal constant factor \(\frac{\sqrt{3}\pi}{2} \approx 2.7\) of the lower bound. In KV-cache quantization, it reports absolute quality neutrality at \(3.5\) bits per channel and marginal quality degradation at \(2.5\) bits per channel.

**LLM post-training quantization and QER**: In the SRR framework, ReVQ refers to the quantize-then-rectify baseline in which a full-precision weight matrix is approximated as
\[
W \approx Q + LR,
\]
with the full rank budget devoted to fitting the quantization residual after quantization [2602.02001]. SRR generalizes this by preserving a top-\(k\) singular subspace before quantization and using the remaining rank \(r-k\) for quantization error reconstruction afterward. In that formulation, ReVQ is exactly the \(k=0\) case. The paper reports consistent perplexity reductions in PTQ and a \(5.9\) percentage-point average gain on GLUE under \(2\)-bit QPEFT.

**Neural image compression**: "Neural Image Compression with Quantization Rectifier" inserts a latent-space predictor between quantization and synthesis, with
\[
\hat y = R(q;\theta) = q + f(q;\theta),
\]
while leaving the entropy model and bitstream unchanged [2403.17236]. The full training objective is
\[
L_{\phi,\psi,\theta} = R + \lambda D(x,\hat x) + \alpha \|y-\hat y\|_2^2.
\]
On Kodak, the paper reports universal rate-distortion gains at unchanged bpp, including for Attn+QR an average PSNR gain of approximately \(+0.17\) dB with a maximum of \(+0.21\) dB, and an average MS-SSIM gain of approximately \(+0.19\) dB with a maximum of \(+0.25\) dB.

**RVQ-based neural audio codecs**: In "Improving Test-Time Performance of RVQ-based Neural Codecs", quantize-then-rectify takes the form of a test-time beam search over RVQ code combinations, replacing greedy level-wise code selection with a search that keeps multiple candidate sums and prunes them by latent-space error \(\|x-\hat x\|_2^2\) [2509.19186]. Greedy RVQ is recovered when the beam width and per-candidate expansion are both \(1\). On LibriTTS with EnCodec at \(6\) kbps, moving from \(B=1\) to \(B=16\) reduces latent quantization error from \(5.096 \pm 0.018\) to \(4.625 \pm 0.017\), improves PESQ from \(2.726 \pm 0.010\) to \(2.850 \pm 0.010\), and raises SI-SNR from \(4.239 \pm 0.137\) to \(4.649 \pm 0.138\).

**Quantized compressed sensing by ReLUs**: In the treatment associated here with [1911.07816], the quantize-then-rectify step is a convex reconstruction program built from rectified linear penalties after one-bit or uniform multi-bit quantization with dither. For one-bit measurements, the functional is
\[
L_q(z) = \frac{1}{m}\sum_{i=1}^{m}\big[-q_i(\langle a_i,z\rangle+\tau_i)\big]_+,
\]
and reconstruction is obtained by minimizing \(L_q(z)\) over a convex low-complexity set \(T\). The paper proves near-optimal uniform recovery guarantees under isotropic, symmetric, \(L\)-subgaussian sensing, with explicit regimes for additive noise and adversarial bit corruptions.

## 6. Scope, limitations, and recurring misconceptions

A recurring misconception is that rectification necessarily requires retraining the full model or changing the compressed representation itself. The surveyed instantiations show otherwise. In efficient VQ-VAE training, the heavy VAE encoder and decoder are frozen and only the latent quantizer and rectifier are trained; in latent-space image compression, the bitstream and entropy model are unchanged because only \(q\) is entropy-coded; in RVQ-based neural codecs, the codebooks and decoder remain unchanged and rectification is purely a test-time search procedure [2507.10547] [2403.17236] [2509.19186].

Another misconception is that ReVQ always rectifies the same error signal. The cases above show multiple targets: latent MSE in VQ-VAE conversion and image compression, inner-product bias in TurboQuant, weight perturbation in LLM PTQ, search suboptimality in RVQ codecs, and measurement inconsistency in compressed sensing [2504.19874] [2602.02001] [1911.07816]. This suggests that ReVQ is best understood as a placement principle—put the corrective operator after the discrete bottleneck—rather than as a fixed architecture.

The limitations are domain-specific. For VQ-VAE training, the method achieves strong quality at \(\le 512\) tokens but does not reach ultra-short sequences such as \(32\) tokens, and the paper attributes this to rectifier design and the exponential growth in required codebook size as tokens decrease; it also states that, for a fixed VAE architecture, there is an inherent upper bound on achievable compression without retraining the decoder [2507.10547]. TurboQuant depends on the high-dimensional near-independence of rotated coordinates and notes reduced efficacy in very low dimensions or under extremely heavy-tailed inputs [2504.19874]. The SRR analysis argues that allocating the entire low-rank budget to post-quantization correction can be suboptimal when dominant activation-scaled directions should instead be preserved before quantization [2602.02001]. The audio beam-search variant introduces a compute-latency trade-off, especially on CPU, and the compressed sensing guarantees depend on explicit assumptions on sensing distributions, dither, convex priors, and corruption regimes [2509.19186] [1911.07816].

Within this landscape, the contribution of "Quantize-then-Rectify: Efficient VQ-VAE Training" is to make the principle operational for visual tokenizers under a particularly stringent efficiency target: exploit a pre-trained VAE’s empirical tolerance to latent perturbation, quantize only up to that tolerance, and use a small post rectifier to recover the remaining error, thereby obtaining competitive reconstruction with orders-of-magnitude less compute [2507.10547].

Source: https://www.emergentmind.com/topics/quantize-then-rectify-revq