---
title: Non-Uniform Quantization Framework (NU-RQ-VAE)
url: https://www.emergentmind.com/topics/non-uniform-quantization-framework-nu-rq-vae
type: topic
---

# Non-Uniform Quantization Framework (NU-RQ-VAE)

Searching arXiv for the cited papers to ground the article in the current record.
Searching for "Autoregressive Image Generation using Residual Quantization" and "CARD: Non-Uniform Quantization of Visual Semantic Unit for Generative Recommendation".
Non-Uniform Quantization Framework (NU-RQ-VAE) denotes a residual-quantization variational autoencoder in which quantization is preceded by a learnable, invertible transformation that explicitly compensates for skewed embedding distributions. In the formulation introduced within CARD, NU-RQ-VAE targets recommendation settings where item semantic embeddings are highly non-uniform, so standard residual quantization suffers from codeword imbalance and generation bias; it therefore learns a transform $\mathcal{T}:\mathbb{R}^m\to\mathbb{R}^m$ that maps the encoder output into a more balanced latent space, performs residual quantization there, and then reconstructs through $\mathcal{T}^{-1}$ [2604.26427]. Conceptually, it extends the residual-quantized VAE substrate of RQ-VAE, which had already shown that multi-stage residual quantization can preserve fidelity while shortening the effective autoregressive sequence length relative to conventional single-stage VQ formulations [2203.01941].

## 1. Residual-quantized VAE as the substrate

RQ-VAE begins from an image input
\[
X\in\mathbb{R}^{H_o\times W_o\times 3},
\]
an encoder $E$ that produces
\[
Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},
\]
and a decoder $G$ that reconstructs $\hat X=G(\hat Z)$. At a spatial location $(h,w)$, the latent vector is denoted
\[
z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.
\]
Instead of single-stage vector quantization, RQ-VAE applies a $D$-stage residual quantizer with initial residual
\[
r^{(0)}=z_e,
\]
followed, for $d=1,\dots,D$, by nearest-code selection,
\[
M_{h,w,d}=k_d=\arg\min_{k\in[1..K]}\bigl\lVert r^{(d-1)}-e(k)\bigr\rVert_2^2,
\]
codeword assignment
\[
q^{(d)}=e\bigl(M_{h,w,d}\bigr),
\]
and residual update
\[
r^{(d)}=r^{(d-1)}-q^{(d)}.
\]
The partial reconstruction is
\[
\hat z^{(d)}=\sum_{i=1}^d q^{(i)},\qquad \hat z=\hat z^{(D)},
\]
with full quantized feature map $\hat Z^{(D)}\equiv\hat Z$ [2203.01941].

This residual formulation underwrites the later non-uniform extension because it decouples representational capacity from latent spatial resolution. The encoder plus residual quantizer yields a discrete tensor
\[
M\in[1..K]^{H\times W\times D}.
\]
Each spatial location is treated as one time step $t=1,\dots,T$ with $T=H\cdot W$, and the autoregressive model predicts the stack
\[
S_t=(M_{t,1},M_{t,2},\dots,M_{t,D}).
\]
The key efficiency point is that inference predicts $D$ codes in parallel per spatial location, so the spatial transformer sees only $T$ context-vectors rather than $T\cdot D$ separate symbols. In the rate-distortion account given for RQ-VAE, a single-stage VQ at resolution $H\times W$ uses $HW\log_2 K$ bits, whereas residual quantization with depth $D$ and codebook size $K$ can in principle partition space into up to $K^D$ regions, making aggressive spatial compression feasible without exponential growth of a single codebook [2203.01941].

## 2. Motivation for non-uniform quantization

The non-uniform extension is motivated by a distributional mismatch. In recommendation scenarios, learned item embeddings $p(\mathbf{z})$ are described as highly skewed: popular or semantically similar items cluster densely, while long-tail items occupy sparse regions. Standard vector-quantized VAEs, including residual variants, minimize average reconstruction error under an implicit assumption of uniform embedding density. When applied directly to such a non-uniform $p(\mathbf{z})$, two failure modes are identified: codeword imbalance, in which a small subset of codebook entries dominates assignments in dense regions, and generation bias, in which frequent codewords are over-chosen during autoregressive decoding, amplifying popularity bias [2604.26427].

NU-RQ-VAE addresses this problem by moving quantization into a transformed coordinate system. Let $\mathbf{h}\in\mathbb{R}^m$ denote the continuous representation produced by an encoder, described in CARD as a small MLP. The objective is to learn an invertible transform
\[
\mathcal{T}:\mathbb{R}^m\to\mathbb{R}^m
\]
such that $\mathbf{d}=\mathcal{T}(\mathbf{h})$ is a “more uniform” latent representation. Quantization is then performed in $\mathbf{d}$-space, after which the model returns to the original representation through $\mathcal{T}^{-1}$. This preserves the residual-quantization machinery while reconditioning the geometry on which nearest-neighbor assignment operates [2604.26427].

Within CARD, this mechanism is paired with a visual semantic unit that unifies textual, visual, and collaborative signals into a structured visual representation prior to encoding. NU-RQ-VAE is therefore not introduced as an isolated quantizer, but as the quantization component of a broader two-stage generative recommendation framework [2604.26427].

## 3. Learnable invertible transformation

CARD specifies two dimension-wise monotonic parameterizations for $\mathcal{T}$, each intended to map individual coordinates toward $[0,1]$ with roughly uniform density. In both cases, transformation parameters are learned jointly with the encoder, decoder, and codebooks [2604.26427].

The first is a Kumaraswamy-based transformation:
\[
\mathcal{T}_i(h_i)=F_{KS}(x_i;a_i,b_i)=1-\bigl(1-x_i^{a_i}\bigr)^{b_i},
\]
with
\[
x_i=\frac{h_i-h_{\min,i}}{h_{\max,i}-h_{\min,i}},\qquad x_i\in[0,1],
\]
and inverse
\[
\mathcal{T}_i^{-1}(y_i)=\Bigl(1-(1-y_i)^{1/b_i}\Bigr)^{1/a_i}\cdot(h_{\max,i}-h_{\min,i})+h_{\min,i}.
\]

The second is a scaled logistic-logit transformation:
\[
\mathcal{T}_i(h_i)=
\frac{\sigma\bigl(\alpha_i(h_i-x_{0,i})\bigr)-\sigma\bigl(\alpha_i(h_{\min,i}-x_{0,i})\bigr)}
{\sigma\bigl(\alpha_i(h_{\max,i}-x_{0,i})\bigr)-\sigma\bigl(\alpha_i(h_{\min,i}-x_{0,i})\bigr)},
\qquad \sigma(u)=(1+e^{-u})^{-1},
\]
with inverse
\[
\mathcal{T}_i^{-1}(y_i)=x_{0,i}+\frac{1}{\alpha_i}\ln\!\Biggl(
\frac{y_i\Delta+\sigma\bigl(\alpha_i(h_{\min,i}-x_{0,i})\bigr)}
{1-\bigl(y_i\Delta+\sigma\bigl(\alpha_i(h_{\min,i}-x_{0,i})\bigr)\bigr)}
\Biggr),
\]
where
\[
\Delta=\sigma\bigl(\alpha_i(h_{\max,i}-x_{0,i})\bigr)-\sigma\bigl(\alpha_i(h_{\min,i}-x_{0,i})\bigr).
\]

In vector form, the transformed and inverse-transformed latents are written
\[
\mathbf{d}=\mathcal{T}(\mathbf{h}),\qquad \hat{\mathbf{h}}=\mathcal{T}^{-1}(\hat{\mathbf{d}}).
\]
The Kumaraswamy and logistic-logit variants are reported separately in the experimental section as NU-RQ-VAE$_K$ and NU-RQ-VAE$_S$ [2604.26427].

## 4. Quantization loop, losses, and training mechanics

After transformation, residual quantization proceeds in $\mathbf{d}$-space. CARD maintains $K$ residual codebooks $\{\mathcal{C}_k\}_{k=1}^K$, each
\[
\mathcal{C}_k=\{\mathbf{e}_j^k\in\mathbb{R}^d\}_{j=1}^N.
\]
With $\mathbf{r}_0=\mathbf{d}$, the recursion for $k=1,\dots,K$ is
\[
c_k=\arg\min_{j\in\{1,\dots,N\}}\|\mathbf{r}_{k-1}-\mathbf{e}_j^k\|^2,\qquad
\mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{e}_{c_k}^k.
\]
The resulting discrete Semantic ID is
\[
\tilde{\mathbf{i}}=[c_1,\dots,c_K],
\]
and the quantized sum in transformed space is
\[
\hat{\mathbf{d}}=\sum_{k=1}^K \mathbf{e}_{c_k}^k.
\]
Decoding then uses
\[
\hat{\mathbf{h}}=\mathcal{T}^{-1}(\hat{\mathbf{d}}),\qquad
\hat{\mathbf{z}}=\mathrm{Decoder}(\hat{\mathbf{h}}).
\]
The backward pass uses a straight-through estimator with gradients
\[
\partial\hat{\mathbf{d}}/\partial\mathbf{r}_{k-1}=\mathbf{I}
\]
[2604.26427].

The quantization and reconstruction objective consists of three terms. The reconstruction term is
\[
\mathcal{L}_{\mathrm{Recon}}=\|\mathbf{z}-\hat{\mathbf{z}}\|^2.
\]
The residual-quantization term is
\[
\mathcal{L}_{\mathrm{RQ}}
=\sum_{k=1}^K
\Bigl\|\mathrm{sg}[\mathbf{r}_{k-1}]-\mathbf{e}_{c_k}^k\Bigr\|^2
+\mu\Bigl\|\mathbf{r}_{k-1}-\mathrm{sg}[\mathbf{e}_{c_k}^k]\Bigr\|^2,
\]
where $\mathrm{sg}[\cdot]$ is stop-gradient and $\mu$ balances codebook versus encoder updates. The non-uniform transformation is regularized by the consistency loss
\[
\mathcal{L}_{\mathrm{NUQ}}=\bigl\|\mathcal{T}^{-1}(\mathcal{T}(\mathbf{h}))-\mathbf{h}\bigr\|^2.
\]
The full objective is
\[
\mathcal{L}_{\mathrm{total}}
=\mathcal{L}_{\mathrm{Recon}}+\mathcal{L}_{\mathrm{RQ}}
+\lambda_{\mathrm{NUQ}}\mathcal{L}_{\mathrm{NUQ}},
\]
with $\lambda_{\mathrm{NUQ}}$ tuned in the range $0.1\sim1.0$ [2604.26427].

The reported training algorithm first obtains $\mathbf{z}_i$ from a pretrained vision-language encoder $f$, computes $\mathbf{h}_i\leftarrow \mathrm{Encoder}(\mathbf{z}_i)$, applies $\mathcal{T}$, performs residual quantization, reconstructs through $\mathcal{T}^{-1}$ and the decoder, and updates encoder, decoder, transform parameters, and codebooks jointly. After convergence of NU-RQ-VAE, the codebooks and $\mathcal{T}$ are fixed; all items are quantized into SIDs; and an autoregressive generator, exemplified by T5, is trained over SID sequences [2604.26427].

## 5. Instantiated architecture and empirical behavior

CARD reports a concrete implementation of NU-RQ-VAE for generative recommendation. The vision-language encoder is SigLIP2 with $m=512$ on $512\times512$ card images. The encoder and decoder are 3-layer MLPs with hidden sizes $[512\to256\to128\to32]$. The codebooks use $K=4$ residual stages, $N=256$ embeddings per codebook, and embedding dimension $d=32$. The autoregressive model is a T5 encoder-decoder with 4 layers, 6 heads, head-dim $=64$, and model-dim $=384$. Optimization uses AdamW with learning rate $10^{-3}$ for NU-RQ-VAE and T5 fine-tuning, batch size $=1024$, and $\lambda_{\mathrm{NUQ}}\in[0.1,1.0]$ [2604.26427].

| Component | Reported configuration |
|---|---|
| Vision-language encoder | SigLIP2, $m=512$ |
| Encoder/Decoder | 3-layer MLPs, $[512\to256\to128\to32]$ |
| Codebooks | $K=4$, $N=256$, $d=32$ |
| AR model | T5, 4 layers, 6 heads, model-dim $384$ |
| Optimization | AdamW, lr $=10^{-3}$, batch size $=1024$ |

The empirical motivation for the non-uniform transformation is supported by codeword-utilization and recommendation metrics. On the Food dataset, the RQ-VAE baseline reports Recall@5 / Recall@10 of $0.0394 / 0.0617$, NU-RQ-VAE$_K$ reports $0.0442 / 0.0693$, and NU-RQ-VAE$_S$ reports $0.0456 / 0.0713$. On Phones, RQ-VAE reports $0.0526 / 0.0794$, while NU-RQ-VAE$_S$ reports $0.0551 / 0.0842$. Figure 7 is summarized as showing an effective number of used codewords of approximately $48$ for TIGER (uniform RQ-VAE), approximately $62$ for CARD without NUT, and approximately $79$ for CARD with NUT [2604.26427].

| Model | Food Recall@5 / Recall@10 | Phones Recall@5 / Recall@10 |
|---|---|---|
| RQ-VAE baseline | $0.0394 / 0.0617$ | $0.0526 / 0.0794$ |
| NU-RQ-VAE$_K$ | $0.0442 / 0.0693$ | — |
| NU-RQ-VAE$_S$ | $0.0456 / 0.0713$ | $0.0551 / 0.0842$ |

The ablation study further reports, on Food, that full CARD$_S$ achieves $0.0547 / 0.0853$, whereas removing the non-uniform transform yields $0.0513 / 0.0795$; removing the visual region yields $0.0485 / 0.0771$; removing the text region yields $0.0472 / 0.0754$; removing the collaborative region yields $0.0525 / 0.0821$; and replacing all “cards” with text-only input yields $0.0456 / 0.0702$ [2604.26427].

## 6. Interpretation, scope, and relation to other non-uniform variants

A central interpretive point is that the term “NU-RQ-VAE” can refer to two distinct, though related, directions. In the RQ-VAE paper, a non-uniform extension is described as a possible modification of the residual quantizer itself: one could introduce stage-specific codebooks $C^{(d)}$ of different sizes $K_d$, or allocate per-depth bit budgets $b_1,\dots,b_D$, with minimal changes to the quantization loop. That possibility is motivated by empirical rate-distortion gains at different residual depths [2203.01941].

However, the published ablations in RQ-VAE also report that separate per-depth codebooks, each of size $K/D$, produce higher reconstruction FID than a single shared $K$-sized codebook. Code-usage histograms across depths show that deeper stages use embeddings of smaller norm, but with substantial overlap, which is presented as evidence that a uniform shared codebook is reused effectively. This creates an important distinction: non-uniformity in the original RQ-VAE discussion is primarily a prospective allocation strategy over residual stages, whereas the implemented NU-RQ-VAE in CARD is a transform-based method that leaves the residual quantization mechanism intact and instead reshapes the latent distribution before quantization [2203.01941].

This distinction also clarifies a common misunderstanding. NU-RQ-VAE in CARD is not merely “RQ-VAE with unequal codebook sizes.” Its defining operation is the insertion of a codebook-agnostic, per-dimension, learnable, invertible transformation $\mathcal{T}$ before residual quantization. Because $\mathcal{T}$ is applied as a pre-quantization layer and only requires closed-form forward and inverse evaluation together with the consistency regularizer $\mathcal{L}_{\mathrm{NUQ}}$, CARD describes it as plug-and-play and reports analogous gains when applied beyond RQ-VAE, including R-VQ, PQ, product RQ, and even diffusion-based quantizers [2604.26427].

Taken together, the two papers position NU-RQ-VAE at the intersection of rate-distortion engineering and distribution-aware quantization. The RQ-VAE substrate provides the coarse-to-fine residual coding and autoregressive sequence-length reduction; CARD adds an explicit mechanism for correcting non-uniform embedding geometry before code assignment. A plausible implication is that future work on NU-RQ-VAE may combine both strands—distribution-shaping transforms and stage-specific allocation—although only the former is instantiated in the CARD results and only the latter is proposed, not validated, in the original RQ-VAE exposition [2203.01941].

Source: https://www.emergentmind.com/topics/non-uniform-quantization-framework-nu-rq-vae