Non-Uniform Quantization Framework (NU-RQ-VAE)

• The paper introduces NU-RQ-VAE, which inserts an invertible transformation before residual quantization to address codeword imbalance and generation bias.
• It reconditions non-uniform latent distributions by mapping encoder outputs into a balanced space, enabling efficient multi-stage residual quantization with reduced autoregressive complexity.
• Empirical evaluations in recommendation tasks show improved recall metrics and effective codeword utilization compared to standard RQ-VAE setups.

Searching arXiv for the cited papers to ground the article in the current record. Searching for "Autoregressive Image Generation using Residual Quantization" and "CARD: Non-Uniform Quantization of Visual Semantic Unit for Generative Recommendation". Non-Uniform Quantization Framework (NU-RQ-VAE) denotes a residual-quantization variational autoencoder in which quantization is preceded by a learnable, invertible transformation that explicitly compensates for skewed embedding distributions. In the formulation introduced within CARD, NU-RQ-VAE targets recommendation settings where item semantic embeddings are highly non-uniform, so standard residual quantization suffers from codeword imbalance and generation bias; it therefore learns a transform $\mathcal{T}:\mathbb{R}^m\to\mathbb{R}^m$ that maps the encoder output into a more balanced latent space, performs residual quantization there, and then reconstructs through $\mathcal{T}^{-1}$ (Wei et al., 29 Apr 2026). Conceptually, it extends the residual-quantized VAE substrate of RQ-VAE, which had already shown that multi-stage residual quantization can preserve fidelity while shortening the effective autoregressive sequence length relative to conventional single-stage VQ formulations (Lee et al., 2022).

1. Residual-quantized VAE as the substrate

RQ-VAE begins from an image input

$X\in\mathbb{R}^{H_o\times W_o\times 3},$

an encoder $E$ that produces

$$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$

and a decoder $G$ that reconstructs $\hat X=G(\hat Z)$. At a spatial location $(h,w)$, the latent vector is denoted

$z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$

Instead of single-stage vector quantization, RQ-VAE applies a $D$-stage residual quantizer with initial residual

$\mathcal{T}^{-1}$0

followed, for $\mathcal{T}^{-1}$1, by nearest-code selection,

$\mathcal{T}^{-1}$2

codeword assignment

$\mathcal{T}^{-1}$3

and residual update

$\mathcal{T}^{-1}$4

The partial reconstruction is

$\mathcal{T}^{-1}$5

with full quantized feature map $\mathcal{T}^{-1}$6 (Lee et al., 2022).

This residual formulation underwrites the later non-uniform extension because it decouples representational capacity from latent spatial resolution. The encoder plus residual quantizer yields a discrete tensor

$\mathcal{T}^{-1}$7

Each spatial location is treated as one time step $\mathcal{T}^{-1}$8 with $\mathcal{T}^{-1}$9, and the autoregressive model predicts the stack

$X\in\mathbb{R}^{H_o\times W_o\times 3},$0

The key efficiency point is that inference predicts $X\in\mathbb{R}^{H_o\times W_o\times 3},$1 codes in parallel per spatial location, so the spatial transformer sees only $X\in\mathbb{R}^{H_o\times W_o\times 3},$2 context-vectors rather than $X\in\mathbb{R}^{H_o\times W_o\times 3},$3 separate symbols. In the rate-distortion account given for RQ-VAE, a single-stage VQ at resolution $X\in\mathbb{R}^{H_o\times W_o\times 3},$4 uses $X\in\mathbb{R}^{H_o\times W_o\times 3},$5 bits, whereas residual quantization with depth $X\in\mathbb{R}^{H_o\times W_o\times 3},$6 and codebook size $X\in\mathbb{R}^{H_o\times W_o\times 3},$7 can in principle partition space into up to $X\in\mathbb{R}^{H_o\times W_o\times 3},$8 regions, making aggressive spatial compression feasible without exponential growth of a single codebook (Lee et al., 2022).

2. Motivation for non-uniform quantization

The non-uniform extension is motivated by a distributional mismatch. In recommendation scenarios, learned item embeddings $X\in\mathbb{R}^{H_o\times W_o\times 3},$9 are described as highly skewed: popular or semantically similar items cluster densely, while long-tail items occupy sparse regions. Standard vector-quantized VAEs, including residual variants, minimize average reconstruction error under an implicit assumption of uniform embedding density. When applied directly to such a non-uniform $E$0, two failure modes are identified: codeword imbalance, in which a small subset of codebook entries dominates assignments in dense regions, and generation bias, in which frequent codewords are over-chosen during autoregressive decoding, amplifying popularity bias (Wei et al., 29 Apr 2026).

NU-RQ-VAE addresses this problem by moving quantization into a transformed coordinate system. Let $E$1 denote the continuous representation produced by an encoder, described in CARD as a small MLP. The objective is to learn an invertible transform

$E$2

such that $E$3 is a “more uniform” latent representation. Quantization is then performed in $E$4-space, after which the model returns to the original representation through $E$5. This preserves the residual-quantization machinery while reconditioning the geometry on which nearest-neighbor assignment operates (Wei et al., 29 Apr 2026).

Within CARD, this mechanism is paired with a visual semantic unit that unifies textual, visual, and collaborative signals into a structured visual representation prior to encoding. NU-RQ-VAE is therefore not introduced as an isolated quantizer, but as the quantization component of a broader two-stage generative recommendation framework (Wei et al., 29 Apr 2026).

3. Learnable invertible transformation

CARD specifies two dimension-wise monotonic parameterizations for $E$6, each intended to map individual coordinates toward $E$7 with roughly uniform density. In both cases, transformation parameters are learned jointly with the encoder, decoder, and codebooks (Wei et al., 29 Apr 2026).

The first is a Kumaraswamy-based transformation: $E$8 with

$E$9

and inverse

$$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$0

The second is a scaled logistic-logit transformation: $$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$1 with inverse

$$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$2

where

$$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$3

In vector form, the transformed and inverse-transformed latents are written

$$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$4

The Kumaraswamy and logistic-logit variants are reported separately in the experimental section as NU-RQ-VAE$$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$5 and NU-RQ-VAE$$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$6 (Wei et al., 29 Apr 2026).

4. Quantization loop, losses, and training mechanics

After transformation, residual quantization proceeds in $$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$7-space. CARD maintains $$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$8 residual codebooks $$Z=E(X)\in\mathbb{R}^{H\times W\times n_z},\qquad H=\tfrac{H_o}{f},\;W=\tfrac{W_o}{f},$$9, each

$G$0

With $G$1, the recursion for $G$2 is

$G$3

The resulting discrete Semantic ID is

$G$4

and the quantized sum in transformed space is

$G$5

Decoding then uses

$G$6

The backward pass uses a straight-through estimator with gradients

$G$7

(Wei et al., 29 Apr 2026).

The quantization and reconstruction objective consists of three terms. The reconstruction term is

$G$8

The residual-quantization term is

$G$9

where $\hat X=G(\hat Z)$0 is stop-gradient and $\hat X=G(\hat Z)$1 balances codebook versus encoder updates. The non-uniform transformation is regularized by the consistency loss

$\hat X=G(\hat Z)$2

The full objective is

$\hat X=G(\hat Z)$3

with $\hat X=G(\hat Z)$4 tuned in the range $\hat X=G(\hat Z)$5 (Wei et al., 29 Apr 2026).

The reported training algorithm first obtains $\hat X=G(\hat Z)$6 from a pretrained vision-language encoder $\hat X=G(\hat Z)$7, computes $\hat X=G(\hat Z)$8, applies $\hat X=G(\hat Z)$9, performs residual quantization, reconstructs through $(h,w)$0 and the decoder, and updates encoder, decoder, transform parameters, and codebooks jointly. After convergence of NU-RQ-VAE, the codebooks and $(h,w)$1 are fixed; all items are quantized into SIDs; and an autoregressive generator, exemplified by T5, is trained over SID sequences (Wei et al., 29 Apr 2026).

5. Instantiated architecture and empirical behavior

CARD reports a concrete implementation of NU-RQ-VAE for generative recommendation. The vision-language encoder is SigLIP2 with $(h,w)$2 on $(h,w)$3 card images. The encoder and decoder are 3-layer MLPs with hidden sizes $(h,w)$4. The codebooks use $(h,w)$5 residual stages, $(h,w)$6 embeddings per codebook, and embedding dimension $(h,w)$7. The autoregressive model is a T5 encoder-decoder with 4 layers, 6 heads, head-dim $(h,w)$8, and model-dim $(h,w)$9. Optimization uses AdamW with learning rate $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$0 for NU-RQ-VAE and T5 fine-tuning, batch size $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$1, and $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$2 (Wei et al., 29 Apr 2026).

Component	Reported configuration
Vision-language encoder	SigLIP2, $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$3
Encoder/Decoder	3-layer MLPs, $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$4
Codebooks	$z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$5, $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$6, $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$7
AR model	T5, 4 layers, 6 heads, model-dim $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$8
Optimization	AdamW, lr $z_e(x)\equiv Z_{hw}\in\mathbb{R}^{n_z}.$9, batch size $D$0

The empirical motivation for the non-uniform transformation is supported by codeword-utilization and recommendation metrics. On the Food dataset, the RQ-VAE baseline reports Recall@5 / Recall@10 of $D$1, NU-RQ-VAE$D$2 reports $D$3, and NU-RQ-VAE$D$4 reports $D$5. On Phones, RQ-VAE reports $D$6, while NU-RQ-VAE$D$7 reports $D$8. Figure 1 is summarized as showing an effective number of used codewords of approximately $D$9 for TIGER (uniform RQ-VAE), approximately $\mathcal{T}^{-1}$00 for CARD without NUT, and approximately $\mathcal{T}^{-1}$01 for CARD with NUT (Wei et al., 29 Apr 2026).

Model	Food Recall@5 / Recall@10	Phones Recall@5 / Recall@10
RQ-VAE baseline	$\mathcal{T}^{-1}$02	$\mathcal{T}^{-1}$03
NU-RQ-VAE$\mathcal{T}^{-1}$04	$\mathcal{T}^{-1}$05	—
NU-RQ-VAE$\mathcal{T}^{-1}$06	$\mathcal{T}^{-1}$07	$\mathcal{T}^{-1}$08

The ablation study further reports, on Food, that full CARD$\mathcal{T}^{-1}$09 achieves $\mathcal{T}^{-1}$10, whereas removing the non-uniform transform yields $\mathcal{T}^{-1}$11; removing the visual region yields $\mathcal{T}^{-1}$12; removing the text region yields $\mathcal{T}^{-1}$13; removing the collaborative region yields $\mathcal{T}^{-1}$14; and replacing all “cards” with text-only input yields $\mathcal{T}^{-1}$15 (Wei et al., 29 Apr 2026).

6. Interpretation, scope, and relation to other non-uniform variants

A central interpretive point is that the term “NU-RQ-VAE” can refer to two distinct, though related, directions. In the RQ-VAE paper, a non-uniform extension is described as a possible modification of the residual quantizer itself: one could introduce stage-specific codebooks $\mathcal{T}^{-1}$16 of different sizes $\mathcal{T}^{-1}$17, or allocate per-depth bit budgets $\mathcal{T}^{-1}$18, with minimal changes to the quantization loop. That possibility is motivated by empirical rate-distortion gains at different residual depths (Lee et al., 2022).

However, the published ablations in RQ-VAE also report that separate per-depth codebooks, each of size $\mathcal{T}^{-1}$19, produce higher reconstruction FID than a single shared $\mathcal{T}^{-1}$20-sized codebook. Code-usage histograms across depths show that deeper stages use embeddings of smaller norm, but with substantial overlap, which is presented as evidence that a uniform shared codebook is reused effectively. This creates an important distinction: non-uniformity in the original RQ-VAE discussion is primarily a prospective allocation strategy over residual stages, whereas the implemented NU-RQ-VAE in CARD is a transform-based method that leaves the residual quantization mechanism intact and instead reshapes the latent distribution before quantization (Lee et al., 2022).

This distinction also clarifies a common misunderstanding. NU-RQ-VAE in CARD is not merely “RQ-VAE with unequal codebook sizes.” Its defining operation is the insertion of a codebook-agnostic, per-dimension, learnable, invertible transformation $\mathcal{T}^{-1}$21 before residual quantization. Because $\mathcal{T}^{-1}$22 is applied as a pre-quantization layer and only requires closed-form forward and inverse evaluation together with the consistency regularizer $\mathcal{T}^{-1}$23, CARD describes it as plug-and-play and reports analogous gains when applied beyond RQ-VAE, including R-VQ, PQ, product RQ, and even diffusion-based quantizers (Wei et al., 29 Apr 2026).

Taken together, the two papers position NU-RQ-VAE at the intersection of rate-distortion engineering and distribution-aware quantization. The RQ-VAE substrate provides the coarse-to-fine residual coding and autoregressive sequence-length reduction; CARD adds an explicit mechanism for correcting non-uniform embedding geometry before code assignment. A plausible implication is that future work on NU-RQ-VAE may combine both strands—distribution-shaping transforms and stage-specific allocation—although only the former is instantiated in the CARD results and only the latter is proposed, not validated, in the original RQ-VAE exposition (Lee et al., 2022).