DRQ: Decoupled Residual Quantization

• DRQ is a method that decouples continuous embedding geometry learning from discrete hierarchical clustering to address issues like codebook underutilization and unstable semantic boundaries.
• It employs a two-stage approach where an encoder-decoder first learns robust continuous representations before hierarchical K-Means assigns discrete Semantic IDs.
• A diagnostic framework based on expected codeword overlap and effective capacity quantifies symbolic robustness, guiding trade-offs in large-scale recommendation systems.

Searching arXiv for the main paper and closely related Semantic ID / residual quantization work to ground the article. Decoupled Residual Quantization (DRQ) is a method for constructing discrete “Semantic IDs” for items that explicitly separates two tasks that standard VQ and RQ methods entangle: learning a good continuous embedding geometry, and learning a good discrete code distribution for robust symbolic identifiers (Wang et al., 1 Jun 2026). In the formulation introduced for recommendation and retrieval, Semantic IDs are short sequences of discrete codes assigned to item embeddings, and their quality is evaluated along multiple axes: symbolic robustness under perturbation, preservation of continuous geometry, and behavior-aware soft matching. DRQ was presented together with a diagnostic framework based on expected codeword overlap and effective codebook capacity, with experiments on a large industrial short-video dataset positioned as a case study rather than a universal benchmark claim (Wang et al., 1 Jun 2026).

1. Problem setting and motivation

Semantic IDs are discrete token sequences assigned to item embeddings so that items can be indexed for fast retrieval, shared across models and tasks, fed as tokens to language-like models, and reused to tie together multimodal and collaborative signals. In the setting described for DRQ, an item embedding $x \in \mathbb{R}^d$ is mapped to a short residual code sequence, for example three levels with $4096$ codes per level (Wang et al., 1 Jun 2026).

The paper isolates three persistent failure modes when standard PQ, RQ, or neural RQ-VAE tokenizers are applied directly to recommendation embeddings. The first is codebook underutilization, also described as index collapse or distribution mismatch. Real item distributions are extremely long-tailed, jointly trained VQ or RQ-VAE methods place implicit pressure toward uniform usage, but gradients flow only through codes currently activated via the Straight-Through Estimator (STE); popular items dominate updates, many codes receive few or no updates, and usage becomes highly skewed. The second is unstable decision boundaries, described as semantic boundary confusion. The latent space is partitioned into Voronoi cells, and small perturbations in retrieval-time representations can move points across nearby boundaries in high-density regions, producing collisions and unstable symbolic IDs. The third is geometric distortion, including dimensional collapse and Euclidean mismatch. PQ imposes axis-aligned, block-wise quantization, while RQ-VAE builds the space from Euclidean residual stages; recommendation embeddings are often highly anisotropic or lie on curved manifolds, so forcing them into flat Euclidean cells compresses meaningful neighborhoods into a small part of the nominal code space (Wang et al., 1 Jun 2026).

A central motivation for DRQ is that these three failure modes are empirically entangled in standard tokenizers. Poor performance can arise from code usage imbalance, from unstable symbolic boundaries, or from the geometry of the latent manifold itself. The paper’s framing is that robust Semantic IDs therefore require diagnostics that separate distributional effects from geometric effects before proposing a new tokenizer design (Wang et al., 1 Jun 2026).

2. Diagnostic framework: overlap and effective capacity

The diagnostic framework models retrieval-time uncertainty as additive isotropic Gaussian noise around each codeword $c_i \in \mathbb{R}^d$, with

$p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$

and prior code usage $\pi_i$, where $\sum_i \pi_i = 1$. The pairwise overlap between two codewords is defined as the integral of the product of their Gaussian densities, and after normalization by self-overlap this yields the scale-free kernel

$$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$

Using this kernel, the paper defines the expected codeword overlap proxy

$$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$

which is interpreted as a normalized confusion proxy rather than a literal probability of crossing a Voronoi boundary (Wang et al., 1 Jun 2026).

The decomposition

$$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$

separates a distribution floor from a geometry-sensitive cross-overlap term. The first term, $\sum_i \pi_i^2$, persists even when all codewords are infinitely separated; if code usage is highly skewed, this term is large. The second term decreases as inter-code distances increase relative to $4096$0, thereby isolating the effect of geometric proximity. In an idealized system with uniform usage and no off-diagonal overlap, the expected overlap is $4096$1, which motivates the definition of effective codebook size

$4096$2

This quantity is interpreted as the effective number of usable, well-separated codes given both usage imbalance and geometry. Larger $4096$3 indicates higher symbolic capacity and greater robustness to perturbations without increasing code length (Wang et al., 1 Jun 2026).

The diagnostic significance of this framework is that semantic boundary confusion can be analyzed quantitatively rather than impressionistically. Large $4096$4 and small $4096$5 can be attributed either to skewed code usage, to tightly packed geometry, or to both. The framework therefore turns the tokenizer quality problem into a multi-factor diagnosis of distributional imbalance and Euclidean constraints in the latent space (Wang et al., 1 Jun 2026).

3. Decoupled Residual Quantization methodology

The defining characteristic of DRQ is its two-stage construction. In the first stage, continuous geometry reconstruction is learned with an encoder–decoder architecture without any discrete quantization constraint. Let $4096$6 denote the encoder and $4096$7 the decoder. For an item embedding $4096$8, the latent is $4096$9, and the reconstruction objective is

$c_i \in \mathbb{R}^d$0

When user behavior is available, the paper optionally adds a contrastive term: $c_i \in \mathbb{R}^d$1 Here $c_i \in \mathbb{R}^d$2 is an InfoNCE loss that pulls co-viewed or semantically similar items closer and pushes random negatives apart. In the reported experiments, the base DRQ-VAE uses $c_i \in \mathbb{R}^d$3, while DRQ-VAE+CL uses $c_i \in \mathbb{R}^d$4 (Wang et al., 1 Jun 2026).

In the second stage, after the encoder is frozen, hierarchical K-Means is applied to the latent vectors $c_i \in \mathbb{R}^d$5. At level 1, K-Means on $c_i \in \mathbb{R}^d$6 produces centroids $c_i \in \mathbb{R}^d$7, labels $c_i \in \mathbb{R}^d$8, and residuals $c_i \in \mathbb{R}^d$9. At each deeper level $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$0, K-Means is run on the previous residuals to obtain $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$1, labels $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$2, and updated residuals $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$3. The final Semantic ID is the tuple $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$4 (Wang et al., 1 Jun 2026).

This construction contrasts with standard RQ-VAE, where the encoder, decoder, and codebooks are jointly trained and the discrete quantization operation sits inside the representation-learning loop via STE. In joint RQ-VAE, popular regions dominate codebook updates, rare codes can remain dead or undertrained, and the encoder is pushed to make latents easier to quantize by a fixed-size codebook, which can induce anisotropic or dimensional collapse. DRQ replaces this coupling with a split objective: first learn the manifold for reconstruction and, optionally, behavior-aware alignment; then cluster that manifold with residual K-Means. Because K-Means recomputes centroids from full-dataset assignments in each Lloyd iteration, the method tends to reduce the update starvation characteristic of STE-based training and allows the latent geometry to be shaped independently of codebook learning (Wang et al., 1 Jun 2026).

4. Geometric and probabilistic properties

The paper evaluates latent geometry using Participation Ratio, entropy-based effective rank, the largest eigenvalue $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$5, and mean absolute cosine between latent dimensions. Raw input embeddings have Participation Ratio approximately $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$6 and effective rank approximately $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$7. Joint RQ-VAE lowers both, and the reported interpretation is that geometry is somewhat compressed and more anisotropic. RQ-KMeans and RQP-VAE are close to raw geometry. DRQ-VAE without contrastive learning is somewhat more compressed than raw, with Participation Ratio approximately $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$8 and effective rank approximately $p_i(x) = \mathcal{N}(x \mid c_i,\ \sigma^2 I_d),$9, but not catastrophically so. DRQ-VAE+CL increases effective dimensionality substantially, with Participation Ratio approximately $\pi_i$0, effective rank approximately $\pi_i$1, and mean absolute cosine $\pi_i$2, which the paper attributes to contrastive uniformity on a curved manifold (Wang et al., 1 Jun 2026).

These observations are tied back to $\pi_i$3 and $\pi_i$4. Flattening the usage distribution $\pi_i$5 reduces the distribution floor $\pi_i$6, while increasing inter-code separation reduces the geometry-sensitive cross-overlap term. DRQ is designed to address both components through different mechanisms: density-adaptive K-Means on the full dataset is used to mitigate the distribution penalty, while the continuous stage is allowed to expand or reshape the manifold without codebook constraints, which is intended to mitigate the geometry penalty (Wang et al., 1 Jun 2026).

The empirical relationship is not monotone across all objectives. According to the reported Table 2, RQP-VAE achieves the lowest $\pi_i$7 and the highest $\pi_i$8 across three levels, owing to EMA updates and dead-code revival that explicitly target distribution flattening. DRQ-VAE and RQ-KMeans occupy an intermediate range, with DRQ slightly better at deeper levels. DRQ-VAE+CL exhibits geometry that is favorable for behavior-aware matching but worse $\pi_i$9 at deeper levels. The paper explicitly interprets this as evidence that geometry favorable for behavior-aware soft matching is not necessarily geometry that minimizes overlap under isotropic noise. No closed-form performance bounds are given, but the overlap derivation is exact and the reported empirical behavior is consistent with the decomposition (Wang et al., 1 Jun 2026).

5. Empirical findings on recommendation and retrieval

The experiments use a large industrial short-video dataset with more than $\sum_i \pi_i = 1$0 million items, raw embedding dimension $\sum_i \pi_i = 1$1, and three codebook levels with $\sum_i \pi_i = 1$2 codes per level, corresponding to nominal $\sum_i \pi_i = 1$3-bit capacity. The comparison set includes RQ-VAE, RQP-VAE, RQ-KMeans, DRQ-VAE, and DRQ-VAE+CL. For robustness analysis, the latent variance $\sum_i \pi_i = 1$4 is estimated, $\sum_i \pi_i = 1$5 is set equal to $\sum_i \pi_i = 1$6, $\sum_i \pi_i = 1$7 and $\sum_i \pi_i = 1$8 are computed per level, and Gaussian noise is also injected into latents to measure code changes empirically (Wang et al., 1 Jun 2026).

On symbolic robustness, RQP-VAE is the strongest model in the reported overlap-based metrics. At level $\sum_i \pi_i = 1$9, it attains $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$0 and $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$1. DRQ-VAE reaches level-$$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$2 $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$3, with levels $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$4 and $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$5 around $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$6–$$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$7; the paper characterizes it as comparable to RQ-KMeans. DRQ-VAE+CL performs well at level $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$8 but shows substantially reduced effective capacity at deeper levels. The stated conclusion is that DRQ improves upon naive RQ-VAE in symbolic capacity, while the distribution-centric modifications of RQP-VAE are more effective when symbolic robustness alone is the target (Wang et al., 1 Jun 2026).

Codebook utilization shows the same pattern. At level $$\mathcal{K}(c_i, c_j) = \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right).$$9, RQ-VAE exhibits severe collapse, with perplexity approximately $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$0, only $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$1 active codes, and Gini approximately $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$2. RQP-VAE is the most balanced, with approximately $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$3 active codes at both measured levels, perplexity approximately $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$4–$$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$5, and Gini approximately $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$6–$$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$7. DRQ-VAE at level $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$8 has approximately $$O_\pi = \sum_{i=1}^K \sum_{j=1}^K \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right),$$9 active codes, perplexity approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$0, and Gini approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$1, which is described as close to RQ-KMeans and RQP-VAE. At level $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$2, DRQ-VAE has approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$3 active codes, perplexity approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$4, and Gini approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$5, indicating residual long-tail imbalance but substantial improvement over RQ-VAE. DRQ-VAE+CL worsens in deeper layers, with perplexity approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$6, only $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$7 active codes, and Gini approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$8 (Wang et al., 1 Jun 2026).

On reconstruction fidelity and retrieval retention, DRQ-VAE is the strongest of the reported methods. Retention is defined as the ratio of hit rate using reconstructed embeddings to hit rate using original embeddings. DRQ-VAE obtains HR@20 approximately $$O_\pi = \sum_i \pi_i^2 + \sum_{i \neq j} \pi_i \pi_j \exp\left(-\frac{\|c_i - c_j\|^2}{4\sigma^2}\right)$$9, HR@50 approximately $\sum_i \pi_i^2$0, HR@100 approximately $\sum_i \pi_i^2$1, and HR@200 approximately $\sum_i \pi_i^2$2, which the paper describes as almost lossless preservation of relative neighbor rankings. The detailed log attributes to DRQ-VAE the lowest MSE ($\sum_i \pi_i^2$3), the highest cosine similarity ($\sum_i \pi_i^2$4), the lowest collision rate ($\sum_i \pi_i^2$5), the largest number of unique IDs ($\sum_i \pi_i^2$6), and the smallest maximum collision bucket ($\sum_i \pi_i^2$7) (Wang et al., 1 Jun 2026).

On behavior-aware soft matching, the paper reports three AUC measures: SID Embedding AUC, Weighted SID Match AUC, and Exact SID Match AUC. DRQ-VAE+CL attains the best SID Embedding AUC ($\sum_i \pi_i^2$8) and Weighted SID Match AUC ($\sum_i \pi_i^2$9), with competitive Exact SID Match AUC ($4096$00). It also slightly exceeds the original embedding in high-cutoff retention at HR@100 and HR@200, which the paper interprets as evidence that collaborative supervision aligns the embedding more closely with observed co-consumption. DRQ-VAE reaches SID Embedding AUC $4096$01 and moderate weighted and exact matching AUC. RQ-VAE attains the best Exact SID Match AUC ($4096$02), but the paper cautions that this is partly driven by heavy code sharing, with higher collision rate, fewer unique IDs, and larger collision buckets; high exact-match AUC in this setting is therefore not necessarily evidence of higher-quality semantics (Wang et al., 1 Jun 2026).

6. Multi-objective interpretation and system trade-offs

A central conclusion of the DRQ paper is that Semantic ID quality is multi-objective rather than reducible to a single tokenizer score. The three explicit objectives are symbolic robustness and capacity, measured by overlap-based and utilization metrics; reconstruction fidelity and continuous retrieval retention, measured by MSE, cosine similarity, and hit-rate retention; and behavior-aware soft matching, measured by SID-based AUCs and high-cutoff retention. The reported experiments show that no single tokenizer dominates across all three objectives (Wang et al., 1 Jun 2026).

Within this framing, RQP-VAE is best on overlap-based symbolic capacity and codebook utilization, DRQ-VAE is best on reconstruction fidelity and near-lossless embedding-level retrieval, and DRQ-VAE+CL is best on soft matching and high-cutoff retrieval. The stated contribution of DRQ is therefore not merely a claim to be a universally superior quantizer, but to separate the controllable factors: the continuous stage can be tuned for geometry and behavior-awareness, while the clustering stage can be tuned for discrete capacity and robustness. The paper characterizes this separation as moving along a better Pareto frontier than a single joint VQ objective can easily provide (Wang et al., 1 Jun 2026).

This also clarifies several common misunderstandings. Low overlap and high $4096$03 do not by themselves guarantee strong downstream recommendation behavior. Conversely, strong exact symbolic match scores can be produced by excessive code reuse and collisions. The empirical evidence in the paper suggests that downstream recommendation quality, especially for soft matching and behavior-aware tasks, depends on the interaction between geometry, supervision, and discretization rather than on symbolic robustness alone (Wang et al., 1 Jun 2026).

7. Relation to prior quantization methods and limitations

The paper positions DRQ against several established families of discrete representation methods. VQ-VAE jointly learns discrete latents with an encoder and decoder, but is described as prone to index collapse on long-tailed, multimodal data. Product quantization is efficient for approximate nearest-neighbor search but imposes rigid axis-aligned structure and is presented as less suitable for hierarchical semantic codes. Residual quantization and RQ-VAE fit hierarchical Semantic IDs more naturally, but still couple joint quantization and continuous learning and thereby inherit collapse and geometric mismatch problems. DRQ differs in using VAE-like latent learning without quantization, followed by clustering on the learned latent space to produce a residual hierarchy without STE or codebook gradients (Wang et al., 1 Jun 2026).

The paper also places DRQ within broader work on semantic hashing, discrete VAEs, and recommendation-oriented Semantic ID frameworks such as TIGER, SE-REC, and OneRec. Its stated novelty lies in combining a diagnostic framework for symbolic robustness and effective capacity with a decoupled design for Semantic IDs in recommendation, rather than treating the tokenizer as a black box. This suggests a shift from purely architectural comparison toward diagnosis of how code usage, boundary confusion, and geometry jointly shape tokenizer behavior (Wang et al., 1 Jun 2026).

The limitations are explicit. All experiments are conducted on a single proprietary industrial short-video dataset with more than $4096$04 million items. The evaluation concerns item-to-item retrieval rather than full production candidate generation and ranking with metrics such as Recall or NDCG. The perturbation model is isotropic Gaussian, whereas real retrieval-time noise may be anisotropic or structured. The quantity $4096$05 is a proxy rather than a literal boundary-crossing probability. Accordingly, the paper presents both the framework and DRQ as a case study, while suggesting that the diagnostic quantities and the design principle of decoupling geometry from discretization may generalize beyond the reported dataset (Wang et al., 1 Jun 2026).