Smoothed Vector Quantization Techniques

• Smoothed vector quantization is a family of methods that modify classical VQ to enhance differentiability and reduce issues like codebook collapse.
• Simplex-based smoothing employs soft assignments, convex mixtures, and vertex concentration to maintain balanced code usage while retaining fidelity.
• Alternative approaches use random rotations and spatial interpolation to smooth input statistics and gradient flows, lowering quantization error and boosting model performance.

Smoothed vector quantization designates a family of methods that modify classical vector quantization (VQ) so that the quantizer is less brittle with respect to optimization, codebook usage, or distortion. In standard deep-learning VQ, an encoder output is replaced by a single nearest codebook vector, which yields a discrete bottleneck but also introduces non-differentiability, codebook collapse, and residual quantization error. Recent work uses the term smoothed in several technically distinct senses: soft simplex assignments and convex mixtures of codewords, regularization that pushes softened assignments toward simplex vertices, random rotations that smooth coordinate statistics before scalar quantization, spatial interpolation-and-pooling around hard assignments, and geometry-aware backward maps that smooth gradient propagation without altering the hard forward assignment (Gautam et al., 2023, Morita, 26 Sep 2025, Zandieh et al., 28 Apr 2025, Li et al., 1 Jan 2026, Fifty et al., 2024).

1. Classical VQ and the rationale for smoothing

In VQ-VAE-style models, an encoder $E_\phi$ maps an input $x$ to a latent $z_e$, and quantization replaces $z_e$ by its nearest codebook element: $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$ A canonical VQ-VAE objective combines reconstruction with codebook and commitment terms: $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$ with $sg[\cdot]$ denoting stop-gradient and $\beta>0$ the commitment weight (Gautam et al., 2023).

The central difficulty is that nearest-neighbor selection is non-differentiable. Classical practice therefore uses the straight-through estimator (STE), which preserves the hard forward pass but substitutes a heuristic backward pass. The literature surveyed here identifies three recurrent pathologies of this construction. First, non-differentiability means the gradient estimator is only a surrogate. Second, codebook collapse arises when a few centroids dominate assignments and updates. Third, hard one-hot assignment is lossy: the residual $\epsilon = z_q - z_e$ is generally nonzero even though STE treats the quantizer as if it were identity in backpropagation (Gautam et al., 2023).

A parallel line of work frames VQ more directly as a rate–distortion problem. With bit budget $B=b\cdot d$, a vector quantizer $x$0 and decoder $x$1 are evaluated by mean-squared error

$x$2

and, for query $x$3, inner-product distortion

$x$4

The information-theoretic lower bounds reported for worst-case unit-norm inputs scale as $x$5 for MSE and $x$6 for inner products (Zandieh et al., 28 Apr 2025).

A useful distinction follows from these formulations. Some smoothed-VQ methods smooth the assignment itself; others smooth the statistics of the input or the backward signal while retaining hard assignment. The term is therefore broader than “soft VQ” in the narrow simplex-mixture sense.

2. Simplex-based smoothing: soft assignments, convex mixtures, and vertex concentration

The most direct form of smoothed VQ replaces a one-hot code index by a point on the probability simplex. In this setup, a codebook $x$7 is combined with a simplex vector $x$8,

$x$9

and the quantized representation becomes

$z_e$0

This formulation preserves differentiability through $z_e$1 and generalizes hard VQ, which is recovered when $z_e$2 is a canonical basis vector $z_e$3 (Morita, 26 Sep 2025).

Soft Convex Quantization (SCQ) instantiates this idea as a convex program. For input $z_e$4, codebook $z_e$5, and nearest-neighbor one-hot assignment $z_e$6, SCQ solves

$z_e$7

The first term minimizes quantization error; the second softly biases the solution toward the VQ one-hot assignment. The quantized output is $z_e$8. When $z_e$9 is one-hot, SCQ reduces to standard VQ; as $z_e$0, the optimum converges to $z_e$1 (Gautam et al., 2023).

The same paper gives a batch formulation. If encoder outputs are flattened as $z_e$2, with codebook $z_e$3 and one-hot assignments $z_e$4, SCQ solves

$z_e$5

subject to $z_e$6 and $z_e$7, so that each column of $z_e$8 lies on the simplex (Gautam et al., 2023).

A later simplex-based approach emphasizes that smoothing should satisfy two criteria simultaneously: assignments should remain close to simplex vertices, and all codes should be used. Its regularizer minimizes the average distance between each canonical vertex $z_e$9 and its $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$0-nearest smoothed assignments: $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$1 The paper studies both squared $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$2 distance,

$$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$3

and cross-entropy,

$$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$4

Because both induce ranking by the magnitude of $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$5, the $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$6-nearest assignments to each vertex can be obtained by top-$$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$7 selection on the corresponding simplex coordinate (Morita, 26 Sep 2025).

This regularizer is partly motivated by a critique of entropy-based usage penalties. Maximizing the perplexity of the mean assignment can be satisfied by diffuse or center-concentrated simplex distributions and therefore does not, by itself, enforce one-hotness. The cited work argues that this creates train–test mismatch when inference uses hard argmax codes (Morita, 26 Sep 2025). A plausible implication is that “balanced usage” and “sharp assignment” are distinct objectives and should not be conflated.

3. Differentiability and optimization through the quantizer

SCQ treats smoothed quantization as a differentiable convex optimization layer. For a single embedding, with equality constraint $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$8 and inequality constraints $$z_q := Q(z_e) = c_k,\quad \text{where } k = \arg\min_{1 \le j \le K}\|z_e - c_j\|.$$9, the Lagrangian is

$$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$0

Its KKT system includes stationarity,

$$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$1

along with primal feasibility, dual feasibility, and complementary slackness $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$2. Differentiation through the forward solution is then performed by implicit differentiation of the KKT conditions, yielding exact gradients through the quantizer up to solver tolerances (Gautam et al., 2023).

For the task loss $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$3 and soft quantized output $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$4, the gradient with respect to the codebook contains both a direct term and a term through the dependence of $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$5 on $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$6: $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$7 The same mechanism transmits gradients back to encoder outputs through $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$8 (Gautam et al., 2023).

Exact interior-point solves are not scalable in the full batch formulation: the reported complexity is cubic in the number of decision variables, $$\mathcal{L}_{\textrm{VQ}} = \|x-\hat{x}\|_2^2 + (1-\beta)\|sg[z_e]-z_q\|_2^2 + \beta\|z_e - sg[z_q]\|_2^2,$$9. SCQ therefore introduces a relaxation. It first solves the unconstrained regularized least-squares problem

$sg[\cdot]$0

a $sg[\cdot]$1 linear solve with cost $sg[\cdot]$2, and then projects each column of $sg[\cdot]$3 onto the simplex by iterating nonnegativity clipping and sum-to-one correction. This removes the need for softmax or temperature parameterization while keeping runtime comparable to practical VQ implementations (Gautam et al., 2023).

A different route to differentiability keeps the hard forward pass unchanged and smooths only the backward map. In the rotation trick, with encoder output $sg[\cdot]$4, selected code $sg[\cdot]$5, normalized vectors $sg[\cdot]$6 and $sg[\cdot]$7, and scale $sg[\cdot]$8, one constructs a rotation $sg[\cdot]$9 that aligns $\beta>0$0 to $\beta>0$1, and defines

$\beta>0$2

Because $\beta>0$3 preserves norms and $\beta>0$4 matches magnitudes, the forward map satisfies $\beta>0$5. During backpropagation, $\beta>0$6 is treated as constant, giving

$\beta>0$7

The resulting gradient is deterministic and geometry-aware: unlike STE, it carries information about the angle and relative norm between $\beta>0$8 and its assigned code (Fifty et al., 2024).

4. Other meanings of smoothing: random rotations and spatial interpolation

TurboQuant uses smoothing in a distributional sense rather than a simplex-mixture sense. For a unit-norm vector $\beta>0$9, it applies a random orthogonal rotation $\epsilon = z_q - z_e$0 to produce $\epsilon = z_q - z_e$1. This transforms any worst-case input into a random point uniformly distributed on the sphere. Each coordinate then has density

$\epsilon = z_q - z_e$2

which converges to $\epsilon = z_q - z_e$3 in high dimensions. The paper argues that the resulting coordinates are nearly independent, justifying identical Lloyd–Max scalar quantizers per coordinate without data-dependent codebook training (Zandieh et al., 28 Apr 2025).

Under this construction, per-coordinate decision thresholds are the midpoints between centroids $\epsilon = z_q - z_e$4: $\epsilon = z_q - z_e$5 The MSE distortion obeys

$\epsilon = z_q - z_e$6

while the two-stage inner-product variant, which adds a 1-bit Quantized Johnson–Lindenstrauss residual quantizer, satisfies

$\epsilon = z_q - z_e$7

These bounds match the optimal $\epsilon = z_q - z_e$8 and $\epsilon = z_q - z_e$9 rates up to constant factors (Zandieh et al., 28 Apr 2025).

LooC introduces yet another meaning of smoothing. It does not use weighted combinations of codevectors. Instead, it splits each feature vector $B=b\cdot d$0 into $B=b\cdot d$1 contiguous segments $B=b\cdot d$2, with $B=b\cdot d$3, quantizes each segment with a single shared codebook, and reconstructs by concatenation: $B=b\cdot d$4 Its smoothing mechanism is a parameter-free extrapolation-by-interpolation (EBI) pipeline: bilinear interpolation to a $B=b\cdot d$5 grid, compositional quantization at all interpolated locations, and $B=b\cdot d$6 average pooling back to the original spatial resolution,

$B=b\cdot d$7

The paper explicitly characterizes this as spatial smoothing rather than soft assignment over the codebook (Li et al., 1 Jan 2026).

These examples show that “smoothed vector quantization” is not a single algorithmic template. In current usage, the smoothing target may be the assignment simplex, the input distribution, the local spatial neighborhood, or the backward Jacobian.

5. Empirical behavior across image, speech, retrieval, and LLM systems

SCQ reports large gains over matched VQ baselines in autoencoding and VQGAN-style settings. On CIFAR-10, across five runs of 50 epochs, SCQ achieved MSE $B=b\cdot d$8, quantization error $B=b\cdot d$9, and perplexity $x$00, compared with VQVAE at $x$01, $x$02, and $x$03, respectively. On GTSRB, SCQ reached MSE $x$04, quantization error $x$05, and perplexity $x$06. In LSUN Church and Classroom experiments with VQGAN losses, SCQGAN consistently improved both reconstruction loss and LPIPS, with the advantage increasing at smaller latent resolutions, which the paper interprets as better information preservation under stronger compression (Gautam et al., 2023).

The simplex-vertex KNN regularizer produces a different empirical profile. On ImageNet autoencoding with latent $x$07 and codebook size $x$08, Softmax + KNN-CE achieved 100% code use, rMSE $x$09, FID $x$10, and IS $x$11; Softmax + PPL, by contrast, had rMSE $x$12, FID $x$13, and IS $x$14. In Wav2Vec 2.0 on LibriSpeech, Softmax + KNN-CE achieved 100% usage for a single $x$15 codebook and for dual $x$16 product quantizers, whereas STE, rotational estimation, and PPL baselines showed severe collapse in the reported configurations (Morita, 26 Sep 2025).

TurboQuant emphasizes online quantization quality and speed rather than learned latent autoencoding. For KV-cache quantization in Llama models, Needle-In-A-Haystack recall was reported as $x$17, matching full precision at compression ratios around $x$18 with 3–4 bits per channel. On LongBench, full cache at 16 bits scored $x$19, TurboQuant at 3.5 bits/channel also scored $x$20, and TurboQuant at 2.5 bits/channel scored $x$21. In nearest-neighbor search, TurboQuant consistently outperformed product quantization and RaBitQ in recall while reducing 4-bit indexing time to $x$22s, $x$23s, and $x$24s at dimensions $x$25, $x$26, and $x$27, respectively (Zandieh et al., 28 Apr 2025).

LooC combines compact codebooks with spatial smoothing. On MNIST, LooC with a $x$28 codebook reported LPIPS $x$29, rFID $x$30, SSIM $x$31, and PSNR $x$32; with $x$33, it improved to LPIPS $x$34, rFID $x$35, SSIM $x$36, and PSNR $x$37. On CIFAR10, the corresponding $x$38 setting reported LPIPS $x$39, rFID $x$40, SSIM $x$41, and PSNR $x$42. On FFHQ $x$43, LooC-VAE $x$44 achieved 100% codebook usage, LPIPS $x$45, rFID $x$46, SSIM $x$47, and PSNR $x$48, while $x$49 further improved to LPIPS $x$50, rFID $x$51, SSIM $x$52, and PSNR $x$53 (Li et al., 1 Jan 2026).

The rotation trick was evaluated across 11 VQ-VAE and VQGAN paradigms. In VQGAN on FFHQ+CelebA-HQ, code usage reportedly increased from about 27% to about 99%, quantization error fell from $x$54 to $x$55, validation loss from $x$56 to $x$57, and r-FID from $x$58 to $x$59. In latent-diffusion VQGAN on ImageNet with latent $x$60 and $x$61, r-FID improved from $x$62 to $x$63, r-IS from $x$64 to $x$65, and usage from about 2% to about 9% (Fifty et al., 2024).

6. Trade-offs, misconceptions, and open technical questions

A persistent misconception is that smoothed VQ is synonymous with softmax or Gumbel-softmax over codewords. The literature here contradicts that identification. SCQ and the KNN-to-vertices method are explicitly simplex-based; TurboQuant smooths coordinate statistics by random rotation; LooC smooths spatially by interpolation and pooling while keeping hard per-segment assignments; the rotation trick smooths only the gradient path (Gautam et al., 2023, Zandieh et al., 28 Apr 2025, Li et al., 1 Jan 2026, Fifty et al., 2024).

Another recurrent issue is the tension between smoothness and discrete fidelity. In SCQ, larger $x$66 yields sparser, more VQ-like assignments, while smaller $x$67 permits denser mixtures and lower quantization error but can reduce compressibility and downstream discrete modeling fidelity. The paper explicitly notes that excessively small $x$68 may produce overly dense mixtures, and suggests post hoc top-1 or top-$x$69 discretization when strict discrete tokens are required, at some performance cost (Gautam et al., 2023). The KNN-regularization paper frames a closely related problem as train–test mismatch: diffuse assignments can satisfy mean-perplexity objectives without resembling the hard argmax codes used at inference (Morita, 26 Sep 2025).

Scalability constraints differ by method. SCQ’s exact differentiable convex optimization is expensive at scale, motivating the regularized linear solve plus simplex projection relaxation (Gautam et al., 2023). TurboQuant is online and data-oblivious but incurs $x$70 dense rotation and projection costs per vector, which the paper identifies as a motivation for structured orthogonal transforms in future work (Zandieh et al., 28 Apr 2025). The KNN regularizer is limited primarily by memory because it must retain large assignment matrices and perform per-code top-$x$71 selection; the paper therefore uses GPU-local KNN and discusses vertex subsampling or product quantization as scaling aids (Morita, 26 Sep 2025). LooC notes that very large interpolation scale $x$72 can weaken local correlations and blur details, and that extreme segment granularity such as $x$73 may cause over-fragmentation (Li et al., 1 Jan 2026). The rotation trick identifies numerical edge cases at $x$74 and when $x$75, where the rotation construction becomes ill-conditioned and requires $x$76-stabilization or a 180-degree fallback (Fifty et al., 2024).

Across these variants, the unifying principle is not a single formula but a shared design goal: preserve more information than hard nearest-neighbor VQ while reducing the optimization and utilization failures that arise from discrete assignment. The methods differ in where the smoothing is imposed, which failure mode they target most directly, and what compromises they make between exact discreteness, computational tractability, and downstream compatibility.