Variational Dequantization Strategies

• Variational dequantization is a technique that augments discrete data with noise to enable continuous space probabilistic modeling.
• Key strategies include uniform, importance-weighted, Rényi, and autoregressive methods, all aimed at tightening the evidence lower bound.
• Empirical results show improved likelihood estimates and cross-domain applicability, notably enhancing both classical flow-based and quantum ML models.

Variational dequantization strategies address the challenge of modeling discrete data using probabilistic models defined on continuous spaces—an essential step for high-fidelity estimation, generation, and representation learning in both classical and quantum machine learning. These strategies formalize the process of stochastic "dequantization," in which observed discrete points are augmented with noise, and a variational lower bound is jointly optimized over both the base model and the dequantizer. The resulting framework unifies uniform noise injection, learned flow-based dequantization, importance-weighted and Rényi bounds, autoregressive dequantization networks, and extensions to discrete latent variable and quantum circuit models. This article surveys core principles, principal variants, their mathematical underpinnings, and their empirical and theoretical impact.

1. Mathematical Framework and ELBO Formulation

The modern variational dequantization paradigm begins with a discrete data sample $x \in \{0,1,\ldots,255\}^D$ (e.g., images or quantized audio) and introduces a continuous "dequantization" variable $u \in [0,1)^D$, defining the continuous variable $z = x + u$. The model posits a continuous density $p_\theta(z)$; the implied marginal likelihood for $x$ is

$$P_\text{model}(x) = \int_{[0,1)^D} p_\theta(x + u) \, du.$$

Direct optimization is intractable, so a variational distribution $q_\phi(u|x)$ is introduced, yielding the evidence lower bound (ELBO)

$$\log P_\text{model}(x) \geq \mathbb{E}_{u \sim q_\phi(u|x)}\left[ \log p_\theta(x+u) - \log q_\phi(u|x) \right] =: L_{\mathrm{VI}}(x).$$

This ELBO recovers uniform dequantization when $q_\phi(u|x)$ is uniform on $[0,1)^D$ and supports trainable dequantizers—such as flows or autoregressive models—when parameterized networks are used (Hoogeboom et al., 2020, Ho et al., 2019).

2. Principal Variational Dequantization Strategies

Four main strategies have emerged, each modifying the variational distribution $u \in [0,1)^D$0 and/or the objective:

Method	$u \in [0,1)^D$1 class	Bound Type
Uniform	Uniform over $u \in [0,1)^D$2	VI
Importance-Weighted	General	IW with $u \in [0,1)^D$3 samples
Rényi-$u \in [0,1)^D$4	General	Rényi bound (order $u \in [0,1)^D$5)
Autoregressive (ARD)	Autoregressive (conditional) flow	VI, IW, or Rényi

Uniform dequantization forces $u \in [0,1)^D$6 to spread probability evenly in each quantization cell. Importance-weighted dequantization tightens the bound via $u \in [0,1)^D$7 samples per input, approaching the true log-likelihood as $u \in [0,1)^D$8. The Rényi bound interpolates between VI ($u \in [0,1)^D$9), IW ($z = x + u$0), and a "max-weight" bound at $z = x + u$1. ARD uses powerful input-conditioned autoregressive models (e.g., masked convolution flows) to represent $z = x + u$2, dramatically shrinking the KL divergence to the true posterior and tightening the ELBO even with $z = x + u$3 (Hoogeboom et al., 2020, Ho et al., 2019).

3. Parameterization, Implementation, and Training

Several architectures have operationalized variational dequantization:

• Flow++ parameterizes $z = x + u$4 as a learned conditional flow: For $z = x + u$5, $z = x + u$6, yielding

$z = x + u$7

and allows fully differentiable ELBO optimization by back-propagating through the reparameterization (Ho et al., 2019).

• Autoregressive Dequantization (ARD): $z = x + u$8 admits complex, multi-modal posterior shapes and is compatible with plug-in into any of the objective types (VI, IW, Rényi, etc.) (Hoogeboom et al., 2020).
• Stochastically Quantized VAEs (SQ-VAE): Generalize variational dequantization to discrete latents. Two latent variables—continuous $z = x + u$9 and discrete $p_\theta(z)$0—are linked, with the ELBO expressing a joint optimization over continuous dequantization and stochastic quantization, leading to improved codebook utilization and data reconstruction (Takida et al., 2022).

The actual training procedure involves Monte Carlo estimation of the ELBO or its variants, with gradient-based updates jointly on model, dequantizer, and where applicable, codebook parameters.

4. Tightness, Trade-offs, and Computational Considerations

A direct comparison of strategies establishes both theoretical and empirical trade-offs:

Objective (with fixed $p_\theta(z)$1)	Tightness	Computational Cost
VI ($p_\theta(z)$2)	Loosest	1 sample
Rényi-$p_\theta(z)$3	Intermediate	$p_\theta(z)$4 samples
IW ($p_\theta(z)$5)	Tighter	$p_\theta(z)$6 samples
ARD	Tightest for $p_\theta(z)$7	Sequential per-pixel/$p_\theta(z)$8 cost

Autoregressive dequantization and its fine-tuned IW variant set empirical benchmarks for state-of-the-art likelihoods (e.g., $p_\theta(z)$9 bpd on CIFAR-10), whereas uniform dequantization is notably suboptimal (Hoogeboom et al., 2020). Computational cost of ARD scales with input dimensionality but requires only sampling (not inversion) at generation time.

5. Empirical Impact and Applications

• Likelihood improvements: Flow-based models with variational dequantization achieve up to $x$00.13 bits/dim improvement over uniform methods, with Flow++ reaching 3.08 bpd vs. 3.35 bpd for Glow on CIFAR-10 (Ho et al., 2019).
• Generalization: Shrinking the train–test ELBO gap (from $x$10.06 bits/dim to $x$20.02) indicates that the model no longer must accommodate degenerate uniform densities.
• Cross-domain applicability: SQ-VAE extends variational dequantization from data space to codebook-based latent VAEs, eliminating codebook collapse and improving codebook perplexity and reconstruction metrics (MSE, FID, mIoU, etc.) across image and speech domains (Takida et al., 2022).

6. Dequantization in Quantum Machine Learning

Within variational quantum machine learning (QML), "dequantization" also refers to the classical simulation and analysis of parameterized quantum circuits (PQC):

• Kernel-based dequantization: Certain PQC-induced kernels admit exact, efficient classical evaluation using tensor-network techniques (e.g., low-bond-dimension symmetric matrix product states) instead of random Fourier features, enabling full classical regression with polynomial complexity (Sweke et al., 31 Mar 2025). This exposes classes of QML models for which quantum advantage is negated by classical tractability.
• Trainability vs. Dequantization Trade-off: Recent formalizations clarify that models can be both gradient-trainable and provably non-dequantizable (i.e., not efficiently simulable by classical algorithms), by embedding BQP-hard circuits within otherwise trainable PQC ansätze (Gil-Fuster et al., 2024). These results sharpen understanding of which architectures truly resist classical surrogates.

7. Theoretical Guarantees, Limitations, and Open Problems

• Theoretical guarantees: In models such as SQ-VAE, as the decoder variance $x$3 decreases, the quantizer's variance $x$4 converges to zero, leading from stochastic exploration to deterministic quantization (self-annealing) (Takida et al., 2022).
• Limitations: Not all variational or kernel-based QML architectures are equally susceptible to efficient dequantization; bond-dimension and data-encoding structure are key. In classical models, the efficiency of ARD or complex flows may be constrained by data dimension or coupling-layer complexity.
• Open questions: Natural, deep-layered variational quantum circuits with both strong inductive biases against dequantization and high practical utility are an ongoing area of investigation (Gil-Fuster et al., 2024).

• "Learning Discrete Distributions by Dequantization" (Hoogeboom et al., 2020)
• "Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design" (Ho et al., 2019)
• "SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization" (Takida et al., 2022)
• "Kernel-based dequantization of variational QML without Random Fourier Features" (Sweke et al., 31 Mar 2025)
• "On the relation between trainability and dequantization of variational quantum learning models" (Gil-Fuster et al., 2024)