Variational Probabilistic Quantization (VPQ)

• VPQ is a framework that leverages uncertainty models and variational objectives to enable probabilistic discretization across neural compression, Bayesian networks, and field theory.
• It replaces deterministic quantization with stochastic relaxations and uncertainty-driven clustering, improving the representation of weights and latent variables.
• Empirical results show VPQ achieves superior compressibility and accuracy on models like ResNets and transformers by iteratively fixing weights based on probabilistic distance metrics.

Searching arXiv for papers using or closely related to “Variational Probabilistic Quantization” and the specific arXiv IDs provided. Search terms used: “Variational Probabilistic Quantization”, “Probabilistic Weight Fixing quantization”, “Variational Bayesian Quantization”, “SQ-VAE stochastic quantization”, and the exact arXiv IDs (Subia-Waud et al., 2023, Yang et al., 2020, Takida et al., 2022, Koide et al., 2013, Li et al., 2 Oct 2025, Borras et al., 11 Dec 2025, Yang et al., 10 Nov 2025). Variational Probabilistic Quantization (VPQ) is not a single universally standardized algorithm in the arXiv literature reviewed here. Rather, the label has been applied to several variational or probabilistic constructions in which quantization, discretization, or field quantization is mediated by uncertainty models, latent distributions, stochastic relaxations, or variational objectives. In neural compression and model quantization, VPQ has denoted uncertainty-aware clustering of weights, stochastic or variational discretization of latent variables, and quantization of variational parameters in Bayesian neural networks; in communications it has denoted probabilistic neural encoders for secret common-randomness extraction; and in stochastic field theory it has been used for the Stochastic Variational Method as an alternative quantization scheme for the complex Klein–Gordon field (Subia-Waud et al., 2023, Takida et al., 2022, Yang et al., 2020, Borras et al., 11 Dec 2025, Li et al., 2 Oct 2025, Koide et al., 2013, Yang et al., 10 Nov 2025).

1. Scope of the term and recurring variational structure

Across these usages, the recurring motif is that quantization is not treated as a purely deterministic nearest-neighbor or thresholding operation. Instead, the discrete or low-cardinality representation is informed by posterior uncertainty, stochastic sampling, variational bounds, or stochastic action principles. In some cases the target of quantization is a neural-network weight tensor; in others it is a latent code, a variational parameter, a common-randomness symbol, or a field configuration (Subia-Waud et al., 2023, Takida et al., 2022, Borras et al., 11 Dec 2025, Li et al., 2 Oct 2025, Koide et al., 2013).

Usage	Quantized object	Variational/probabilistic mechanism
PWFN / neural weight sharing	Network weights $w_i$	Mean-field Gaussian posterior $$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$
SQ-VAE	Discrete latent code indices $s$	Softmax posterior with temperature $\tau$ and self-annealed stochastic quantization
VBQ	Continuous latent coordinates $z_i$	Quantile-space discrete search weighted by posterior uncertainty
QBNN VPQ	Variational parameters $\mu,\log\sigma^2$	Quantizers inserted inside the SVI ELBO with STE
Secret-CR VPQ	Encoder outputs $W,V\in\{1,\dots,|\mathcal W|\}$	Variational bounds on $I(W;Z)$ and adversarial training
SVM as quantization scheme	Stochastic field modes $C_{i,\mathbf k}$	Stochastic action principle leading to the functional Schrödinger equation
VAEVQ	Continuous visual latents and codebook	VAE ELBO plus RCS and DCR around nearest-neighbor quantization

A common misconception is to read VPQ as if it named one settled technique. The surveyed literature indicates the opposite: the same acronym has been used for distinct research programs. Another common source of confusion is the meaning of “quantization” itself. In (Koide et al., 2013), quantization means field quantization in the sense of quantum theory; in (Subia-Waud et al., 2023, Takida et al., 2022, Yang et al., 2020, Borras et al., 11 Dec 2025), and (Yang et al., 10 Nov 2025), it means compression-oriented discretization or low-precision representation; and in (Li et al., 2 Oct 2025) it means mapping correlated observations to discrete random variables with agreement, uniformity, and secrecy constraints.

2. Weight-space VPQ: probabilistic weight fixing and uncertainty-aware clustering

In (Subia-Waud et al., 2023), the method is introduced as “Probabilistic Weight Fixing” (PWFN) and is based on a fully factorized Gaussian variational posterior over network weights,

$$q(w;\theta)=\prod_{i=1}^N \mathcal N(w_i\mid \mu_i,\sigma_i^2),$$

with likelihood $$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$0. The usual ELBO is written as

$$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$1

but in practice the prior KL is absorbed into a regularizer. To prevent $$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$2, the training loss adds a “noise-resilience” term

$$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$3

so that

$$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$4

In experiments, $$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$5 and $$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$6.

The central quantization mechanism is a separate “fixing” stage applied after or between variational updates. A codebook $$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$7, initially chosen as powers-of-two and additive combinations, is used to fix each $$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$8 to a cluster center according to the distance-in-$$q(w;\theta)=\prod_i \mathcal N(w_i\mid \mu_i,\sigma_i^2)$$9 metric

$s$0

The paper presents this as a variational relaxation of discrete assignments: rather than introducing hard latent indicators $s$1 during training, all weights remain Gaussian and clustering is deferred to the fixing step. The description explicitly notes that one may view this as the $s$2 limit of a Gumbel-Softmax relaxation,

$s$3

while the implemented method uses the direct $s$4 criterion.

The initialization is also uncertainty-aware. The means $s$5 are initialized to the pre-trained point weights. The $s$6 are initialized by a small parabolic function of the distance from the nearest power-of-two:

$s$7

for $s$8, followed by clamping to $s$9.

The fixing procedure alternates $\tau$0 rounds; each round fixes a fraction $\tau$1 of the weights and retrains the remaining free weights for $\tau$2 epochs, for a total of $\tau$3 epochs. The optimizer is SGD with momentum $\tau$4, learning rate $\tau$5, batch-size $\tau$6, and $\tau$7RTX8000 GPUs. The threshold is $\tau$8 in units of $\tau$9, and the schedule $z_i$0 follows Subia-Waud & Dasmahapatra 2022. No architecture-specific tuning is required beyond choosing the same $z_i$1, and the method is reported to work on ResNets-18/34/50, DenseNet-161, and DeiT-Small/Tiny.

Empirically, the method “demonstrates superior compressibility and higher accuracy compared to state-of-the-art methods” on both ResNet models and transformer-based architectures (Subia-Waud et al., 2023). On ImageNet, the reported Top-1 results are $z_i$2 for ResNet-18 with $z_i$3 unique values, $z_i$4 for ResNet-34 with $z_i$5 unique values, $z_i$6 for ResNet-50 with $z_i$7 unique values, $z_i$8 for DeiT-Small with $z_i$9 unique values, $\mu,\log\sigma^2$0 for DeiT-Tiny with $\mu,\log\sigma^2$1 unique values, and $\mu,\log\sigma^2$2 for DenseNet-161