Entropy-Regularized Quantization Loss

• Entropy-Regularized Quantization Loss is an objective that couples a distortion measure with an entropy term to control coding rate and quantizer output complexity.
• It spans multiple domains—from classical rate–distortion theory and entropic optimal transport to neural compression and low-bit quantization—each using tailored entropy measures.
• Key insights include explicit asymptotic rate–distortion laws, the benefits of soft quantization for differentiability, and techniques to manage assignment softness and geometric constraints.

Searching arXiv for recent and foundational papers on entropy-regularized quantization, entropic OT, and learned compression to ground the article. Entropy-regularized quantization loss denotes a family of objective functions that couple a quantization distortion term with an entropy-derived regularizer, typically to control codebook usage, coding rate, assignment softness, or representational structure under discretization. Across information theory, optimal transport, learned compression, quantization-aware training, and soft quantization, the recurring form is a rate–distortion trade-off in which quantization quality is balanced against Shannon entropy, Rényi entropy, coding length, or Kullback–Leibler divergence (Kreitmeier et al., 2011, Koch et al., 2015, Lakshmanan et al., 2023, Zhang et al., 2024). In the strict classical setting, the entropy term constrains or penalizes the entropy of the quantizer output; in modern neural settings it may instead regularize latent symbol distributions, transport couplings, feature covariance structure, or conditional source entropy as a surrogate linked to latent entropy minimization (Arvinte et al., 2021, Viehmann, 2019, Pang et al., 19 Sep 2025, Zhang et al., 2024).

1. Classical information-theoretic formulation

In entropy-constrained quantization, the basic ingredients are a source, a quantizer, a distortion measure, and an entropy functional on the quantizer output. For scalar quantization with source $X$, quantizer $q$, and $r$-th power distortion, the distortion is

$$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$

while the entropy of the quantizer output is computed from the probabilities of codecells (Kreitmeier et al., 2011). For Rényi entropy of order $\alpha\in(0,\infty)\setminus\{1\}$,

$$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$

with the limiting cases $H_1(p)=-\sum_i p_i\log p_i$ and $H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$ recovering Shannon entropy and log-cardinality, respectively (Kreitmeier et al., 2011).

The constrained optimization problem is

$$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$

which includes fixed-rate quantization when $\alpha=0$ and Shannon-entropy–constrained quantization when $q$0 (Kreitmeier et al., 2011). A standard Lagrangian view is

$q$1

and this is the prototypical entropy-regularized quantization loss in information theory (Kreitmeier et al., 2011).

A closely related formulation appears in entropy-constrained vector quantization. For a $q$2-dimensional source $q$3, a single-letter vector quantizer $q$4, and distortion $q$5, the minimal achievable output entropy under distortion constraint is

$q$6

and the Lagrangian form is

$q$7

This places the entropy penalty directly on the discrete quantizer output, so that the regularizer is interpretable as coding rate under optimal lossless coding (Koch et al., 2015).

These formulations establish the canonical meaning of entropy-regularized quantization loss: an objective of the form “distortion plus an entropy term,” with the entropy measuring either codebook cardinality, Shannon entropy, or Rényi entropy of the quantized representation (Kreitmeier et al., 2011, Koch et al., 2015).

2. High-rate asymptotics and structural implications

In the high-rate regime, entropy-regularized quantization admits explicit asymptotic structure. For scalar quantization with Rényi entropy constraint, if

$q$8

then

$q$9

and

$r$0

under the regularity conditions stated for weakly unimodal densities and finite moments (Kreitmeier et al., 2011). This yields the asymptotic rate–distortion law

$r$1

which is the high-rate behavior of the constrained problem and therefore of the corresponding Lagrangian entropy-regularized loss (Kreitmeier et al., 2011).

A notable feature of the Rényi formulation is the appearance of an “entropy density” that generalizes the classical point density of fixed-rate quantization. For an interval $r$2,

$r$3

and the corresponding distortion density is governed by the same $r$4 term (Kreitmeier et al., 2011). This implies that, asymptotically, local contributions to entropy and local contributions to distortion are aligned.

For entropy-constrained vector quantization, converse bounds identify a source-independent excess-rate constant above the Shannon rate–distortion function. If

$r$5

then

$r$6

and in one dimension this converges to the output entropy achieved by a uniform quantizer, recovering the Gish–Pierce result (Koch et al., 2015). In the scalar case, any sequence of asymptotically optimal almost-regular quantizers must converge to a uniform quantizer as the allowed distortion tends to zero (Koch et al., 2015).

This suggests that, in the low-distortion high-resolution regime, the entropy regularizer does not merely penalize complexity abstractly; it imposes sharp geometric constraints on cell sizes, point density, and the feasible asymptotic shape of optimal quantizers (Kreitmeier et al., 2011, Koch et al., 2015).

3. Entropic optimal transport as a soft quantization loss

A second major lineage defines entropy-regularized quantization through entropic optimal transport. On a finite space, the entropy-regularized transport cost between histograms $r$7 and $r$8 with cost matrix $r$9 is

$$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$0

where

$$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$1

is the relative entropy of the coupling (Bigot et al., 2017). In the implementation-oriented formulation of Cuturi-style regularization,

$$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$2

with $$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$3, and the loss is

$$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$4

(Viehmann, 2019). The entropy term makes the problem strictly convex, yields a unique dense transport plan, and admits Sinkhorn iterations (Viehmann, 2019, Bigot et al., 2017).

When one measure represents data and the other a finite codebook or prototype distribution, this becomes an entropy-regularized quantization loss. With data points $$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$5, codewords $$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$6, and cost

$$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$7

one obtains

$$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$8

where $$D_\mu(q) = \mathbb{E}|X-q(X)|^r = \int |x-q(x)|^r\,\mu(dx),$$9 is a soft assignment of data points to codewords (Viehmann, 2019). As $\alpha\in(0,\infty)\setminus\{1\}$0, assignments become harder; as $\alpha\in(0,\infty)\setminus\{1\}$1 increases, assignments become more diffuse (Viehmann, 2019).

A more directly quantization-oriented continuous formulation appears in soft quantization via entropic regularization. For a target measure $\alpha\in(0,\infty)\setminus\{1\}$2, a discrete quantizer

$\alpha\in(0,\infty)\setminus\{1\}$3

and regularization parameter $\alpha\in(0,\infty)\setminus\{1\}$4, the master problem is

$\alpha\in(0,\infty)\setminus\{1\}$5

and the induced loss is

$\alpha\in(0,\infty)\setminus\{1\}$6

For discrete $\alpha\in(0,\infty)\setminus\{1\}$7, the per-sample soft assignment is

$\alpha\in(0,\infty)\setminus\{1\}$8

and the loss is a smooth approximation of $\alpha\in(0,\infty)\setminus\{1\}$9 (Lakshmanan et al., 2023).

This OT-based viewpoint treats entropy regularization as a way to soften the quantizer assignment map itself. Rather than penalizing only the entropy of output symbols, it regularizes the coupling between source and codebook, producing differentiable soft assignments, smoother optimization, and a tunable interpolation between hard quantization and diffuse matching (Viehmann, 2019, Lakshmanan et al., 2023, Bigot et al., 2017).

4. Learned compression and neural rate–distortion objectives

In learned compression, entropy-regularized quantization loss is usually a rate–distortion objective where the rate is an estimate of latent entropy and the distortion is reconstruction error. In neural image compression, the standard loss is

$$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$0

with discrete latent $$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$1 obtained by quantizing an intermediate representation $$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$2, and $$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$3 a learned entropy model (Zhang et al., 2024). The entropy term is therefore a differentiable surrogate for the coding rate of the quantized latent.

The work on an information-theoretic regularizer for lossy neural image compression derives an identity linking latent entropy to conditional source entropy: $$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$4 for transform coding (Zhang et al., 2024). Since directly modeling $$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$5 is difficult, the proposed regularized objective is

$$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$6

which adds a negative conditional source entropy term as a structural regularizer (Zhang et al., 2024). The regularizer is used only in training and imposes no inference overheads (Zhang et al., 2024).

A different learned quantization setting appears in wideband soft-bit quantization for communication systems. There, the loss is

$$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$7

where $$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$8 is a sample-weighted MSE on reconstructed soft bits and $$H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^\infty p_i^\alpha,$$9 is a differentiable entropy surrogate for a fixed scalar codebook (Arvinte et al., 2021). The surrogate is

$H_1(p)=-\sum_i p_i\log p_i$0

with soft assignments

$H_1(p)=-\sum_i p_i\log p_i$1

and empirical codeword frequencies $H_1(p)=-\sum_i p_i\log p_i$2 estimated from the hard-quantized latent $H_1(p)=-\sum_i p_i\log p_i$3 (Arvinte et al., 2021). This is a direct neural instantiation of $H_1(p)=-\sum_i p_i\log p_i$4, where the entropy term is estimated minibatch-wise via soft histogramming.

These settings share the same rate–distortion template but differ in where entropy is imposed. Neural image compression penalizes latent entropy through learned entropy models and may add a conditional source-entropy regularizer (Zhang et al., 2024). Communication-oriented soft-bit quantization penalizes the entropy of a fixed latent codebook via soft assignments (Arvinte et al., 2021). In both cases, the quantizer is embedded in a larger differentiable system, so entropy regularization is used both for compression and for optimization stability.

5. Representation-level regularization in low-bit neural quantization

In quantization-aware training for neural networks, entropy-related terms can regularize not only quantized weights or latent codeword usage but also internal feature representations. MEC-Quant formulates the problem from a coding-length perspective rather than directly estimating differential entropy of high-dimensional features (Pang et al., 19 Sep 2025).

For a batch of features

$H_1(p)=-\sum_i p_i\log p_i$5

the minimal lossy coding length surrogate is

$H_1(p)=-\sum_i p_i\log p_i$6

where $H_1(p)=-\sum_i p_i\log p_i$7 is an upper bound on the allowed average distortion (Pang et al., 19 Sep 2025). The total training objective is

$H_1(p)=-\sum_i p_i\log p_i$8

with the entropy surrogate applied to backbone features $H_1(p)=-\sum_i p_i\log p_i$9 and the task loss defined either by cross-entropy or KL distillation (Pang et al., 19 Sep 2025).

Because the log-det term is expensive, MEC-Quant uses a Taylor expansion,

$H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$0

and then a Mixture-of-Experts reformulation

$H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$1

to better handle long-tailed spectra in deep features (Pang et al., 19 Sep 2025). The regularization strength is scheduled rather than fixed: $H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$2 This schedule is reported to improve accuracy over constant $H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$3 on ResNet-18 W2A4 CIFAR-10, from 87.85% to 88.26% (Pang et al., 19 Sep 2025).

The paper interprets quantization at extremely low bitwidth as introducing bias, feature collapse, and concentrated singular values, and uses the coding-length surrogate to shape the feature spectrum (Pang et al., 19 Sep 2025). It also introduces a “rectified eigenvalue entropy” for collapse analysis and reports that MEC-Quant significantly alleviates collapse relative to LSQ across W4A4, W2A4, and W2A2 (Pang et al., 19 Sep 2025).

This suggests a broader interpretation of entropy-regularized quantization loss in deep networks: the entropy term need not be attached only to symbol histograms or transport plans. It can be imposed on internal representations through information-theoretic surrogates such as minimal coding length, with the goal of improving generalization under quantization noise (Pang et al., 19 Sep 2025).

6. Variants, limits, and recurrent themes

Across these formulations, several recurring variants appear.

Variant	Distortion term	Entropy-related term
Classical entropy-constrained quantization	$H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$4 or $H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$5	$H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$6 or $H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$7
Entropic OT quantization	$H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$8 or $H_0(p)=\log(\mathrm{card}\{i:p_i>0\})$9	$$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$0 or $$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$1
Learned compression	Reconstruction distortion $$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$2	$$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$3, or $$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$4 regularization
Low-bit neural QAT	Task loss	Minimal coding length surrogate $$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$5

A first recurring theme is the difference between hard and soft quantization. Classical fixed-rate or entropy-constrained quantization uses hard partitions and discrete output entropy (Kreitmeier et al., 2011, Koch et al., 2015). Entropic OT and soft quantization replace hard nearest-neighbor assignment with Gibbs-like soft assignments and log-sum-exp smooth minima (Viehmann, 2019, Lakshmanan et al., 2023). Neural surrogates often use softmax assignment to codebook entries for differentiability (Arvinte et al., 2021).

A second theme is the difference between entropy of outputs and entropy of couplings or representations. In information theory, the regularizer is the entropy of the quantizer output distribution itself (Kreitmeier et al., 2011, Koch et al., 2015). In OT, it is the entropy or KL divergence of the transport plan (Viehmann, 2019, Bigot et al., 2017). In neural compression and QAT, it may be latent entropy, coding length, or conditional source entropy used as a structural proxy (Zhang et al., 2024, Pang et al., 19 Sep 2025).

A third theme is the existence of small-regularization limits. In soft quantization using entropic regularization,

$$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$6

(Lakshmanan et al., 2023). For divergence-regularized OT, $$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$7, and for entropic regularization under quadratic cost the leading bias is

$$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$8

under the moment assumptions stated in the theorem (Eckstein et al., 2022). This indicates that the entropy term is a controlled approximation device as well as a regularizer.

A fourth theme is collapse under large regularization. In soft quantization using entropic regularization, there exists $$D_\alpha(R)=\inf\{D_\mu(q): q\in\mathcal{Q},\, H_\alpha(q)\le R\},\quad R\ge 0,$$9 such that for every $\alpha=0$0, the best approximation reduces to a Dirac measure $\alpha=0$1, where $\alpha=0$2 is a center of the measure $\alpha=0$3 with respect to $\alpha=0$4 (Lakshmanan et al., 2023). In neural settings, overly strong entropy or coding regularization can likewise over-compress latents or harm task performance, which is why regularization schedules or carefully tuned strengths are used (Arvinte et al., 2021, Pang et al., 19 Sep 2025, Zhang et al., 2024).

7. Interpretation, misconceptions, and scope

A common misconception is that entropy-regularized quantization loss always means “maximize entropy.” The literature does not support a single sign convention. In classical entropy-constrained quantization and learned compression, the entropy term is penalized because lower output entropy implies lower coding rate (Kreitmeier et al., 2011, Arvinte et al., 2021, Zhang et al., 2024). In entropic OT, the regularizer is added to the coupling objective in a way that favors higher-entropy transport plans relative to the unregularized optimum (Viehmann, 2019, Bigot et al., 2017). In maximum-entropy image quantization, the optimization is performed over a Gibbs distribution over quantized images, and the relevant free-energy view is $\alpha=0$5 rather than a direct penalty on codeword entropy (Lakhal et al., 2022).

A second misconception is that all entropy regularizers target the same object. The target may be a quantizer output pmf, a transport plan, a latent code histogram, a feature covariance spectrum, or a conditional source model (Kreitmeier et al., 2011, Viehmann, 2019, Arvinte et al., 2021, Pang et al., 19 Sep 2025, Zhang et al., 2024). This difference is substantive, because it changes both the optimization geometry and the interpretation of the regularizer.

A third misconception is that entropy regularization is purely heuristic. Several of the formulations are tied directly to information-theoretic equalities or rate–distortion arguments. Rényi-constrained high-rate theory specifies explicit asymptotic coefficients and density laws (Kreitmeier et al., 2011). Converse bounds for entropy-constrained quantization quantify unavoidable excess rate above the Shannon limit (Koch et al., 2015). Entropic OT admits sharp convergence rates as regularization vanishes (Eckstein et al., 2022). Conditional source-entropy regularization in neural compression is derived from exact identities linking $\alpha=0$6, $\alpha=0$7, and $\alpha=0$8 (Zhang et al., 2024).

A plausible implication is that “entropy-regularized quantization loss” is best understood not as a single loss but as a design pattern: pair a quantization or approximation objective with an information-theoretic term whose role is to control coding rate, smooth assignments, improve differentiability, or shape the geometry of learned representations. The specific entropy functional and the variable it acts upon determine the regime—classical quantization theory, entropic OT, learned compression, or low-bit neural optimization—in which the loss operates (Kreitmeier et al., 2011, Lakshmanan et al., 2023, Zhang et al., 2024, Pang et al., 19 Sep 2025).