Entropy-Constrained Minimum Divergence

• Entropy-constrained minimum divergence is a variational framework that minimizes divergence measures while enforcing entropy or coding-length constraints.
• It generalizes classical KL minimization to include Rényi, Tsallis, and alpha-divergences, leading to power-law optimizers and non-exponential solutions.
• Dual representations and minimax formulations within this framework enable robust, tractable optimization for applications in importance sampling, compression, and deep model training.

Entropy-constrained minimum divergence denotes a family of variational problems in which a probability distribution, model, or coupling is selected by minimizing a divergence while obeying an entropy-related constraint, or equivalently by optimizing a Lagrangian that trades divergence against entropy or code length. In the classical finite-alphabet setting, this includes minimum Kullback–Leibler divergence under linear statistical constraints, which is equivalent to maximum entropy when the reference measure is uniform. Beyond the classical case, the literature extends the paradigm to relative $\alpha$-entropies and Rényi/Tsallis entropy, robust importance-sampling design under entropy-bounded admissible targets, minimum relative entropy under general integral constraints, and rate–distortion or MDL formulations in which empirical Shannon entropy acts as a coding penalty rather than a side condition (Kumar et al., 2014).

1. Classical formulation and information-geometric interpretation

The canonical formulation fixes a reference distribution $Q$ and minimizes a divergence over a constrained set of distributions. In the finite-alphabet setting studied in "Relative $\alpha$-Entropy Minimizers Subject to Linear Statistical Constraints" (Kumar et al., 2014), the constraint set is a linear family

$$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$

and the basic optimization problem is

$\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$

For $\alpha \to 1$, the divergence reduces to ordinary relative entropy,

$$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$

so the classical minimum-KL projection appears as a limit case (Kumar et al., 2014).

When the reference $Q$ is uniform, the constrained minimum-divergence problem is exactly a maximum-entropy problem. The same paper states that for the uniform distribution $U$,

$\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$

so minimizing $Q$0 over a constraint set is equivalent to maximizing Shannon entropy $Q$1. In the generalized setting,

$Q$2

with

$Q$3

so minimum relative $Q$4-entropy under linear constraints becomes maximum Rényi entropy, and equivalently maximum Tsallis entropy when the same optimizer is viewed through the monotone relation between the two entropies (Kumar et al., 2014).

The projection geometry is a central structural feature. For $Q$5, the optimizer $Q$6 satisfies the exact Pythagorean equality

$Q$7

whereas for $Q$8 the corresponding relation is generally an inequality,

$Q$9

with equality if the support of the optimizer matches the support of the family. This places entropy-constrained minimum divergence in direct continuity with Euclidean projection and classical information projection theory (Kumar et al., 2014).

2. Generalized entropy principles: Rényi, Tsallis, and power-law optimizers

Replacing KL divergence by relative $\alpha$0-entropy changes both the optimizer and the geometry. In the same finite-alphabet theory, the relative $\alpha$1-entropy is

$\alpha$2

with $\alpha$3 and equality iff $\alpha$4, and

$\alpha$5

(Kumar et al., 2014).

The optimizer is no longer exponential in general. For $\alpha$6, the $\alpha$7-projection takes the power-law form

$\alpha$8

whereas for $\alpha$9 the solution acquires a truncation operator,

$$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$0

The paper identifies this as the exact non-KL analogue of the exponential family: linear constraints plus KL divergence lead to an exponential family, while linear constraints plus relative $$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$1-entropy lead to an $$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$2-power-law family (Kumar et al., 2014).

A closely related continuous-density formulation appears in "On some entropy functionals derived from Rényi information divergence" (0805.0129). There the objective is to minimize Rényi divergence relative to a reference $$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$3,

$$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$4

under either a standard mean constraint $$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$5 or a generalized escort-mean constraint

$$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$6

The corresponding optimizers are generalized power-law deformations of the reference measure,

$$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$7

with $$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$8 for the classical mean constraint and $$\mathbb{L}=\left\{P\in\mathcal{P}(\mathbb{X}) : \sum_x P(x)f_i(x)=0,\quad i=1,\dots,k \right\},$$9 for the generalized mean constraint (0805.0129).

That paper also makes the entropy–divergence equivalence explicit by defining Rényi $\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$0-entropy as

$\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$1

Thus maximum Rényi $\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$2-entropy is exactly minimum Rényi divergence relative to $\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$3. For the generalized constraint, it further derives a Pythagorean identity,

$\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$4

which gives a direct projection interpretation for the escort-constrained optimizer (0805.0129).

A more recent superfamily appears in "Characterization of Generalized Alpha-Beta Divergence and Associated Entropy Measures" (Roy et al., 7 Jul 2025). The generalized alpha-beta divergence

$\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$5

is shown to subsume AB divergence, logarithmic AB divergence, power divergence, density power divergence, logarithmic density power divergence, S-divergence, logarithmic S-divergence, $\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$6-divergence, and $\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$7-divergence. Its associated entropy is derived directly from divergence to a uniform measure,

$\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$8

and the paper provides sufficient conditions for concavity of this entropy and convexity of the divergence in one argument (Roy et al., 7 Jul 2025). This suggests a broad generalized framework in which an entropy constraint can be reinterpreted as a constraint on divergence from uniformity.

3. Dual representations and integral constraints

A second major strand of the subject concerns duality. "Dual Representation of Minimum Divergence Under Integral Constraints" (Shekhar et al., 22 Mar 2026) studies problems of the form

$\min_{P\in\mathbb{L}} \mathscr{I}_{\alpha}(P,Q).$9

on $\alpha \to 1$0, with $\alpha \to 1$1 continuous and $\alpha \to 1$2 closed and convex. A defining feature is that the minimization is in the second argument,

$\alpha \to 1$3

rather than the more common $\alpha \to 1$4 (Shekhar et al., 22 Mar 2026).

In the mean-constrained KL case,

$\alpha \to 1$5

the paper proves the dual formula

$\alpha \to 1$6

with

$\alpha \to 1$7

For arbitrary integral constraints, the KL dual becomes

$\alpha \to 1$8

These formulas convert infinite-dimensional constrained minimum-divergence problems into finite-dimensional concave dual programs (Shekhar et al., 22 Mar 2026).

The same paper extends the construction to a broad class of $\alpha \to 1$9-divergences. For

$$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$0

the dual is

$$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$1

where

$$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$2

A plausible implication is that entropy-constrained minimum divergence problems become tractable whenever the entropy constraint can be encoded as an integral constraint or as divergence to a reference that itself admits such a dualization (Shekhar et al., 22 Mar 2026).

An axiomatic perspective on entropy–divergence pairing is given in "Entropy and relative entropy from information-theoretic principles" (Gour et al., 2020). It proves a bijection between entropies and relative entropies continuous in the second argument, with the map

$$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$3

For uniform reference, an entropy constraint is therefore equivalent to a divergence-to-uniform constraint. The same paper also establishes the universal bounds

$$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$4

for every relative entropy in its axiomatic class. This does not solve a generic entropy-constrained projection problem, but it gives structural lower and upper envelopes for any such optimization (Gour et al., 2020).

4. Minimax and robust formulations

Entropy-constrained minimum divergence also appears in minimax form. "Entropy minimizing distributions are worst-case optimal importance proposals" (Cérou et al., 2022) considers importance sampling with reference proposal $$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$5, unknown target $$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$6, and proposal $$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$7, with cost measured by relative entropy

$$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$8

The key quantity is the information projection of $$\mathscr{I}(P\|Q)=\sum_{x\in\mathbb{X}} P(x)\log \frac{P(x)}{Q(x)},$$9 onto a convex admissible class $Q$0,

$Q$1

and the worst-case log-cost functional

$Q$2

Under the paper’s half-space approximation assumption, $Q$3 uniquely minimizes this worst-case constrained KL cost and satisfies

$Q$4

and for every $Q$5,

$Q$6

This yields a precise minimax statement: targets are constrained by an entropy budget relative to $Q$7, worst-case target-to-proposal KL is minimized, and the optimizer is the reference-relative entropy minimizer over the admissible convex class (Cérou et al., 2022).

A related but older concentration-based interpretation appears in "Entropy Concentration and the Empirical Coding Game" (0809.1017). There the maximum-entropy or minimum-relative-entropy solution

$Q$8

is shown to describe the asymptotic conditional law obtained by drawing from $Q$9 and conditioning on the empirical moment constraint $U$0. The strong conditional limit theorem states that if $U$1, then

$U$2

converges weakly to

$U$3

This does not impose an entropy bound directly, but it supplies a probabilistic interpretation of constrained minimum divergence: the KL minimizer is the effective law of typical constrained samples (0809.1017).

The same general theme extends to structural constraints. "Complex-Valued Random Vectors and Channels: Entropy, Divergence, and Capacity" (Tauboeck, 2011) proves that the circular analog $U$4 of a complex random vector $U$5 is exactly the reverse-KL projection onto the family of circular laws,

$U$6

and identifies the minimum divergence as a conditional entropy. This is not a standard entropy-constrained problem, but it shows that minimum-divergence projection can collapse to an entropy quantity under symmetry constraints (Tauboeck, 2011).

5. Rate, MDL, and entropy as coding cost

In coding and model compression, the relevant entropy is often not the entropy of a data distribution but the Shannon entropy of a codebook or empirical symbol histogram. "Entropy-Constrained Training of Deep Neural Networks" (Wiedemann et al., 2018) gives an explicit entropy-constrained MDL objective for a discrete-weight network with alphabet

$U$7

empirical weight histogram

$U$8

and network entropy

$U$9

The exact compression-aware training problem is

$\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$0

Equivalently, in constrained form,

$\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$1

Here the entropy term is explicitly a rate or codelength surrogate: the model bit-size is approximated by

$\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$2

and for large $\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$3, $\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$4 dominates (Wiedemann et al., 2018).

The paper does not formulate the main discrete objective as a minimum-KL problem. Its core formulation is an MDL or rate–distortion Lagrangian. However, after continuous relaxation it shows that the entropy regularizer can be written as

$\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$5

where $\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$6 is a factorized distribution over symbol assignments and $\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$7 is a prior induced by the average symbol usage. Thus the relaxed entropy penalty becomes a cross-entropy or $\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$8 coding term under a learned shared symbol prior (Wiedemann et al., 2018).

This coding-theoretic reading places entropy-constrained minimum divergence in direct contact with MDL. A plausible implication is that when entropy is interpreted as exact or asymptotically exact codelength, the divergence term need not be KL between probability models; it can instead be negative log-likelihood or prediction loss, with entropy acting as a rate budget. In that sense, entropy-constrained minimum divergence, rate–distortion optimization, and MDL occupy adjacent but non-identical positions in the same variational landscape (Wiedemann et al., 2018).

A related but distinct coupling-based rate formulation appears in "Cross-Domain Lossy Compression via Constrained Minimum Entropy Coupling" (Nguyen et al., 11 May 2026). There the objective is not KL minimization but

$\mathscr{I}(P\|U)=\log |\mathbb{X}| - H(P),$9

subject to a target-marginal constraint and an entropy-rate constraint

$Q$00

Because $Q$01 and $Q$02 are fixed when marginals are fixed, maximizing $Q$03 is equivalent to minimizing joint entropy under fixed marginals. This places the work in an entropy-constrained distribution-matching family that is close to, but not identical with, entropy-constrained minimum-divergence formulations (Nguyen et al., 11 May 2026).

6. Scope, adjacent problems, and limitations

The literature uses the phrase in more than one sense, and not all nearby problems are strictly entropy-constrained minimum divergence. "On the Renyi Divergence, Joint Range of Relative Entropies, and a Channel Coding Theorem" (Sason, 2015) studies

$Q$04

which is constrained divergence minimization under a total variation constraint, not under an entropy constraint. The problem is nevertheless methodologically relevant because it yields exact binary reduction, KKT characterization, and support-line geometry for feasible divergence regions (Sason, 2015).

Likewise, "Entropy-Based Dimension-Free Convergence and Loss-Adaptive Schedules for Diffusion Models" (Aghapour et al., 29 Jan 2026) does not set up an explicit entropy-constrained optimization. Instead, it proves an entropy-controlled upper bound on KL discretization error,

$Q$05

leading to a dimension-free bound of order $Q$06 up to endpoint factors for diffusion-model discretization. Entropy here is a complexity parameter controlling divergence, not a constraint variable (Aghapour et al., 29 Jan 2026).

Recent work on generalized divergences similarly provides machinery rather than complete projection theory. The generalized alpha-beta framework supplies validity conditions, an associated entropy, lower semicontinuity, convexity results, and an approximate Pythagorean identity, but it does not yet derive explicit KKT systems or full forward/reverse projection theorems for

$Q$07

(Roy et al., 7 Jul 2025). A plausible implication is that future work will unify generalized entropy constraints, divergence projections, and robust estimation within a single geometric theory.

Across these strands, several distinctions are stable. First, entropy may denote Shannon entropy of a feasible distribution, Rényi or Tsallis entropy induced by a generalized divergence, empirical entropy of a codebook, or conditional entropy in coupling and log-loss formulations. Second, the divergence may be forward KL, reverse KL, relative $Q$08-entropy, Rényi divergence, an $Q$09-divergence in the second argument, or even a task loss plus coding penalty rather than a literal divergence. Third, “constrained” may mean an explicit hard entropy budget, a Lagrangian rate term, a linear expectation constraint equivalent to maximum entropy under uniform reference, or a minimax admissible set defined by a divergence-to-reference budget. The subject is therefore best understood as a broad information-theoretic family of projection, coding, and robust optimization principles rather than a single canonical optimization template (Kumar et al., 2014).