Entropy Projection: Theory & Applications

• Entropy projection is a method that fuses an entropy-based criterion with a projection operator to optimize distributions and preserve structural information.
• It underlies applications in information geometry, statistical projection pursuit, machine learning regularization, and numerical methods for stability.
• Various formulations—from divergence minimization to projection-induced corrections—offer practical benefits in estimation accuracy and robust system design.

Entropy projection denotes a family of constructions in which an entropy, a relative entropy, or an entropy-related functional is coupled to a projection operator or projection problem. In information geometry, it usually means minimizing a divergence such as Kullback–Leibler divergence or relative $\alpha$-entropy over a constraint family, producing forward or reverse projections (Kumar et al., 2014, Kumar et al., 2014). In statistics, it appears in projection pursuit through negative Shannon entropy or relative-entropy minimization (Cui et al., 2019, Touboul, 2010). In machine learning, it can mean pushing a classifier’s predictive distribution toward the uniform distribution through a KL penalty, which is equivalent to entropy maximization (Cha et al., 2020). Other literatures use the phrase differently: projection entropy is an entropy of a projected feature allocation for clustering words (Fidaner et al., 2014); “entropy projection” can denote the harmful flattening of a bidimensional field into a one-dimensional string before compression-based entropy estimation (Filho et al., 2022); and in entropy-stable discontinuous Galerkin methods it denotes projection of entropy variables into a discrete approximation space before reconstructing conservative variables (Chan et al., 2022). The term is therefore best understood as a family resemblance rather than a single universally standardized construction.

1. Scope and principal meanings

Across the cited literature, entropy projection combines two ideas: a projection step and an entropy-based criterion. The projection may act on distributions, coordinates, conditional laws, feature allocations, discrete entropy variables, or raw data representations. The entropy quantity may be Shannon entropy, differential entropy, Kullback–Leibler divergence, relative $\alpha$-entropy, Von Neumann entropy, or a specialized statistic such as projection entropy on feature allocations (Kumar et al., 2014, Fidaner et al., 2014, Chan et al., 2022).

Domain	Projection object	Entropy role
Information geometry	Probability measure onto constraint family	Divergence minimization
Projection pursuit	Linear subspace or direction	Low-entropy or non-Gaussian index
Continual learning	Classifier output distribution	KL-to-uniform, hence entropy maximization
Text clustering	Feature allocation restricted to a word set	Entropy of segmentedness
2D entropy estimation	Image flattened to 1D stream	Projection causes information loss
Entropy-stable DG	Entropy variables into polynomial space	Enables discrete entropy stability

A recurring misconception is that entropy projection always refers to Euclidean projection followed by entropy evaluation. The literature instead shows several non-equivalent patterns: an information projection in divergence geometry, a projection pursuit index defined by entropy, a projection-induced bias in entropy estimation, or a numerical projection used to preserve an entropy law. The common structure is not the algebraic form of the projection, but the use of projection to enforce, reveal, estimate, or preserve entropy-related structure.

2. Information-geometric projection theory

The most systematic theory appears in the work on relative $\alpha$-entropy $\mathscr I_\alpha$, a one-parameter generalization of Kullback–Leibler divergence. If $P,Q$ have densities $p,q$, one representation is

$$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$

and $$\lim_{\alpha\to 1}\mathscr{I}_{\alpha}(P,Q)=\mathscr{I}(P\|Q)$$. The forward projection problem is

$\min_{P\in\mathbb E}\mathscr I_\alpha(P,R),$

with minimizer called the forward $\mathscr I_\alpha$-projection of $\alpha$0 on $\alpha$1. For closed convex sets of densities in $\alpha$2, the projection exists uniquely; it satisfies a Pythagorean property,

$\alpha$3

and for linear families its density takes a power-law form rather than the exponential-family form familiar from the KL case (Kumar et al., 2014).

This framework directly generalizes maximum Rényi and Tsallis entropy principles. For a uniform reference $\alpha$4 on a finite alphabet,

$\alpha$5

so minimizing $\alpha$6 is equivalent to maximizing Rényi entropy, and by monotonicity also Tsallis entropy. In linear families $\alpha$7, the forward projection yields a power-law family; under moment or covariance constraints, the maximizing distribution is of $\alpha$8-Gaussian or Student-type power-law form (Kumar et al., 2014).

The reverse problem interchanges the arguments: $\alpha$9 This reverse $\alpha$0-projection is the form relevant for estimation and constrained compression. Its geometry is subtler. For $\alpha$1, reverse projections on log-convex families are unique; for $\alpha$2, uniqueness can fail, and the paper gives an explicit example with two distinct global minima. A central result is an orthogonality relation between linear families and $\alpha$3-power-law families, which converts a reverse projection into a forward projection on a related linear family. For $\alpha$4, the construction may require an extended family $\alpha$5 because the minimizer need not lie in the closure of the original power-law family (Kumar et al., 2014).

The same projection viewpoint reappears in a broader divergence-setting treatment of likelihood maximization. Reverse projections of $\alpha$6, $\alpha$7, $\alpha$8, and $\alpha$9 onto appropriate parametric families correspond to ordinary or robustified likelihood estimators, while the associated projection theorems show equivalence between geometric projection, estimating equations, and sufficiency-based reductions to linear constraints (Gayen et al., 2017).

3. Projection pursuit and entropy as an index of non-Gaussian structure

In projection pursuit, entropy supplies a projection index that prefers non-Gaussian low-dimensional views. One formulation uses the negative standardized Shannon entropy of a one-dimensional projection $\mathscr I_\alpha$0,

$\mathscr I_\alpha$1

where $\mathscr I_\alpha$2 is the density of $\mathscr I_\alpha$3 and $\mathscr I_\alpha$4 is the standard normal density. The rationale is classical: Gaussian distributions maximize entropy under common constraints, while most random high-dimensional projections look approximately Gaussian; projections with lower entropy or greater non-Gaussianity are therefore treated as “interesting.” The paper further gives approximations based on cumulants,

$\mathscr I_\alpha$5

and on non-polynomial functions such as $\mathscr I_\alpha$6. In the scRNA-seq application considered there, however, PCA produced better cell-type clustering than the entropy-based projection pursuit method (Cui et al., 2019).

A different projection-pursuit line uses relative entropy minimization for high-dimensional density estimation. There, an elliptical baseline density $\mathscr I_\alpha$7 with the same mean and covariance as the target density $\mathscr I_\alpha$8 is iteratively corrected along selected projection directions. The updated density has the form

$\mathscr I_\alpha$9

and the method seeks directions through KL-based criteria. The paper states that the same vector $P,Q$0 solves several apparently different optimization problems, including Huber-style analytic and synthetic formulations and the paper’s own criterion based on minimizing a KL divergence in the opposite direction. If the KL divergence reaches zero after $P,Q$1 steps, the density is exactly factorized; otherwise the sequence yields a product approximation (Touboul, 2010).

Local refinement of promising projections can also be based directly on estimated differential entropy. In that approach, invariant coordinate selection provides an initial orthogonal projection $P,Q$2, and a local search on the orthogonal group minimizes a kernel-based entropy estimate

$P,Q$3

The optimization is performed by gradient descent in antisymmetric coordinates via $P,Q$4, with an Armijo–Goldstein rule for step selection. The resulting procedure is explicitly designed to sharpen “almost interesting” ICS projections that become informative after small rotations (Duembgen et al., 2021).

4. Machine learning, regularization, and closed-loop distributional control

In continual learning, classifier-projection regularization interprets entropy maximization as a projection of the classifier output toward the uniform distribution. If $P,Q$5 is the softmax output and $P,Q$6 is the uniform distribution on the $P,Q$7-class simplex, the regularized loss is

$P,Q$8

Because

$P,Q$9

minimizing the KL term is equivalent to maximizing Shannon entropy of the output distribution. The paper formulates this as a classifier projection onto a KL-divergence ball centered at the uniform distribution and invokes the KL Pythagorean theorem to argue that such projection can keep new-task classifiers closer, in KL and cross-entropy sense, to previous-task classifiers. The reported motivation is reduced catastrophic forgetting, improved plasticity, and wider local minima (Cha et al., 2020).

A more general closed-loop framework appears in Entropy-Reservoir Bregman Projection. There, self-referential learning is modeled as a stochastic projection sequence in distribution space. At each round, an empirical distribution $p,q$0 from the current model is mixed with an external reservoir,

$p,q$1

and then approximately projected back onto a model manifold: $p,q$2 Without reservoir coupling, the paper proves an entropy-contraction bound

$p,q$3

while positive coupling yields a nontrivial floor

$p,q$4

The framework interprets real-data mixing, entropy bonuses, knowledge distillation, RLHF, and retrieval-augmented generation as distinct reservoir choices and coupling coefficients (Chen, 16 Dec 2025).

These machine-learning usages differ from classical information projection in one important respect. The projection target is not merely a static feasible set; it is often a regularizing distribution such as $p,q$5 or a time-dependent mixed target $p,q$6. The entropy term is therefore used not only for inference under constraints, but also for stability-plasticity control, flattening of predictive distributions, and prevention of closed-loop entropy collapse.

5. Projected objects, projected data, and entropy estimation

A distinct usage is projection entropy, introduced as the entropy of a projected feature allocation. In the word-clustering setting, a literary text is reduced to a feature allocation $p,q$7, where each block $p,q$8 is a paragraph-level set of word types. For a subset of words $p,q$9, the projection $$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$0 restricts attention to blocks relevant to $$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$1, and projection entropy is defined by

$$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$2

The statistic measures segmentedness: low $$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$3 means the words in $$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$4 tend to occur in the same paragraphs, high $$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$5 means they are split across paragraphs in different combinations, and full overlap yields $$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$6. Entropy agglomeration then performs greedy hierarchical clustering by repeatedly merging the pair of clusters whose union has the smallest projection entropy. Applied to Ulysses, this recovered antonyms, pronoun and verb inflections, reciprocals, and thematic associations (Fidaner et al., 2014).

In bidimensional entropy estimation, “entropy projection” has almost the opposite meaning. There the problematic step is flattening a 2D image into a 1D string before applying a compressor-based entropy estimator. The paper argues that such projection destroys or blurs genuinely bidimensional correlations, makes the estimate path-dependent, can create spurious long-range correlations, and leads to systematic overestimation of entropy for long-range correlated images. The practical conclusion is that block entropies remain superior, while compressors that avoid dimensionality reduction, especially Jpeg-ls, perform better than projection-based compressors but still do not match block-entropy estimates (Filho et al., 2022).

Projection can also be beneficial when it preserves the relevant geometry. For large low-rank density matrices, random projection is used to reduce dimension before estimating Von Neumann entropy from approximate singular values,

$$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$7

If the projector satisfies a Johnson–Lindenstrauss guarantee, the projected singular values remain close enough to the original spectrum to control the entropy error. The cited bound is

$$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$8

with the stated probability conditions. The novelty there is that local random quantum circuits approximating unitary 2-designs can serve as such random projectors (Kumaran et al., 2023).

Taken together, these cases show that entropy projection is not uniformly beneficial or harmful. Projection entropy on feature allocations is itself the target statistic; one-dimensional flattening of 2D spatial data degrades entropy estimation; and Johnson–Lindenstrauss-type random projection can preserve enough spectral structure to approximate Von Neumann entropy. This suggests that the decisive issue is whether the projection respects the correlation or spectral structure on which the entropy depends.

6. Numerical discretization and entropy-preserving projection steps

In high-order entropy-stable discontinuous Galerkin methods, entropy projection is a discrete representation device rather than a divergence minimization. The procedure is: compute entropy variables from the current polynomial solution at quadrature nodes, project those entropy variables into the polynomial approximation space, and map the projected entropy variables back to conservative variables before evaluating the flux-differencing residual. With quadrature-based projector

$$\mathscr{I}_{\alpha}(P,Q) = \frac{\alpha}{1-\alpha}\log \int \frac{p}{\|p\|}\left(\frac{q}{\|q\|}\right)^{\alpha-1} d\mu,$$9

the sequence is

$$\lim_{\alpha\to 1}\mathscr{I}_{\alpha}(P,Q)=\mathscr{I}(P\|Q)$$0

This projected conservative state, not the original nodal state, is used in the entropy-stable residual. The purpose is to make the discrete entropy proof work for non-collocated quadrature and modal formulations (Chan et al., 2022).

The same paper reports that entropy-projection-based schemes are empirically more robust for under-resolved variable-density Euler and MHD flows than collocation-type entropy-stable schemes. The reported pattern is that Gauss DG or modal entropy projection methods more often run to the final time, while collocation DGSEM or collocation SBP methods crash earlier in Kelvin–Helmholtz, Rayleigh–Taylor, Richtmeyer–Meshkov, and related tests. The authors do not claim a complete theory for the robustness improvement, but they investigate projection error in entropy variables, sensitivity near vacuum, interface spikes in projected states, and altered dissipation as possible explanations (Chan et al., 2022).

A broader Runge–Kutta projection framework treats entropy as one instance of an auxiliary admissibility criterion. Quasi-orthogonal projection methods append a post-step correction

$$\lim_{\alpha\to 1}\mathscr{I}_{\alpha}(P,Q)=\mathscr{I}(P\|Q)$$1

where $$\lim_{\alpha\to 1}\mathscr{I}_{\alpha}(P,Q)=\mathscr{I}(P\|Q)$$2 is the component of the invariant gradient lying in the span of the RK stage derivatives. The paper states explicitly that it does not derive a standalone entropy-stable flux formulation; rather, if entropy is modeled as a conserved invariant $$\lim_{\alpha\to 1}\mathscr{I}_{\alpha}(P,Q)=\mathscr{I}(P\|Q)$$3, the method projects to the entropy level set, and if entropy is modeled as dissipative, it projects to an RK-consistent discrete dissipation law (Najafian et al., 2024).

These numerical usages clarify a final distinction. In information geometry, projection is a variational statement about proximity under a divergence. In entropy-stable time integration and DG discretization, projection is a structure-preserving correction inside a finite-dimensional approximation. The entropy criterion is then tied to admissibility, stability, or robustness of the discretized dynamics, not to model fitting or statistical inference.