Entropic Gradient Descent (EGD)

• Entropic Gradient Descent (EGD) is a first-order optimization method that uses the geometry induced by negative entropy to produce multiplicative, exponentiated updates.
• It enforces simplex and positivity constraints natively, eliminating the need for explicit Euclidean projections and enabling efficient updates in various domains.
• EGD is applied in areas such as hyperspectral unmixing, variational inference, and deep learning, with robust convergence guarantees and extensions to measure-valued and information-geometric frameworks.

Entropic Gradient Descent (EGD) denotes a family of first-order optimization methods in which descent is performed in the geometry induced by negative entropy rather than the Euclidean metric. In its standard form, EGD is mirror descent with the negative-entropy mirror map, so the associated Bregman divergence is the Kullback–Leibler divergence and the resulting update is multiplicative or exponentiated rather than additive (Zouaoui et al., 2022). In a distinct but related deep-learning usage, EGD also refers to Entropy-SGD, where the algorithm descends not on the raw loss but on a local-entropy smoothing of that loss, thereby favoring wide, flat minima (Bardi et al., 2022).

1. Terminological scope and core meaning

In the optimization literature represented here, EGD is most commonly synonymous with entropic mirror descent, entropic gradient, exponentiated gradient, and multiplicative weights. The underlying construction uses the negative entropy

$h(x)=\sum_{i=1}^d x_i \ln x_i$

on a simplex or positive orthant, and the mirror step replaces Euclidean projection by a KL-proximal update (Zouaoui et al., 2022). In this sense, EGD is a geometric method: the “entropic” aspect lies in the mirror map and the induced Bregman geometry.

A recurrent source of confusion is that some deep-learning papers use “Entropic Gradient Descent” to denote Entropy-SGD, which instead modifies the objective by introducing a local-entropy functional

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$

and then performs descent on $\phi_\gamma$ rather than on $\phi$ itself (Bardi et al., 2022). The two usages are related by their entropic character, but they act at different levels: one changes the optimization geometry, the other smooths the objective.

The same exponentiated mechanism also appears in measure-valued optimization. In $(\alpha,\Gamma)$-descent, choosing $\Gamma_\alpha(v)=e^{-\eta v}$ recovers infinite-dimensional Entropic Mirror Descent, with updates of the form

$$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$

and, in the discrete case, a normalized multiplicative update on mixture weights (Daudel et al., 2020). This suggests that EGD is best understood as a broad entropic descent principle instantiated on simplices, positive orthants, matrices of row-wise distributions, or spaces of probability measures.

2. Bregman and information-geometric foundations

A general formulation is given by the generalized Bregman framework, where the iteration

$$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$

with $p^k\in \partial J(u^k)$, reduces to mirror descent when $J$ is smooth (Benning et al., 2016). Specializing $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$0 to the negative entropy $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$1 yields entropic mirror descent, for which

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$2

On the simplex, this coincides with the KL divergence (Benning et al., 2016).

For a differentiable objective $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$3 on the simplex $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$4, the mirror step has the primal form

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$5

and solving the subproblem gives the normalized exponentiated update

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$6

Equivalently,

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$7

which preserves nonnegativity and the sum-to-one constraint automatically (Zouaoui et al., 2022).

An information-geometric interpretation is available on the positive orthant $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$8. With

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$9

the induced Fisher–Rao metric is

$\phi_\gamma$0

and the Riemannian gradient is

$\phi_\gamma$1

Using the $\phi_\gamma$2-Exp retraction

$\phi_\gamma$3

Riemannian gradient descent becomes

$\phi_\gamma$4

which is exactly exponentiated gradient on $\phi_\gamma$5 (Elshiaty et al., 7 Apr 2025). In this reading, EGD is simultaneously mirror descent, natural gradient descent under $\phi_\gamma$6, and multiplicative descent in primal coordinates.

3. Canonical update rules and constraint handling

The central computational advantage of EGD is that simplex or positivity constraints are enforced by the update itself. In simplex-constrained problems, the softmax or normalized exponentiated form ensures that iterates remain feasible, thereby avoiding Euclidean projections, active-set logic, or sorting-based routines (Zouaoui et al., 2022). In the positive orthant, multiplicative updates preserve strict positivity as long as the initialization is strictly positive (Elshiaty et al., 7 Apr 2025).

This property is especially important in blind hyperspectral unmixing via archetypal analysis. In "Entropic Descent Archetypal Analysis for Blind Hyperspectral Unmixing" (Zouaoui et al., 2022), the data matrix $\phi_\gamma$7 is modeled as

$\phi_\gamma$8

with abundance columns $\phi_\gamma$9 and archetype-coefficient columns $\phi$0. The blind AA objective

$\phi$1

is convex in $\phi$2 for fixed $\phi$3 and convex in $\phi$4 for fixed $\phi$5, and the algorithm alternates entropic updates on the two blocks. The update for $\phi$6 is

$\phi$7

and the update for $\phi$8 is

$\phi$9

with column-wise softmax enforcing the simplex constraints (Zouaoui et al., 2022). Each inner update costs $(\alpha,\Gamma)$0, and with $(\alpha,\Gamma)$1 outer iterations and $(\alpha,\Gamma)$2 inner steps the total cost is $(\alpha,\Gamma)$3. The paper uses $(\alpha,\Gamma)$4, $(\alpha,\Gamma)$5, reports that this is fast on a single GPU, and recommends $(\alpha,\Gamma)$6 runs in the ensembling stage (Zouaoui et al., 2022).

A closely related matrix-simplex construction appears in adversarial optimization for LLMs. There the relaxed one-hot matrix $(\alpha,\Gamma)$7 has rows constrained to $(\alpha,\Gamma)$8, and the update is

$(\alpha,\Gamma)$9

where the KL/Bregman projection $\Gamma_\alpha(v)=e^{-\eta v}$0 is simply row normalization (Biswas et al., 14 May 2025). The paper emphasizes that this is an intrinsic optimization over token distributions rather than an optimization in embedding space followed by an extrinsic projection to tokens.

4. Convergence theory

Within the generalized Bregman framework, entropic mirror descent inherits sufficient-decrease and global-convergence results under structural assumptions on the objective. For linearized Bregman iteration, if there exists a proper, lower semi-continuous, convex $\Gamma_\alpha(v)=e^{-\eta v}$1 such that $\Gamma_\alpha(v)=e^{-\eta v}$2 is convex, the iteration satisfies a sufficient decrease property; under additional strong-convexity conditions on $\Gamma_\alpha(v)=e^{-\eta v}$3 and $\Gamma_\alpha(v)=e^{-\eta v}$4, one obtains a gradient bound; and under a generalized Kurdyka–Łojasiewicz property, strongly convergent subsequences exist, while in finite dimensions the full sequence converges (Benning et al., 2016). Specializing $\Gamma_\alpha(v)=e^{-\eta v}$5 to the negative entropy yields the corresponding EGD statements.

The measure-valued $\Gamma_\alpha(v)=e^{-\eta v}$6-framework gives more explicit performance guarantees for the entropic choice $\Gamma_\alpha(v)=e^{-\eta v}$7. Under Hypotheses A-1 and A-2, the exact update satisfies

$\Gamma_\alpha(v)=e^{-\eta v}$8

with equality iff $\Gamma_\alpha(v)=e^{-\eta v}$9 is a fixed point. Under additional smoothness and boundedness conditions, the paper proves an $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$0 rate in the exact setting and an $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$1 stochastic bound for averaged iterates (Daudel et al., 2020). The same paper also identifies a limitation: in high dimension, entropic mirror descent can under-discriminate when the values of $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$2 cluster tightly, and support mismatch for $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$3 can lead to divergence; this motivates the alternative Power Descent transform (Daudel et al., 2020).

On the positive orthant, a different route to convergence is developed through information geometry. Interpreting EG as Riemannian gradient descent with the $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$4-Exp retraction, "Information Geometry of Exponentiated Gradient: Convergence beyond L-Smoothness" (Elshiaty et al., 7 Apr 2025) proves finite termination of Riemannian Armijo line search and global convergence under smoothness of $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$5, without assuming $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$6-smoothness or relative smoothness. Every accumulation point is a critical point, and if $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$7 is convex the iterates converge to a global minimizer (Elshiaty et al., 7 Apr 2025).

For the row-simplex LLM attack formulation, the update with KL projection is shown to converge to a critical point when $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$8 is differentiable with Lipschitz continuous gradient and the learning rates are sufficiently small (Biswas et al., 14 May 2025). The theorem is explicitly limited to the EG-with-projection iteration; the paper states that it does not cover ReLU-based architectures or the Adam-accelerated EGD variant (Biswas et al., 14 May 2025).

5. Representative application domains

Blind hyperspectral unmixing provides one of the clearest large-scale demonstrations of EGD as a practical optimizer. In EDAA, endmembers are expressed as convex combinations of pixels already present in the image, so both abundances and archetype coefficients lie on simplices. Across six real datasets—Samson, Jasper Ridge, Urban4/6, APEX, and WDC—the method achieves the best overall abundance RMSE, with SAD results that are competitive and frequently second-best or best; the paper further reports processing times of approximately $$d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},$$9 s per Urban6 run and approximately $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$0 s total for $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$1 runs and model selection (Zouaoui et al., 2022). The same study attributes part of this performance to the fact that EGD is projection-free, GPU-friendly, and fast enough to support ensembling and coherence-based model selection.

In variational inference over measures, EGD appears as an update on mixture weights or on the measure itself. For discrete support $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$2, the update

$$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$3

optimizes an $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$4-divergence objective over the simplex of mixture coefficients (Daudel et al., 2020). The stochastic formulation is notable because it updates the weights of a mixture model using only samples $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$5, without requiring knowledge of the distribution of the variational parameters. At the same time, the paper reports that in toy Gaussian-mixture experiments the $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$6-Mirror method fails to learn in dimensions $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$7 and $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$8, while $$u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},$$9-Power remains stable, and that on Bayesian logistic regression for Covertype the $p^k\in \partial J(u^k)$0-Power method improves predictive likelihood and accuracy relative to an adaptive importance-sampling baseline (Daudel et al., 2020).

Entropic-gradient ideas also appear in semi-discrete optimal transport, although here the “entropic” component lies in the objective rather than the mirror geometry. DRAG performs stochastic gradient descent on the entropically smoothed semi-dual

$p^k\in \partial J(u^k)$1

with single-sample gradient

$p^k\in \partial J(u^k)$2

and uses a decreasing regularization schedule $p^k\in \partial J(u^k)$3 together with Polyak–Ruppert averaging (Genans et al., 31 Oct 2025). The theoretical result is an unbiased $p^k\in \partial J(u^k)$4 sample and iteration complexity for OT cost and potential estimation, and an $p^k\in \partial J(u^k)$5 rate for the OT map (Genans et al., 31 Oct 2025). This is not mirror-descent EGD in the strict sense, but it is an entropic-gradient method in the sense of optimizing an entropically smoothed objective.

A further application is adversarial optimization for LLMs. By optimizing directly over distributions on vocabulary tokens, the exponentiated update remains intrinsic to the simplex of one-hot relaxations. The paper evaluates five open-source LLMs on four openly available datasets and reports that EGD achieves higher attack success rates with greater efficiency than three other state-of-the-art jailbreaking techniques (Biswas et al., 14 May 2025).

6. Local-entropy EGD in deep learning

In the deep-learning line of work, EGD denotes descent on a local-entropy objective rather than mirror descent on a simplex. Given a base loss $p^k\in \partial J(u^k)$6, the local entropy at anchor $p^k\in \partial J(u^k)$7 is

$p^k\in \partial J(u^k)$8

and its gradient is

$p^k\in \partial J(u^k)$9

where $J$0 is the invariant Gibbs measure of the inner Langevin dynamics with $J$1 frozen (Bardi et al., 2022). The practical algorithm samples an auxiliary variable $J$2 with a short inner Langevin loop and updates the slow variable in the direction of $J$3, thereby approximating $J$4 (Bardi et al., 2022).

The control-theoretic homogenization analysis embeds the Chaudhari et al. two-scale stochastic system into a broader singularly perturbed control problem and shows that, under assumptions including Lipschitz $J$5 and $J$6, the uncontrolled entropic system converges to the effective ODE

$J$7

while the corresponding value functions converge to those of an effective control problem with averaged coefficients (Bardi et al., 2022). This places Entropy-SGD on a rigorous averaging foundation and interprets tuning parameters such as the learning-rate control $J$8, anchoring strength $J$9, inverse temperature $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$00, and noise as controls in the homogenized system.

Empirically, Entropy-SGD and Replicated-SGD are reported to “consistently improve the generalization error for common architectures (e.g. ResNet, EfficientNet)” (Pittorino et al., 2020). The same paper uses local entropy and local energy as flatness diagnostics and states that an easy-to-compute flatness measure shows a clear correlation with test accuracy (Pittorino et al., 2020). The theoretical and empirical accounts therefore converge on the same interpretation: local-entropy EGD biases optimization toward wide flat minima rather than merely accelerating constrained first-order descent.

7. Generalizations, limitations, and research directions

Several recent works generalize classical EGD by replacing the Shannon mirror map with other trace-form entropies and deformed logarithms. "Generalized Exponentiated Gradient Algorithms Using the Euler Two-Parameter Logarithm" (Cichocki, 21 Feb 2025) defines

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$01

with inverse $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$02, and derives the generalized mirror update

$$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$03

"Mirror Descent and Novel Exponentiated Gradient Algorithms Using Trace-Form Entropies and Deformed Logarithms" (Cichocki et al., 11 Mar 2025) develops analogous constructions for Tsallis, Kaniadakis, Sharma–Taneja–Mittal, Tempesta, Kaniadakis–Scarfone, and Euler/Borges–Roditi entropies. The common principle is to modify the Bregman geometry so that the multiplicative step better matches heavy tails, sparsity, or problem-specific curvature (Cichocki, 21 Feb 2025, Cichocki et al., 11 Mar 2025).

The literature also makes clear that EGD is not uniformly superior across all regimes. In $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$04-divergence minimization, entropic mirror descent can fail as dimension grows, whereas Power Descent is reported to remain robust (Daudel et al., 2020). In the LLM attack setting, the convergence theorem excludes ReLU-based models and does not extend to the Adam-accelerated EGD used in practice (Biswas et al., 14 May 2025). In the information-geometric analysis on $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$05, the method requires strictly positive initialization and the paper does not provide explicit complexity or convergence-rate bounds (Elshiaty et al., 7 Apr 2025). In the homogenization view of Entropy-SGD, the assumptions of globally Lipschitz $$\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,$$06 and strong dissipativity may be restrictive for practical deep networks (Bardi et al., 2022).

Taken together, these results suggest a precise contemporary picture. EGD is not a single algorithm but a family of entropic descent mechanisms. In one branch, it is mirror descent with negative entropy, producing exponentiated updates that respect simplex or positivity constraints by construction. In another, it is descent on a local-entropy smoothing of the loss, implemented through a two-scale stochastic process that biases training toward wide flat minima. Around these two cores, current work extends the method to measures, positive-orthant information geometry, adversarial token optimization, semi-discrete optimal transport, and generalized trace-form entropies (Zouaoui et al., 2022).