---
title: Entropic Gradient Descent (EGD)
url: https://www.emergentmind.com/topics/entropic-gradient-descent-egd
type: topic
---

# Entropic Gradient Descent (EGD)

Entropic Gradient Descent (EGD) denotes a family of first-order optimization methods in which descent is performed in the geometry induced by negative entropy rather than the Euclidean metric. In its standard form, EGD is mirror descent with the negative-entropy mirror map, so the associated Bregman divergence is the Kullback–Leibler divergence and the resulting update is multiplicative or exponentiated rather than additive [2209.11002]. In a distinct but related deep-learning usage, EGD also refers to Entropy-SGD, where the algorithm descends not on the raw loss but on a local-entropy smoothing of that loss, thereby favoring wide, flat minima [2209.05564].

## 1. Terminological scope and core meaning

In the optimization literature represented here, EGD is most commonly synonymous with **entropic mirror descent**, **entropic gradient**, **exponentiated gradient**, and **multiplicative weights**. The underlying construction uses the negative entropy
$$
h(x)=\sum_{i=1}^d x_i \ln x_i
$$
on a simplex or positive orthant, and the mirror step replaces Euclidean projection by a KL-proximal update [2209.11002]. In this sense, EGD is a geometric method: the “entropic” aspect lies in the mirror map and the induced Bregman geometry.

A recurrent source of confusion is that some deep-learning papers use “Entropic Gradient Descent” to denote **Entropy-SGD**, which instead modifies the objective by introducing a local-entropy functional
$$
\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\big(-\beta \phi(y)-\tfrac{\beta}{2\gamma}\|y-x\|^2\big)\,dy,
$$
and then performs descent on \(\phi_\gamma\) rather than on \(\phi\) itself [2209.05564]. The two usages are related by their entropic character, but they act at different levels: one changes the optimization geometry, the other smooths the objective.

The same exponentiated mechanism also appears in measure-valued optimization. In \((\alpha,\Gamma)\)-descent, choosing \(\Gamma_\alpha(v)=e^{-\eta v}\) recovers infinite-dimensional Entropic Mirror Descent, with updates of the form
$$
d\mu_{t+1}(\theta)\propto d\mu_t(\theta)\,\exp\{-\eta B_\alpha^{\mu_t}(\theta)\},
$$
and, in the discrete case, a normalized multiplicative update on mixture weights [2005.10618]. This suggests that EGD is best understood as a broad entropic descent principle instantiated on simplices, positive orthants, matrices of row-wise distributions, or spaces of probability measures.

## 2. Bregman and information-geometric foundations

A general formulation is given by the generalized Bregman framework, where the iteration
$$
u^{k+1}=\arg\min_{u\in\operatorname{dom}(J)}\left\{\tau^k\langle u-u^k,\nabla E(u^k)\rangle + D_J^{p^k}(u,u^k)\right\},
$$
with \(p^k\in \partial J(u^k)\), reduces to mirror descent when \(J\) is smooth [1612.02506]. Specializing \(J\) to the negative entropy \(\Phi(x)=\sum_i x_i\log x_i\) yields entropic mirror descent, for which
$$
D_\Phi(x,y)=\sum_{i=1}^n\Big[x_i\log\frac{x_i}{y_i}-(x_i-y_i)\Big].
$$
On the simplex, this coincides with the KL divergence [1612.02506].

For a differentiable objective \(f\) on the simplex \(\Delta_d\), the mirror step has the primal form
$$
\alpha^{k+1}
= \arg\min_{\alpha \in \Delta_d}
\left\{ \nabla f(\alpha^k)^\top (\alpha-\alpha^k)+\frac{1}{\eta_k}D_h(\alpha,\alpha^k)\right\},
$$
and solving the subproblem gives the normalized exponentiated update
$$
\alpha^{k+1}_j=
\frac{\alpha^k_j\exp(-\eta_k \nabla f(\alpha^k)_j)}
{\sum_{\ell=1}^d \alpha^k_\ell \exp(-\eta_k \nabla f(\alpha^k)_\ell)}.
$$
Equivalently,
$$
\alpha^{k+1}=\operatorname{softmax}\!\left(\log(\alpha^k)-\eta_k\nabla f(\alpha^k)\right),
$$
which preserves nonnegativity and the sum-to-one constraint automatically [2209.11002].

An information-geometric interpretation is available on the positive orthant \(\mathbb{R}^n_{++}\). With
$$
\psi(x)=\sum_{i=1}^n(x_i\log x_i-x_i),
$$
the induced Fisher–Rao metric is
$$
g(x)=\nabla^2\psi(x)=\operatorname{Diag}(1/x_1,\dots,1/x_n),
$$
and the Riemannian gradient is
$$
\operatorname{grad} f(x)=x\odot \nabla f(x).
$$
Using the \(e\)-Exp retraction
$$
\operatorname{Exp}^e_x(v)=x\odot \exp(v/x),
$$
Riemannian gradient descent becomes
$$
x_{k+1}=x_k\odot \exp(-\eta_k \nabla f(x_k)),
$$
which is exactly exponentiated gradient on \(\mathbb{R}^n_{++}\) [2504.05136]. In this reading, EGD is simultaneously mirror descent, natural gradient descent under \(g(x)=\operatorname{Diag}(1/x)\), and multiplicative descent in primal coordinates.

## 3. Canonical update rules and constraint handling

The central computational advantage of EGD is that simplex or positivity constraints are enforced by the update itself. In simplex-constrained problems, the softmax or normalized exponentiated form ensures that iterates remain feasible, thereby avoiding Euclidean projections, active-set logic, or sorting-based routines [2209.11002]. In the positive orthant, multiplicative updates preserve strict positivity as long as the initialization is strictly positive [2504.05136].

This property is especially important in blind hyperspectral unmixing via archetypal analysis. In "Entropic Descent Archetypal Analysis for Blind Hyperspectral Unmixing" [2209.11002], the data matrix \(Y\in\mathbb{R}^{L\times N}\) is modeled as
$$
Y=EA+N,\qquad E=YB,
$$
with abundance columns \(A_{(:,i)}\in \Delta_p\) and archetype-coefficient columns \(B_{(:,j)}\in \Delta_N\). The blind AA objective
$$
\min_{A,B}\;\frac12\|Y-YBA\|_F^2
$$
is convex in \(A\) for fixed \(B\) and convex in \(B\) for fixed \(A\), and the algorithm alternates entropic updates on the two blocks. The update for \(A\) is
$$
A^{k+1}=\operatorname{softmax}\!\left(\log(A^k)+\eta_1 E^\top(Y-EA^k)\right),
$$
and the update for \(B\) is
$$
B^{k+1}=\operatorname{softmax}\!\left(\log(B^k)+\eta_2 Y^\top(Y-YB^kA)A^\top\right),
$$
with column-wise softmax enforcing the simplex constraints [2209.11002]. Each inner update costs \(O(LNp)\), and with \(T\) outer iterations and \(K_1+K_2\) inner steps the total cost is \(O(T(K_1+K_2)LNp)\). The paper uses \(T=100\), \(K_1=K_2=5\), reports that this is fast on a single GPU, and recommends \(M\ge 50\) runs in the ensembling stage [2209.11002].

A closely related matrix-simplex construction appears in adversarial optimization for large language models. There the relaxed one-hot matrix \(\tilde X\in [0,1]^{L\times |\mathbb{T}|}\) has rows constrained to \(\Delta^{|\mathbb{T}|}\), and the update is
$$
\tilde X_t = P_{KL}\!\big(\tilde X_{t-1}\odot \exp(-\eta \nabla F(\tilde X_{t-1}))\big),
$$
where the KL/Bregman projection \(P_{KL}\) is simply row normalization [2505.09820]. The paper emphasizes that this is an intrinsic optimization over token distributions rather than an optimization in embedding space followed by an extrinsic projection to tokens.

## 4. Convergence theory

Within the generalized Bregman framework, entropic mirror descent inherits sufficient-decrease and global-convergence results under structural assumptions on the objective. For linearized Bregman iteration, if there exists a proper, lower semi-continuous, convex \(F\) such that \(G:=F-E\) is convex, the iteration satisfies a sufficient decrease property; under additional strong-convexity conditions on \(J\) and \(J^*\), one obtains a gradient bound; and under a generalized Kurdyka–Łojasiewicz property, strongly convergent subsequences exist, while in finite dimensions the full sequence converges [1612.02506]. Specializing \(J\) to the negative entropy yields the corresponding EGD statements.

The measure-valued \((\alpha,\Gamma)\)-framework gives more explicit performance guarantees for the entropic choice \(\Gamma_\alpha(v)=e^{-\eta v}\). Under Hypotheses A-1 and A-2, the exact update satisfies
$$
\Psi_\alpha(T_{\alpha,\Gamma}(\mu))\le \Psi_\alpha(\mu),
$$
with equality iff \(\mu\) is a fixed point. Under additional smoothness and boundedness conditions, the paper proves an \(O(1/N)\) rate in the exact setting and an \(O(1/\sqrt{N})\) stochastic bound for averaged iterates [2005.10618]. The same paper also identifies a limitation: in high dimension, entropic mirror descent can under-discriminate when the values of \(B_\alpha^{\mu_t}\) cluster tightly, and support mismatch for \(\alpha\ge 1\) can lead to divergence; this motivates the alternative Power Descent transform [2005.10618].

On the positive orthant, a different route to convergence is developed through information geometry. Interpreting EG as Riemannian gradient descent with the \(e\)-Exp retraction, "Information Geometry of Exponentiated Gradient: Convergence beyond L-Smoothness" [2504.05136] proves finite termination of Riemannian Armijo line search and global convergence under smoothness of \(f\), without assuming \(L\)-smoothness or relative smoothness. Every accumulation point is a critical point, and if \(f\) is convex the iterates converge to a global minimizer [2504.05136].

For the row-simplex LLM attack formulation, the update with KL projection is shown to converge to a critical point when \(F\) is differentiable with Lipschitz continuous gradient and the learning rates are sufficiently small [2505.09820]. The theorem is explicitly limited to the EG-with-projection iteration; the paper states that it does not cover ReLU-based architectures or the Adam-accelerated EGD variant [2505.09820].

## 5. Representative application domains

Blind hyperspectral unmixing provides one of the clearest large-scale demonstrations of EGD as a practical optimizer. In EDAA, endmembers are expressed as convex combinations of pixels already present in the image, so both abundances and archetype coefficients lie on simplices. Across six real datasets—Samson, Jasper Ridge, Urban4/6, APEX, and WDC—the method achieves the best overall abundance RMSE, with SAD results that are competitive and frequently second-best or best; the paper further reports processing times of approximately \(1.5\) s per Urban6 run and approximately \(75\) s total for \(50\) runs and model selection [2209.11002]. The same study attributes part of this performance to the fact that EGD is projection-free, GPU-friendly, and fast enough to support ensembling and coherence-based model selection.

In variational inference over measures, EGD appears as an update on mixture weights or on the measure itself. For discrete support \(\Theta=\{\theta_1,\dots,\theta_J\}\), the update
$$
\lambda_{j,t+1}\propto \lambda_{j,t}\exp\{-\eta B_\alpha^{\mu_t}(\theta_j)\}
$$
optimizes an \(\alpha\)-divergence objective over the simplex of mixture coefficients [2005.10618]. The stochastic formulation is notable because it updates the weights of a mixture model using only samples \(Y\sim \mu_t k\), without requiring knowledge of the distribution of the variational parameters. At the same time, the paper reports that in toy Gaussian-mixture experiments the \(0.5\)-Mirror method fails to learn in dimensions \(16\) and \(32\), while \(0.5\)-Power remains stable, and that on Bayesian logistic regression for Covertype the \(0.5\)-Power method improves predictive likelihood and accuracy relative to an adaptive importance-sampling baseline [2005.10618].

Entropic-gradient ideas also appear in semi-discrete optimal transport, although here the “entropic” component lies in the objective rather than the mirror geometry. DRAG performs stochastic gradient descent on the entropically smoothed semi-dual
$$
H_\epsilon(g)=\mathbb{E}_{X\sim \mu}[h_\epsilon(X,g)],
$$
with single-sample gradient
$$
\frac{\partial h_\epsilon(X,g)}{\partial g_j}=-w_j+\chi_j^\epsilon(X,g),
$$
and uses a decreasing regularization schedule \(\epsilon_t=t^{-a}\) together with Polyak–Ruppert averaging [2510.27340]. The theoretical result is an unbiased \(O(1/t)\) sample and iteration complexity for OT cost and potential estimation, and an \(O(1/\sqrt{t})\) rate for the OT map [2510.27340]. This is not mirror-descent EGD in the strict sense, but it is an entropic-gradient method in the sense of optimizing an entropically smoothed objective.

A further application is adversarial optimization for language models. By optimizing directly over distributions on vocabulary tokens, the exponentiated update remains intrinsic to the simplex of one-hot relaxations. The paper evaluates five open-source LLMs on four openly available datasets and reports that EGD achieves higher attack success rates with greater efficiency than three other state-of-the-art jailbreaking techniques [2505.09820].

## 6. Local-entropy EGD in deep learning

In the deep-learning line of work, EGD denotes descent on a **local-entropy** objective rather than mirror descent on a simplex. Given a base loss \(\phi\), the local entropy at anchor \(x\) is
$$
\phi_\gamma(x)= -\frac{1}{\beta}\log \int_{\mathbb{R}^n}\exp\Big(-\beta \phi(y)-\frac{\beta}{2\gamma}\|y-x\|^2\Big)\,dy,
$$
and its gradient is
$$
\nabla \phi_\gamma(x)=\int_{\mathbb{R}^n}\frac{x-y}{\gamma}\,\rho^\infty(dy;x),
$$
where \(\rho^\infty(dy;x)\propto \exp(-\beta \Phi(y,x))\,dy\) is the invariant Gibbs measure of the inner Langevin dynamics with \(x\) frozen [2209.05564]. The practical algorithm samples an auxiliary variable \(Y\) with a short inner Langevin loop and updates the slow variable in the direction of \(x-\mathbb{E}_{\rho^\infty}[Y]\), thereby approximating \(-\nabla \phi_\gamma(x)\) [2209.05564].

The control-theoretic homogenization analysis embeds the Chaudhari et al. two-scale stochastic system into a broader singularly perturbed control problem and shows that, under assumptions including Lipschitz \(\nabla \phi\) and \(\gamma<1/L\), the uncontrolled entropic system converges to the effective ODE
$$
d\hat X_s=-\nabla \phi_\gamma(\hat X_s)\,ds,
$$
while the corresponding value functions converge to those of an effective control problem with averaged coefficients [2209.05564]. This places Entropy-SGD on a rigorous averaging foundation and interprets tuning parameters such as the learning-rate control \(u_s\), anchoring strength \(\gamma\), inverse temperature \(\beta\), and noise as controls in the homogenized system.

Empirically, Entropy-SGD and Replicated-SGD are reported to “consistently improve the generalization error for common architectures (e.g. ResNet, EfficientNet)” [2006.07897]. The same paper uses local entropy and local energy as flatness diagnostics and states that an easy-to-compute flatness measure shows a clear correlation with test accuracy [2006.07897]. The theoretical and empirical accounts therefore converge on the same interpretation: local-entropy EGD biases optimization toward wide flat minima rather than merely accelerating constrained first-order descent.

## 7. Generalizations, limitations, and research directions

Several recent works generalize classical EGD by replacing the Shannon mirror map with other trace-form entropies and deformed logarithms. "Generalized Exponentiated Gradient Algorithms Using the Euler Two-Parameter Logarithm" [2502.17500] defines
$$
\log^E_{a,b}(x)=\frac{x^a-x^b}{a-b},
$$
with inverse \(\exp^E_{a,b}\), and derives the generalized mirror update
$$
x_{t+1}=\exp^E_{a,b}\!\big(\log^E_{a,b}(x_t)-\eta_t g_t\big).
$$
"Mirror Descent and Novel Exponentiated Gradient Algorithms Using Trace-Form Entropies and Deformed Logarithms" [2503.08748] develops analogous constructions for Tsallis, Kaniadakis, Sharma–Taneja–Mittal, Tempesta, Kaniadakis–Scarfone, and Euler/Borges–Roditi entropies. The common principle is to modify the Bregman geometry so that the multiplicative step better matches heavy tails, sparsity, or problem-specific curvature [2502.17500, 2503.08748].

The literature also makes clear that EGD is not uniformly superior across all regimes. In \(\alpha\)-divergence minimization, entropic mirror descent can fail as dimension grows, whereas Power Descent is reported to remain robust [2005.10618]. In the LLM attack setting, the convergence theorem excludes ReLU-based models and does not extend to the Adam-accelerated EGD used in practice [2505.09820]. In the information-geometric analysis on \(\mathbb{R}^n_{++}\), the method requires strictly positive initialization and the paper does not provide explicit complexity or convergence-rate bounds [2504.05136]. In the homogenization view of Entropy-SGD, the assumptions of globally Lipschitz \(\nabla \phi\) and strong dissipativity may be restrictive for practical deep networks [2209.05564].

Taken together, these results suggest a precise contemporary picture. EGD is not a single algorithm but a family of entropic descent mechanisms. In one branch, it is mirror descent with negative entropy, producing exponentiated updates that respect simplex or positivity constraints by construction. In another, it is descent on a local-entropy smoothing of the loss, implemented through a two-scale stochastic process that biases training toward wide flat minima. Around these two cores, current work extends the method to measures, positive-orthant information geometry, adversarial token optimization, semi-discrete optimal transport, and generalized trace-form entropies [2209.11002].

Source: https://www.emergentmind.com/topics/entropic-gradient-descent-egd