Entropic Proximal Method

• Entropic proximal methods are algorithms that use entropy-driven Bregman divergences to enforce feasibility and ensure convergence in structured convex optimization.
• They enable closed-form multiplicative updates in tasks such as image segmentation, optimal transport, and linear programming within probability and nonnegative frameworks.
• Leveraging strong convergence guarantees and efficient parallel computations, these methods reduce memory use and computational cost in large-scale, high-dimensional problems.

The entropic proximal method refers to a class of algorithms rooted in Bregman proximal point techniques with entropy (typically Boltzmann–Shannon or Kullback–Leibler–type) divergence as the regularization or geometry-inducing term. These methods have become central for efficiently solving large-scale, structured optimization and inference tasks in convex and variational settings, notably where solution domains are probability simplices or nonnegative cones. The approach leverages the properties of entropic regularizers to obtain computationally tractable projections, closed-form multiplicative updates, automatic domain invariance, and strong convergence, particularly suited for high-dimensional, memory-constrained, and GPU-accelerated environments.

1. Mathematical Foundations and Proximal Operators

The core principle is the use of a Bregman proximal step where the Bregman divergence is generated by a Legendre function associated with entropy. On the positive orthant, the Boltzmann–Shannon entropy $\phi(x) = \sum_{i} x_i \ln x_i$ yields the Kullback–Leibler divergence: $$D_\phi(x, y) = \sum_{i} \big( x_i \ln \frac{x_i}{y_i} - x_i + y_i \big)$$ This divergence replaces the traditional squared Euclidean distance in proximal algorithms, fundamentally changing the update geometry. The entropic proximal operator for a convex function $F$ and step $c$ is defined as

$$\operatorname{prox}_{c D_g}(u) = \arg\min_{v \in \mathcal{C}}\left[ F(v) + c D_g(v, u) \right]$$

where $\mathcal{C}$ represents the constraint set—for example, a probability simplex.

In problems such as convex-relaxed multi-label segmentation (Potts model), the entropic-proximal update for label variables $u$ admits a closed-form multiplicative rescaling followed by normalization: $$u_L^{k+1}(x) = \frac{u_L^k(x) \exp\big( - (D_L(x) + \operatorname{div} q_L^k(x))/c \big)}{\sum_M u_M^k(x)\exp\big( - (D_M(x) + \operatorname{div} q_M^k(x))/c \big)}$$ This directly enforces simplex constraints $u_L \ge 0, \sum_L u_L = 1$ at every iteration and obviates explicit simplex projection (Baxter et al., 2015).

2. Algorithmic Frameworks and Update Derivations

Entropic proximal methods are derived in multiple problem domains by regularizing classical variational energies or KKT conditions with an entropic Bregman term. In continuous max-flow problems, the non-smooth “pseudo-flow” energy

$$E_{\text{pseudo}}(q) = \int_\Omega \min_L \left[ D_L(x) + \operatorname{div} q_L(x) \right]\mathrm{d}x$$

is regularized via adding $c D_g(u, v^k)$, yielding strongly convex subproblems in $u$. The variation w.r.t. $u$ gives the multiplicative update above, and the variation w.r.t. dual fluxes $q_L$ gives (projected) gradient ascent steps. The entire algorithm alternates entropic-multiplicative “primal” steps with “dual” projection steps, and convergence is inherited from Bregman-proximal and (generalized) gradient ascent theory (Baxter et al., 2015).

In the context of Kullback–proximal generalizations of the EM algorithm, entropic proximity is used to penalize deviation from the previous parameter iterate (in expected complete-data KL divergence): $$\theta^{k+1} = \arg\max_\theta \left( l_y(\theta) - \beta_k I_y(\theta, \theta^k) \right)$$ where $I_y(\theta, \theta^k)$ is the conditional KL divergence under the current and previous parameters, and $\beta_k$ is a relaxation parameter controlling the effective step-size and regularization (Chrétien et al., 2012).

Other domains—linear optimization with entropic constraints (Briceño-Arias et al., 12 Jun 2025), entropy–energy variational interpolation (Bauschke et al., 2018), and large-scale linear programming with multi-marginal structure (Chu et al., 2020)—derive analogous update rules, always exploiting the closed-form or efficiently computable structure provided by the entropic divergence.

3. Convergence, Complexity, and Regularizing Effects

Entropic proximal point methods inherit strong global convergence properties from the convexity of the underlying regularizer and the strict feasibility of the iterates. Each outer iteration solves a strongly convex (in the primal variable) and smooth (in dual variables, where applicable) subproblem. The theory of Bregman proximal methods ensures monotonicity of the objective, existence and uniqueness of cluster points, and—under additional assumptions—global or ergodic rates, typically $O(1/n)$ in objective value for linear or entropy-regularized problems (Briceño-Arias et al., 12 Jun 2025, Chrétien et al., 2012).

The entropic regularizer provides both smoothing (making non-smooth min/max or LP energies differentiable) and domain invariance (all iterates remain positive and feasible with respect to simplex or conic constraints). Memory requirements per iteration are significantly reduced due to the implicit variable representations enabled by the entropic pseudo-flow formulation—for example, in continuous max-flow, memory is cut by $\sim 20\%-30\%$ depending on model structure (Baxter et al., 2015).

The per-iteration computational cost is dominated by vector-wise (or tensor-wise) exponentials, multiplications, and elementwise normalizations, all of which are efficient on modern parallel architectures (GPUs or multicore CPUs), and scale as $O(NK)$ for $N$ points and $K$ labels.

4. Practical Implementations and Applications

The entropic proximal method is deployed in a wide range of optimization and inference settings:

• Large-scale image and volume segmentation: In continuous max-flow models, the entropic Bregman-proximal pseudo-flow method allows very large instances to be solved efficiently on commodity GPUs, thanks to reduced memory, pointwise update structure, and lack of global synchronization (Baxter et al., 2015).
• Structured LPs in optimal transport and tomography: In the iEPPA framework, very large LPs with block constraints and multi-marginal structure are efficiently handled by dual block coordinate descent on the entropic-proximal subproblem, dramatically outperforming classical simplex, interior-point, and Sinkhorn-style regularization for moderate to high accuracy targets (Chu et al., 2020).
• Probability-constrained and entropic metric-constrained convex programs: For linear objectives with entropic (KL-type) constraints, as arise in game theory and information theory, the Bregman-proximal gradient iterates (and their fixed-point realization) provide fast, provably convergent solvers that recover classical Blahut–Arimoto-type algorithms as special cases (Briceño-Arias et al., 12 Jun 2025).

In all cases, the entropic divergence serves as a “soft barrier,” ensuring feasibility (non-negativity, summation constraints) without explicit projection. Early stopping yields feasible approximations at any stage, advantageous in real-time or massively parallel contexts.

5. Theoretical and Algorithmic Connections

The entropic proximal method synthesizes multiple traditions:

• Bregman Proximal Point Algorithms: The replacement of the classical squared Euclidean distance with an entropy-like Bregman divergence is foundational (Baxter et al., 2015, Chrétien et al., 2012).
• EM and Kullback–Proximal Framework: The EM algorithm is a special case with unit KL penalty (β=1), and Kullback–proximal iterations generalize to arbitrary β, obtaining both acceleration and constraint enforcement effects (Chrétien et al., 2012).
• Entropy–Energy Interpolation: The proximal averaging of entropy with quadratic energy functionals produces a one-parameter homotopy between hard barrier and $L^2$ penalization regimes, enabling differentiable models that interpolate between regularization philosophies (Bauschke et al., 2018).
• Sinkhorn, Dykstra, Bregman Iterative Scaling: In transport and matching problems, the entropic proximal method connects, but is not identical, to classical entropic regularization (Sinkhorn, Dykstra–KL): it allows larger proximal parameters, bypassing issues of vanishing step-size, numerical underflow, and poor conditioning while preserving rapid convergence and warm-start capability (Chu et al., 2020).

6. Implementation Considerations and Limitations

While entropic proximal methods offer computational and theoretical advantages, several technical challenges must be addressed:

• Highly anisotropic or nearly singular inputs can cause numerical instabilities (overflow, underflow) in multiplicative update and normalization steps, especially when the regularization parameter approaches degenerate limits; this is handled by high-precision arithmetic, symbolic simplification, and homotopy continuation for sensitive cases (Bauschke et al., 2018).
• Proximal averages with entropy produce analytic expressions involving the non-elementary Lambert $W$ function, affecting implementation practicality for certain classes of interpolation (Bauschke et al., 2018).
• In block-structured LPs, classical stopping conditions may be infeasible to verify; instead, approximations using primal residuals, Bregman gaps, and feasiblized "pull-back" mappings provide numerically stable stopping criteria (Chu et al., 2020).

When applied with appropriate scaling and stabilization techniques, the entropic proximal method enables high-accuracy solutions to problems with $10^8+$ variables and constraints within practical time and memory budgets on contemporary hardware (Chu et al., 2020).

---

References:

• "A Proximal Bregman Projection Approach to Continuous Max‐Flow Problems Using Entropic Distances" (Baxter et al., 2015)
• "Bregman proximal gradient method for linear optimization under entropic constraints" (Briceño-Arias et al., 12 Jun 2025)
• "On EM algorithms and their proximal generalizations" (Chrétien et al., 2012)
• "Proximal Averages for Minimization of Entropy Functionals" (Bauschke et al., 2018)
• "An efficient implementable inexact entropic proximal point algorithm for a class of linear programming problems" (Chu et al., 2020)