---
title: Bregman Proximal Gradient Descent
url: https://www.emergentmind.com/topics/bregman-proximal-gradient-descent-bpgd
type: topic
---

# Bregman Proximal Gradient Descent

Bregman Proximal Gradient Descent (BPGD) is a first-order optimization algorithm for solving composite minimization problems that generalizes classical proximal gradient methods by replacing the Euclidean quadratic regularization with a general Bregman divergence. This extension enables efficient optimization for both convex and nonconvex objective functions, particularly when the differentiable part of the objective lacks global Lipschitz continuity. BPGD provides rigorous convergence guarantees under the "relative smoothness" condition, supports a wide class of geometries through the choice of Legendre kernel functions, and unifies multiple schemes—including mirror descent, proximal gradient, and NoLips methods—under a single framework [1711.01136], [1503.05601].

## 1. Mathematical Foundations: Bregman Distance, Relative Smoothness, and Problem Setting

Let $w : \mathbb{R}^n \rightarrow (-\infty, +\infty]$ denote a Legendre function—proper, closed, strictly convex, and differentiable on an open domain $\operatorname{int}\,\operatorname{dom}w \ne \emptyset$. The associated Bregman distance is
\[
D_w(x, y) = w(x) - w(y) - \langle \nabla w(y), x - y \rangle,
\]
which satisfies $D_w(x, y) \ge 0$ with equality iff $x = y$, but in general is asymmetric.

BPGD aims to solve composite problems of the form
\[
\min_{x \in Q} \Phi(x) := f(x) + g(x)
\]
with
- $Q = \operatorname{dom}w$, a closed convex set with nonempty interior,
- $f$ convex and continuously differentiable on $\operatorname{int}\,\operatorname{dom}w$,
- $g$ proper, closed, convex, and possibly nonsmooth, with nonempty intersection of its domain with $\operatorname{int}\,\operatorname{dom}w$.

**Relative smoothness** replaces the usual Lipschitz gradient requirement: $f$ is $L$-smooth relative to $w$ if
\[
f(y) \leq f(x) + \langle \nabla f(x), y - x \rangle + L D_w(y, x) \qquad \forall x, y \in \operatorname{int}\,\operatorname{dom}w
\]
[1711.01136], [1503.05601], [2202.02388].

## 2. Algorithmic Framework and Iteration

The canonical BPGD iteration is
\[
x_{k+1} = \arg\min_{x \in Q} \left\{ \langle \nabla f(x_k), x - x_k \rangle + \frac{1}{\alpha} D_w(x, x_k) + g(x) \right\},
\]
where the step-size $\alpha > 0$ must satisfy $\alpha \leq 1/L$ for sublinear or linear convergence, depending on additional assumptions. In operator-theoretic terms, this step is a **Bregman-proximal mapping**:
\[
x_{k+1} = \mathrm{prox}^{\,w}_{\alpha g}(x_k - \alpha \nabla f(x_k)).
\]
Special cases include:
- **Euclidean proximal gradient method**: $w(x) = \frac{1}{2}\|x\|^2$, recovering standard forward-backward splitting.
- **NoLips/mirror descent**: $g \equiv 0$ or $g$ as indicator of a convex set; the iteration reduces to Bregman mirror descent [1711.01136], [1503.05601], [2202.02388], [1712.04045].

Block-wise and two-reference variants, as in B2B methods [1912.07527], decouple the Bregman geometry to obtain closed-form projections in high-dimensional or structured problems.

## 3. Convergence Theory: Sublinear, Linear, and KL Rates

### Sublinear convergence
Under relative smoothness (no strong convexity, no growth condition), for step-size $\alpha \leq 1/L$,
\[
\Phi(x_k) - \Phi^* \leq \frac{1}{\alpha k} D_w(x^*, x_0).
\]
This generalizes the $O(1/k)$ rate of classical proximal gradient descent to the Bregman setting [1711.01136].

### Linear convergence (Bregman-distance growth condition)
If, in addition, the following Bregman growth condition holds:
\[
\Phi(x) - \Phi^* \geq \mu \cdot \inf_{z \in X} D_w(z, x),
\]
with $X = \{x : \Phi(x) = \Phi^*\}$, then (for $\alpha \leq 1/L$) the Lyapunov function
\[
\mathcal{E}(x) := \Phi(x) - \Phi^* + \mu \inf_{z \in X} D_w(z, x)
\]
contracts as
\[
\mathcal{E}(x_{k+1}) \leq \frac{1}{1 + \alpha \mu} \mathcal{E}(x_k),
\]
implying $R$-linear convergence in both function value and Bregman distance to the optimal set [1711.01136].

### Extension to nonconvex objectives and KL inequality
For general nonconvex settings, convergence to stationary points and finite-length property of the sequence are obtained under the Kurdyka–Łojasiewicz (KL) property for appropriate Lyapunov functions. Rates determined by the KL exponent yield sublinear or local linear convergence [1904.11295], [2012.13161], [1707.02278].

## 4. Variants and Extensions: Stochastic, Inertial, Block-wise, and Multilevel

- **Stochastic BPGD**: Replaces the gradient with a stochastic estimator and leverages variance reduction; under kernel-conditioning, achieves $O(\sqrt{n}\epsilon^{-1})$ sample complexity matching lower bounds for nonconvex finite-sum objectives [2401.03155], [2306.14522].
- **Inertial/Extrapolated BPGD**: Adds a Nesterov-type extrapolation step $y^k = x^k + \beta_k(x^k - x^{k-1})$ with adaptive $\beta_k$, provably accelerating convergence over non-inertial BPGD [1904.11295], [1904.04436].
- **Block-Coordinate and Two-Reference BPGD**: Alternates updates in blocks using Bregman divergences adapted per block, shown to accelerate and admit closed-form updates in matrix factorization and NMF [1912.07527], [1905.09050].
- **Multilevel BPGD**: For large-scale problems with hierarchical structure, the ML-BPGD framework recursively constructs lower-dimensional surrogates, achieves global linear rates under Bregman PL-type conditions [2506.03950].
- **Delay-tolerant Distributed BPGD**: Supports totally asynchronous parallelization and is robust to unbounded communication delays [2404.17190].

## 5. Applications and Empirical Performance

BPGD and its stochastic/inertial variants have demonstrated superior convergence in:

| Problem Class                 | Bregman Kernel     | Key Outcomes                                              |
|-------------------------------|--------------------|-----------------------------------------------------------|
| Poisson linear inverse        | Burg entropy       | Robust to non-Lipschitz $\nabla f$; closed-form updates   |
| Phase retrieval, robust stats | Quartic/polynomial | Outperforms generic methods in iteration/time             |
| Deep linear networks/NMF      | Structured kernel  | Avoids variable bias; closed-form updates; faster conv.   |
| Variational inference         | log-partition      | Mirror steps = moment projection; monotonic decrease      |
| Image reconstruction, games   | KL, Tsallis        | Superior scalability; matches classical mirror-descent    |

Plug-and-play priors and deep-unfolding frameworks exploiting BPGD (PnP-BPGM) outperform Euclidean analogues in Poisson image deblurring and other tasks [2202.02388]. BPGD admits step sizes and kernels tailored to domain constraints and fits well in structured or non-Euclidean spaces [2506.10849].

## 6. Practical Considerations: Choice of Bregman Kernel, Step-size, and Regularization

### Bregman kernel selection
- **Quadratic:** $w(x) = \frac{1}{2}\|x\|^2$, Euclidean geometry.
- **KL/entropy:** $w(x) = \sum x_i \log x_i$, for simplex or probability constraints.
- **Burg:** $w(x) = -\sum \log x_i$, for positive-orthant constraints.
- **Problem-specific polynomials:** To handle non-Lipschitz objectives, e.g., deep nets.

### Step-size control
- **Fixed step:** For known $L$, take $\alpha \le 1/L$.
- **Line-search / backtracking:** When $L$ unknown, backtrack to enforce local descent in Bregman divergence [1503.05601], [1712.04045].
- **Adaptive rules:** In block or stochastic variants, adaptive $\alpha_k$ maintains contraction [2401.03155].

### Regularization structure
BPGD supports nonsmooth convex $g$ (e.g., $\ell_1$, indicator functions), which is handled through proximal operators with respect to the Bregman kernel.

## 7. Connections to Existing Algorithms and Theoretical Unification

The BPGD framework subsumes:
- Classical proximal gradient (when $w$ is quadratic).
- Mirror descent/NoLips (when $g \equiv 0$, general $w$).
- Incremental and aggregated methods (as in the PLIAG scheme), distributed and block-coordinate algorithms [1711.01136], [2404.17190], [1912.07527].
- Proximal point and model-based minimization schemes, including the "MAP" property, which extends $L$-smooth-adaptivity to composite nonsmooth models [2012.13161].

The framework establishes convergence theory in regimes far beyond Euclidean settings, covering nonconvex, nonsmooth, and large-scale optimization, and aligns with modern needs in machine learning, signal processing, and variational inference.

---

**References:**
- [1711.01136] Proximal-Like Incremental Aggregated Gradient Method with Linear Convergence under Bregman Distance Growth Conditions
- [1503.05601] A Simple Convergence Analysis of Bregman Proximal Gradient Algorithm
- [2202.02388] Bregman Plug-and-Play Priors
- [1712.04045] Choose your path wisely: gradient descent in a Bregman distance framework
- [1912.07527] Leveraging Two Reference Functions in Block Bregman Proximal Gradient Descent for Non-convex and Non-Lipschitz Problems
- [1904.11295] Bregman Proximal Gradient Algorithm with Extrapolation for a class of Nonconvex Nonsmooth Minimization Problems
- [2012.13161] Global Convergence of Model Function Based Bregman Proximal Minimization Algorithms
- [1707.02278] Non-smooth Non-convex Bregman Minimization: Unification and new Algorithms
- [2404.17190] Delay-tolerant distributed Bregman proximal algorithms
- [2506.10849] Bregman proximal gradient method for linear optimization under entropic constraints
- [2401.03155] Stochastic Bregman Proximal Gradient Method Revisited: Kernel Conditioning and Painless Variance Reduction
- [2306.14522] Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning
- [1905.09050] Beyond Alternating Updates for Matrix Factorization with Inertial Bregman Proximal Gradient Algorithms
- [2506.03950] Multilevel Bregman Proximal Gradient Descent
- [2211.04776] Regularized Rényi divergence minimization through Bregman proximal gradient algorithms
- [1910.03638] Bregman Proximal Framework for Deep Linear Neural Networks

Source: https://www.emergentmind.com/topics/bregman-proximal-gradient-descent-bpgd