---
title: 'FDGM-AA: Fenchel Dual Gradient with Anderson Acceleration'
url: https://www.emergentmind.com/topics/fenchel-dual-gradient-method-with-anderson-acceleration-fdgm-aa
type: topic
---

# FDGM-AA: Fenchel Dual Gradient with Anderson Acceleration

The Fenchel Dual Gradient Method with Anderson Acceleration (FDGM-AA) is an optimization approach developed to solve distributed constrained optimization problems over time-varying networks. FDGM-AA integrates Anderson Acceleration (AA), originally designed for fixed-point iteration acceleration, into the Fenchel dual gradient paradigm by embedding local, edge-wise AA steps within the standard distributed gradient method, supplemented with a safeguard mechanism to ensure convergence. This formulation is particularly targeted at consensus problems with local constraints, where the agents' communication topology varies over time [2601.12398].

## 1. Problem Formulation and Fenchel Duality

The optimization setting is a distributed consensus problem over a network of $n$ agents:
\[
\min_{x_1,\dots,x_n\in\R^d} \sum_{i=1}^n f_i(x_i) \qquad \text{s.t.} \quad x_1 = x_2 = \ldots = x_n
\]
where each $f_i: \R^d \to \R \cup \{+\infty\}$ is $\mu$-strongly convex, possibly non-smooth (e.g., indicator functions for constraint sets). Under standard conditions (nonempty intersection of $\operatorname{dom} f_i$), strong duality holds.

The Fenchel conjugate $d_i$ is defined as:
\[
d_i(w_i) := \max_{x_i \in \R^d} \left\{ w_i^\top x_i - f_i(x_i) \right\}
\]
and the Fenchel dual becomes
\[
\min_{w_1,\ldots,w_n \in \R^d} \sum_{i=1}^n d_i(w_i) \qquad \text{s.t.} \quad \sum_{i=1}^n w_i = 0
\]
Each $d_i$ is differentiable and $L$-smooth with $L = 1/\mu$; the gradient mapping is $\nabla d_i(w_i) = \arg\max_{x_i}\{ w_i^\top x_i - f_i(x_i) \}$, and one recovers the primal optimum by $x^*_i = \nabla d_i(w^*_i)$ [2601.12398].

## 2. Standard Fenchel Dual Gradient Method (FDGM)

FDGM, adapted to time-varying undirected network graphs $G^k=(\mathcal{V},\mathcal{E}^k)$ at each iteration $k$, computes the dual update as:
\[
\mathbf{w}^{k+1} = \mathbf{w}^k - \beta (H_{G^k} \otimes I_d) \nabla D(\mathbf{w}^k)
\]
where $H_{G^k}$ is the weighted network Laplacian, and $\mathbf{w}$ is the concatenation of all dual variables. In componentwise form:
\[
w_i^{k+1} = w_i^k - \beta \sum_{j \in \mathcal{N}_i^k} h_{ij}^k \left( \nabla d_i(w_i^k) - \nabla d_j(w_j^k) \right)
\]
Provided $\beta \in (0, 1/L)$ and with $B$-connectivity (the union of network graphs over any $B$ consecutive steps is connected), FDGM achieves exact consensus with dual error $O(1/k)$ and primal error $O(1/\sqrt{k})$ rates [2601.12398].

## 3. Reformulation as Local Edge Subproblems

FDGM's global update can be understood as a collection of local two-node Fenchel dual subproblems. For each edge $\{i,j\}$, intermediate "gossip-variables" are introduced:
\[
\begin{aligned}
w_{ij}^{k+1/2} &= w_i^k - \beta \left( \nabla d_i(w_i^k) - \nabla d_j(w_j^k) \right) \\
w_{ji}^{k+1/2} &= w_j^k - \beta \left( \nabla d_j(w_j^k) - \nabla d_i(w_i^k) \right)
\end{aligned}
\]
The aggregation
\[
w_i^{k+1} = \left(1-\sum_{j \in \mathcal{N}_i^k} h_{ij}^k\right)w_i^k + \sum_{j \in \mathcal{N}_i^k} h_{ij}^k w_{ij}^{k+1/2}
\]
corresponds to performing a single projected gradient step for each two-node subproblem:
\[
\min_{w_{ij}, w_{ji}} \quad d_i(w_{ij}) + d_j(w_{ji}) \quad \text{s.t. } w_{ij} + w_{ji} = w_i^k + w_j^k
\]
In standard FDGM, these subproblems are solved inexactly with one gradient step; the FDGM-AA approach introduces acceleration at this local level [2601.12398].

## 4. Anderson Acceleration in Distributed Local Updates

Anderson Acceleration (AA) is integrated into FDGM by embedding edge-wise AA within each two-node subproblem. The procedure consists of:

**a) Dual-gradient linearization**: Each $(i,j)$ edge maintains a history of past iterates and gradients. Form affine combinations using coefficients $\alpha_{ij}^{t,k}$ over recent history, both to build a candidate next iterate and to approximate the gradient.

**b) Coefficient determination via approximate KKT**: The coefficients $\alpha_{ij}^k, \alpha_{ji}^k$ are determined by minimizing the mismatch between approximate gradients $\Vert D_{ij}^k \alpha_{ij}^k - D_{ji}^k \alpha_{ji}^k \Vert^2$, under the constraints that enforce consistency and normalization.

**c) Anderson-type half-step update**: The extrapolated local iterates $(\bar w_{ij}^{k+1/2}, \bar w_{ji}^{k+1/2})$ are constructed using the AA linearization and scaled differences of past gradients:
\[
\bar w_{ij}^{k+1/2} = \tilde w_{ij}^{k+1/2} - \beta (D_{ij}^k\,\alpha_{ij}^k - D_{ji}^k\,\alpha_{ji}^k)
\]
and analogously for $\bar w_{ji}^{k+1/2}$.

**d) Safe-guard/fallback mechanism**: Since $d_i$ is typically only smooth (not affine), AA steps need not ensure descent. A sufficient-descent safeguard is imposed: accept $(\bar w_{ij}^{k+1/2}, \bar w_{ji}^{k+1/2})$ only if a specific decrease in the sum $d_i + d_j$ is verified, otherwise revert to the standard gradient step. This guarantees monotonicity in the global dual objective [2601.12398].

The local AA-enabled solutions are then re-aggregated as in standard FDGM, thus preserving distributedness at every step.

## 5. Convergence Properties

Global convergence of FDGM-AA is established under the following assumptions: each $f_i$ is $\mu$-strongly convex; the graphs $\{G^k\}$ ensure $B$-connectivity; step size $\beta \in (0,1/L)$; and edge weights satisfy $\sum_j h_{ij}^k \le 1$.

The safeguard ensures that at each edge, the local dual objective decreases by at least a constant times the squared norm of gradient differences. Summing over all edges and aggregating across $B$ network steps, one obtains a contraction in the global dual gap:
\[
D(\mathbf{w}^k) - D^* = O(1/k)
\]
for the dual objective, and
\[
\|\mathbf{x}^k - \mathbf{x}^*\| = O(1/\sqrt{k})
\]
for the primal variable convergence. The method yields $O(1/\sqrt{k})$ convergence for the primal sequence and $O(1/k)$ for the dual, matching the best-known rates for distributed first-order methods in this regime [2601.12398].

## 6. Empirical Performance and Comparative Evaluation

FDGM-AA was tested on distributed $\ell_2$-regularized logistic regression problems with ball constraints, set over $n=30$ agents, each holding $M=20$ samples in $\R^{20}$, and local variable constraints. The methods compared include FDGM-AA, vanilla FDGM, distributed projected subgradient, and proximal minimization. The metric is average squared primal error $\frac{1}{n}\sum_{i} \|x_i^k - x^*\|^2$ as a function of iteration.

FDGM-AA achieves consistently superior performance to all benchmarks. The acceleration compared to vanilla FDGM increases as the number of iterations and the AA history length $m$ are raised. The algorithm demonstrates robustness to longer network periods $B$ and to weaker regularization $\lambda$ [2601.12398].

## 7. Summary and Theoretical Significance

FDGM-AA constitutes an advance in the distributed optimization of convex, possibly non-smooth, functions over time-varying networks by merging Anderson-type accelerated extrapolation into every two-node dual subproblem, regulated by a simple yet effective safeguard mechanism. This hybrid preserves distributed implementability and maintains global $O(1/k)$ dual and $O(1/\sqrt{k})$ primal convergence rates, while yielding significant speedup and robustness in empirical scenarios involving time-varying topologies and constrained optimization [2601.12398].

Source: https://www.emergentmind.com/topics/fenchel-dual-gradient-method-with-anderson-acceleration-fdgm-aa