---
title: 'SPG-AHOC: Smoothed Proximal Gradient for DAG Recovery'
url: https://www.emergentmind.com/topics/smoothed-proximal-gradient-spg-ahoc
type: topic
---

# SPG-AHOC: Smoothed Proximal Gradient for DAG Recovery

Smoothed Proximal Gradient (SPG-AHOC) is an optimization framework that combines the machinery of the smoothed proximal gradient (SPG) method with new structural constraints, primarily for exact support recovery in high-dimensional, constrained estimation settings. Originally developed for structured sparse regression, SPG has recently been adapted to causal discovery with the SPG-AHOC algorithm, enabling finite-iteration support identification in linear structural equation models under acyclicity constraints [2601.18189]. The integration of smoothed regularization, non-trivial manifold identification, and constraint formulation achieves superior structure recovery compared to traditional continuous optimization approaches.

## 1. Fundamental Principles of Smoothed Proximal Gradient Methods

The SPG framework addresses optimization of composite objectives where a smooth data term is combined with a nonsmooth regularizer, such as in structured sparse regression. A distinctive feature is the use of a smoothing surrogate, typically via Nesterov’s technique, for the nonseparable and nonsmooth regularizer. The general form is:
\[
\min_{\beta} f(\beta) := g(\beta) + \Omega(\beta) + \lambda\|\beta\|_1
\]
where $g$ is smooth (e.g., squared loss) and $\Omega$ encodes structured sparsity (overlapping group lasso, graph-fused lasso, etc.). Smoothing replaces $\Omega$ with a differentiable approximation $\Omega_\mu$, yielding a regularized problem amenable to first-order methods with controlled bias:
\[
\Omega_\mu(\beta) = \max_{\alpha \in Q} \left[\alpha^\top C \beta - \mu d(\alpha)\right]
\]
with $\nabla \Omega_\mu(\beta) = C^\top \alpha^*$ and Lipschitz constant $L = L_g + L_\Omega$ [1005.4717, 1202.3708].

The use of acceleration strategies, e.g., FISTA-type extrapolation, further boosts convergence rates to $O(1/\varepsilon)$ for objective value error, surpassing classical subgradient schemes [1202.3708, 2112.01114].

## 2. Hybrid-Order Acyclicity Constraint and Problem Formulation

In the causal discovery setting, SPG-AHOC extends the conventional SPG machinery to optimization over the space of weighted adjacency matrices $W \in \mathbb{R}^{d \times d}$ for linear structural equation models (SEMs). The optimization problem is:
\[
\min_{W} L(W;X) + \lambda_1 \|W\|_1 \quad \text{subject to} \ h_\text{AHOC}(W) = 0
\]
with $L(W;X)$ the least-squares data fit and $\|W\|_1$ promoting sparsity [2601.18189]. Critical to the approach is the Hybrid-Order Acyclicity Constraint (AHOC), which unifies quadratic and $\ell_1$-core terms:
\[
M(W; \alpha) = \alpha (W \circ W) + (1-\alpha) |W|
\]
\[
B(W; \alpha, \epsilon) = \frac{M(W; \alpha)}{\|M(W; \alpha)\|_F + \epsilon}
\]
\[
h_\text{AHOC}(W; \alpha, \epsilon) = \mathrm{tr} \exp(B(W; \alpha, \epsilon)) - d
\]
AHOC exhibits essential mathematical stability properties: it yields a non-vanishing gradient near $W=0$ and uniform boundedness near cyclic regions, unlike alternatives such as $h_\text{exp}(W) = \mathrm{tr} \exp (W \circ W) - d$ [2601.18189].

## 3. Smoothed Proximal Gradient Algorithm with AHOC Constraint

SPG-AHOC operates by replacing the nonsmooth $|W|$ within $h_\text{AHOC}$ with a smooth surrogate:
\[
\mathrm{smooth\_abs}(x; \delta) = \sqrt{x^2 + \delta^2}
\]
yielding the smoothed core
\[
\tilde{M}(W; \alpha, \delta) = \alpha (W \circ W) + (1-\alpha) \sqrt{W^2 + \delta^2}
\]
and consequently a smooth, Lipschitz-continuous constraint function $\tilde{h}_\text{AHOC}(W; \alpha, \epsilon, \delta)$.

The constraint is enforced via an augmented Lagrangian approach:
\[
F_t(W) = L(W; X) + \mu_t \tilde{h}_\text{AHOC}(W) + \frac{\rho_t}{2} [\tilde{h}_\text{AHOC}(W)]^2 + \lambda_1 \|W\|_1
\]
At each iteration, a proximal gradient update is performed on the differentiable part, followed by coordinatewise soft-thresholding to enforce sparsity:
```python
# Update step (compact notation)
G = ∇f(W_k)
Z = W_k - η * G
W_{k+1} = soft_threshold(Z, η*λ1)
```
where $f$ encapsulates the smooth loss plus constraint penalty. The step-size $\eta$ is adapted by backtracking to ensure a sufficient decrease of the objective [2601.18189].

This iterative scheme leverages the “manifold identification” property: after finitely many iterations, the support (zero pattern) of $W$ stabilizes exactly, owing to the geometric structure of the proximal operator and constraint [2601.18189].

## 4. Finite-Time Oracle Property and Theoretical Guarantees

A central advance of SPG-AHOC is the finite-time oracle property. Under standard Lasso-type identifiability conditions—local restricted strong convexity, irrepresentability, and a minimal signal strength (beta-min)—it is proven:
- There exists a critical neighborhood such that for small enough smoothing parameter $\delta < \delta^*$, after a finite iteration $K$, all subsequent iterates $W_k$ ($k \geq K$) exactly recover the support and signs of the true DAG:
  - $\mathrm{supp}(W_k) = \mathrm{supp}(W^*)$
  - $\mathrm{sign}(W_k) = \mathrm{sign}(W^*)$
  with high probability $1 - O(\exp(-cN))$ [2601.18189].

The argument proceeds in two stages: first, establishing primal–dual statistical consistency of the underlying estimator, second, leveraging the strict feasibility gap in the dual certificate to show explicit finite $K$ for support stabilization. The smoothing bias $O(\delta)$ is quantifiably controlled.

## 5. Numerical and Empirical Performance

SPG-AHOC demonstrates strong empirical advantages relative to earlier continuous DAG structure learning algorithms such as NOTEARS and DAGMA:
- On benchmark synthetic and real graph datasets, SPG-AHOC achieves Structural Hamming Distance (SHD) at least on par with or superior to NOTEARS/DAGMA.
- Sparsity in the returned $W$ is markedly improved: NOTEARS typically duplicates fully dense matrices (requiring heuristic post-thresholding), while SPG-AHOC produces 98–99.8% exact zeros, fully eliminating the need for post hoc support selection.
- Computationally, for moderate $d$ ($d \leq 200$), SPG-AHOC is 2–5x faster than NOTEARS and competitive with DAGMA, attributed to active set restriction induced by early topological locking [2601.18189].

## 6. Comparison to Other SPG Applications and Extensions

Core advances in SPG-AHOC build upon earlier SPG innovations for structured sparse regression and clustering:
- General SPG approaches smooth composite nonsmooth penalties and leverage FISTA-type momentum [1202.3708, 1005.4717].
- The Smoothing Proximal Gradient with Extrapolation (SPGE) method achieves convergence rates $o(1/k^{1-\sigma})$ and reduced zig-zag in nonconvex $\ell_0$-relaxation [2112.01114].
- Convex clustering via SPG requires only $O(knp)$ per iteration and achieves comparable accuracy to interior-point and AMA/ADMM solvers but at drastically reduced computation and memory [2006.12592].

A recurring insight is that controlled smoothing enables exact recovery and sharply improves the interpretability of first-order optimization output—no longer requiring dense solutions or hand-thresholding. Extensions of SPG-AHOC may incorporate adaptive restarts, alternative constraint stabilization (e.g., line-search for $L$), or asynchronous coordinate updates [2112.01114].

## 7. Limitations and Open Directions

Despite the theoretical and empirical strengths, several limitations persist:
- Parameter selection (e.g., step-size, smoothing parameter, constraint weight) can be delicate and problem-specific, requiring explicit calibration for optimal practical results [2112.01114].
- Sublinear convergence remains fundamental to these first-order methods; further improvements hinge on structural properties (e.g., strong convexity or Kurdyka-Łojasiewicz conditions).
- While SPG-AHOC's finite-time identification holds under standard assumptions, behavior in the presence of model misspecification or strong correlations requires further investigation [2601.18189].

Potential extensions include adaptive restart schemes, improved line-search for step-size selection, handling nonconvex differentiable loss terms, and distributed or stochastic variants for large-scale DAG recovery.

---

**References**:  
[1005.4717]: "Smoothing proximal gradient method for general structured sparse regression"  
[1202.3708]: "Smoothing Proximal Gradient Method for General Structured Sparse Learning"  
[2006.12592]: "An Efficient Smoothing Proximal Gradient Algorithm for Convex Clustering"  
[2112.01114]: "The Smoothing Proximal Gradient Algorithm with Extrapolation for the Relaxation of $\ell_0$ Regularization Problem"  
[2601.18189]: "Smooth, Sparse, and Stable: Finite-Time Exact Skeleton Recovery via Smoothed Proximal Gradients"

Source: https://www.emergentmind.com/topics/smoothed-proximal-gradient-spg-ahoc