Dual Spectral Projected Gradient (DSPG)

• Dual Spectral Projected Gradient (DSPG) is a first-order algorithm designed for solving dual log-determinant semidefinite programming problems with linear constraints and nonsmooth regularizers.
• It utilizes spectral projected gradients with Barzilai–Borwein step-size selection and a nonmonotone line search to efficiently converge to first-order optimality.
• The method scales to high-dimensional settings by reducing per-iteration costs while outperforming interior-point methods in speed and accuracy for sparse covariance and graphical model estimation.

The Dual Spectral Projected Gradient (DSPG) method is a first-order algorithm designed for efficiently solving the dual of log-determinant semidefinite programming (SDP) problems subject to linear equality constraints and nonsmooth convex regularizers, with core applications in sparse Gaussian graphical model selection, covariance estimation, and related high-dimensional inference problems. DSPG generalizes the spectral projected gradient framework of Birgin et al. to log-determinant optimization, enabling the rapid solution of both standard and structured covariance selection SDP instances at scales not tractable by interior-point or conventional first-order methods (Nakagaki et al., 2018, Namchaisiri et al., 2024).

1. Problem Formulation

The primary domain of DSPG is the regularized log-determinant SDP, frequently appearing in graphical lasso-type estimation. The canonical primal form is

$$\min_{X\in S^n} f(X) := -\mu\log\det X + \langle C, X\rangle + \langle \rho, |X| \rangle \quad\text{s.t.}\quad \mathcal{A}(X)=b,\ X\succ0,$$

where $S^n$ denotes the set of real symmetric $n\times n$ matrices, $C\in S^n$ encodes sample information (typically empirical covariance), $\mu>0$ is a log-barrier parameter, $\rho\ge0$ are element-wise regularization weights, and $\mathcal{A}:S^n\to\mathbb{R}^m$ encodes linear equality constraints.

The dual problem introduces variables $y\in\mathbb{R}^m$ for equality constraints and $W\in S^n$ for the elementwise box constraints $|X_{ij}|\le\rho_{ij}$: $S^n$0

$S^n$1

The gradient of $S^n$2 is

$S^n$3

Extensions accommodate structured penalties such as block-wise, group-wise, or hidden cluster $S^n$4-like terms by generalizing the dual with additional dual variables and projections, as in (Namchaisiri et al., 2024, Namchaisiri et al., 2024).

2. Algorithmic Framework

DSPG is a nonmonotone projected gradient algorithm using Barzilai–Borwein (BB) step-size selection and line search, formulated for the dual SDP:

1. Initialization: Select $S^n$5 or, for generalized settings, $S^n$6 so that dual feasibility and $S^n$7 are satisfied. Set algorithmic parameters $S^n$8.
2. Stopping Test: Compute the projected gradient direction:

$S^n$9

where $n\times n$0 is the projection onto the feasible set (e.g., box and LMI constraints). If $n\times n$1, terminate.

1. Spectral Step and Search Direction: Compute BB-scaled search direction

$n\times n$2

with $n\times n$3 determined by BB update.

1. Dual Feasibility Safeguard: Ensure $n\times n$4 by restricting step sizes using the minimum eigenvalue of the direction in the transformed metric.
2. Nonmonotone Line Search: Seek the maximal $n\times n$5 (by geometric reduction, e.g., $n\times n$6) satisfying

$n\times n$7

for globalization.

1. Update: Set $n\times n$8 and update $n\times n$9 via BB-type rule.

Table: Key steps and operations in DSPG (generalized form)

Step	Operation	Dominant Cost per Iteration
Gradient Eval	$C\in S^n$0, compute $C\in S^n$1	Cholesky factorization $C\in S^n$2
Projections	Box, ball, and custom projections	$C\in S^n$3 – $C\in S^n$4 (PAVA, sorting)
Line Search	Feasibility safeguard, function evaluations	$C\in S^n$5 per trial

3. Projection Operators and Special Structures

Projection onto box constraints is component-wise truncation: $C\in S^n$6 In advanced settings with hidden clustering, an auxiliary variable $C\in S^n$7 is introduced, constrained to lie in the image of $C\in S^n$8 for suitably bounded $C\in S^n$9. Projection onto this set reduces to isotonic regression (ordered $\mu>0$0-regression), efficiently solved via the pool-adjacent-violators algorithm (PAVA), which incurs $\mu>0$1 complexity for $\mu>0$2 variables (Namchaisiri et al., 2024).

For generalized regularizers (block, group, multitask), projections onto $\mu>0$3-balls or block vehicle sets are performed separately for each variable block, often via closed-form or efficient sorting-based routines (Namchaisiri et al., 2024).

4. Convergence Properties

Convergence of DSPG is established under standard conditions: surjectivity of $\mu>0$4, strict primal and dual feasibility, and bounded level sets. Key properties include:

• All iterates remain in a compact level set of the dual objective $\mu>0$5.
• The search direction is a true ascent direction when nonzero.
• The line search always results in step size $\mu>0$6 bounded from below by a positive minimum.
• BB step-sizes remain within given bounds.
• Either finite termination occurs, or the projected gradient vanishes asymptotically ($\mu>0$7), ensuring first-order optimality.
• Under convex-concave structure and dual-primal strong duality (Slater’s condition), the dual optimizer reconstructs the primal optimizer uniquely.
• No global linear convergence rate is claimed; local linear convergence is possible under local strong concavity and smoothness (Nakagaki et al., 2018, Namchaisiri et al., 2024).

5. Computational Complexity and Scalability

The main per-iteration cost stems from a Cholesky factorization of an $\mu>0$8 matrix ($\mu>0$9) and the subsequent cost of projections. For structure-exploiting cases (e.g., when $\rho\ge0$0 or regularization operators are sparse or block-diagonal), this cost can be reduced. Projection operations scale as $\rho\ge0$1 (component-wise constraints) or $\rho\ge0$2 for isotonic regression in hidden clustering models (Namchaisiri et al., 2024). Overall memory requirements are modest, allowing the method to scale to large problem instances ($\rho\ge0$3 up to 4000–5000).

6. Numerical Performance

Empirical benchmarks report that DSPG solves standard sparse and structured covariance selection SDPs with $\rho\ge0$4 up to 5000 in $\rho\ge0$5–$\rho\ge0$6 iterations, achieving primal-dual gaps below $\rho\ge0$7 and outperforming inexact primal-dual interior-point, adaptive spectral projected gradient (ASPG), and Nesterov’s smooth method in wall-clock time, especially for moderate to high-accuracy requirements. For hidden clusters, isotonic projection reduces total runtime by several orders of magnitude compared to direct approaches (Nakagaki et al., 2018, Namchaisiri et al., 2024, Namchaisiri et al., 2024). DSPG is also competitive or superior to specialized solvers such as QUIC on gene expression and structured multitask data, particularly when extended with block or multitask regularizers.

7. Implementation and Practical Considerations

DSPG is parameterized by $\rho\ge0$8, with typical values $\rho\ge0$9 to $\mathcal{A}:S^n\to\mathbb{R}^m$0, $\mathcal{A}:S^n\to\mathbb{R}^m$1, $\mathcal{A}:S^n\to\mathbb{R}^m$2–$\mathcal{A}:S^n\to\mathbb{R}^m$3, and BB step bounds from $\mathcal{A}:S^n\to\mathbb{R}^m$4 to $\mathcal{A}:S^n\to\mathbb{R}^m$5. Initializing with dual-feasible $\mathcal{A}:S^n\to\mathbb{R}^m$6 and maintaining the positivity constraint via Cholesky-based step size control is essential. For large-scale or structured cases, exploiting sparsity in $\mathcal{A}:S^n\to\mathbb{R}^m$7 and leveraging efficient projection routines (including PAVA and fast sorting for block norms) is critical for performance. Recommended stopping tolerance is $\mathcal{A}:S^n\to\mathbb{R}^m$8, and safeguards against near-singular updates are advised (Namchaisiri et al., 2024, Namchaisiri et al., 2024).

DSPG’s flexibility enables application across a range of log-det SDP problems: standard graphical lasso, hidden-structure precision matrix recovery, multitask graphical model learning, and block/group regularized structure learning. It is particularly suited to problems where moderate to high numerical precision is needed without incurring the cost of explicit KKT system formation, and where the structure of constraints allows efficient projections (Nakagaki et al., 2018, Namchaisiri et al., 2024, Namchaisiri et al., 2024).