Dual Iterative Hard Thresholding (DIHT)

• Dual Iterative Hard Thresholding (DIHT) is an algorithmic framework that tackles NP-hard sparsity constrained minimization problems using Lagrangian duality.
• It employs a projected super-gradient ascent with a hard-thresholding operation to recover sparse solutions, ensuring reliable parameter and support recovery.
• Empirical evaluations on synthetic and real-world datasets demonstrate that DIHT converges faster and recovers sparse structures more accurately than traditional IHT methods.

Dual Iterative Hard Thresholding (DIHT) is a class of algorithms for solving sparsity-constrained minimization problems in machine learning, specifically addressing non-convex and NP-hard contexts by leveraging Lagrangian duality theory. DIHT extends the classical Iterative Hard Thresholding (IHT) paradigm from the primal optimization setting to the dual, providing both theoretical guarantees and practical gains for high-dimensional learning problems where feature selection and model interpretability via sparse solutions are critical (Liu et al., 2017).

1. Primal Sparsity-Constrained Minimization

The primary focus of DIHT is the $\ell_0$-constrained, $\ell_2$-regularized empirical risk minimization problem:

$$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$

Here, $(x_i, y_i)_{i=1}^N$ are training data, $l(\cdot,\cdot)$ is a convex loss (e.g., squared, hinge, or Huber), $\lambda > 0$ is the regularization parameter, and the $\ell_0$ constraint limits $w$ to $k$ nonzero entries. This combinatorial constraint renders the optimization problem non-convex and NP-hard, precluding direct application of standard convex optimization tools.

2. Sparse Lagrangian Duality Theory

2.1 Conjugate-Based Saddle Reformulation

By rewriting each loss using its convex conjugate, $$l_i(u) = \max_{\alpha_i \in \mathcal{F}} \{ \alpha_i u - l_i^*(\alpha_i) \}$$, the primal objective can be expressed in a saddle-point formulation:

$\ell_2$0

with

$\ell_2$1

2.2 Sparse Saddle-Point and Mini–Max Theorems

A pair $\ell_2$2 is a $\ell_2$3-sparse saddle point if $\ell_2$4 and for all $\ell_2$5, $\ell_2$6:

$\ell_2$7

The central results are:

• $\ell_2$8 is a sparse saddle point if and only if:
  • $\ell_2$9 solves the primal minimization,
  • $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$0,
  • $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$1,
  • where $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$2 is the hard-thresholding operator retaining the $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$3 largest-magnitude entries.

The mini–max theorem guarantees duality equivalence if and only if such a sparse saddle exists: $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$4

2.3 Dual Objective and Strong Duality

The dual objective is: $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$5 with explicit solution: $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$6

$$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$7

The dual functional $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$8 is concave in $$\min_{w\in\mathbb{R}^d} P(w) := \frac{1}{N} \sum_{i=1}^N l(w^\top x_i, y_i) + \frac{\lambda}{2}\|w\|^2 \quad \text{subject to} \ \|w\|_0 \leq k$$9. Sparse strong duality holds: if $(x_i, y_i)_{i=1}^N$0 satisfy the saddle conditions, then $(x_i, y_i)_{i=1}^N$1.

3. Dual Iterative Hard Thresholding Algorithms

3.1 Deterministic (Batch) DIHT

DIHT maximizes $(x_i, y_i)_{i=1}^N$2 via projected super-gradient ascent with primal recovery steps. The core algorithm is:

• Initialize $(x_i, y_i)_{i=1}^N$3, $(x_i, y_i)_{i=1}^N$4.
• For $(x_i, y_i)_{i=1}^N$5:

  • For each $(x_i, y_i)_{i=1}^N$6:

    $(x_i, y_i)_{i=1}^N$7

    $(x_i, y_i)_{i=1}^N$8

  • Primal recovery:

    $(x_i, y_i)_{i=1}^N$9

• Return $l(\cdot,\cdot)$0.

The hard-thresholding step $l(\cdot,\cdot)$1 ensures every iterate remains $l(\cdot,\cdot)$2-sparse.

3.2 Stochastic Variant (SDIHT)

Indices are partitioned into $l(\cdot,\cdot)$3 blocks $l(\cdot,\cdot)$4. At each iteration, a random block $l(\cdot,\cdot)$5 is chosen, and only the corresponding $l(\cdot,\cdot)$6's are updated:

• For $l(\cdot,\cdot)$7:

  $l(\cdot,\cdot)$8

• Maintain

  $l(\cdot,\cdot)$9

• $\lambda > 0$0

4. Convergence Theory

4.1 Non-Asymptotic Rates for DIHT

Assume each $\lambda > 0$1 is $\lambda > 0$2-smooth. With step-size

$\lambda > 0$3

where $\lambda > 0$4 is a data-dependent constant, the following hold:

• Dual estimation error: $\lambda > 0$5, $\lambda > 0$6.
• Support recovery: If a margin $\lambda > 0$7 around $\lambda > 0$8’s smallest nonzero exists, support is exactly recovered after $\lambda > 0$9.
• Primal-dual gap: $\ell_0$0 after $\ell_0$1.

4.2 RIP-Free Convergence

Unlike primal IHT methods, which require Restricted Isometry Property (RIP) assumptions on $\ell_0$2, DIHT does not need RIP conditions. Convergence only assumes smoothness and boundedness of feature vectors. This demonstrates invariance to data isometry that distinguishes DIHT from previous approaches.

4.3 SDIHT Convergence

Under the same smoothness conditions, SDIHT achieves:

• Expected error: $\ell_0$3
• With probability at least $\ell_0$4, support recovery and primal-dual gap $\ell_0$5 in $\ell_0$6.

5. Empirical Results and Comparison

5.1 Synthetic Sparse Regression

With $\ell_0$7 or $\ell_0$8, true sparsity $\ell_0$9, and varied $w$0, DIHT was benchmarked against classical IHT and Hard Thresholding Pursuit (HTP) on squared loss. Metrics included parameter estimation error $w$1 and support recovery rate. DIHT achieved lower error and higher probability of success, especially when $w$2.

5.2 Large-Scale Real Data

Datasets such as RCV1 ($w$3K, $w$4M) and News20 ($w$5M, $w$6K) were used. Comparing DIHT and SDIHT with IHT, HTP, and SVR-GHT under Huber and hinge loss, DIHT/SDIHT achieved an order of magnitude faster convergence to the same primal sub-optimality and closed the primal–dual gap non-asymptotically. This confirms superior model estimation accuracy and computational efficiency.

6. Significance and Theoretical Implications

DIHT establishes a new paradigm by formulating a sparse Lagrangian duality for $w$7-constrained problems and optimizing the resultant non-smooth concave dual via projected super-gradient ascent. This enables sublinear convergence rates $w$8 for parameter recovery and primal–dual optimality, without RIP constraints. Empirical results corroborate theoretical predictions, showing both faster convergence and enhanced accuracy compared to primal IHT-style algorithms (Liu et al., 2017). A plausible implication is that DIHT broadens the scope of tractable sparse learning by relaxing isometry assumptions, facilitating applications to a wider range of data regimes.