Iterative Thresholding with Inversion Algorithm

• ITI is a sparse signal recovery method that employs iterative hard-thresholding integrated with inversion and null-space tuning for enhanced performance.
• The algorithm leverages an eigenvalue-based suboptimal feedback approach to significantly reduce computational complexity while maintaining high accuracy.
• Empirical studies show that ITI achieves comparable NMSE to projection-based methods with dramatically lower runtime in large-scale compressive sensing settings.

Iterative Thresholding with Inversion (ITI) is a framework for sparse signal recovery from undersampled linear measurements. The method is fundamentally characterized by the incorporation of an inversion step—either exact or suboptimal—within an iterative hard-thresholding process, coupled with additional null-space correction. ITI, also referred to as Null-Space Tuning with Hard Thresholding and Feedback (NST+HT+FB), is notable for achieving recovery accuracy comparable to projection-based algorithms while offering significantly improved computational scalability, particularly when leveraging an eigenvalue-based suboptimal feedback strategy. This makes ITI applicable to large-scale compressive sensing and related inverse problems (Song et al., 2017).

1. Mathematical Formulation and Algorithmic Structure

The ITI algorithm addresses the canonical sparse recovery model:

$y = A x + e,$

where $x \in \mathbb{R}^n$ is $k$-sparse, $A \in \mathbb{R}^{m \times n}$ ($m \ll n$) is the measurement matrix, $y \in \mathbb{R}^m$ are the measurements, and $e \in \mathbb{R}^m$ is noise. Sparsity is enforced by a hard thresholding operator $H_k$, which retains only the $k$ largest (in magnitude) entries of a vector.

Each ITI iteration proceeds as follows:

1. Feedback Step: Compute a proxy

$$g^{(t)} = x^{(t)} + A^\top (A A^\top)^{-1}(y - A x^{(t)}).$$

2. Hard Thresholding: Compute sparse proxy

$x \in \mathbb{R}^n$0

3. Feasibility Tuning: Project back to the feasible set using the orthogonal projector onto the null space of $x \in \mathbb{R}^n$1,

$x \in \mathbb{R}^n$2

Alternatively, a merged update is available:

$x \in \mathbb{R}^n$3

These iterations ensure that recovered vectors gradually approach sparsity and measurement consistency as defined by $x \in \mathbb{R}^n$4. The null-space tuning operator is given by

$x \in \mathbb{R}^n$5

which maps arbitrary updates back to the feasible set (Song et al., 2017).

2. Suboptimal Feedback and Eigenvalue-Based Approximation

In standard ITI or NST+HT+FB, support-based inversion operations such as inversion of $x \in \mathbb{R}^n$6 (where $x \in \mathbb{R}^n$7 indexes the active support) are required, incurring $x \in \mathbb{R}^n$8 computational cost per iteration. For large-scale or high-sparsity settings, this cost is prohibitive.

To reduce complexity, the NST+HT+subOptFB variant replaces the explicit inversion with an eigenvalue-based scalar approximation. Specifically, under the restricted isometry property (RIP) and preconditioned RIP (P-RIP), the Gram matrix $x \in \mathbb{R}^n$9 has eigenvalues in $k$0, with $k$1 being an appropriate RIP constant. Its inverse is replaced by $k$2, where $k$3 is chosen to control contraction.

The update becomes:

• Identify $k$4 as the $k$5 largest-magnitude indices of $k$6.
• For entries in $k$7, set

$k$8

and $k$9.

• Update as before:

$A \in \mathbb{R}^{m \times n}$0

This version preserves the key properties of ITI while reducing per-iteration complexity from $A \in \mathbb{R}^{m \times n}$1 to $A \in \mathbb{R}^{m \times n}$2 (Song et al., 2017).

3. Convergence Guarantees

The convergence analysis employs the preconditioned restricted isometry property (P-RIP). For every $A \in \mathbb{R}^{m \times n}$3-sparse vector $A \in \mathbb{R}^{m \times n}$4,

$A \in \mathbb{R}^{m \times n}$5

where $A \in \mathbb{R}^{m \times n}$6 is the P-RIP constant.

Theoretical results show that for $A \in \mathbb{R}^{m \times n}$7 with $A \in \mathbb{R}^{m \times n}$8 $A \in \mathbb{R}^{m \times n}$9-sparse and $m \ll n$0, if $m \ll n$1, and $m \ll n$2 is sufficiently large (dependent on $m \ll n$3 and related constants), then the iterates $m \ll n$4 satisfy

$m \ll n$5

with explicit forms for $m \ll n$6 and $m \ll n$7 provided in (Song et al., 2017).

In the special case where $m \ll n$8 is a Parseval frame ($m \ll n$9), one may guarantee $y \in \mathbb{R}^m$0 by choosing $y \in \mathbb{R}^m$1 whenever $y \in \mathbb{R}^m$2.

4. Computational Complexity

The computational cost per iteration is summarized as follows:

Algorithm Variant	Per-Iteration Complexity	Dominant Costs
NST+HT+FB (exact feedback)	$y \in \mathbb{R}^m$3	Gram matrix inversion on active support ($y \in \mathbb{R}^m$4)
NST+HT+subOptFB (suboptimal)	$y \in \mathbb{R}^m$5	Matrix-vector multiply ($y \in \mathbb{R}^m$6); thresholding ($y \in \mathbb{R}^m$7)
HTP (Hard-Threshold Pursuit)	$y \in \mathbb{R}^m$8	Inversion of $y \in \mathbb{R}^m$9 matrix

The eigenvalue-based suboptimal feedback approach eliminates the per-iteration $e \in \mathbb{R}^m$0 bottleneck, making the method scalable to problem sizes with $e \in \mathbb{R}^m$1 and large $e \in \mathbb{R}^m$2. The complexity is ultimately controlled by the cost of matrix-vector multiplication (Song et al., 2017).

5. Adaptive Iterative Thresholding Without Sparsity Knowledge

The adaptive algorithm (AdptNST+HT+subOptFB) addresses scenarios where the true sparsity level $e \in \mathbb{R}^m$3 is unknown. The procedure incrementally increases the support size at each iteration. At iteration $e \in \mathbb{R}^m$4:

1. Form proxy $e \in \mathbb{R}^m$5.
2. Let $e \in \mathbb{R}^m$6 be the indices of the $e \in \mathbb{R}^m$7 largest magnitudes of $e \in \mathbb{R}^m$8.
3. Set $e \in \mathbb{R}^m$9, others zero.
4. Null-space tune: $H_k$0.

Convergence holds under analogous P-RIP conditions, with geometric error decay after support recovery in $H_k$1 iterations, where $H_k$2 is the true sparsity level (Song et al., 2017).

6. Empirical Performance and Comparative Analysis

Extensive simulations reported in (Song et al., 2017) include problem sizes up to $H_k$3, measurement ratios $H_k$4 up to $H_k$5, and sparsity ratios up to $H_k$6. The ITI variants are benchmarked against:

• NST+HT+FB (exact inversion)
• NST+HT+subOptFB (suboptimal, eigenvalue-based feedback)
• HTP (Hard-Thresholding Pursuit)
• GHTP (graded HTP)
• Adaptive versions: AdptNST+HT+FB, AdptNST+HT+subOptFB

Key findings include:

• The NMSE (normalized mean-square error) achieved by NST+HT+subOptFB is on par with exact feedback and HTP across a broad range of undersampling/sparsity regimes.
• The runtime of NST+HT+subOptFB grows linearly in $H_k$7 and $H_k$8 and is one order of magnitude faster than HTP and exact-feedback NST+HT+FB at $H_k$9. At $k$0, $k$1, $k$2, NST+HT+subOptFB is 3–5× faster than exact-feedback NST+HT+FB.
• The adaptive variant AdptNST+HT+subOptFB achieves comparable NMSE to its competitors but with the lowest per-iteration cost and overall runtime.

7. Significance and Applicability

The ITI framework with null-space tuning and eigenvalue-based suboptimal feedback is distinguished by its combination of rapid geometric convergence and low per-iteration complexity. Particularly under mild P-RIP conditions, this makes ITI a compelling choice for sparse recovery in very large-scale systems, such as large-dimensional compressive sensing, where conventional inversion-based schemes become infeasible (Song et al., 2017).