Greedy Matching Pursuit Algorithms

• Matching pursuit is a family of algorithms that constructs sparse representations by iteratively selecting dictionary atoms with the highest correlation to the residual.
• Advanced variants incorporate orthogonal projection, optimal MAP-based selection, and tree-search strategies to improve recovery precision and noise robustness.
• Extensions for large-scale, distributed, and specialized applications ensure efficient signal approximation in high-dimensional and practical scenarios.

A greedy algorithm based on matching pursuits refers to a family of algorithms that sequentially construct sparse signal or function representations via iterative selection of elements (“atoms”) from an overcomplete dictionary. Each iteration greedily chooses the atom that yields the largest instantaneous improvement according to a problem-specific criterion—classically, the absolute correlation with the residual signal. This approach underlies many efficient algorithms for high-dimensional approximation, compressed sensing, and sparse recovery, and has been extended by optimizing selection rules, projection steps, and search structures. Recent research has advanced the formal underpinnings, introduced optimal dictionary-aware selection rules, varied tree search structures, and generalized core principles to convex objectives and large-scale distributed settings.

1. Matching Pursuit Framework and Classical Algorithms

Classical matching pursuit (MP) iteratively represents a signal $y$ with respect to a dictionary $\{\phi_j\}$ by repeatedly selecting the atom with maximal absolute correlation to the current residual $r^{(k)}$, updating the representation, and recalculating the residual: $$j^\ast = \arg\max_j |\langle r^{(k)}, \phi_j \rangle|,\quad r^{(k+1)} = r^{(k)} - \langle r^{(k)}, \phi_{j^\ast}\rangle \phi_{j^\ast}$$ Orthogonal Matching Pursuit (OMP) improves this by orthogonally projecting $y$ onto the span of all selected atoms at each iteration, ensuring the residual is always orthogonal to the active set. This iterative greedy construction is central to sparse approximation and recovery in underdetermined linear systems.

Variants including Subspace Pursuit (SP), Compressive Sampling Matching Pursuit (CoSaMP), Iterative Hard Thresholding (IHT), and their “with replacement” versions, further leverage multi-atom selection, pruning, and more sophisticated support updates to improve recovery and robustness (0812.2202, Chen et al., 2012).

2. Optimal and Posterior-Inspired Selection: The Bit-wise MAP Approach

Optimal greedy support identification can be formalized as a bit-wise maximum a posteriori (B-MAP) detection problem. At each iteration $k$, B-MAP seeks the non-selected index $i$ that maximizes the log posterior probability: $$\hat i_k = \arg\max_{i\notin \widehat S^{(k-1)}} \log P(s_i = 1 \mid y, s_j = 1\,\forall j \in \widehat S^{(k-1)})$$ This requires marginalizing over remaining feasible support configurations, making direct computation intractable.

To resolve this, a B-MAP proxy is derived using Jensen-KL lower-bounds, yielding an efficiently computable criterion: $$\gamma_j^{(k)} = (1 - \lambda_k)\log\frac{p_j}{1-p_j} + \frac{1}{\sigma^2} \phi_j^T r^{(k)} - \frac{\tau^{(k)}}{2\sigma^2}\|\phi_j\|_2^2$$ where the parameters encode prior, current residuals, and penalize atom norms. This reduces to OMP when only the linear correlation term is used, and to MAP-OMP when a log-ratio is employed. The B-MAP proxy significantly improves support identification under the condition of previous correct selections (Chae et al., 2019).

3. Extensions: Advanced Greedy and Tree-Search Strategies

Classical greedy algorithms are limited by their myopic, single-candidate selection. Several approaches mitigate this via parallel branch exploration and structured tree searches.

Matching Pursuit with Tree Pruning (TMP): TMP performs an initial pre-selection of promising dictionary indices, then systematically builds a search tree of candidate supports using noncausal completions and residual-based pruning to negotiate the combinatorial search space efficiently. Under RIP-type conditions and appropriate parameter settings, TMP achieves exact or near-oracle recovery performance, significantly outperforming OMP in both noiseless and noisy regimes (Lee et al., 2014).

Greedy Sparse Reconstruction Algorithms (GSRA): GSRA further employs both OMP and SP in pre-selection to form a robust initial support, expands candidate supports via a controlled “hope-tree,” and finally refines through a decreasing subspace pursuit (Li et al., 2017). In empirical studies, GSRA enjoys higher recovery thresholds and improved noise robustness.

Self-Projected Matching Pursuit (SPMP): For large-scale or redundant dictionaries, SPMP mimics OMP’s orthogonal projection property using only matching pursuit-style operations on the active set. This allows OMP-level accuracy at dramatically reduced memory requirements, given well-posed least-squares subproblems (Rebollo-Neira et al., 2016).

4. Statistical and Submodular Analyses of Greedy Selection

Recent advances generalize greedy matching pursuit in relation to set functions and submodularity:

• Submodular Matching Pursuit (SMP) poses dictionary selection as maximizing a submodular-in-expectation function. Under expected submodularity, standard greedy selection guarantees a $(1-1/e)$ performance bound with respect to the optimum. Most practical matching pursuit variants, including OMP, can be viewed as single-instance or “weak submodularity” approximations of the SMP rule (Tohidi et al., 2023).
• In convex settings, the Chebyshev Greedy Algorithm in Banach spaces extends OMP-style logic to arbitrary smooth convex objectives. Convergence rates depend jointly on the smoothness of the objective and incoherence properties of the dictionary, yielding Lebesgue-type inequalities for approximation error (Temlyakov, 2013).

5. Large-Scale, Distributed, and Specialized Greedy Matching Pursuits

A. Large-Scale Data

For massive dictionaries or implicit transforms, specialized algorithms leverage operator-based representations, FFT acceleration, or Conjugate Gradient (CG) for least-squares projections, allowing OMP and SP to scale to signals orders of magnitude larger than conventional implementations (Hsieh et al., 2015).

B. Distributed Algorithms

In sensor networks or parallelized environments, local variants of OMP and SP exchange support estimates and use inter-node voting, enabling robust, communication-efficient distributed reconstruction close to centralized performance (Sundman et al., 2013).

C. Specialized Application Structures

Greedy matching pursuits have been reengineered for convolutional sparse coding with local-overlap ($\ell_{0,\infty}$) constraints, achieving strong performance in text inpainting and salt-and-pepper noise removal tasks (Plaut et al., 2018). Enhancements such as equiprobable selection strategies maintain high entropy in dictionary learning and improve denoising and reconstruction (Sandin et al., 2016).

6. Theoretical Rate Limits and Fundamental Complexity

Recent work has sharpened the known upper and lower bounds for approximation rate of pure matching pursuit. For a target function in the variation space associated with the dictionary, the residual error after $n$ steps decays no faster than $n^{-\alpha^*}$ for $\alpha^*\approx 0.182$, as established by the construction of worst-case dictionaries with recursively designed hiding directions. This is suboptimal compared to the $n^{-1/2}$ decay achieved by orthogonal greedy algorithms, and the gap cannot be closed without modifying the basic MP framework (Klusowski et al., 2023).

The table summarizes complexity characteristics for major greedy matching pursuit families:

Algorithm	Storage	Per-iteration Complexity	Oracle Guarantee
MP/OMP	O(mn), O(mk)	O(mn)	No (except for OMP)
SP/CoSaMP/B-MAP variants	O(mn), O(mk)	O(mn) + O(mk^2)	Yes (with RIP)
TMP, GSRA	O(mn), O(mk)	>O(mn) (depends on tree)	Yes (with RIP)
SPMP	O(m + k)	O(mn) + small overhead	Yes (well-posed proj.)

Here, $m$ is the signal/measurement dimension, $n$ the dictionary size, $k$ the sparsity level.

7. Practical Implications and Future Research Directions

Greedy matching pursuit algorithms remain a cornerstone for high-dimensional sparse modeling because of their algorithmic simplicity, computational efficiency, and their capacity to be refined by statistical, geometric, and application-aware insights. Recent developments integrating optimal posterior-inspired criteria, tree searches, operator-based computation, and distributed protocols push greedy matching pursuits closer to the statistical and computational efficiency of convex relaxations—often at lower cost and complexity.

Open directions include: formal recovery guarantees under highly structured or adversarial dictionaries, unification with explicit convex relaxation methods (as in loss function-based OMP (Mohammad-Taheri et al., 2023)), and adaptive subdictionary or multi-bandit strategies for atom selection in nonstationary regimes (Rakotomamonjy et al., 2015).

References:

(Chae et al., 2019, Tohidi et al., 2023, Rebollo-Neira et al., 2016, Plaut et al., 2018, Lee et al., 2014, Li et al., 2017, Hsieh et al., 2015, Sundman et al., 2013, Temlyakov, 2013, Palumbo et al., 2022, Klusowski et al., 2023, Rakotomamonjy et al., 2015, 0812.2202, Chen et al., 2012, Sandin et al., 2016).