Orthogonal Matching Pursuit (OMP)

• Orthogonal Matching Pursuit (OMP) is a greedy algorithm for sparse recovery that iteratively selects the column most correlated with the current residual.
• The method employs Residual Ratio Thresholding (RRT) as a tuning-free stopping rule, eliminating the need for prior knowledge of sparsity or noise levels.
• OMP and its RRT adaptation have proven effective in compressed sensing, imaging, and outlier detection, offering robust support recovery under theoretical guarantees.

Orthogonal Matching Pursuit (OMP) is a canonical greedy algorithm designed for support recovery and sparse approximation in high-dimensional linear regression and compressed sensing. OMP iteratively identifies the active support of a sparse vector by sequentially selecting the column of the design or sensing matrix that is maximally correlated with the current residual, updating the support estimate, and recomputing the least-squares fit at each step. This mechanism underpins a wide array of theoretical results and practical successes in sparse signal processing, statistical learning, imaging, and related domains.

1. Algorithmic Framework and Standard OMP Procedure

OMP operates on the linear model

$$\mathbf{y} = \mathbf{X}\boldsymbol\beta + \mathbf{w},$$

where $\mathbf{X}\in\mathbb{R}^{n\times p}$ has unit-norm columns, $\boldsymbol{\beta}\in\mathbb{R}^p$ is assumed $k_0$-sparse ($|\mathcal{S}|=k_0$ for the support $\mathcal{S} = \{j : \beta_j\neq0\}$), and $\mathbf{w}\sim\mathcal{N}(0,\sigma^2 I)$. OMP is initialized with residual $r^0 = y$ and support $S^0 = \emptyset$, proceeding by:

1. At iteration $k$, select $$t_k = \arg\max_{j\notin S^{k-1}} |\mathbf{X}_j^T r^{k-1}|$$;
2. Update $S^k = S^{k-1}\cup\{t_k\}$;
3. Compute the restricted least-squares solution $\widehat{\beta}_{S^k} = \mathbf{X}_{S^k}^{+}y$, setting the complement to zero;
4. Update the residual $r^k = y - \mathbf{X}\widehat{\beta}$;
5. Iterate until a stopping criterion is satisfied.

Common stopping conditions require knowledge of either $k_0$ (fixed number of steps) or noise variance (residual norm below a threshold).

2. Motivations for Tuning-Free Recovery

In practical scenarios, neither the sparsity level $k_0$ nor the noise variance $\sigma^2$ are typically available, and attempts to estimate them can be unreliable or computationally intensive, e.g., via cross-validation or stability selection. Conventional OMP thus depends on prior knowledge that is not robustly attainable, limiting its utility in many real-world signal and model selection problems (Kallummil et al., 2018).

This motivates the development of data-adaptive methodologies capable of support recovery and error control without such prior statistical information.

3. Residual Ratio Thresholding (RRT): Principle and Implementation

Residual Ratio Thresholding (RRT) introduces an alternative to classical stopping criteria. Instead of monitoring the residual norm itself,

$||r^k||_2,$

RRT tracks the ratio

$$RR(k) = \frac{||r^k||_2}{||r^{k-1}||_2}, \quad k=1,2,\ldots,k_{\max},$$

with $k_{\max}\geq k_0$. The iteration is halted at the maximal $k$ for which $RR(k)$ falls below a non-adaptive threshold $\Gamma_{RRT}^{\alpha}(k)$, where

$$\Gamma_{RRT}^\alpha(k) = \sqrt{F^{-1}_{\frac{n-k}{2},\,0.5}\left(\frac{\alpha}{k_{\max}(p-k+1)}\right)},$$

and $F_{a,b}(\cdot)$ denotes the CDF of the Beta distribution.

The RRT-OMP procedure is:

• Run standard OMP up to $k_{\max}$ steps, recording $RR(k)$;
• For each $k$, compute $\Gamma_{RRT}^\alpha(k)$;
• Set $$k_\text{RRT} = \max\{k : RR(k) \leq \Gamma_{RRT}^\alpha(k)\}$$, or increase $\alpha$ until non-empty;
• Report $\widehat{S}=S^{k_{RRT}}$, $\widehat{\beta} = \mathbf{X}_{\widehat{S}}^{+} y$.

Parameters are typically chosen as $\alpha=1/\log n$ and $k_{max}=\lfloor (n+1)/2 \rfloor$.

4. Support Recovery Guarantees and Sample Complexity

Assuming the standard restricted isometry constant (RIC) condition $\delta_{k_0+1}<1/\sqrt{k_0+1}$, RRT-OMP achieves deterministic recovery analogously to OMP with known $k_0$ or $\sigma^2$, up to a mild additional SNR requirement in finite samples. Specifically,

• Finite sample result (Theorem 3): If

$$\epsilon_\sigma = \sigma\sqrt{n+2\sqrt{n\log n}} < \min\{\epsilon_{omp}, \epsilon_{rrt}\},$$

with $$\epsilon_{rrt} = [\Gamma_{RRT}^{\alpha}(k_0)\,\sqrt{1 - \delta_{k_0}}\,\beta_{\min}]/[1+\Gamma_{RRT}^{\alpha}(k_0)]$$, then $P(\widehat{S}=S)\geq 1-1/n-\alpha$.

• Asymptotic regime (Theorem 4): When $\log p = o(n)$, $k_0/n \to c < 1/2$, and $\alpha\to 0$ slowly (e.g., $\alpha=1/\log n$), $\Gamma_{RRT}^{\alpha}(k_0)\to 1$, so the SNR slack vanishes, and the procedure is large-sample consistent under the same RIC as optimal OMP.
• High-SNR error control (Theorem 6): As $\sigma^2\to 0$, the probability of missed discoveries $\to 0$, and the false-positive rate (and overall error) is bounded by $\alpha$.

5. Empirical Performance and Computational Considerations

Comprehensive numerical simulations validate the theoretical claims:

• In both moderate $(n=200, p=300/900, k_0=6, \text{SNR}\simeq 3)$ and small-sample $(n=32, p=64, k_0=3)$ regimes, RRT using $\alpha=1/\log n$ or $1/\sqrt{n}$ closely tracks the $\ell_2$-error and support recovery rate of OMP with known $k_0$ or $\sigma^2$, while outperforming OMP+CV and low-complexity alternatives such as LAT.
• Empirical probability of exact recovery versus SNR is similar to oracle OMP, and the high-SNR false-alarm rate is always bounded by $\alpha$.
• On real data for outlier detection (Stack-loss, AR2000, Stars, Brain–Body–Weight), RRT identifies the same key outliers as classical robust methods with drastically lower computational burden.

Regarding implementation, running OMP up to $k_{max}$ steps requires at most $O(k_{max} p n)$ arithmetic operations, only marginally exceeding the standard (oracle) variant, and obviates any need for cross-validation or expensive grid search strategies.

6. Relation to Other Greedy and Convex Procedures

RRT as a stopping rule is applicable to any greedy, monotonic procedure with nested supports (e.g., OMP, OLS, thresholded variants). Its operational simplicity (two hyperparameters $\alpha$, $k_{max}$, both with recommended default choices) and tuning insensitivity make it attractive for settings where parameter-free or plug-and-play solutions are needed.

The additional SNR requirement over optimal OMP is negligible in the high-dimensional limit ($\log p=o(n)$) and is, in practice, offset by a significant gain in reliability and reduced computational overhead compared to cross-validation or stability-selection-based selection.

Extension of such data-driven stopping rules to algorithms with non-monotonic support patterns (e.g., LASSO, Subspace Pursuit, CoSaMP) remains an open problem.

7. Implications for Practice and Open Challenges

Residual Ratio Thresholding transforms OMP into a data-driven sparse recovery framework requiring no prior model knowledge, while retaining the structural guarantees derived from RIC and achieving optimal support recovery up to an asymptotically negligible SNR surplus.

Key practical recommendations:

• Set $k_{max}=\lfloor (n+1)/2 \rfloor$, $\alpha=1/\log n$; in most applications, asymptotic “tuning-free” operation is achieved over broad $\alpha$ ranges.
• The method is directly applicable to high-dimensional regression, variable selection, signal processing, and any context demanding sparse recovery.
• The lack of reliance on parameter selection not only simplifies deployment but also improves robustness against user mis-specification and model uncertainty.

Further research is required to generalize RRT principles to non-monotone greedy algorithms and convex relaxations, as well as to extend empirical support in regimes with ultrahigh-dimensional or highly structured design matrices.