Greedy Forward Selection Overview

• Greedy forward selection is an iterative, bottom-up approach that incrementally adds features based on a locally optimal gain criterion to improve model performance.
• It forms the basis of methods like Forward Regression and Orthogonal Matching Pursuit, effectively reducing error and enhancing sparsity in high-dimensional settings.
• The approach offers theoretical guarantees via submodularity ratios and scalable approximations, making it vital for applications in regression, sensor selection, and network pruning.

Greedy forward selection is an iterative, bottom-up approach for subset selection and sparse modeling, in which candidate variables (features, atoms, measurements, or other primitives) are incorporated one at a time according to a locally optimal criterion. At each step, the method identifies the unselected candidate whose addition maximizes a defined gain—such as improvement in an objective function, reduction in error, or marginal likelihood—and then permanently augments the current set with this candidate. The process repeats until a stopping rule based on cardinality, statistical significance, information criterion, or lack of improvement is met. Greedy forward selection is fundamental to high-dimensional statistics, machine learning, and signal processing, and is the basis for classical algorithms such as Forward Regression, Orthogonal Matching Pursuit, and many practical selection routines in regression, sparse approximation, and experimental design.

1. Canonical Algorithms and Selection Criteria

The form of greedy forward selection is highly contingent on the statistical, approximation, or information objective. Classic examples include:

• Forward Regression (FR): At each step, the candidate predictor that most increases $R^2$ or reduces sum of squared residuals (SSR) when appended to the current model is selected. This maximizes $f(S) = R^2(\text{target}; X_S)$ or $-SSR(S)$ (Cheng et al., 2015).
• Orthogonal Matching Pursuit (OMP): Chooses the atom with the largest correlation to the current residual, then projects onto the span of the selected atoms and repeats (Yasuda et al., 2022).
• Lasso/ℓ₁-regularized variants: Some selection variants incorporate convex surrogates such as the Lasso; at each step, a coordinate is chosen based on maximum amplitude in the Lasso solution with the rest held fixed. For example, Single ℓ₁ Selection (SLS) adds the index with maximum amplitude from the active set of a constrained Lasso (Mhenni et al., 2021).
• Variance-explained and information gain: In unsupervised settings, the criterion may be variance explained by selected principal variables (e.g., Forward Selection Component Analysis, FSCA) (Zocco et al., 2021) or marginal mutual information.
• Nonlinear or neural models: Extensions to differentiable models may use attention weights, as in Sequential Attention Feature Selection (Yasuda et al., 2022).

The typical workflow involves (i) initializing an empty set $S$, (ii) repeatedly evaluating some marginal gain for all $j \notin S$, selecting $j^* = \arg\max_{j\notin S} \text{Gain}(j; S)$, and (iii) updating $S \leftarrow S \cup \{j^*\}$, stopping per a criterion (size, significance, AIC/BIC, or lack of improvement).

2. Theoretical Guarantees and the Submodularity Ratio

If the target function is submodular—i.e., exhibits diminishing returns—the classic Nemhauser bound guarantees $(1 - 1/e)$-approximate maximization relative to the global optimum. However, most natural objectives (e.g., $R^2$ in subset regression, variance explained with correlated features) are not fully submodular. To quantify this, the submodularity ratio $\gamma_{U,k}$ is introduced: $$\gamma_{U,k} = \min_{L\subseteq U, S: |S| \leq k, S \cap L = \emptyset} \frac{\sum_{j \in S}[f(L \cup \{j\}) - f(L)]}{f(L \cup S) - f(L)}$$ This parameter interpolates between non-submodular and submodular regimes (Das et al., 2011, Khanna et al., 2017).

Greedy forward selection achieves the guarantee: $$f(S_{\text{greedy}}) \geq (1 - e^{-\gamma_{U,k}}) \max_{|S| \leq k} f(S)$$ For regression $R^2$, $\gamma_{U,k}$ may be strictly greater than classical spectral bounds (sparse-eigenvalues, mutual coherence), yielding improved, instance-dependent guarantees (Das et al., 2011).

For weakly submodular objectives (i.e., $\gamma > 0$), this framework extends to diverse selection problems and distributed/stochastic accelerations, preserving approximation ratios that can approach the classical $1 - 1/e$ in favorable regimes (Khanna et al., 2017).

3. Extensions: Model Classes, Nonlinearity, and Constraints

Greedy forward selection supports broad generalizations:

• Sparse High-Dimensional Models: In regression with $p \gg n$, forward selection and its multi-variable per step variants (Greedy Forward Regression) satisfy the sure-screening property and selection consistency under restricted eigenvalue conditions (Cheng et al., 2015, Chen et al., 7 Jul 2025).
• Atomic-norm and structured selection: The conditional gradient (Frank–Wolfe) method with atomic-norm constraints yields a greedy forward step—adding the atom most aligned with the negative gradient—generalizing to infinite or structured dictionaries. In particular, CoGEnT incorporates both forward and backward steps for atomic-norm minimization (Rao et al., 2014).
• General convex objectives: Forward-backward greedy variants (FoBa-obj, FoBa-gdt) for convex smooth functions alternate between forward coordinate selection (via marginal reduction or gradient magnitude) and pruning of weak coordinates, achieving support recovery and near-oracle error rates under restricted strong convexity (Liu et al., 2013).
• Feature group and fairness constraints: Under partition matroid or downward-closed constraints (e.g., fairness-aware groupings), forward selection remains applicable, with provable guarantees using extended adaptive query models (FASTFW) (Quinzan et al., 2022).
• Sensor and measurement selection: For heterogeneous sensor networks, greedy selection under cardinality constraints yields provable performance (at least $1/2$-approximate for submodular surrogates), and specialized frame-potential surrogates yield bounds directly on estimation MSE (Majumder et al., 2023).
• Network and model pruning: Greedy forward selection enables provable pruning of neural networks by sequentially adding the most important neurons or filters selected by marginal loss reduction, yielding subnetworks with lower risk than directly trained small models under overparameterization (Ye et al., 2020).

4. Algorithmic Efficiency and Large-Scale Approximations

While greedy forward selection is algorithmically tractable, its naive form has per-step complexity $O(p)$ in candidate evaluation (or $O(pn)$ with $n$ data points), and, if implemented via exact retraining (e.g., for cross-validated error or LOO estimates), this may be prohibitive for large $p$ or $n$ (Pahikkala et al., 2010). Several strategies address scalability:

• Rank-one and matrix-inversion updates: In regularized least squares (greedy RLS), rank-one updates via Sherman–Morrison–Woodbury formulas reduce LOO error computation to $O(m)$ per feature (Pahikkala et al., 2010).
• Stochastic/delayed evaluation: Stochastic Greedy and Lazy variants of forward selection reduce the number of explicit marginal gain evaluations, using random subsamples or maintaining upper bounds and updating only when necessary (Zocco et al., 2021, Khanna et al., 2017).
• Parallel and distributed computation: Partitioning the dataset and performing local greedy selection, then consolidating, yields sublinear scaling in wall-clock time for large $p$ (Khanna et al., 2017).
• Batch addition: Multi-variable addition per iteration (e.g., GFR, J>1) reduces the number of steps required to achieve a given model size, at minimal penalty in false inclusions (Cheng et al., 2015).
• Gradient-surrogate and adaptive sequencing: For objectives with efficient gradients, marginal gains can be approximated with gradient norms and selected in adaptive, highly-parallel rounds (Quinzan et al., 2022).

5. Applications and Empirical Performance

Greedy forward selection is widely used in:

• Variable selection in classical and high-dimensional regression (linear, GLM, nonparametric, and varying-coefficient models) (Cheng et al., 2015, Cheng et al., 2014, Chen et al., 7 Jul 2025);
• Measurement and sensor subset selection in signal processing and experimental design (Zocco et al., 2021, Majumder et al., 2023);
• Model selection and feature engineering in tabular and mixed-data scenarios (Amballa et al., 2024);
• Sparse dictionary learning and atomic-norm regularization in compressive sensing (Rao et al., 2014);
• Deep network pruning and efficient architecture search (Ye et al., 2020).

Empirical results consistently show that, especially in high-correlation or high-dimensional regimes, greedy forward methods with enhanced selection steps (e.g., SLS, Gram-Schmidt-based, BIC/EBIC stop, or backward pruning) can outperform pure $\ell_1$-regularization or random selection, with lower false positive rates, improved predictive performance, and dramatic runtime/computational scalability improvements (Mhenni et al., 2021, Chen et al., 7 Jul 2025, Cheng et al., 2015). Approximations based on stochastic or distributed variants yield performance within negligible loss of the optimum and often considerable speedups (Khanna et al., 2017). In unsupervised settings (variance explained), the “lazy” greedy approach nearly matches the exact greedy path while reducing computational effort (Zocco et al., 2021).

6. Limitations, Optimality, and Practical Considerations

While greedy forward selection enjoys broad applicability and strong empirical performance, several limitations and theoretical caveats remain:

• Nonglobal optimality: Greedy selection is not globally optimal for arbitrary objectives, especially for non-submodular or highly nonconvex landscapes. Path-dependence and the absence of look-ahead can lead to suboptimal inclusion/exclusion.
• Correlation and identifiability: In ultra-high-dimensional settings with near-collinear features or under weak signal regimes, specific variants (GFR, GSFR, FoBa) require careful tuning of step size, selection thresholds, and stopping rules to guarantee recovery or avoid false positives (Cheng et al., 2015, Chen et al., 7 Jul 2025, Liu et al., 2013).
• Assumption sensitivity: Theoretical guarantees depend on the restricted eigenvalue, incoherence, or submodularity ratio being bounded away from zero; in degenerate or adversarial configurations, performance degrades (Das et al., 2011, Khanna et al., 2017).
• Stopping rule impact: Over- or under-selection is possible if significance, AIC/BIC, or EBIC thresholds are not rigorously calibrated, particularly when model complexity or sample size is misspecified (Cheng et al., 2014, Cheng et al., 2015).
• Missing data and structured constraints: Extensions to missing data leverage adaptive awarding criteria and multiple imputation combined with gradient-based selection; theoretical consistency for these settings is an open problem (Lee, 2022).

Empirically, greedy forward selection with adaptive shrinkage (e.g., FLASH) strictly enlarges the design class for which variable selection is consistent compared to Lasso or Forward Selection, and hybrid forward-backward approaches offer improved bias-variance tradeoff and parsimony (Radchenko et al., 2011).

Greedy forward selection is a foundational approach for interpretable, scalable, and theoretically robust subset selection across a range of statistical, signal processing, and machine learning tasks. Its tractability and adaptability explain its prevalence and continued innovation in modern high-dimensional and structured data domains.