Error-Guided Feature Selection

• Error-Guided Feature Selection (EGFS) is a framework that uses prediction error signals—such as validation loss or reprojection error—to determine the relevance of features, masks, or subsets.
• EGFS spans multiple methodologies, including NBC wrappers, neural greedy pursuit, error-based knockoffs, and stability-aware approaches, each tailoring error evaluation to the task.
• Empirical studies demonstrate that EGFS enhances model accuracy and robust error control, while addressing challenges like computational cost and model coupling.

Error-Guided Feature Selection (EGFS) denotes a class of feature-selection procedures in which the selection signal is derived directly from an error functional rather than from a purely filter-based relevance score. Across published uses, the relevant error may be an estimated classification error probability, a validation loss, an error increase induced by feature replacement with knockoffs, a reprojection error in geometric vision, or a multi-environment objective combining mean prediction error with cross-environment error variance. In that sense, EGFS is less a single algorithm than a unifying design principle: features, masks, or subsets are retained insofar as they improve predictive behavior under the error notion chosen for the task (Rabenoro et al., 2015, Das et al., 2022, Zhao et al., 2022, Liu et al., 2024, Malarkkan et al., 21 Mar 2026).

1. Conceptual scope

EGFS appears in several technically distinct forms. In wrapper methods for a Naive Bayes classifier (NBC), candidate subsets are evaluated by the classifier’s own estimated error probability (Rabenoro et al., 2015). In neural greedy pursuit, features are added sequentially according to the validation loss of a neural predictor retrained on each candidate subset (Das et al., 2022). In model-X knockoff inference, importance is defined by the change in prediction error when a feature is replaced by its knockoff, and this error-guided statistic is coupled to FDR, FDP, or $k$-FWER control (Zhao et al., 2022). In scene coordinate regression (SCR), EGFS is a training-time mechanism that uses reprojection error to determine where in the image to learn from, rather than which tabular variables to keep (Liu et al., 2024). In stable feature selection under distribution shift, CGDFS can be read as a Bayesian, invariance-aware EGFS method because its posterior over masks is driven by mean prediction error and cross-environment error variance (Malarkkan et al., 21 Mar 2026).

Setting	Error signal	Selection output
NBC wrappers (Rabenoro et al., 2015)	Estimated classification error probability	Binary indicator subset
Neural greedy pursuit (Das et al., 2022)	Validation loss after retraining	Ordered feature subset
Error-based knockoffs (Zhao et al., 2022)	Error increase under knockoff replacement	Controlled selected set
SCR masking (Liu et al., 2024)	Reprojection error and confidence	Error-guided spatial mask
CGDFS (Malarkkan et al., 21 Mar 2026)	Mean error plus cross-environment error variance	Posterior over masks and top-$k$ subset

A central consequence is that EGFS is model-coupled. The error signal is not external to the predictor: it is generated by the predictor, by its posterior probabilities, by its validation loss, by its geometric residuals, or by a stability-aware likelihood constructed from its environment-specific losses. This differentiates EGFS from filter methods based only on mutual information, marginal correlation, impurity decrease, or other classifier-agnostic scores.

2. Error objectives and mathematical forms

In the NBC wrapper formulation, the feature subset $S$ is evaluated by the average estimated error probability

$$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$

The paper’s main point is that this quantity is a finer-grained search criterion than raw $0$–$1$ error, because posterior probabilities can change even when discrete decisions do not (Rabenoro et al., 2015).

In neural greedy pursuit, the guiding quantity is the validation loss of a predictor trained on a candidate subset. For a subset $\mathcal{S}$, the fitted parameters satisfy

$$\hat{\theta}^{\mathcal{S}} \in \underset{\theta}{\arg\min} \left[ \sum_{j=1}^{J} \mathcal{L}\big( \mathbf{t}_j, f_\theta(\mathbf{x}_j^{\mathcal{S}}) \big) + \mathcal{R}(\theta) \right],$$

and the feature added at each step is the one minimizing the resulting validation loss $\mathcal{L}^{\mathcal{S}}$. The loss may be MSE for regression or cross-entropy for classification (Das et al., 2022).

In error-based knockoff inference, the predictor $f$ is first trained on one data split. Feature importance is then defined through the error difference

$k$0

where $k$1 replaces $k$2 by its knockoff. The statistic

$k$3

measures how often prediction error increases when feature $k$4 is replaced. For irrelevant features, the paper shows that $k$5, which is what makes exact p-values and stepdown procedures possible (Zhao et al., 2022).

In SCR, the guiding error is geometric rather than classification- or regression-based. For pixel or patch $k$6, the reprojection error is

$k$7

EGFS ranks pixels by this quantity, uses the lowest $k$8 of pixels as point prompts for SAM, and refines the resulting mask using a learned confidence map. The paper uses $k$9 and a confidence regularization weight $S$0 (Liu et al., 2024).

In CGDFS, the core error functional is explicitly stability-aware. For environment-specific validation loss

$S$1

the method defines the mean error

$S$2

and the error variance across environments

$S$3

These are combined as

$S$4

with likelihood $S$5. This makes the EGFS signal simultaneously accuracy-driven and invariance-aware (Malarkkan et al., 21 Mar 2026).

3. Search, inference, and mask construction

The mechanisms used to exploit these error signals differ substantially.

In the NBC setting, the search space consists of subsets of binary indicators, and the paper studies greedy forward selection, greedy backward elimination, and floating forward–backward or backward–forward searches. The state is a subset $S$6, neighbors are obtained by adding or removing one feature, and the evaluation function is either $S$7–$S$8 error or the average NBC-estimated error probability. Because the NBC log-likelihood is additive in the features, the paper emphasizes efficient incremental updates and an overall $S$9 cost for a full greedy search (Rabenoro et al., 2015).

In neural greedy pursuit, the search is sequential and bottom-up. Starting from $$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$0, the method evaluates every candidate augmentation $$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$1, retrains a neural predictor, computes validation loss, and selects

$$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$2

The output is both a subset and a descending ranking of feature importance, since earlier selections correspond to larger marginal error reductions (Das et al., 2022).

Error-based knockoffs avoid combinatorial subset search. After data splitting and predictor fitting, each feature is perturbed independently by knockoff replacement on the evaluation split, producing per-feature statistics $$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$3. Selection then proceeds either through the standard knockoff threshold for FDR control or through p-value-based stepdown procedures for FDP or $$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$4-FWER control. This is a one-shot EGFS mechanism rather than a greedy or stochastic search (Zhao et al., 2022).

SCR-based EGFS is iterative masking rather than subset search. The model first trains for several epochs, then computes reprojection error maps and confidence maps. Low-error points become SAM prompts, SAM expands them into a coarse semantic mask, and confidence thresholding refines the mask according to

$$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$5

Training then samples only from pixels with $$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$6. The full schedule is 20 epochs with mask recomputation every 5 epochs (Liu et al., 2024).

CGDFS replaces deterministic search with approximate posterior inference over continuous masks $$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$7. A diffusion model is trained as a prior over plausible masks from a mask pool, and guided annealed Langevin dynamics updates

$$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$8

Final discrete selection is obtained by top-$$\mathrm{PErr}(S) = \frac{1}{N}\sum_{i=1}^N \hat{P}_S(Y \neq Y_i \mid B_i) = 1 - \frac{1}{N}\sum_{i=1}^{N} \hat{P}_S(Y = Y_i \mid B_i).$$9 discretization of posterior samples and aggregation through posterior inclusion frequencies $0$0 (Malarkkan et al., 21 Mar 2026).

A plausible implication is that EGFS should be understood at two levels: as an objective principle centered on error, and as a family of optimization or inference schemes that can be greedy, probabilistic, controlled-testing-based, or mask-iterative depending on the application.

4. Representative domains and empirical behavior

In redundant binary indicator selection for NBC, the reported setting contains $0$1 binary indicators derived from expert-designed scores on simulated time-series data. Wrapper methods outperform filter methods, and probability-guided wrapper search improves upon raw-error-guided wrapper search. In the reported table, the best result is the backward–forward floating search guided by “Probability,” with 122 features and test error $0$2, compared with $0$3 for the MI filter and $0$4 for mRMR (Rabenoro et al., 2015).

In neural greedy pursuit, the artificial benchmark contains 5 true features among 500. Reported results include FPSR $0$5 and FNSR $0$6 for NGP with SSFN, and FPSR $0$7 and FNSR $0$8 for NGP with MLP. The paper states that NGP achieves a 58% improvement in FPSR over Drop-one-out on the artificial data. On MNIST, when using only the top 40% of selected features and $0$9 training samples for selection, NGP + SSFN reports 86.45% test accuracy and NGP + CNN 86.25%, compared with 73.60% for LASSO and 68.85% for SHAP + DeepLIFT (Das et al., 2022).

In error-based knockoff inference, the empirical emphasis is not only predictive relevance but also controlled false discoveries. In a simulated nonlinear single-index model with $1$0, $1$1, and $1$2 up to 2000, the error-based method in FDR mode achieves average FDR near the target $1$3, while the FDP and $1$4-FWER variants control $1$5 and the maximum number of false discoveries more tightly. The paper also reports that power increases monotonically with $1$6 and that better predictive models, such as DNNs with lower MSE, yield higher power while maintaining control (Zhao et al., 2022).

In SCR, EGFS changes both efficiency and localization accuracy. On Cambridge Landmarks, ACE trains in approximately 5 min, whereas EGFS trains in 12 min with model size 4.5MB, still far below DSAC* at approximately 15 hours. Reported average localization error is 17 cm / 0.3° for EGFS (single), matching ACE (quad) and improving over ACE (single) at 25 cm / 0.4°. On Indoor6, EGFS (single) reports 56.1% of test frames with $1$7cm / $1$8 errors, versus 35.6% for ACE (single), and EGFS (dual) reports 61.2% (Liu et al., 2024).

In CGDFS, the evaluation spans 12 tabular datasets, including 6 classification and 6 regression tasks constructed to exhibit distribution shift. The paper reports that CGDFS is best on all 6 classification datasets and all 6 regression datasets. Example values include Spam Base F1-macro $1$9, UCI Credit $\mathcal{S}$0, Activity $\mathcal{S}$1, and Boston Housing MSE $\mathcal{S}$2 versus ICP $\mathcal{S}$3 and LASSO $\mathcal{S}$4. The ablation also reports that removing the diffusion prior increases Boston Housing MSE from 15.1 to 36.3 (Malarkkan et al., 21 Mar 2026).

Taken together, these results indicate that EGFS has been effective in at least four distinct technical regimes: redundant indicator selection, nonlinear subset recovery, controlled high-dimensional inference, and geometry-aware sampling for localization. This suggests that the common ingredient is not a particular architecture but the decision to use task-relevant error as the supervisory signal for selection.

5. Robustness, stability, and common misconceptions

A recurring misconception is that EGFS is equivalent to minimizing in-sample predictive loss on pooled data. Some published EGFS instances do exactly that only in a restricted sense; others extend the guiding error far beyond ordinary empirical risk. CGDFS explicitly augments mean error with cross-environment error variance, so that the objective is $\mathcal{S}$5, not just single-distribution error. The paper states that causal invariance serves as a soft inductive bias rather than explicit causal discovery (Malarkkan et al., 21 Mar 2026).

A second misconception is that low-error points alone are sufficient. The SCR work directly tests this claim by comparing naïve thresholding of reprojection error against full EGFS masks. On Cambridge, the best direct thresholding gives about 20–21 cm / 0.4°, whereas EGFS masks yield 17 cm / 0.3°; on Indoor6, direct thresholds yield approximately 42–44%, while EGFS masks yield 56.1%. The reported interpretation is that low-error pixels may be spatially isolated, whereas EGFS gains from expanding seeds into coherent semantic regions via SAM and then pruning with confidence (Liu et al., 2024).

A third misconception is that error guidance and statistical control are opposed. Error-based knockoffs show the contrary: importance can be defined by prediction error shifts while still achieving finite-sample control of FDR, FDP, or $\mathcal{S}$6-FWER. The exact binomial null for irrelevant features is what allows this combination of model flexibility and controlled inference (Zhao et al., 2022).

A fourth misconception is that EGFS necessarily produces a single deterministic optimum. The NBC wrappers and NGP are deterministic search procedures, but CGDFS treats selection as approximate posterior inference over masks and aggregates posterior inclusion frequencies, while SCR updates masks iteratively as the model changes. This suggests that EGFS includes both point-estimate and uncertainty-aware formulations (Rabenoro et al., 2015, Das et al., 2022).

6. Limitations and research directions

The limitations reported in the literature are heterogeneous and domain-specific. In NBC-based EGFS, the error estimate is internal to the NBC and inherits its independence assumptions; the paper also notes that log conditional likelihood gives very poor results in this setting, which underscores the sensitivity of EGFS to the choice of error proxy (Rabenoro et al., 2015). In neural greedy pursuit, retraining a model for each candidate subset can be expensive, the method is greedy and does not remove early false positives, and its feature recovery exhibits a phase transition behavior that depends on the training data size (Das et al., 2022). In error-based knockoffs, the method requires knockoff generation, data splitting, and sufficiently accurate modeling of the covariate distribution; poor predictive models reduce power even when error control is preserved (Zhao et al., 2022). In SCR, mask quality depends on SAM and EfficientViT-SAM-L0, mask generation adds overhead, and the method remains scene-specific rather than cross-scene (Liu et al., 2024). In CGDFS, the authors note approximate inference, reliance on observed environments, dependence on the quality of the mask pool and prior, and scalability limits for “moderate-dimensional” tabular data (Malarkkan et al., 21 Mar 2026).

Several open directions are stated explicitly or follow naturally from these results. CGDFS asks how to design other error-based invariance terms beyond simple variance, how to learn the trade-off parameter $\mathcal{S}$7 adaptively, how to scale diffusion-based priors to very high-dimensional feature spaces, and how to incorporate task-specific constraints such as cost or fairness (Malarkkan et al., 21 Mar 2026). The SCR work suggests that the broader pattern of “error as a prompt” could transfer to depth estimation, SLAM / SfM, and other pixel-wise regression tasks where a meaningful per-pixel geometric error can be computed (Liu et al., 2024). Error-based knockoffs suggest a route toward EGFS methods that remain model-agnostic on $\mathcal{S}$8 while preserving rigorous inferential guarantees, especially if learned knockoff generators become sufficiently accurate (Zhao et al., 2022). Neural greedy pursuit suggests that error-guided subset selection remains attractive when $\mathcal{S}$9, but that better search schemes than purely forward greedy pursuit may be worth investigating (Das et al., 2022).

Across these directions, the stable common core of EGFS is clear: selection is delegated to an error signal that is deemed structurally relevant to the task, and the technical challenge is to convert that signal into a search, mask, posterior, or inferential procedure that is computationally tractable and robust under the regime of interest.