Ensemble Feature Selection Overview

Updated 6 July 2026

EFS is the ensemble analogue of feature selection that aggregates multiple selector outputs to improve robustness and reproducibility.
It employs strategies like data and function perturbation to create homogeneous, heterogeneous, and hybrid ensembles for diverse research applications.
EFS enhances predictive performance in fields such as biomarker discovery, remote sensing, and cybersecurity while mitigating instability in high-dimensional datasets.

Ensemble Feature Selection (EFS) is the feature-selection analogue of ensemble learning: rather than relying on a single selector or a single run of one selector, it combines multiple feature-selection outputs or multiple selected feature subsets to obtain a final ranking or subset that is intended to be more robust than any individual constituent. In the literature, EFS is motivated primarily by instability of feature selection under high dimensionality, small-sample regimes, class imbalance, censoring, and correlated variables, and by the observation that distinct selectors or distinct data perturbations often return different but plausible feature sets. Reported objectives therefore include improved stability, maintained or improved predictive performance, better reproducibility for biomarker discovery, and, in some formulations, deliberate induction of diversity among downstream base learners (Zhang et al., 2018, Colombelli et al., 2021, Jenul et al., 2021).

1. Definition and scope

A recurrent formulation in the literature distinguishes two broad EFS strategies. Data perturbation applies the same selector repeatedly to perturbed versions of the data, typically bootstrap resamples, and aggregates the resulting ranked lists or selected subsets. Function perturbation applies multiple different selectors to the same dataset and aggregates their outputs. These are often termed homogeneous and heterogeneous ensemble feature selection, respectively; hybrid schemes combine both (Zhang et al., 2018).

A second, related usage treats EFS not as an ensemble of selectors but as a mechanism for constructing diverse base learners through different feature subsets. In land-cover mapping, for example, the same Gaussian SVM was trained on different Landsat band subsets, and the resulting maps were fused by majority vote; here the selected feature subsets are the source of classifier diversity rather than the final object of aggregation (0811.2016). A further extension appears in end-to-end deep systems, where multiple feature-selection models are trained jointly and regularized to discover sparse, minimally overlapping groups of predictive features without predefined groups (Imrie et al., 2022).

Strategy	Mechanism	Representative papers
Homogeneous EFS	Same selector on bootstrap or resampled datasets, then aggregation	(Zhang et al., 2018, Spooner et al., 2022)
Heterogeneous EFS	Different selectors on the same data, then aggregation	(Zhang et al., 2018, Polewko-Klim et al., 2020)
Hybrid EFS	Data perturbation and function perturbation in a two-level design	(Colombelli et al., 2021, Zhang et al., 2018)
Feature-subspace ensemble construction	Different selected feature subsets define diverse base learners	(0811.2016, Madasu et al., 2019)
Model-integrated or end-to-end EFS	Feature selection learned jointly with downstream ensemble behavior	(Grósz et al., 2022, Imrie et al., 2022, Jenul et al., 2021, Stanchev et al., 1 Jul 2026)

This breadth suggests that EFS is better understood as a family of ensemble principles applied to feature selection than as a single algorithmic template.

2. Principal design strategies

In homogeneous EFS, repeated resampling is used to expose the variability of a selector. EFSIS generates $M$ bootstrap datasets, applies each ranker to every bootstrap sample, estimates a stability score $S_n$ for each ranker, and then aggregates rankings with a stability-weighted rank product. Its final cross-ranker aggregation is written as

$R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$

where $R_{f,n}$ is the aggregated rank score of feature $f$ from ranker $n$ and $S_n$ is the stability of that ranker (Zhang et al., 2018).

Hybrid schemes extend this idea by injecting both data-level and method-level diversity. In Hyb-EFS for transcriptomic biomarker discovery, $n=50$ bootstrap samples are combined with $m=5$ feature-selection methods, producing $n \times m$ rankings. The first aggregation stage combines methods within each bootstrap; the second aggregation stage combines the resulting bootstrap-level consensus rankings. The design was introduced to “disrupt associations of good performance with a single dataset, single algorithm, or a specific combination of both,” thereby improving reproducibility of candidate biomarkers (Colombelli et al., 2021).

Other formulations treat the ensemble outputs as probabilistic evidence. UBayFS constructs an ensemble of $S_n$ 0 elementary feature selectors, aggregates their binary outputs into $S_n$ 1, and places a Dirichlet-type prior on latent feature importances $S_n$ 2. The posterior is

$S_n$ 3

so user knowledge enters as prior pseudo-counts while ensemble votes contribute through a multinomial likelihood (Jenul et al., 2021).

Deep-learning variants replace explicit rank aggregation by joint end-to-end training. CompFS uses an ensemble of feature-selection models with relaxed Bernoulli gates, per-group sparsity penalties, an ensemble prediction loss, and an inter-group overlap penalty to recover predictive groups rather than merely individual variables (Imrie et al., 2022). In paralinguistics, an ensemble of deep models is first trained on the full feature set; feature importance is then estimated by output gradients or loss gradients, and a smaller ensemble is retrained on the selected subset (Grósz et al., 2022).

3. Aggregation, thresholding, and stability

Aggregation is the central operation in EFS, and the literature uses several families of operators. Rank-product aggregation appears in EFSIS; Borda aggregation appears in Hyb-EFS, where the score for feature $S_n$ 4 is

$S_n$ 5

with $S_n$ 6 the rank position of feature $S_n$ 7 in ranking $S_n$ 8. Hyb-EFS also introduces a stability-weighted aggregation variant in which each ranking’s contribution is weighted by a transformed Kuncheva Index (Colombelli et al., 2021).

In clinical survival analysis, homogeneous ensembles of six base selectors are aggregated by Mean Rank, Mean Weight, Robust Rank Aggregation (RRA), the Threshold Algorithm (TA), or MedRank (MA). RRA simultaneously aggregates and thresholds by assigning $S_n$ 9-values and selecting features with $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 0. TA and MA are sequential algorithms that maintain top- $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 1 candidates, with $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 2 set to the mean length of the 50 feature subsets generated in each ensemble (Spooner et al., 2022).

Thresholding is a separate problem in many EFS pipelines. A fixed threshold on aggregated ranks or scores can be operationally convenient, but it offers no guarantee that the final feature set contains only relevant features. Data-driven alternatives therefore use the empirical distribution of ensemble scores. In Alzheimer’s disease biomarker discovery, three such thresholds were developed: a 75% quantile threshold, a kernel density estimation threshold that treats the largest density peak as the irrelevant-feature cluster and selects features above the next local minimum, and a random probes threshold that retains features ranked above the best permuted probe (Spooner et al., 2022).

Stability is measured in several distinct ways. Hyb-EFS uses the Kuncheva Index for pairwise subset consistency across resamples; a value above 0.5 is described as indicating high stability (Colombelli et al., 2021). Fast stability estimation for ensemble selectors uses pairwise Jaccard similarity,

$R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 3

and models ensemble behavior with a simulator parameterized by the number of useful features $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 4 and a selector uncertainty parameter $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 5 (Onda et al., 2021). In survival analysis, Somol and Novovičová’s relative weighted consistency is used because subset sizes can vary across ensemble runs (Spooner et al., 2022). By contrast, the land-cover study used pairwise Cohen’s $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 6 agreement and its variance as proxies for “in-ensemble diversity,” with diversity inferred inversely from agreement (0811.2016).

4. Coupling EFS with predictive models

Many EFS systems are not standalone rankers but front ends to downstream predictive ensembles. In phishing detection, Information Gain ranks 87 URL, content, and external-service features; the top 20 are retained; five classifiers—RF, KNN, SVM, LR, and NB—are retrained on the reduced set; and a weighted voting ensemble combines their class probabilities using weights derived from training AUC on the selected features (Bidabadi et al., 2022).

In pan-cancer transcriptome classification, a multi-view feature-selection framework partitions 36,017 RNA features by biological type, applies Boruta to each partition, merges confirmed features, applies Boruta again, repeats the entire process across 12 values of Random Forest max_depth, and ranks genes by occurrence frequency across runs. The resulting ranked sets Rank12 through Rank1 are then evaluated with LR, SVM, XGBoost, KNN, RF, and two parallel ensembles: mvEns based on max voting and avEns based on averaged class probabilities (Chowdhury et al., 12 Jan 2025).

Feature-subspace ensembling is particularly explicit in remote sensing and text classification. In land-cover mapping, exhaustive search over Landsat band combinations is scored by Bhattacharyya distance, divergence, or transformed divergence; the best five band combinations under each criterion are used to define five Gaussian SVM base classifiers; and majority voting produces the final land-cover map (0811.2016). In sentiment analysis, the pipeline is TF-IDF $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 7 feature selection $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 8 classifier training $R_f=\prod_{n=1}^{N}(R_{f,n})^{(1-S_n)},$ 9 optional ensemble, and Random Subspace is the most EFS-like method because it trains each learner on a different feature subset of the selected vocabulary (Madasu et al., 2019).

Other works embed feature selection more deeply in the predictive architecture. The cardiotocography study first combines feature selectors—most successfully Information Gain with Correlation-based feature selection—and then trains multiple polynomial-kernel SVMs whose outputs are fused by majority voting (Silwattananusarn et al., 2020). In intrusion detection, CFS-BA performs feature-space reduction and then feeds a heterogeneous ensemble of C4.5, Random Forest, and Forest PA whose probability vectors are combined by average-of-probabilities voting (Zhou et al., 2019). Time-series forecasting introduces a hierarchical ensemble-based method in which one feature subset generates an initial prediction and later subsets sequentially rescale and refine that prediction, explicitly modeling feature co-dependency across hierarchy levels (Tumay et al., 2023).

A more unusual adaptation appears in quantum benchmarking. There, EFS does not rank covariates but selects a compact subset of experimentally executable survival-probability circuits from a pool of 8192 candidates, and ridge regression learns a linear estimator of process infidelity for involutory multi-qubit gates, including non-Clifford CCZ (Stanchev et al., 1 Jul 2026).

5. Empirical behavior across application domains

Reported empirical patterns are mixed but informative. In biomarker discovery and high-dimensional omics, hybrid and stability-aware ensembles are repeatedly associated with improved reproducibility. EFSIS is reported to give both high prediction accuracy and stability across six microarray cancer datasets (Zhang et al., 2018). Hyb-EFS and Het-EFS attenuate the large performance variation observed for most single selectors and homogeneous ensembles across distinct datasets, and the hybrid design yields greater biological plausibility, including strong enrichment for cancer-related genes and pathways (Colombelli et al., 2021). The TCGA pan-cancer study reports its best performance with Rank10 features and avEns, achieving 97.11% accuracy, 0.9996 weighted AUC, and an average accuracy of 90.33% on a difficult 12-cancer subset (Chowdhury et al., 12 Jan 2025).

Security applications emphasize dimensionality reduction and detection efficiency. EMFFS combines Information Gain, Gain Ratio, Chi-squared, and ReliefF by retaining only features appearing in at least three of four top-14 lists, reducing NSL-KDD from 41 to 13 features and achieving 99.67% accuracy, 99.76% detection rate, and 0.78 s model-building time with J48 (1807.10443). A related hybrid filter-wrapper design on KDDCUP99 uses Information Gain to order features and a KNN wrapper with $R_{f,n}$ 0 to retain only additions that improve accuracy, with reported gains from roughly the low-70% range using all 41 features to roughly the low-90% range using selected subsets (0912.1014). CFS-BA-Ensemble reports 10, 8, and 13 selected features on NSL-KDD, AWID, and CIC-IDS2017, respectively, together with large gains in accuracy and major reductions in model-building time relative to the full feature sets (Zhou et al., 2019).

In NLP, speech, and forecasting, EFS often functions as a computationally efficient alternative to larger architectures. For sentiment analysis, Chi-Square and Count Difference are the strongest feature-selection methods, MNB and LR are the strongest base learners, and Random Subspace > Bagging across datasets and metrics (Madasu et al., 2019). In paralinguistics, output-gradient-selected spectral features produce 25–32% reductions in inference time while maintaining competitive results with larger-footprint neural ensembles (Grósz et al., 2022). In phishing detection, the top-20 Information Gain subset substantially improves weaker learners, especially SVM, which rises from 44.6% to 91.64% accuracy after feature selection and standardization (Bidabadi et al., 2022). In non-stationary time-series forecasting, the hierarchical ensemble-based method is reported to perform best on both synthetic and M4 hourly data and to be much faster than wrapper search (Tumay et al., 2023).

These results suggest that EFS is particularly attractive when interpretability, computational efficiency, and reproducibility are valued alongside predictive performance.

6. Limitations, controversies, and terminological ambiguity

The EFS literature does not support a universal claim that ensembling feature selection automatically improves prediction. On correlated high-dimensional TCGA RNA-seq data, models built on feature sets obtained from an ensemble of four selectors were no better than models developed on feature sets obtained from individual algorithms, even though the best selector varied by dataset (Polewko-Klim et al., 2020). In land-cover mapping, ensemble accuracy correlated positively with pairwise classifier agreement ( $R_{f,n}$ 1) and negatively with the variance of pairwise agreement ( $R_{f,n}$ 2), so “more measured diversity” was associated with lower accuracy rather than higher accuracy (0811.2016). The cardiotocography study likewise states that feature selection does not automatically improve performance, and that the best result arose only for a specific combination of Information Gain, Correlation-based feature selection, and SVM ensemble (Silwattananusarn et al., 2020).

Threshold choice and computational cost are persistent difficulties. Fixed thresholds are described as fragile and dataset-dependent in survival-analysis EFS, while data-driven thresholds such as RRA, KDE, and random probes can behave differently depending on sparsity, correlation structure, and selector type (Spooner et al., 2022). Hybrid and stability-aware frameworks are computationally expensive because they combine multiple bootstraps, multiple selectors, and multiple aggregation stages; fast stability estimation has therefore been proposed to reduce the number of real selector runs from $R_{f,n}$ 3 to $R_{f,n}$ 4 by calibrating a simulator of weak selectors (Onda et al., 2021).

A further source of confusion is terminological. In building energy simulations, EFS explicitly means Exhaustive Feature Selection, a wrapper that tests all combinations of features with LDA accuracy as the criterion (Shahsavari et al., 2021). In subspace clustering, EFS means Exact Feature Selection, namely recovery of a support set containing only points from the same subspace as the query point, analyzed for OMP and sparse recovery methods (Dyer et al., 2013). In the ensemble-feature-selection literature proper, by contrast, EFS denotes aggregation across selectors, resamples, or feature subsets. Distinguishing these usages is essential, because the underlying objectives, mathematical formulations, and evaluation criteria are entirely different.