Active Selection of Classification Features

• ASCF is an adaptive feature selection method that differentiates between cheap selection variables and expensive classification features to maximize model utility.
• It employs sequential mechanisms—such as uncertainty-driven, mutual-information, and conditional-dependence strategies—to dynamically select features based on current model state and budget limitations.
• Empirical results demonstrate that ASCF improves early performance under limited acquisition budgets in domains like MRI and high-dimensional biological data, while also enabling class-aware and real-time feature selection.

Searching arXiv for recent and foundational papers on ASCF and related active feature selection. arxiv_search(query="\"Active Selection of Classification Features\" OR active feature selection classification", max_results=10, sort_by="relevance") Searching arXiv for the specific ASCF paper and closely related active feature selection work. arxiv_search(query="(Kok et al., 2021) OR (Schnapp et al., 2020) OR (Chen et al., 2021) OR (Mirzaei et al., 2023) OR (Zhang et al., 2014)", max_results=10, sort_by="relevance") Active Selection of Classification Features (ASCF) denotes a family of classification procedures in which feature choice is adaptive rather than fixed: the method sequentially selects features, class-specific feature subsets, or instances whose expensive classification features should be acquired, using current model state, auxiliary variables, or class-wise discriminative criteria to maximize downstream classification utility. In its explicit formulation, ASCF is the complementary problem in which a primary classifier $f:x\to y$ depends on expensive-to-acquire variables, while readily available selection variables are used to decide which instances should receive those expensive measurements (Kok et al., 2021). A broader literature instantiates the same idea through active mutual-information feature ranking, class-separability-driven sequential filters, test-phase feature acquisition, and active discovery of minimal gene panels; this suggests that ASCF is best understood as an umbrella for adaptive, utility-driven feature selection in classification (Schnapp et al., 2020, Zhang et al., 2014, Mirzaei et al., 2023, Chen et al., 2021).

1. Definition and problem boundary

In the explicit ASCF setting, the central distinction is between two feature spaces. The classifier is trained on expensive classification features, whereas selection is driven by cheap auxiliary variables already available for all candidate instances. The task is to choose those instances whose acquisition of expensive classification features and subsequent inclusion into the acquired set will improve the primary classifier most under a limited acquisition budget (Kok et al., 2021). This makes ASCF complementary to standard pool-based active learning, where labels are expensive but features are already available, and also distinct from ordinary active feature acquisition, where the same feature space is used both for selection and for prediction.

The same boundary also separates ASCF from global, class-independent feature selection. In class-independent selection, one global subset or ranking is returned for all classes. Class-specific selection instead allows each class $c$ to have its own subset $S_c \subseteq F$, and pairwise class-specific schemes allow distinct subsets for class pairs $(p,q)$ (Aguilar-Ruiz, 2024). In that sense, ASCF includes not only acquisition problems but also adaptive decompositions in which the relevant features depend on the current class hypothesis or ambiguity pattern.

A common misconception is that ASCF is synonymous with either active learning or ordinary feature selection. The literature instead separates at least three regimes: active selection of instances whose expensive features should be acquired, active use of a limited label budget to identify informative features, and test-time sequential selection of features for a particular instance (Kok et al., 2021, Schnapp et al., 2020, Mirzaei et al., 2023). The unifying element is adaptivity: feature relevance is not treated as static, but as contingent on the current selected set, the current uncertainty state, or the current class decomposition.

2. Formal settings and utility functions

The canonical ASCF formulation maintains a candidate pool and an acquired set. The acquired set contains instances for which both cheap selection variables and expensive classification features are known; the candidate pool contains instances for which only the cheap variables are known. The primary classifier operates on the expensive feature space, and an auxiliary regressor $h$ maps cheap variables to expensive ones. At each iteration, ASCF selects the candidate $j$ maximizing a utility $U_j$, acquires its expensive features, moves it into the acquired set, retrains, and repeats (Kok et al., 2021).

Two concrete utilities have been proposed in this setting. U-ASCF is unsupervised and uses the average feature-level variance across a bootstrap ensemble of regressors $h_\theta:\sel\to\cf$; candidates whose expensive features are most uncertain under imputation are preferred. S-ASCF is supervised and combines a probabilistic classifier $f:\cf\to y$ with the regressor $h$, using the utility

$c$0

where $c$1 is the estimated misclassification probability and $c$2 (Kok et al., 2021). The first criterion is uncertainty-driven; the second is misclassification-driven.

A second formal setting replaces expensive feature acquisition by expensive labels. In active feature selection for the mutual-information criterion, an unlabeled sample $c$3 is available, a labeling oracle may be queried at most $c$4 times, and the goal is to select a subset of $c$5 features with large mutual information $c$6 while using far fewer labels than the dataset size (Schnapp et al., 2020). Here the active decision is not which expensive feature vector to acquire, but which labels to request so that the top-$c$7 features under the mutual-information criterion can be identified efficiently.

A third formal setting is per-instance test-phase acquisition. Given training data, a budget $c$8, and a test instance, the objective is to find a subset $c$9 such that

$S_c \subseteq F$0

with uniform feature costs in the reported implementation (Mirzaei et al., 2023). This is still ASCF in the sense that the feature subset used for classification is chosen sequentially and adaptively, but the adaptation occurs at inference time rather than during dataset construction.

3. Sequential selection mechanisms

One major ASCF mechanism is active mutual-information estimation. The algorithm AFS maintains empirical estimates of conditional entropies $S_c \subseteq F$1, exact Clopper–Pearson confidence intervals for the Bernoulli parameters $S_c \subseteq F$2, and upper and lower confidence bounds $S_c \subseteq F$3. At each round it constructs a disputed feature set

$S_c \subseteq F$4

where $S_c \subseteq F$5 is the current top-$S_c \subseteq F$6 set and $S_c \subseteq F$7 is an optimistic–pessimistic alternative using the confidence bounds. Example selection is then driven by a score that combines feature-wise allocation weights with pairwise bias correction terms $S_c \subseteq F$8, and the selected label updates all feature estimates simultaneously (Schnapp et al., 2020). The design is explicitly inspired by combinatorial pure-exploration bandits.

A second mechanism is sequential conditional-dependence ranking. DEA-CS treats each feature as a decision-making unit, decomposes the class variable into binary one-versus-complement labels $S_c \subseteq F$9, computes conditional dependence scores

$(p,q)$0

and evaluates the resulting output vector with super-efficiency DEA. The selected feature is the one with maximal efficiency score, and the conditioning set $(p,q)$1 is then updated for the next iteration (Zhang et al., 2014). This produces a forward, data-driven filter in which relevance and redundancy are recomputed after each feature choice.

A third mechanism is test-phase sequential acquisition. In fast classification with sequential feature selection, the next feature is chosen by Fisher score on the current filtered training set, the value of that feature is acquired for the test instance, and the training set is then filtered by Euclidean distance on the already acquired features. This loop continues until the feature budget is exhausted or the filtered training set becomes empty; prediction is the most frequent label among the remaining training instances (Mirzaei et al., 2023). The method is explicitly instance-wise: different test points can induce different acquisition orders and different subsets.

These mechanisms differ in what is actively sampled—labels, features, or instances—but all share the same structural pattern. A current state is maintained, the utility of each candidate feature is evaluated conditional on that state, a single choice is made, and the state is updated before the next decision. That recurrent dependence on already selected information is the defining procedural trait of ASCF.

4. Class-aware and weak-supervision formulations

Class-aware ASCF replaces a single global relevance score with class-indexed or class-pair-indexed structures. In class-specific feature selection, each feature receives a different relevance score for each class, producing per-class subsets $(p,q)$2. One-versus-all and one-versus-each constructions yield class-by-feature matrices, while the Deep One-versus-Each strategy retains all pairwise relevance scores $(p,q)$3 instead of aggregating them away. From these pairwise scores, a class-specific relevance matrix $(p,q)$4 is built, with diagonal entries $(p,q)$5 containing features relevant for class $(p,q)$6 against all others and off-diagonal entries $(p,q)$7 containing additional features specific to the pair $(p,q)$8 (Aguilar-Ruiz, 2024). This supports decomposable two-layer and three-layer classification schemes.

DEA-CS reaches a related class-aware effect through a different route. Instead of a single $(p,q)$9 or $h$0, it retains per-label conditional dependence scores and lets DEA combine them as multiple outputs. This means that a feature that is highly discriminative for a subset of classes can remain highly ranked even when a scalar aggregate would dilute that effect (Zhang et al., 2014). In both cases, class separability is treated as structured and heterogeneous rather than globally homogeneous.

A weak-supervision perspective appears in the formulation that labels the features rather than the samples. Here the only required supervision is the sign of each feature—whether it has, on average, stronger values over positive samples than over negatives. Features with negative empirical sign are flipped by $h$1. Classifier learning then solves either a supervised constrained least-squares problem

$h$2

or an almost unsupervised covariance-maximization problem

$h$3

subject to $h$4 and $h$5 (Leordeanu et al., 2015). The KKT analysis shows that, generically, the optimal solution has exactly $h$6 non-zero entries, all equal to $h$7. This yields a sparse equal-weight ensemble selected with extremely limited label information.

Taken together, these class-aware and weak-supervision variants broaden ASCF beyond simple forward search. They show that adaptive feature selection can be conditioned on class identity, class-pair ambiguity, or even a 1-bit supervision signal per feature, while still producing compact discriminative subsets and decomposable decision structures.

5. Empirical behavior and application domains

The empirical profile of ASCF is strongly budget-dependent. In the explicit expensive-feature-acquisition setting, both U-ASCF and S-ASCF often improve over random acquisition early in the learning curve, when only a small number of expensive instances can be collected. On the MRI case study, S-ASCF is reported as consistently and significantly better than random sampling for almost all acquisition steps in both dataset configurations, reaching peak performance much earlier than random selection; U-ASCF also improves over random, but less consistently (Kok et al., 2021). This is precisely the regime in which acquisition-aware feature selection has practical value.

Under limited label budgets, active feature selection for mutual information shows a similar pattern. AFS significantly outperforms RANDOM, CORESET, and DWUS for moderate and large $h$8, especially at small-to-moderate label budgets, while for $h$9 it is often similar to RANDOM and CORESET (Schnapp et al., 2020). The relevant point is not only that fewer labels are used, but that label queries are directed toward the features whose ranking relative to the top-$j$0 set is still disputed.

In very high-dimensional biological data, ActiveSVM identifies minimal gene sets by repeatedly examining misclassified cells. The method is reported to enable $j$1 cell-type classification accuracy across a variety of datasets, and on the PBMC dataset it achieves at least $j$2 test accuracy with only 15 genes while using only 298 unique cells (Chen et al., 2021). It also scales to a mouse brain dataset with over a million cells because it never needs to use all genes and all cells simultaneously during selection.

At test time, sequential feature acquisition can trade accuracy against latency. The lazy sequential method for active feature acquisition is reported to achieve competitive accuracy results compared to existing methods while significantly outperforming them in terms of speed, with average test-time per instance orders of magnitude below the reported RL baselines on the CUBE, Forest, and Synthetic datasets (Mirzaei et al., 2023). This shows that ASCF is not restricted to training-set design; it can also be a deployment-time mechanism.

Class-aware ASCF is additionally motivated by explainability. Class-specific methods can support explanations of the form “these features are why the model predicts class $j$3,” and the CSR matrix provides a direct bridge from feature selection to decomposable multiclass architectures (Aguilar-Ruiz, 2024). This suggests that ASCF is relevant not only for cost reduction but also for local interpretability and modular error analysis.

6. Limitations, statistical cautions, and open directions

ASCF inherits limitations from each of its concrete instantiations. In expensive-feature acquisition, the method depends on how well cheap selection variables predict expensive classification features; if the auxiliary regression $j$4 is weak, both U-ASCF and S-ASCF degrade. S-ASCF also assumes labels are already available for all candidates, all acquisitions are treated as equal-cost, and the acquired set is intentionally biased toward informative instances rather than population representativeness (Kok et al., 2021). These are structural assumptions, not implementation details.

In sequential filter formulations, the conditioning set can become statistically problematic. DEA-CS explicitly notes sample inefficiency as the selected set grows: reliable estimation of $j$5 can require exponentially more data, many conditional mutual information estimates become zero, and the algorithm may stop prematurely. The method also incurs linear-program overhead, and the radial input-oriented super-efficiency DEA measure does not consider slacks (Zhang et al., 2014). In mutual-information AFS, the paper provides no PAC-style guarantee on recovering the true top-$j$6 features and does not address redundancy beyond the univariate mutual-information criterion (Schnapp et al., 2020).

Test-phase ASCF has its own restrictions. The fast lazy method assumes uniform feature costs, uses Fisher scores that are univariate, and has no explicit confidence-based stopping rule; acquisition continues until the budget is exhausted or the filtered training set becomes empty (Mirzaei et al., 2023). Class-specific selection remains architecturally incomplete as well: surveyed literature is described as typically implementing an $j$7 strategy—$j$8 subsets and $j$9 classifiers—while no surveyed method realizes a $U_j$0 strategy, a single classifier that internally uses different feature subsets per class (Aguilar-Ruiz, 2024).

A more fundamental caution concerns finite-data ROC-based subset ranking. For binary classification on finite alphabets, performance estimation for a fixed classifier can be made arbitrarily accurate given sufficient data, independent of feature-set size, but the Neyman–Pearson likelihood-ranking procedure used to recover the ROC is highly sensitive to density-estimation errors. In the worst case, guaranteeing that the estimated performance curve equals the true ROC may require data exponential in the size of the feature set (Coetzee et al., 2013). For ASCF, this means that large-subset ranking based on NP-designed ROCs can become statistically meaningless even when the performance of any given classifier can still be estimated accurately.

The open directions exposed by the literature are therefore coherent. They include formal decision policies for when to move from diagonal to pairwise class-specific features, integration with sequential feature acquisition and dynamic early exit, heterogeneous feature and label costs, better handling of feature redundancy in active mutual-information frameworks, and scalable compression of class-pair relevance structures in very large $U_j$1 and $U_j$2 regimes (Aguilar-Ruiz, 2024, Schnapp et al., 2020, Kok et al., 2021). A plausible implication is that future ASCF systems will combine multiple strands already present in the literature: utility-based acquisition, class-specific structure, uncertainty-aware sampling, and explicit budgeted stopping rules.