Infinite Feature Selection

• Infinite Feature Selection (Inf-FS) is a graph-based method that represents features as nodes in a weighted graph, scoring them by summing infinite-length paths to capture relevance and redundancy.
• It employs matrix power-series identities and absorbing Markov chain concepts to compute a closed-form solution, efficiently aggregating feature interactions without explicit subset enumeration.
• Extensions like Supervised IFS and ILFS integrate label information and latent-variable models, enhancing performance in diverse applications from classification benchmarks to dynamic visual tracking.

Searching arXiv for relevant papers on Infinite Feature Selection and closely related variants. Infinite Feature Selection (Inf-FS) is a graph-based filtering approach to feature selection in which features are represented as nodes of a weighted graph and feature subsets are interpreted as paths on that graph. Its central construction evaluates paths of arbitrary length, including the limit to infinitely long walks, by exploiting matrix power-series identities or, in a later formulation, absorbing Markov chain fundamentals. The resulting ranking assigns a score to each feature according to its participation in many high-weight paths, thereby operationalizing the joint criteria of relevance and redundancy in a closed form (Roffo et al., 2020). Subsequent work introduced supervised and improved unsupervised variants, including Supervised Infinite Feature Selection (SIFS) (Eskandari et al., 2017), and related latent-variable extensions such as Infinite Latent Feature Selection (ILFS), which learns edge weights through a PLSA-inspired generative model (Roffo et al., 2017). Inf-FS has also been embedded into application-specific systems, including dynamic visual tracking pipelines (Roffo et al., 2016).

1. Core formulation

Given a set of features $F=\{f_1,\dots,f_m\}$, Inf-FS forms a complete undirected graph $G=(V,E)$ with one node per feature and an adjacency matrix $A\in\mathbb{R}^{m\times m}$ whose entries are pairwise energies,

$a_{ij}=e(\{f_i,f_j\}),$

where $e(\cdot)$ is a pairwise measure such as correlation or mutual information (Eskandari et al., 2017). In the 2020 graph-based formulation, the same idea is written as

$A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$

with $\varphi$ encoding pairwise relevance and redundancy relations between features (Roffo et al., 2020).

For a fixed walk length $\ell$, the score of feature $f_i$ is obtained from the $\ell$-th matrix power: $G=(V,E)$0 which counts all possibly cyclic walks of length $G=(V,E)$1 starting at $G=(V,E)$2 (Eskandari et al., 2017). The decisive step in Inf-FS is to sum over all path lengths rather than fixing a subset cardinality. Introducing a damping factor $G=(V,E)$3 such that $G=(V,E)$4, where $G=(V,E)$5 denotes the spectral radius, yields the convergent series

$G=(V,E)$6

The final feature score vector is then

$G=(V,E)$7

and ranking features by descending $G=(V,E)$8 produces the Inf-FS ordering (Eskandari et al., 2017).

The 2020 exposition gives the equivalent form

$G=(V,E)$9

and interprets higher scores as evidence that a feature participates in many high-weight paths, meaning that it is both relevant and non-redundant (Roffo et al., 2020). This suggests that Inf-FS should be understood not as a local pairwise filter alone, but as a global-ranking scheme in which each feature is evaluated through its position in a recursively weighted network of feature interactions.

2. Interpretations of the infinite-path score

The original and later formulations emphasize a matrix power-series interpretation. Any path $A\in\mathbb{R}^{m\times m}$0 of length $A\in\mathbb{R}^{m\times m}$1 has weight equal to the product of its edge weights, and summing the weights of all length-$A\in\mathbb{R}^{m\times m}$2 paths between two nodes gives $A\in\mathbb{R}^{m\times m}$3 (Roffo et al., 2020). The infinite series over $A\in\mathbb{R}^{m\times m}$4 captures the contribution of subsets of all sizes without explicit combinatorial enumeration.

A second interpretation, formalized in the 2020 treatment, views $A\in\mathbb{R}^{m\times m}$5 as the transient-to-transient transition matrix of an absorbing Markov chain after suitable row-stochastic normalization. The canonical transition matrix is

$A\in\mathbb{R}^{m\times m}$6

and its fundamental matrix is $A\in\mathbb{R}^{m\times m}$7. Up to an additive identity, this coincides with $A\in\mathbb{R}^{m\times m}$8, so $A\in\mathbb{R}^{m\times m}$9 can be read as the expected number of visits to state $a_{ij}=e(\{f_i,f_j\}),$0 starting from $a_{ij}=e(\{f_i,f_j\}),$1 before absorption (Roffo et al., 2020). Under this interpretation, the ranking reflects expected visit counts through the graph induced by feature relations.

The formulation also explains why infinite-length aggregation is computationally tractable. Rather than enumerating subsets or truncating at a chosen path length, the method reduces the entire aggregation to a single matrix inverse. In the theoretical overview accompanying SIFS, computing $a_{ij}=e(\{f_i,f_j\}),$2 takes $a_{ij}=e(\{f_i,f_j\}),$3 time with fast algorithms, or $a_{ij}=e(\{f_i,f_j\}),$4 naively, and requires $a_{ij}=e(\{f_i,f_j\}),$5 memory (Eskandari et al., 2017). The 2020 graph-based paper states a total cost of $a_{ij}=e(\{f_i,f_j\}),$6, where $a_{ij}=e(\{f_i,f_j\}),$7 is the number of samples, because adjacency construction and inversion dominate runtime (Roffo et al., 2020).

3. Adjacency design: relevance and redundancy

The practical behavior of Inf-FS depends primarily on how the adjacency matrix is defined. In the 2020 formulation, two variants are specified explicitly (Roffo et al., 2020).

For unsupervised Inf-FS$a_{ij}=e(\{f_i,f_j\}),$8,

$a_{ij}=e(\{f_i,f_j\}),$9

where $e(\cdot)$0 is the normalized standard deviation of feature $e(\cdot)$1, and $e(\cdot)$2 controls the trade-off between relevance and redundancy (Roffo et al., 2020).

For supervised Inf-FS$e(\cdot)$3, each feature receives a scalar summary

$e(\cdot)$4

followed by the combination

$e(\cdot)$5

Here, Fisher-type separation, normalized mutual information with the label, and marginal variability are combined into a feature importance prior, which is then lifted to pairwise affinities by an outer product (Roffo et al., 2020).

The SIFS paper proposed a different supervised adjacency specifically aimed at incorporating label relevance via mutual information and penalizing redundancy via Spearman’s rank correlation: $e(\cdot)$6

$e(\cdot)$7

This construction departs from the outer-product form and instead defines edge weights directly from pairwise combinations of relevance and redundancy (Eskandari et al., 2017).

The same work also introduced an improved unsupervised adjacency, denoted mIFS, based on feature standard deviation and average mutual information: $e(\cdot)$8

$e(\cdot)$9

Empirically, the paper reports that normalizing features into $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$0, using MI-based $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$1, and tuning $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$2 gives substantial gains over the original IFS adjacency in unsupervised settings (Eskandari et al., 2017).

4. Supervised and latent-variable extensions

SIFS extends the original Infinite Feature Selection to supervised learning while retaining the same infinite-walk scoring machinery. The algorithmic outline is: preprocess data, compute feature-feature and feature-label statistics, build the adjacency matrix, scale $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$3 so that $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$4, compute $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$5, multiply by the all-ones vector, and rank features in descending order of the resulting scores (Eskandari et al., 2017). For supervised learning, the practical recommendation is to standardize features, apply the supervised adjacency, tune the trade-off $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$6 by cross-validation with typical values $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$7–$A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$8, and scale the adjacency matrix by $A(i,j)=\varphi(\mathbf v_i,\mathbf v_j)>0,$9 or similarly to ensure convergence (Eskandari et al., 2017).

A distinct but closely related development is ILFS, which retains the infinite-path ranking but replaces hand-crafted edge weights with weights learned from a supervised latent-variable model (Roffo et al., 2017). ILFS introduces a binary latent variable

$\varphi$0

representing relevancy versus irrelevancy, and assumes

$\varphi$1

The posterior relevance of a feature is

$\varphi$2

leading to the closed-form adjacency

$\varphi$3

The ranking stage is then

$\varphi$4

which is formally analogous to Inf-FS (Roffo et al., 2017).

ILFS estimates its latent parameters with EM over token-feature counts $\varphi$5, using E-step and M-step updates for $\varphi$6, $\varphi$7, and $\varphi$8, and iterates until the likelihood increase falls below a small threshold or after a fixed number of steps (Roffo et al., 2017). The paper explicitly contrasts ILFS with original Inf-FS: both methods rank features by summing contributions of all walks through a node, but Inf-FS uses an unsupervised hand-crafted edge weight whereas ILFS learns posterior relevancy in a supervised PLSA-style framework (Roffo et al., 2017).

A plausible implication is that the Inf-FS family is better described as a scoring architecture than as a single adjacency prescription. Across these variants, the invariant element is the infinite-walk ranking operator; the changing element is the definition or estimation of edge weights.

5. Algorithmic procedure and subset extraction

Across the formulations in the provided literature, the operational pipeline is stable. In SIFS and mIFS, the recommended sequence is: normalize or standardize data as appropriate, compute the relevant statistics, build $\varphi$9, choose $\ell$0, compute $\ell$1, multiply by the all-ones vector, and sort the scores (Eskandari et al., 2017). The 2020 paper gives the same high-level pseudocode for Inf-FS$\ell$2, with $\ell$3, followed by sorting and a mean-shift step on the score vector (Roffo et al., 2020).

The subset-selection strategy is an important addition in the 2020 treatment. Once the score vector $\ell$4 has been sorted in descending order, the method applies one-dimensional mean-shift clustering to the score values and retains the cluster containing the highest-scoring feature. All features in that cluster form the final subset (Roffo et al., 2020). This differs from protocols in which the number of retained features is fixed a priori, and it addresses a recurring practical issue in filter methods: converting a ranking into a subset without wrapper-style search.

For very large feature sets, the SIFS paper notes a limitation directly tied to the closed-form computation: for $\ell$5, $\ell$6 memory becomes a concern, and blockwise or low-rank approximations should be considered (Eskandari et al., 2017). This suggests that the main scalability bottleneck of Inf-FS lies not in the infinite-path idea itself, but in dense adjacency storage and inversion.

6. Empirical evaluation and reported performance

The empirical record reported in the supplied papers spans supervised and unsupervised classification, image recognition, text and genomics benchmarks, and real-time visual tracking.

In the SIFS study, evaluation was conducted on 14 high-dimensional benchmarks, including USPS, GINA, Gisette, Colon, Lung181, DLBCL, Prostate, Arcene, REGED0, MARTI0, Madelon, Sido0, and PASCAL VOC 2007/2012, using a linear SVM with 5-fold cross-validation for $\ell$7 and $\ell$8. Metrics were classification accuracy, reported as average and maximum over top-$\ell$9 features, and mean Average Precision for VOC (Eskandari et al., 2017). On 12 binary datasets, the reported average accuracies were:

Method	Avg accuracy
IFS (original)	83.97%
mRMR	88.72%
mIFS	85.75%
SIFS	90.61%

The paper states that SIFS is better than mRMR on 10 of 12 datasets and improves by $f_i$0 on average, while using supervised label information in the adjacency yields approximately a $f_i$1 absolute gain over original IFS (Eskandari et al., 2017). On PASCAL VOC 2007, the reported mAP values are no-FS $f_i$2, mRMR $f_i$3, and SIFS $f_i$4; on VOC 2012, they are no-FS $f_i$5, mRMR $f_i$6, and SIFS $f_i$7 (Eskandari et al., 2017). For unsupervised tasks, replacing Spearman redundancy with MI-based redundancy plus STD relevance is reported to outperform vanilla IFS by up to $f_i$8 on some datasets (Eskandari et al., 2017).

The 2020 graph-based Inf-FS paper broadens the benchmark range to 11 datasets, including DNA microarrays, NIPS-challenge datasets, and CNN features from PASCAL VOC 2007 and Caltech-101, and compares against 18 methods covering filters, wrappers, embedded methods, and graph-based approaches (Roffo et al., 2020). The protocol uses 70% train / 30% test splits, cross-validation of classifier $f_i$9, repeated 20 times with random splits, and reports mean accuracy or mean Average Precision (Roffo et al., 2020). The main findings reported are that Inf-FS$\ell$0 outperforms other unsupervised filters in nearly all settings, Inf-FS$\ell$1 consistently leads supervised filters across a wide range of feature budgets, both variants rival or exceed the best wrapper and embedded methods despite being classifier-agnostic and one-shot, and the automatic clustering strategy yields accuracy close to the best fixed-budget result with fewer features than LASSO (Roffo et al., 2020).

ILFS was tested on 10 benchmarks spanning cancer microarray data, handwritten/text/sentiment datasets, and PASCAL VOC 2007 and 2012. Evaluation used mAP or classification accuracy via one-vs-rest SVM, repeated random splits, and statistical testing with Student’s $\ell$2-test and binomial tests (Roffo et al., 2017). The summary reports that ILFS attains the highest mAP/AUC on all 10 benchmarks, is particularly robust on DEXTER, POLARITY, and small-sample, large-feature genomics settings, and ranks among the top three methods across all object classes in PASCAL VOC deep-feature experiments, with the highest overall mAP (Roffo et al., 2017).

7. Applications, misconceptions, and practical considerations

Inf-FS has been used not only as an offline filter for high-dimensional classification but also as a component in online adaptive systems. In DFST, a visual tracker based on Adaptive colour Names embeds a feature-selection mechanism derived from Inf-FS to adaptively select the top-ranked discriminative features at each update (Roffo et al., 2016). In that setting, the tracker operates on $\ell$3 color-name channels and constructs a supervised affinity from Fisher score, two-sample $\ell$4-test $\ell$5-value, and Pearson correlation with the class label: $\ell$6 The same infinite-walk computation,

$\ell$7

produces per-frame rankings (Roffo et al., 2016). The reported tracker settings use $\ell$8 selected features in practice, and the paper states that embedding the supervised Inf-FS stage yields a significant qualitative gain in accuracy and precision over the fixed-feature ACT baseline, especially under heavy background clutter, distractors, and illumination changes (Roffo et al., 2016).

One common misconception is to treat Inf-FS as exclusively unsupervised. The literature provided here contradicts that view directly: supervised Inf-FS variants appear both in the general feature-selection context (Eskandari et al., 2017, Roffo et al., 2020) and in application-specific adaptations such as DFST (Roffo et al., 2016). Another misconception is that “infinite” implies computational intractability. In the Inf-FS framework, the opposite claim is made explicitly: going to infinite allows the method to constrain the computational complexity of the selection process by replacing subset enumeration with a closed-form inverse (Roffo et al., 2020).

At the same time, the method is not free of constraints. Its dense graph representation entails $\ell$9 memory, and its closed-form solution requires matrix inversion, which becomes burdensome on very large feature sets (Eskandari et al., 2017). Practical recommendations in the supervised extension therefore include using the closed-form inversion for speed and numerical stability, choosing $G=(V,E)$00 or a similar value to ensure convergence, and considering blockwise or low-rank approximations when $G=(V,E)$01 is very large (Eskandari et al., 2017).

Taken together, these works position Infinite Feature Selection as a family of graph-based ranking methods unified by an infinite-path scoring operator. Within that family, differences in adjacency construction—hand-crafted, supervised, or latent-variable learned—determine how relevance and redundancy are encoded, while the ranking itself remains a global sum over all damped walks through the feature graph (Roffo et al., 2020).