Pathwise Feature Selection (PFS)

• Pathwise Feature Selection (PFS) is a family of methods that selects features along trajectories or graph paths, emphasizing local structures and global ranking.
• It integrates approaches like local graph estimation, global path aggregation via matrix power series, and distributed stepwise selection to address scalability and false discovery control.
• These techniques yield interpretable outputs such as weighted adjacency matrices, nested selected sets, and instance-specific acquisition trajectories, facilitating practical application in high-dimensional settings.

Pathwise Feature Selection (PFS) denotes a class of feature-selection procedures in which the basic selection object is not only an isolated feature score, but a path, trajectory, or path-indexed family of selected sets. In its most explicit current usage, PFS is a local graph estimation method that starts from a target set $V_0$, iteratively applies nodewise feature selection, and propagates edgewise uncertainty along graph paths to recover a target-centered subgraph (Melikechi et al., 23 Jul 2025). In adjacent literatures, closely related ideas appear under different names: feature subsets represented as paths in a feature graph and summed to infinite length (Roffo et al., 2020), forward-backward stepwise trajectories computed in distributed form (Tsamardinos et al., 2017), threshold paths integrated by stability selection (Melikechi et al., 2024), and instance-wise acquisition trajectories trained with pathwise gradients (Aronsson et al., 6 May 2026). The term therefore has both a narrow meaning, tied to local graph estimation, and a broader methodological meaning covering several non-equivalent path-based formulations.

1. Terminological scope and core distinctions

PFS is not a single standardized label across the literature. The 2025 paper on local graph estimation explicitly presents pathwise feature selection as an alternative to full-graph estimation when the scientific goal is recovery of local structure around variables of interest (Melikechi et al., 23 Jul 2025). By contrast, earlier and parallel work uses other names while employing clearly path-based constructions. "Infinite Feature Selection" treats a subset of features as a path in a feature graph and scores each feature through its participation in paths of arbitrary length (Roffo et al., 2020). PFBP is a massively parallel redesign of forward-backward selection in which the selected set evolves along a greedy stepwise path with additions, deletions, and reruns (Tsamardinos et al., 2017). IPSS defines a path through nested thresholded feature sets $\lambda \mapsto \hat S_\lambda$ and integrates stability over that threshold continuum (Melikechi et al., 2024). NM-PPG treats feature selection as a sequential acquisition trajectory $m_0 \to m_1 \to \cdots$ and optimizes it with pathwise gradients (Aronsson et al., 6 May 2026).

Path notion	Representative formulation	Representative paper
Graph path over features	Subsets scored as weighted paths, aggregated by $\sum_{l=1}^{\infty}(rA)^l$	(Roffo et al., 2020)
Stepwise selection path	Forward additions and backward deletions of $\mathbf S$ across iterations and runs	(Tsamardinos et al., 2017)
Threshold path	$\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$ integrated over $\Lambda$	(Melikechi et al., 2024)
Local graph path	Paths from targets with cumulative edge $q$-values	(Melikechi et al., 23 Jul 2025)
Acquisition trajectory	Instance-wise mask sequence with STOP action	(Aronsson et al., 6 May 2026)

This diversity makes several common misconceptions important to avoid. PFS is not synonymous with a lasso or regularization path: both PFBP and local-graph PFS are driven by conditional-independence or $q$-value decisions rather than coefficient trajectories (Tsamardinos et al., 2017, Melikechi et al., 23 Jul 2025). It is also not necessarily a monotone or nested model sequence: PFBP includes backward deletions, so the full path of selected sets is not guaranteed to remain nested over an entire run (Tsamardinos et al., 2017). Conversely, pathwise need not mean sequential one-feature-at-a-time entry. Inf-FS does not follow one path at all; it performs a one-shot global aggregation over all possible paths encoded by matrix powers (Roffo et al., 2020).

2. PFS as local graph estimation

In its narrow and explicit sense, PFS is a method for estimating only the subgraph near a user-specified target set $V_0$, rather than recovering a full conditional independence graph on all $\lambda \mapsto \hat S_\lambda$0 variables (Melikechi et al., 23 Jul 2025). Let $\lambda \mapsto \hat S_\lambda$1 have undirected conditional independence graph $\lambda \mapsto \hat S_\lambda$2. For graph distance $\lambda \mapsto \hat S_\lambda$3, the radius-$\lambda \mapsto \hat S_\lambda$4 ball around $\lambda \mapsto \hat S_\lambda$5 is

$\lambda \mapsto \hat S_\lambda$6

with sphere $\lambda \mapsto \hat S_\lambda$7. The associated local edge set is

$\lambda \mapsto \hat S_\lambda$8

so the local graph is $\lambda \mapsto \hat S_\lambda$9 (Melikechi et al., 23 Jul 2025).

The algorithm grows this local graph outward one layer at a time. It initializes the current frontier and visited set at the targets,

$m_0 \to m_1 \to \cdots$0

and a weighted adjacency matrix

$m_0 \to m_1 \to \cdots$1

At each radius $m_0 \to m_1 \to \cdots$2, every node $m_0 \to m_1 \to \cdots$3 is treated as a response variable, and nodewise feature selection is used to compute edgewise $m_0 \to m_1 \to \cdots$4-values $m_0 \to m_1 \to \cdots$5 for all $m_0 \to m_1 \to \cdots$6. If $m_0 \to m_1 \to \cdots$7, the edge weight is updated symmetrically as

$m_0 \to m_1 \to \cdots$8

The method then computes, for each unvisited node, the lightest estimated path of length $m_0 \to m_1 \to \cdots$9 from $\sum_{l=1}^{\infty}(rA)^l$0,

$\sum_{l=1}^{\infty}(rA)^l$1

and admits only nodes satisfying

$\sum_{l=1}^{\infty}(rA)^l$2

The new frontier is therefore restricted by cumulative path uncertainty rather than by edgewise screening alone (Melikechi et al., 23 Jul 2025).

The central theorem is path-specific. For a path $\sum_{l=1}^{\infty}(rA)^l$3 with distinct nodes, under Storey-type assumptions for each nodewise family and independence of $\sum_{l=1}^{\infty}(rA)^l$4 along the path, the probability that the selected path is not a true graph path is bounded by the sum of its edge $\sum_{l=1}^{\infty}(rA)^l$5-values: $\sum_{l=1}^{\infty}(rA)^l$6 This is a finite-sample path discovery guarantee, not a full local-graph FDR guarantee (Melikechi et al., 23 Jul 2025). The distinction is fundamental: the controlled discovery unit is the path, and the local scientific claim is that a retained node is connected to the targets through at least one low-uncertainty chain.

The method is designed for settings with $\sum_{l=1}^{\infty}(rA)^l$7 i.i.d. samples, possibly large $\sum_{l=1}^{\infty}(rA)^l$8, mixed data types, nonlinear relationships, and a small target set of primary scientific interest (Melikechi et al., 23 Jul 2025). Its implementation uses IPSS to generate edgewise $\sum_{l=1}^{\infty}(rA)^l$9-values, but the framework is modular in the sense that any nodewise selector producing appropriate $\mathbf S$0-values may be substituted. This yields an output that is explicitly interpretable: a weighted local adjacency matrix $\mathbf S$1, radius layers around $\mathbf S$2, edge-specific $\mathbf S$3-values, and lightest paths from the targets. The main limitation stressed by the paper is that deeper paths accumulate uncertainty additively, and the theorem controls path non-membership probability rather than exact local edge FDP (Melikechi et al., 23 Jul 2025).

3. Graph-based path aggregation and global ranking

A different pathwise formulation appears in "Infinite Feature Selection" (Roffo et al., 2020). Here, the feature set is encoded as a weighted, undirected, fully connected graph $\mathbf S$4, in which node $\mathbf S$5 corresponds to feature $\mathbf S$6, and edge weight

$\mathbf S$7

expresses confidence that both $\mathbf S$8 and $\mathbf S$9 are good candidates to be selected together. A path

$\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$0

is interpreted as a subset of $\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$1 features, and its weight is the product of edge weights,

$\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$2

If $\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$3 is the set of all paths of length $\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$4 from $\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$5 to $\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$6, the total contribution of all such paths is

$\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$7

The score of feature $\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$8 at fixed path length $\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}$9 is

$\Lambda$0

and aggregation over all lengths yields

$\Lambda$1

To ensure convergence, the paper introduces a damping factor $\Lambda$2 and uses the regularized sum

$\Lambda$3

with

$\Lambda$4

Thus each feature is ranked by its total participation in weighted paths of all lengths up to infinity (Roffo et al., 2020).

The graph construction differs between the unsupervised and supervised variants. In Inf-FS$\Lambda$5,

$\Lambda$6

where $\Lambda$7 and $\Lambda$8, so high-variance and weakly correlated pairs receive large weight. In Inf-FS$\Lambda$9, a feature-level score

$q$0

is formed from a Fisher criterion, mutual information, and standard deviation, and the edge factorizes as

$q$1

The unsupervised version therefore encodes redundancy explicitly through pairwise Spearman correlation, whereas the supervised version uses factorized node relevance (Roffo et al., 2020).

This method is pathwise in the sense of global path aggregation rather than sequential search. It does not trace one route through subset space; it effectively sums over all paths by matrix power series. The resulting procedure is a filter method that produces a global ranking, after which one either keeps a prescribed top $q$2 features or uses the proposed automatic cutoff based on 1D Mean Shift with automatic bandwidth selection (Roffo et al., 2020). The paper argues that the closed form reduces complexity from $q$3 to $q$4, and reports complexity $q$5 for Inf-FS$q$6 and $q$7 for Inf-FS$q$8 (Roffo et al., 2020). This formulation is therefore pathwise without being stepwise.

4. Stepwise trajectories and distributed path construction

PFBP, or Parallel, Forward-Backward with Pruning, is a Big-Data extension of forward-backward selection and is naturally interpreted as a pathwise method in the classical stepwise sense (Tsamardinos et al., 2017). Its path is the ordered evolution of the selected set $q$9 across forward additions, backward deletions, and multiple runs. In a forward phase, the algorithm repeatedly adds the variable with the smallest conditional-independence $q$0-value given the current $q$1, provided that $q$2-value is $q$3. In a backward phase, it repeatedly removes the selected variable with the largest $q$4-value conditional on $q$5, provided that $q$6-value is $q$7 (Tsamardinos et al., 2017).

The paper’s central contribution is operational rather than conceptual: it shows how to preserve this stepwise path semantics when both the number of samples and the number of features are too large for ordinary centralized model fitting. The data matrix is partitioned both by rows and by columns into blocks $q$8. Workers run local conditional-independence tests on the blocks, returning local $q$9-values and log-likelihood contributions. These are combined centrally by Fisher’s combined probability test,

$V_0$0

which is distributed as $V_0$1 with $V_0$2 degrees of freedom (Tsamardinos et al., 2017). The path state therefore still depends on the current selected set $V_0$3, but the evidence for each path step is assembled from local summaries rather than from repeated global optimization.

Three pruning heuristics define how aggressively the path is shortened during computation. Early Dropping removes variables from future forward iterations in the current run if

$V_0$4

Early Stopping removes variables from the current iteration if

$V_0$5

Early Return truncates the current competition and commits to the current best if the likelihood-based criterion indicates that the best feature is probably good enough (Tsamardinos et al., 2017). The recommended thresholds are $V_0$6, $V_0$7, $V_0$8, $V_0$9, and minimum $\lambda \mapsto \hat S_\lambda$00. These heuristics are described as asymptotically sound and are the main reason the algorithm achieves low communication cost and large-scale viability (Tsamardinos et al., 2017).

The pathwise interpretation is precise. Within a forward phase, the selected sets are nested: $\lambda \mapsto \hat S_\lambda$01 Over the full algorithm, however, the path is not monotone because backward deletion may shrink $\lambda \mapsto \hat S_\lambda$02, and reruns may allow previously dropped variables to re-enter contention (Tsamardinos et al., 2017). This distinguishes PFBP from purely monotone forward stagewise procedures.

The theoretical guarantees are formulated as Markov blanket recovery results under an independence oracle and faithfulness assumptions. If the distribution can be faithfully represented by a Bayesian network, then PFBP with two runs identifies the Markov blanket of the target $\lambda \mapsto \hat S_\lambda$03. If the distribution can be faithfully represented by a directed maximal ancestral graph, then PFBP with no limit on the number of runs identifies the Markov blanket of $\lambda \mapsto \hat S_\lambda$04 (Tsamardinos et al., 2017). This is stronger than a guarantee about a locally optimal greedy subset; it states that the iterative path, together with reruns after dropping, converges to the correct blanket under the stated assumptions.

5. Threshold paths, stability integration, and false discovery control

A third meaning of pathwise selection appears in integrated path stability selection (IPSS) (Melikechi et al., 2024). IPSS begins with an arbitrary feature importance function

$\lambda \mapsto \hat S_\lambda$05

and converts it into a path of nested selection sets by thresholding: $\lambda \mapsto \hat S_\lambda$06 The path is therefore the family $\lambda \mapsto \hat S_\lambda$07: large $\lambda \mapsto \hat S_\lambda$08 selects few features, and smaller $\lambda \mapsto \hat S_\lambda$09 admits more features (Melikechi et al., 2024). This is not a regularization path in the lasso sense and not a sequential add-one-feature path. It is a threshold path over ranked scores.

IPSS then applies disjoint half-subsampling. Across $\lambda \mapsto \hat S_\lambda$10 iterations there are $\lambda \mapsto \hat S_\lambda$11 half-samples, and the empirical selection frequency of feature $\lambda \mapsto \hat S_\lambda$12 at threshold $\lambda \mapsto \hat S_\lambda$13 is

$\lambda \mapsto \hat S_\lambda$14

This yields a stability curve $\lambda \mapsto \hat S_\lambda$15. Rather than choosing one threshold, IPSS integrates stability over a threshold interval $\lambda \mapsto \hat S_\lambda$16: $\lambda \mapsto \hat S_\lambda$17 using

$\lambda \mapsto \hat S_\lambda$18

The corresponding efp score is

$\lambda \mapsto \hat S_\lambda$19

and the final selection at target expected false positives $\lambda \mapsto \hat S_\lambda$20 is

$\lambda \mapsto \hat S_\lambda$21

Under the paper’s simultaneous-selection condition on null features, this construction satisfies finite-sample expected-false-positive control: $\lambda \mapsto \hat S_\lambda$22 Approximate FDR control is then obtained via

$\lambda \mapsto \hat S_\lambda$23

leading to empirical $\lambda \mapsto \hat S_\lambda$24-value estimation and the rule to choose the largest $\lambda \mapsto \hat S_\lambda$25 such that

$\lambda \mapsto \hat S_\lambda$26

for target FDR level $\lambda \mapsto \hat S_\lambda$27 (Melikechi et al., 2024).

The framework is nonparametric whenever the base importance score is nonparametric. The paper emphasizes two special cases: IPSSGB, which uses gradient boosting importance scores from XGBoost, and IPSSRF, which uses random forest importance scores from scikit-learn (Melikechi et al., 2024). The default measure is

$\lambda \mapsto \hat S_\lambda$28

so integration is performed on a log scale; the paper uses $\lambda \mapsto \hat S_\lambda$29 for IPSSGB, $\lambda \mapsto \hat S_\lambda$30 for IPSSRF, and usually $\lambda \mapsto \hat S_\lambda$31 thresholds (Melikechi et al., 2024). In the PFS taxonomy, IPSS is best understood as a pathwise thresholding-and-aggregation procedure with finite-sample $\lambda \mapsto \hat S_\lambda$32 control, rather than as a classical stepwise selector.

6. Instance-wise sequential selection and common structural themes

A fourth pathwise interpretation arises in active feature acquisition, where the selected subset is instance-dependent and order-dependent (Aronsson et al., 6 May 2026). NM-PPG formulates this setting as a finite-horizon POMDP. For each instance $\lambda \mapsto \hat S_\lambda$33, the state of acquired features is encoded by a binary mask

$\lambda \mapsto \hat S_\lambda$34

and the observation is the masked representation

$\lambda \mapsto \hat S_\lambda$35

The action space is

$\lambda \mapsto \hat S_\lambda$36

where $\lambda \mapsto \hat S_\lambda$37 acquire individual features and $\lambda \mapsto \hat S_\lambda$38 is STOP. The objective is to minimize expected prediction loss plus acquisition cost,

$\lambda \mapsto \hat S_\lambda$39

with stopping either chosen by the policy or forced at horizon $\lambda \mapsto \hat S_\lambda$40 (Aronsson et al., 6 May 2026).

The pathwise aspect is twofold. First, the subset is built along a trajectory

$\lambda \mapsto \hat S_\lambda$41

so later feature choices depend on earlier observed values. Second, training uses pathwise gradients through a differentiable relaxation of the full acquisition rollout. With Gumbel-Softmax, the relaxed action is

$\lambda \mapsto \hat S_\lambda$42

which induces a relaxed feature-selection vector $\lambda \mapsto \hat S_\lambda$43, a soft mask update

$\lambda \mapsto \hat S_\lambda$44

and a survival mass

$\lambda \mapsto \hat S_\lambda$45

The relaxed full-path objective is

$\lambda \mapsto \hat S_\lambda$46

so gradients propagate from later losses back through earlier mask updates and action logits (Aronsson et al., 6 May 2026).

To align training with discrete deployment, the paper introduces a straight-through rollout: $\lambda \mapsto \hat S_\lambda$47 The forward pass therefore follows hard acquisitions, while the backward pass uses the soft relaxation. Entropy regularization and staged temperature sharpening are added for stability (Aronsson et al., 6 May 2026). This is pathwise selection in a fully instance-wise sense: there is no global subset and no global feature ranking.

Taken together, these formulations show that PFS is best understood as a family of methods organized around a path object, but the meaning of that object varies substantially. In local graph estimation, the path is a chain from target variables through edges with additive $\lambda \mapsto \hat S_\lambda$48-value uncertainty (Melikechi et al., 23 Jul 2025). In Inf-FS, it is a combinatorial path through a feature graph, aggregated globally in closed form (Roffo et al., 2020). In PFBP, it is the stepwise evolution of $\lambda \mapsto \hat S_\lambda$49 under conditional-independence decisions and pruning (Tsamardinos et al., 2017). In IPSS, it is a threshold continuum over ranked scores with integrated stability (Melikechi et al., 2024). In NM-PPG, it is a sequential, cost-sensitive, instance-specific acquisition trajectory (Aronsson et al., 6 May 2026). A plausible implication is that the term is most precise when accompanied by the path semantics being used: graph path, stepwise path, threshold path, target-centered local path, or acquisition trajectory.