---
title: Pathwise Feature Selection (PFS)
url: https://www.emergentmind.com/topics/pathwise-feature-selection-pfs
type: topic
---

# Pathwise Feature Selection (PFS)

Pathwise Feature Selection (PFS) denotes a class of feature-selection procedures in which the basic selection object is not only an isolated feature score, but a path, trajectory, or path-indexed family of selected sets. In its most explicit current usage, PFS is a local graph estimation method that starts from a target set \(V_0\), iteratively applies nodewise feature selection, and propagates edgewise uncertainty along graph paths to recover a target-centered subgraph [2507.17172]. In adjacent literatures, closely related ideas appear under different names: feature subsets represented as paths in a feature graph and summed to infinite length [2006.08184], forward-backward stepwise trajectories computed in distributed form [1708.07178], threshold paths integrated by stability selection [2410.02208], and instance-wise acquisition trajectories trained with pathwise gradients [2605.05511]. The term therefore has both a narrow meaning, tied to local graph estimation, and a broader methodological meaning covering several non-equivalent path-based formulations.

## 1. Terminological scope and core distinctions

PFS is not a single standardized label across the literature. The 2025 paper on local graph estimation explicitly presents **pathwise feature selection** as an alternative to full-graph estimation when the scientific goal is recovery of local structure around variables of interest [2507.17172]. By contrast, earlier and parallel work uses other names while employing clearly path-based constructions. "Infinite Feature Selection" treats a subset of features as a path in a feature graph and scores each feature through its participation in paths of arbitrary length [2006.08184]. PFBP is a massively parallel redesign of forward-backward selection in which the selected set evolves along a greedy stepwise path with additions, deletions, and reruns [1708.07178]. IPSS defines a path through nested thresholded feature sets \(\lambda \mapsto \hat S_\lambda\) and integrates stability over that threshold continuum [2410.02208]. NM-PPG treats feature selection as a sequential acquisition trajectory \(m_0 \to m_1 \to \cdots\) and optimizes it with pathwise gradients [2605.05511].

| Path notion | Representative formulation | Representative paper |
|---|---|---|
| Graph path over features | Subsets scored as weighted paths, aggregated by \(\sum_{l=1}^{\infty}(rA)^l\) | [2006.08184] |
| Stepwise selection path | Forward additions and backward deletions of \(\mathbf S\) across iterations and runs | [1708.07178] |
| Threshold path | \(\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}\) integrated over \(\Lambda\) | [2410.02208] |
| Local graph path | Paths from targets with cumulative edge \(q\)-values | [2507.17172] |
| Acquisition trajectory | Instance-wise mask sequence with STOP action | [2605.05511] |

This diversity makes several common misconceptions important to avoid. PFS is not synonymous with a lasso or regularization path: both PFBP and local-graph PFS are driven by conditional-independence or \(q\)-value decisions rather than coefficient trajectories [1708.07178; 2507.17172]. It is also not necessarily a monotone or nested model sequence: PFBP includes backward deletions, so the full path of selected sets is not guaranteed to remain nested over an entire run [1708.07178]. Conversely, pathwise need not mean sequential one-feature-at-a-time entry. Inf-FS does not follow one path at all; it performs a one-shot global aggregation over all possible paths encoded by matrix powers [2006.08184].

## 2. PFS as local graph estimation

In its narrow and explicit sense, PFS is a method for estimating only the subgraph near a user-specified target set \(V_0\), rather than recovering a full conditional independence graph on all \(p\) variables [2507.17172]. Let \(X=(X_1,\dots,X_p)\) have undirected conditional independence graph \(G=(V,E)\). For graph distance \(d(j,k)\), the radius-\(r\) ball around \(V_0\) is
\[
B_0(V_0)=V_0,\qquad B_r(V_0)=\{k\in V:\min_{j\in V_0} d(j,k)\le r\},
\]
with sphere \(S_r(V_0)=B_r(V_0)\setminus B_{r-1}(V_0)\). The associated local edge set is
\[
E_r(V_0)=\{(j,k)\in E: j\in B_{r-1}(V_0)\ \text{or}\ k\in B_{r-1}(V_0)\},
\]
so the local graph is \(G_r(V_0)=(B_r(V_0),E_r(V_0))\) [2507.17172].

The algorithm grows this local graph outward one layer at a time. It initializes the current frontier and visited set at the targets,
\[
S\gets V_0,\qquad B\gets V_0,
\]
and a weighted adjacency matrix
\[
Q\gets \mathbf 1\in\mathbb R^{p\times p}.
\]
At each radius \(r\), every node \(j\in S\) is treated as a response variable, and nodewise feature selection is used to compute edgewise \(q\)-values \(q_j(k)\) for all \(k\neq j\). If \(q_j(k)\le q_r^*\), the edge weight is updated symmetrically as
\[
Q_{jk}\gets Q_{kj}\gets \min\{q_j(k),Q_{kj}\}.
\]
The method then computes, for each unvisited node, the lightest estimated path of length \(r\) from \(V_0\),
\[
d_Q(V_0,j,r)=\min\left\{\sum_{s=0}^{r-1} Q_{j_s,j_{s+1}} : (j_0,\dots,j_r)\in \widehat{\mathcal J}_r(V_0),\ j_r=j\right\},
\]
and admits only nodes satisfying
\[
d_Q(V_0,j,r)\le q_{\mathrm{path}^*}.
\]
The new frontier is therefore restricted by cumulative path uncertainty rather than by edgewise screening alone [2507.17172].

The central theorem is path-specific. For a path \(j=(j_0,\dots,j_r)\) with distinct nodes, under Storey-type assumptions for each nodewise family and independence of \((T_{j_s}(j_{s+1}),H_{j_s}(j_{s+1}))\) along the path, the probability that the selected path is not a true graph path is bounded by the sum of its edge \(q\)-values:
\[
\widetilde q(j)\le \sum_{s=0}^{r-1} q_{j_s}(j_{s+1}).
\]
This is a finite-sample path discovery guarantee, not a full local-graph FDR guarantee [2507.17172]. The distinction is fundamental: the controlled discovery unit is the path, and the local scientific claim is that a retained node is connected to the targets through at least one low-uncertainty chain.

The method is designed for settings with \(n\) i.i.d. samples, possibly large \(p\), mixed data types, nonlinear relationships, and a small target set of primary scientific interest [2507.17172]. Its implementation uses IPSS to generate edgewise \(q\)-values, but the framework is modular in the sense that any nodewise selector producing appropriate \(q\)-values may be substituted. This yields an output that is explicitly interpretable: a weighted local adjacency matrix \(Q\), radius layers around \(V_0\), edge-specific \(q\)-values, and lightest paths from the targets. The main limitation stressed by the paper is that deeper paths accumulate uncertainty additively, and the theorem controls path non-membership probability rather than exact local edge FDP [2507.17172].

## 3. Graph-based path aggregation and global ranking

A different pathwise formulation appears in "Infinite Feature Selection" [2006.08184]. Here, the feature set is encoded as a weighted, undirected, fully connected graph \(G=(V,E)\), in which node \(\vec v_i\) corresponds to feature \(f_i\), and edge weight
\[
A(i,j)=\varphi(\vec v_i,\vec v_j)
\]
expresses confidence that both \(f_i\) and \(f_j\) are good candidates to be selected together. A path
\[
\gamma=\{\vec v_0=i,\vec v_1,\ldots,\vec v_l=j\}
\]
is interpreted as a subset of \(l+1\) features, and its weight is the product of edge weights,
\[
\mathcal{\pi}_\gamma=\prod_{k=0}^{l-1} A(\vec v_k,\vec v_{k+1}).
\]
If \(\mathbb P^l_{i,j}\) is the set of all paths of length \(l\) from \(i\) to \(j\), the total contribution of all such paths is
\[
R_l(i,j)=\sum_{\gamma\in\mathbb P^l_{i,j}} \mathcal{\pi}_\gamma,
\qquad R_l=A^l.
\]
The score of feature \(i\) at fixed path length \(l\) is
\[
c_l(i)=\sum_{j\in V}A^l(i,j),
\]
and aggregation over all lengths yields
\[
c(i)=\sum_{l=1}^{\infty} c_l(i).
\]
To ensure convergence, the paper introduces a damping factor \(r\) and uses the regularized sum
\[
\check C=(I-rA)^{-1}-I,\qquad \check c=\check C e,
\]
with
\[
r=\frac{0.9}{\rho(A)}.
\]
Thus each feature is ranked by its total participation in weighted paths of all lengths up to infinity [2006.08184].

The graph construction differs between the unsupervised and supervised variants. In Inf-FS\(_U\),
\[
\varphi_U(\vec v_i,\vec v_j)=\alpha E_{ij}+(1-\alpha)\overline{corr}_{ij},
\]
where \(E_{ij}=\max(\sigma_i,\sigma_j)\) and \(\overline{corr}_{ij}=1-|Spearman(f_i,f_j)|\), so high-variance and weakly correlated pairs receive large weight. In Inf-FS\(_S\), a feature-level score
\[
s_i=h_i\alpha_1+m_i\alpha_2+\sigma_i\alpha_3,\qquad \sum_k \alpha_k=1,
\]
is formed from a Fisher criterion, mutual information, and standard deviation, and the edge factorizes as
\[
A(i,j)=s_is_j.
\]
The unsupervised version therefore encodes redundancy explicitly through pairwise Spearman correlation, whereas the supervised version uses factorized node relevance [2006.08184].

This method is pathwise in the sense of **global path aggregation** rather than sequential search. It does not trace one route through subset space; it effectively sums over all paths by matrix power series. The resulting procedure is a filter method that produces a global ranking, after which one either keeps a prescribed top \(b\) features or uses the proposed automatic cutoff based on **1D Mean Shift with automatic bandwidth selection** [2006.08184]. The paper argues that the closed form reduces complexity from \(\mathcal O(n^3 l T)\) to \(\mathcal O(n^3 T)\), and reports complexity \(\mathcal O(n^3(1+T))\) for Inf-FS\(_U\) and \(\mathcal O(T^2+n^3(1+T))\) for Inf-FS\(_S\) [2006.08184]. This formulation is therefore pathwise without being stepwise.

## 4. Stepwise trajectories and distributed path construction

PFBP, or **Parallel, Forward-Backward with Pruning**, is a Big-Data extension of forward-backward selection and is naturally interpreted as a pathwise method in the classical stepwise sense [1708.07178]. Its path is the ordered evolution of the selected set \(\mathbf S\) across forward additions, backward deletions, and multiple runs. In a forward phase, the algorithm repeatedly adds the variable with the smallest conditional-independence \(p\)-value given the current \(\mathbf S\), provided that \(p\)-value is \(\le \alpha\). In a backward phase, it repeatedly removes the selected variable with the largest \(p\)-value conditional on \(\mathbf S\setminus\{V\}\), provided that \(p\)-value is \(>\alpha\) [1708.07178].

The paper’s central contribution is operational rather than conceptual: it shows how to preserve this stepwise path semantics when both the number of samples and the number of features are too large for ordinary centralized model fitting. The data matrix is partitioned both by rows and by columns into blocks \(D_{i,j}\). Workers run local conditional-independence tests on the blocks, returning local \(p\)-values and log-likelihood contributions. These are combined centrally by Fisher’s combined probability test,
\[
-2\sum_{i=1}^{K}\log(p_i),
\]
which is distributed as \(\chi^2\) with \(2K\) degrees of freedom [1708.07178]. The path state therefore still depends on the current selected set \(\mathbf S\), but the evidence for each path step is assembled from local summaries rather than from repeated global optimization.

Three pruning heuristics define how aggressively the path is shortened during computation. **Early Dropping** removes variables from future forward iterations in the current run if
\[
P(\pi_j\ge \alpha)>P_{drop}.
\]
**Early Stopping** removes variables from the current iteration if
\[
P(\pi_j>\pi_{best})>P_{stop}.
\]
**Early Return** truncates the current competition and commits to the current best if the likelihood-based criterion indicates that the best feature is probably good enough [1708.07178]. The recommended thresholds are \(P_{drop}=0.99\), \(P_{stop}=0.99\), \(P_{return}=0.95\), \(tol=0.9\), and minimum \(B=500\). These heuristics are described as asymptotically sound and are the main reason the algorithm achieves low communication cost and large-scale viability [1708.07178].

The pathwise interpretation is precise. Within a forward phase, the selected sets are nested:
\[
\emptyset \subset \mathbf S_1 \subset \mathbf S_2 \subset \cdots.
\]
Over the full algorithm, however, the path is not monotone because backward deletion may shrink \(\mathbf S\), and reruns may allow previously dropped variables to re-enter contention [1708.07178]. This distinguishes PFBP from purely monotone forward stagewise procedures.

The theoretical guarantees are formulated as Markov blanket recovery results under an independence oracle and faithfulness assumptions. If the distribution can be faithfully represented by a Bayesian network, then PFBP with two runs identifies the Markov blanket of the target \(T\). If the distribution can be faithfully represented by a directed maximal ancestral graph, then PFBP with no limit on the number of runs identifies the Markov blanket of \(T\) [1708.07178]. This is stronger than a guarantee about a locally optimal greedy subset; it states that the iterative path, together with reruns after dropping, converges to the correct blanket under the stated assumptions.

## 5. Threshold paths, stability integration, and false discovery control

A third meaning of pathwise selection appears in **integrated path stability selection (IPSS)** [2410.02208]. IPSS begins with an arbitrary feature importance function
\[
\Phi_{Z_{1:n}}:\{1,\dots,p\}\to\mathbb R
\]
and converts it into a path of nested selection sets by thresholding:
\[
\hat S_\lambda=\{j:\Phi(j)\ge \lambda\}.
\]
The path is therefore the family \(\lambda\mapsto \hat S_\lambda\): large \(\lambda\) selects few features, and smaller \(\lambda\) admits more features [2410.02208]. This is not a regularization path in the lasso sense and not a sequential add-one-feature path. It is a threshold path over ranked scores.

IPSS then applies disjoint half-subsampling. Across \(B\) iterations there are \(2B\) half-samples, and the empirical selection frequency of feature \(j\) at threshold \(\lambda\) is
\[
\hat\pi_j(\lambda)=\frac{1}{2B}\sum_{b=1}^{2B}\mathbf 1\big(j\in \hat S_\lambda(Z_{A_b})\big).
\]
This yields a stability curve \(\lambda\mapsto \hat\pi_j(\lambda)\). Rather than choosing one threshold, IPSS integrates stability over a threshold interval \(\Lambda\):
\[
\hat S_{\mathrm{IPSS}(\tau)}=
\left\{j:\int_\Lambda f(\hat\pi_j(\lambda))\,\mu(d\lambda)\ge \tau\right\},
\]
using
\[
f(x)=(2x-1)^3\mathbf 1(x\ge 0.5).
\]
The corresponding efp score is
\[
efp_{Z_{1:n}}(j)=
\min\left\{
\frac{\mathcal I(\Lambda)}
{\int_\Lambda f(\hat\pi_j(\lambda))\,\mu(d\lambda)},
\,p
\right\},
\]
and the final selection at target expected false positives \(t\) is
\[
\hat S(t)=\{j:efp_{Z_{1:n}}(j)\le t\}.
\]
Under the paper’s simultaneous-selection condition on null features, this construction satisfies finite-sample expected-false-positive control:
\[
\mathrm E(\mathrm{FP}(t))\le t.
\]
Approximate FDR control is then obtained via
\[
\mathrm{pFDR}(t)\approx \mathrm{FDR}(t)\approx
\frac{\mathrm E(\mathrm{FP}(t))}{\mathrm E|\hat S(t)|}
\le \frac{t}{\mathrm E|\hat S(t)|},
\]
leading to empirical \(q\)-value estimation and the rule to choose the largest \(\hat S(t)\) such that
\[
\frac{t}{|\hat S(t)|}\le \alpha
\]
for target FDR level \(\alpha\) [2410.02208].

The framework is nonparametric whenever the base importance score is nonparametric. The paper emphasizes two special cases: **IPSSGB**, which uses gradient boosting importance scores from XGBoost, and **IPSSRF**, which uses random forest importance scores from scikit-learn [2410.02208]. The default measure is
\[
\mu(d\lambda)=z^{-1}\lambda^{-1}d\lambda,
\]
so integration is performed on a log scale; the paper uses \(B=100\) for IPSSGB, \(B=50\) for IPSSRF, and usually \(K=100\) thresholds [2410.02208]. In the PFS taxonomy, IPSS is best understood as a pathwise thresholding-and-aggregation procedure with finite-sample \(\mathrm E(\mathrm{FP})\) control, rather than as a classical stepwise selector.

## 6. Instance-wise sequential selection and common structural themes

A fourth pathwise interpretation arises in **active feature acquisition**, where the selected subset is instance-dependent and order-dependent [2605.05511]. NM-PPG formulates this setting as a finite-horizon POMDP. For each instance \(x\), the state of acquired features is encoded by a binary mask
\[
m_t\in\{0,1\}^d,
\]
and the observation is the masked representation
\[
x(m_t)=(m_t\odot x,\; m_t)\in\mathbb R^{2d}.
\]
The action space is
\[
\mathcal A=\{1,\dots,d,d+1\},
\]
where \(1,\dots,d\) acquire individual features and \(d+1\) is STOP. The objective is to minimize expected prediction loss plus acquisition cost,
\[
\mathbb E\left[
\ell\!\left(f_\phi(x(m_{t_\theta(x)})),y\right)+\alpha\,c(m_{t_\theta(x)})
\right],
\]
with stopping either chosen by the policy or forced at horizon \(k\) [2605.05511].

The pathwise aspect is twofold. First, the subset is built along a trajectory
\[
m_0\to m_1\to \cdots \to m_{t_\theta(x)},
\]
so later feature choices depend on earlier observed values. Second, training uses pathwise gradients through a differentiable relaxation of the full acquisition rollout. With Gumbel-Softmax, the relaxed action is
\[
\tilde a_t=
\operatorname{softmax}\!\left(
\frac{(z_t/\tau_{\mathrm{hard}})+\varepsilon_t}{\tau_{\mathrm{soft}}}
\right),
\]
which induces a relaxed feature-selection vector \(\tilde r_t\), a soft mask update
\[
\tilde m_{t+1}=\tilde m_t+(1-\tilde m_t)\odot \tilde r_t,
\]
and a survival mass
\[
\tilde s_{t+1}=\tilde s_t(1-\tilde a_{t,d+1}).
\]
The relaxed full-path objective is
\[
\tilde G(x,y,\theta,\varepsilon)=
\sum_{t=0}^{k-1}\tilde s_t
\Big(
\alpha (1-\tilde a_{t,d+1})c(\tilde r_t)+
\tilde a_{t,d+1}\tilde \ell_t
\Big)
+\tilde s_k\tilde \ell_k,
\]
so gradients propagate from later losses back through earlier mask updates and action logits [2605.05511].

To align training with discrete deployment, the paper introduces a straight-through rollout:
\[
\bar r_t=r_t-\mathrm{sg}(\tilde r_t)+\tilde r_t,\qquad
\bar m_{t+1}=m_{t+1}-\mathrm{sg}(\tilde m_{t+1})+\tilde m_{t+1}.
\]
The forward pass therefore follows hard acquisitions, while the backward pass uses the soft relaxation. Entropy regularization and staged temperature sharpening are added for stability [2605.05511]. This is pathwise selection in a fully instance-wise sense: there is no global subset and no global feature ranking.

Taken together, these formulations show that PFS is best understood as a family of methods organized around a path object, but the meaning of that object varies substantially. In local graph estimation, the path is a chain from target variables through edges with additive \(q\)-value uncertainty [2507.17172]. In Inf-FS, it is a combinatorial path through a feature graph, aggregated globally in closed form [2006.08184]. In PFBP, it is the stepwise evolution of \(\mathbf S\) under conditional-independence decisions and pruning [1708.07178]. In IPSS, it is a threshold continuum over ranked scores with integrated stability [2410.02208]. In NM-PPG, it is a sequential, cost-sensitive, instance-specific acquisition trajectory [2605.05511]. A plausible implication is that the term is most precise when accompanied by the path semantics being used: graph path, stepwise path, threshold path, target-centered local path, or acquisition trajectory.

Source: https://www.emergentmind.com/topics/pathwise-feature-selection-pfs