Saliency-Driven Column Selection

• Saliency-driven column selection is a technique that quantifies the importance of each column to retain critical information while reducing data dimensionality.
• It leverages gradient-based measures in deep models and group-lasso penalties in inverse regression to assess and select key features efficiently.
• Empirical results show that these methods can substantially lower feature counts while maintaining or enhancing predictive performance in high-dimensional contexts.

Saliency-driven column selection encompasses a family of techniques whereby salient or informative columns (features or structural basis vectors) are identified and retained for downstream analysis, with the goal of achieving dimensionality reduction, interpretability, or computational tractability. The notion of "saliency" is operationalized via instance- or population-level scores, which guide systematic elimination or retention of matrix columns according to optimization, ranking, or penalization schemas. These approaches are prominent both in modern feature selection in supervised learning (Cancela et al., 2019) and recently in sufficient dimension reduction for inverse regression frameworks (Jin et al., 2024).

1. Conceptual Foundations

Saliency-driven column selection is grounded in the principle that, for high-dimensional data problems, most columns of the data matrix or parameter matrices are redundant, non-informative, or even detrimental to representation learning or statistical estimation. By quantifying the "saliency" of each column—whether as its local contribution to predictive performance or its role in spanning a target subspace—one constructs a ranking or subset selection that preserves essential information with substantially reduced dimensionality.

In supervised feature selection, saliency may refer to the sensitivity of the predictive loss to perturbation of inputs (gradient-based measures). In structured inverse regression, columns of higher-order moment matrices are evaluated for their contribution to the central subspace of interest, facilitating subspace recovery without exhaustively utilizing all generated moments.

2. Mathematical Formulations of Saliency

The construction of a column-wise saliency score depends on the context and the model class.

Saliency in Feature Selection via Deep Models

Given data $X \in \mathbb{R}^{N \times R}$, model $f(X; \Theta)$, and target $Y$, saliency for feature $j$ in instance $i$ is:

$$\sigma^{(i)}_j = \left| \frac{\partial g(\tilde{y}^{(i)}, y^{(i)})}{\partial x^{(i)}_j} \right|$$

where $g$ is a gain function that is large when the model's output aligns well with the target, e.g., $$g_{\mathrm{MSE}} = \frac{\alpha}{\ell_{\mathrm{MSE}} + \varepsilon}$$ for regression or a cross-entropy-based function for classification (Cancela et al., 2019).

Saliency in Sufficient Dimension Reduction

For SDR, the candidate matrix $M \in \mathbb{R}^{p \times q}$ (e.g., stacking conditional mean or covariance differences) may possess far more columns than necessary for its column space to capture the central subspace $S_{Y|X}$. Salient columns are those whose inclusion into submatrices $M_F$ maximally reduces the subspace estimation error, as measured by population or bootstrap criteria:

$$g(F) = \sum_{b=1}^B ||\Pi(\beta_F)^{(b)} - \Pi(\beta_F)||_F^2 + c n |F|$$

where $\Pi(\cdot)$ projects onto the column space, and $F$ is the subset of columns considered (Jin et al., 2024).

3. Adaptive Saliency-Driven Selection Algorithms

Deep Saliency-Based Feature Selection (SFS)

The SFS procedure is iterative and embedded, repeatedly retraining models while masking the least-salient features. At each loop, features with the lowest aggregate saliency (after normalization and class- or instance-level accumulation) are removed. The algorithm guarantees that column saliency is continuously re-evaluated as the active set shrinks, promoting robustness to feature collinearity.

Pseudocode skeleton for SFS:

while n_alive > ε + 1: mask features with lowest σ_avg retrain model, accumulate saliency reorder features by σ_avg drop bottom γ fraction return ranking

(Cancela et al., 2019)

Forward Selection in Inverse Regression

High-dimensional SDR realizes column selection via group-lasso penalization and forward addition. Each column's initial saliency is given by the ℓ₂ norm of a penalized estimate. In each step, the algorithm adds the column whose residual (after projection onto the current subspace) has the highest ℓ₂ norm, iterating until the desired subspace dimension is attained.

Pseudocode skeleton for forward column selection:

F = ∅ for k in 1..D: for i not in F: r_i = ||(I - Π(F)) M_i^init||_2 pick j with largest r_i if r_j small: break F = F ∪ {j} return R = F

(Jin et al., 2024)

4. Theoretical Guarantees and Analysis

For both feature selection and inverse regression, saliency-driven column selection benefits from strong consistency and error control under appropriate regularity and signal conditions.

• In SFS, stability emerges with multiple random initializations and careful tuning of $\gamma$, mitigating variance from non-convex model landscapes (Cancela et al., 2019).
• In high-dimensional SDR, theoretical results guarantee subspace recovery and active set recovery, provided the minimum signal strength and incoherence conditions are met, and the penalty parameters in group-lasso are appropriately selected (Jin et al., 2024).

The efficiency of these methods is reflected in oracle inequalities and empirical performance, with the adaptive forward selection matching minimax optimality in several scenarios.

5. Empirical Performance and Comparative Results

Empirical studies confirm that saliency-driven column selection matches or outperforms classical feature selection (e.g., LASSO, Elastic Net, ReliefF) and outperforms conventional SDR (e.g., SC-SIR, TC-SAVE) in critical high-dimensional and high-correlation regimes.

Method	SVM Acc (%)	#Features	Reference
Baseline (all)	88.0	10,000	(Cancela et al., 2019)
LASSO	76.0	~1,200	(Cancela et al., 2019)
SFS (γ=0.975)	88.0	4,870	(Cancela et al., 2019)

In SDR, methods such as SCS-SAVE and SCS-ENS retrieve the central subspace and active set accurately even for large $p$ with weakly sparse covariance structures, while classical methods fail when the population covariance is not near-sparse (Jin et al., 2024).

In vision tasks (e.g., MNIST), SFS achieves >99% accuracy with less than half of the original pixels, outperforming or matching baseline classifiers (Cancela et al., 2019).

6. Implementation Considerations and Scalability

Saliency-driven methods are model-agnostic, provided gradient information can propagate to the raw input for SFS, or group-lasso penalties can be efficiently computed for SDR approaches. Complexity is governed by the cost of repeated model retrainings and gradient computations (SFS: $O(RN)$), or by the cost of group-lasso estimation and orthogonal projections (SDR: overall $O(p^2n)$ for q=O(p)). The methods are naturally parallelizable and support embedded or post-hoc workflows.

Practical efficiency is further improved by:

• Moderate masking rates ($\gamma$) to speed up convergence.
• Reusable code via standard deep learning libraries or convex optimization toolkits.
• Intermediate screening (pre-grouping, importance pruning) to handle ultra-high-dimensional matrix constructions (Jin et al., 2024, Cancela et al., 2019).

7. Extensions and Broader Implications

Saliency-driven column selection has applications that extend beyond classical feature selection and SDR, including principal component analysis (PCA) acceleration, canonical correlation analysis (CCA), and any setting where parameter matrix dimensionality poses computational or statistical challenges. Adaptive pooling and construction of candidate column sets—potentially via polynomial or nonlinear expansion—extend the flexibility for tailored subspace recovery.

Discussion in (Jin et al., 2024) highlights the potential for convex relaxations of forward selection and sophisticated weighting schemes (e.g., low-rank or sparse column weighting) to further enhance selection optimality. A plausible implication is that such methods offer a pathway toward unified, scalable, and interpretable feature/subspace selection across modalities and statistical paradigms.