Variable Selection Networks (VSN)

• Variable Selection Networks (VSN) are computational architectures that use graph-informed, adaptive shrinkage to perform structured variable selection in high-dimensional contexts.
• They integrate a two-layer hierarchical Bayesian model with an EM algorithm for efficient posterior estimation, facilitating robust inference even with hundreds of thousands of covariates.
• Empirical studies in genomics show that VSN methods improve prediction accuracy and recover biologically meaningful networks compared to traditional variable selection techniques.

Variable Selection Networks (VSN) refer to computational architectures for variable selection that exploit known structural relationships among covariates by coupling local shrinkage (regularization) parameters across a graph representing variable dependencies. VSNs naturally arise in Bayesian variable selection for structured high-dimensional data, notably in genomics and related contexts, where covariates (e.g., genes) can be connected via biological pathways or other network data. The VSN framework implements adaptive, structured shrinkage by combining a hierarchical Bayesian prior model—which aligns shrinkage strengths for “neighboring” variables in a known graph—with scalable inference algorithms.

1. Model Architecture and Notation

A VSN is constructed on a regression framework with observed data $\{(y_i, x_i)\}_{i=1}^n$ where $y_i \in \mathbb{R}$ and $x_i \in \mathbb{R}^p$. In matrix notation:

$y = X\beta + \epsilon$

with $y$ ($n \times 1$), $X$ ($n \times p$), $\beta = (\beta_1, ..., \beta_p)'$ ($p \times 1$), and $y_i \in \mathbb{R}$0. The primary objective is to select a sparse subset among the $y_i \in \mathbb{R}$1 predictors, using additional information about covariate structure.

VSN architectures incorporate this structure via a known undirected graph $y_i \in \mathbb{R}$2 with $y_i \in \mathbb{R}$3 and edges $y_i \in \mathbb{R}$4. The $y_i \in \mathbb{R}$5 covariates each correspond to a node. The adjacency matrix $y_i \in \mathbb{R}$6 iff $y_i \in \mathbb{R}$7; otherwise, $y_i \in \mathbb{R}$8. This graph informs the smoothing of local shrinkage parameters associated with regression coefficients $y_i \in \mathbb{R}$9 (Chang et al., 2016).

2. Prior Specification and Hierarchical Shrinkage Formulation

VSN methodology employs a two-layer hierarchical prior:

• Layer 1: Each coefficient $x_i \in \mathbb{R}^p$0 has a Laplace (double-exponential) prior,

$x_i \in \mathbb{R}^p$1

where $x_i \in \mathbb{R}^p$2 is a local shrinkage parameter.

• Layer 2: The log-shrinkage parameters $x_i \in \mathbb{R}^p$3 are assigned a Gaussian Markov random field (GMRF) prior,

$x_i \in \mathbb{R}^p$4

with $x_i \in \mathbb{R}^p$5 a graph-Laplacian–like precision matrix: $x_i \in \mathbb{R}^p$6, $x_i \in \mathbb{R}^p$7 for $x_i \in \mathbb{R}^p$8, $x_i \in \mathbb{R}^p$9 if edge exists.

Integrating out $y = X\beta + \epsilon$0 leads to:

$y = X\beta + \epsilon$1

The effect is network-smoothed sparsity: connected variables in $y = X\beta + \epsilon$2 tend to receive similar shrinkage, thus grouping or structuring the selection process (Chang et al., 2016).

3. Expectation Maximization for Posterior Mode Estimation

Inference in VSNs leverages the EM algorithm to maximize the posterior:

$y = X\beta + \epsilon$3

The EM formulation:

• E-step: Updates latent edge variables $y = X\beta + \epsilon$4 as

$y = X\beta + \epsilon$5

• M-step: Maximizes a "complete-data" Q-function in three blocks:
  • $y = X\beta + \epsilon$6-update: Weighted Lasso solve

  $y = X\beta + \epsilon$7

  where $y = X\beta + \epsilon$8. - $y = X\beta + \epsilon$9-update: Solves $y$0, explicitly,

  $y$1 - $y$2-update: (Diagonal-approximate) Newton step on

  $y$3

This results in efficiency per iteration: E-step $y$4, $y$5-update $y$6 (with $y$7 active), and $y$8-updates $y$9. For large sparse graphs and $n \times 1$0, minutes of computation on standard hardware suffice (Chang et al., 2016).

4. Theoretical Properties and Oracle Guarantees

The VSN approach features theoretical oracle properties for both fixed and diverging dimension regimes given appropriately tuned hyperparameters $n \times 1$1:

• Fixed $n \times 1$2, $n \times 1$3: MAP estimator $n \times 1$4 achieves variable selection consistency ($n \times 1$5 for the true active set $n \times 1$6), and

$n \times 1$7

under minimal signal and limiting covariance conditions.

• Diverging $n \times 1$8: For $n \times 1$9, $X$0, similar consistency and asymptotic normality are established under specific eigenvalue and signal assumptions on $X$1.

These results demonstrate the VSN framework's statistical validity in structured high-dimensional selection (Chang et al., 2016).

5. Empirical Performance and Benchmarking

VSN methodology has been empirically evaluated in both simulations and real genomic datasets using variants of the EMVS algorithm.

Simulation design: $X$2, $X$3 signals, $X$4 with graph $X$5 reflecting overlapping pathway architectures. Competing methods: Lasso, adaptive-Lasso, EMVS, BVS-MRF, EMVSS, EMSH (no structure), and EMSHS (network-smoothed, the VSN variant).

Key results:

• With a correct or ideal $X$6, EMSHS (i.e., the VSN instantiation) exhibits lowest mean-squared prediction error (MSPE) and optimal true/false positive rates.
• Under graph mis-specification, EMSHS remains robust and outperforms all competitive structured methods.
• Only EMSHS/EMSH scale successfully to $X$7 variables; competing approaches fail or timeout.

Application to cancer genomics: On glioblastoma survival ($X$8, $X$9 genes; 332 pathway graph from KEGG), the accelerated-failure-time model with EMSHS obtains the lowest 5-fold CV MSPE (0.975 vs. Lasso at 0.986). Gene selection by EMSHS recovers TOM1L1, RANBP17, BRD7, and the Wnt pathway, consistent with biological literature (Chang et al., 2016).

6. Connection to Network Representation and Extensions

A Variable Selection Network can be abstracted as a two-layer graph:

• Layer 1: Nodes are regression coefficients $n \times p$0 (target weights).
• Layer 2: Nodes are local log-shrinkage parameters $n \times p$1, coupled by a GMRF over the edges $n \times p$2 of $n \times p$3.
• Edges: Connect $n \times p$4 to encourage similarity for neighboring shrinkage parameters.

EM inference alternately propagates updates on edge variables (E-step) and node weights (M-step), aligning with the networked view.

Extensions are plausible: multi-layer hierarchical priors (e.g., pathways of pathways), non-Gaussian Markov fields, or embedding within neural network architectures where shrinkage parameters are outputs of an upstream GCN, while preserving rigorous sparsity and network smoothing. This suggests scope for integrating VSNs with deep learning pipelines for high-throughput variable selection (Chang et al., 2016).