LLM Sparsity Prior in Feature Selection

• The paper introduces LSP as a Bayesian mechanism that transforms LLM-derived ordinal weights into adaptive prior inclusion probabilities in spike-and-slab models.
• It employs two hyperparameters—global sparsity and weight concentration—to balance LLM influence while preserving model robustness.
• LSP outperforms LLM-Lasso by decoupling LLM guidance from the penalty term, thereby mitigating errors from poor-quality weight inputs.

Searching arXiv for the cited papers to ground the article and verify bibliographic details. LLM Sparsity Prior (LSP) denotes a Bayesian mechanism for incorporating LLM-generated feature-importance information into sparse variable-selection models through prior inclusion probabilities rather than through hard reweighting of the optimization objective (Skinner et al., 21 May 2026). In its primary formulation, LSP uses ordinal weights produced by an LLM and transforms them into feature-specific inclusion probabilities within Spike-and-Slab and Spike-and-Slab Lasso models, with two interpretable hyperparameters controlling global sparsity and weight concentration (Skinner et al., 21 May 2026). Related inference literature uses the language of a sparsity prior in a broader, engineering sense: SparseInfer provides a training-free, sign-bit-based mechanism that functions as an LSP-like activation sparsity prior for ReLU-fied LLMs (Shin et al., 2024), while WiSparse introduces a weight-aware sparsity prior and mixed-granularity allocation for training-free activation sparsity during inference (Chen et al., 16 Feb 2026). The term therefore has both a formal Bayesian meaning in variable selection and a looser inductive-bias meaning in LLM inference systems.

1. Conceptual scope and terminology

In the 2026 paper "LLM Sparsity Prior for Robust Feature Selection" (Skinner et al., 21 May 2026), LSP is introduced to address the use of LLM-generated prior information in high-dimensional variable selection. The core design choice is to treat LLM outputs as soft prior knowledge rather than as hard penalties. Instead of directly forcing the optimization procedure to privilege LLM-preferred variables, LSP embeds LLM-derived ordinal weights into prior inclusion probabilities.

The same phrase, or closely related phrasing, appears in later efficient-inference work with a different meaning. SparseInfer is described as best understood as an LSP-like, training-free approximation to an “LLM Sparsity Prior” for ReLU-fied LLMs, rather than a learned prior in the probabilistic sense (Shin et al., 2024). WiSparse similarly frames its method as introducing a weight-aware sparsity prior and a global prior over block sensitivity, but in the context of channel selection and runtime sparsification rather than Bayesian posterior inference (Chen et al., 16 Feb 2026).

This suggests that "LLM Sparsity Prior" has become polysemous. In the strict sense, it refers to a hierarchical Bayesian prior over feature inclusion probabilities. In a broader systems sense, it refers to an inductive bias or runtime prior used to decide which activations or channels can be skipped safely during LLM inference.

2. Bayesian LSP for robust feature selection

The formal LSP framework begins with ordinal LLM-generated weights

$w \in \{1,\ldots,K\}^p,$

where larger values indicate that a feature is more likely to matter (Skinner et al., 21 May 2026). These weights are transformed into inclusion probabilities for latent feature indicators: $$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$ The defining transformation is

$$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$

where $s \in (0,1)$ is the global sparsity level and $\eta \in (0,\eta_{\max})$ is the weight concentration or contrast parameter (Skinner et al., 21 May 2026).

This construction separates two roles that are often entangled in weighted sparse methods. The parameter $s$ governs the baseline inclusion level, analogous to the usual sparsity prior in spike-and-slab models. The parameter $\eta$ controls how strongly the LLM ranking differentiates features. As $\eta \to 0$,

$\theta_j \to s,$

so the model ignores the weights and reduces to a standard sparse Bayesian model (Skinner et al., 21 May 2026). Larger $\eta$ amplifies differences between high-ranked and low-ranked features.

LSP is designed to plug into both Spike-and-Slab and Spike-and-Slab Lasso formulations. For standard linear regression,

$$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$0

the Spike-and-Slab prior is

$$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$1

The Spike-and-Slab Lasso prior is

$$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$2

In both cases, LSP modifies the prior inclusion probabilities $$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$3 rather than the likelihood or objective (Skinner et al., 21 May 2026).

3. Motivation: robustness relative to LLM-Lasso

LSP was introduced because existing LLM-based feature-selection methods were argued to be sensitive to the quality of LLM-generated weights. The paper contrasts LSP with LLM-Lasso, which uses weights inside a weighted Lasso objective: $$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$4 In this setup, larger weights reduce the penalty and encourage inclusion (Skinner et al., 21 May 2026).

The stated concern is that LLM-Lasso ties the LLM directly to the optimization objective, so inaccurate weights can distort the fit. The paper further argues that even with cross-validation over $$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$5, moderate-quality weights can mislead the tuning procedure into a poor solution (Skinner et al., 21 May 2026). LSP addresses this by relocating the LLM contribution from the penalty term to the prior inclusion probabilities.

The central robustness mechanism is hierarchical rather than deterministic. Instead of fixing the degree of trust placed in the LLM, LSP lets the model infer how much the weights should matter. The claimed failure mode therefore differs from that of LLM-Lasso: LLM-Lasso can be substantially worse than plain Lasso when weights are poor, whereas LSP is designed so that performance is lower-bounded by the corresponding baseline (Skinner et al., 21 May 2026).

A plausible implication is that LSP reframes LLM guidance as a calibration problem rather than an objective-shaping commitment. That distinction is central to its interpretation as a prior rather than a penalty heuristic.

4. Hierarchical specification and weight-quality framework

The robustness of LSP is driven by hyperpriors on the two governing parameters. The prior on global sparsity is

$$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$6

For the weight concentration parameter, the paper recommends a zero-inflated discrete uniform prior: $$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$7 where $$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$8 is a point mass at zero, $$\gamma_j \mid \theta_j \sim \text{Bernoulli}(\theta_j), \qquad j=1,\ldots,p.$$9 is the prior probability of falling back to no LLM influence, and $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$0 is a finite grid of positive candidate values (Skinner et al., 21 May 2026). If posterior mass concentrates at $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$1, the model effectively reverts to an uninformative sparse baseline.

The same paper also introduces a formal framework for quantifying the quality of LLM-generated weights relative to the true active set. Let $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$2 be the true inclusion vector and $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$3 the generated weights. The first metric is the $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$4 weight agreement: $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$5 The second is a pairwise ranking agreement defined through

$$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$6

and

$$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$7

The paper interprets $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$8 as near-perfect alignment, $$\theta_j = s \, \frac{w_j^\eta}{\frac{1}{p}\sum_{k=1}^p w_k^\eta},$$9 as no real information, and $s \in (0,1)$0 as weights that actively oppose the truth (Skinner et al., 21 May 2026).

These metrics are not part of the prior itself, but they are integral to the paper’s claim of robustness across weight regimes. They provide a controlled way to examine whether LSP benefits only from high-quality LLM guidance or can also recover gracefully when the guidance is poor.

5. Estimation, prompting, and empirical behavior

LSP is presented as compatible with standard Bayesian variable-selection algorithms. For Spike-and-Slab, the paper adapts Add-Delete-Swap MCMC by modifying the acceptance ratio to account for heterogeneous $s \in (0,1)$1, and it also uses the LLM weights to initialize $s \in (0,1)$2 (Skinner et al., 21 May 2026). If $s \in (0,1)$3 has the zero-inflated discrete prior, it is updated within Gibbs/MCMC by evaluating its posterior on the discrete grid. For Spike-and-Slab Lasso, coordinate descent is run over $s \in (0,1)$4 values, and for each $s \in (0,1)$5, the $s \in (0,1)$6 that maximizes the joint posterior is selected (Skinner et al., 21 May 2026).

The prompt engineering strategy is deliberately ordinal rather than probabilistic. Instead of asking the LLM for explicit inclusion probabilities, the model is asked to produce integer importance scores from 1 to 5. In the Acute Kidney Injury application, the prompt has five modules: Background, Task, Constraints, Scoring rubric, and Formatting (Skinner et al., 21 May 2026). The prompt further instructs the model to prefer physiology over mere measurement frequency, use conservative scores for early post-op windows, cap certain aggregated summaries to avoid redundancy, distinguish $s \in (0,1)$7, $s \in (0,1)$8, $s \in (0,1)$9, and $\eta \in (0,\eta_{\max})$0 appropriately, and assign higher scores only in the closest time window to the outcome (Skinner et al., 21 May 2026).

The empirical setup includes simulations with $\eta \in (0,\eta_{\max})$1 predictors, 20 active features, correlated Gaussian design, and two low-data settings, $\eta \in (0,\eta_{\max})$2 and $\eta \in (0,\eta_{\max})$3 (Skinner et al., 21 May 2026). Weight quality is varied across

$\eta \in (0,\eta_{\max})$4

The reported findings are that all LLM-informed methods improve as weight quality improves, LSP is robust across all weight regimes, and LLM-Lasso is unstable and can perform substantially worse than ordinary Lasso when weights are only moderate or poor (Skinner et al., 21 May 2026).

In the Baylor College of Medicine cardiothoracic surgery EMR cohort for predicting AKI, the data comprise patients from 2017–2022, five subpopulations, sample sizes roughly $\eta \in (0,\eta_{\max})$5 to $\eta \in (0,\eta_{\max})$6, about $\eta \in (0,\eta_{\max})$7 to $\eta \in (0,\eta_{\max})$8 EMR-derived predictors, and an outcome defined as the ratio of postoperative creatinine at 60 hours to baseline creatinine (Skinner et al., 21 May 2026). The LLM prompting uses GPT-5.2o, zero-shot, greedy decoding with temperature 0. LSP (SS) improves over standard Spike-and-Slab in all five subgroups, with average MSE reduction about 5% across subsets, while LSP (SSL) improves over standard Spike-and-Slab Lasso in four of five subsets (Skinner et al., 21 May 2026). Using the median probability model, LSP (SS) selects Max Creatinine Ratio (25–36h), Intraoperative Red Blood Cell Transfusion, and Max Creatinine Ratio (13–24h), with the paper emphasizing that LSP identifies RBC transfusion, a known AKI risk factor, which baseline Spike-and-Slab misses (Skinner et al., 21 May 2026).

The paper also reports robustness to prompt variation. LSP (SS) beats baseline in 24 out of 25 cases across prompt alternatives and multiple LLM draws, and the advantage of LSP grows as sample size decreases when the AKI data are subsampled to $\eta \in (0,\eta_{\max})$9 (Skinner et al., 21 May 2026).

6. LSP-like priors in efficient LLM inference

Although the formal Bayesian LSP is a feature-selection framework, two later LLM inference papers use closely related prior concepts in a different technical domain.

SparseInfer targets LLM decoding for ReLU-fied LLMs, where MLPs dominate runtime and ReLU induces activation sparsity in

$s$0

Its predictor estimates whether a row of $s$1 will be negative or positive without computing the full dot product. For each row $s$2, it looks only at the sign bits of $s$3 and $s$4, uses XOR to infer the sign of each elementwise product, counts predicted positive versus negative products, and predicts the dot product output to be negative if negative signs dominate (Shin et al., 2024). The basic rule is

$s$5

with a tunable conservative version

$s$6

Predicted negative outputs are treated as zeros after ReLU, creating a runtime sparsity mask used to skip GEMV rows (Shin et al., 2024).

The paper explicitly states that SparseInfer does not learn a sparsity prior from data the way DEJAVU does. Instead, it approximates activation sparsity online using only sign bits and a statistical assumption that the sign of the dot product $s$7 can often be inferred from the majority sign of elementwise products (Shin et al., 2024). It is therefore characterized as a training-free, sign-distribution prior rather than a learned or Bayesian prior.

WiSparse uses a different sparsity prior in efficient inference. It argues that activation magnitude alone can mis-rank channels because a small activation may still align with a highly important weight column, and that sparsity sensitivity varies non-monotonically across blocks (Chen et al., 16 Feb 2026). Its saliency score for channel $s$8 in layer $s$9 is

$\eta$0

A channel is kept if

$\eta$1

with the threshold chosen according to the desired keep ratio (Chen et al., 16 Feb 2026). WiSparse also distributes a global sparsity budget across Transformer blocks via evolutionary search and refines sparsity within blocks by minimizing reconstruction error.

The paper explicitly frames this as an inductive bias: a local prior over channel importance based on activation magnitude and weight-column norm, a global prior over block sensitivity, and a coarse-to-fine prior structure that protects fragile regions and allocates sparsity where it hurts least (Chen et al., 16 Feb 2026). This usage differs from Bayesian LSP, but both share the idea that sparsity decisions should encode structured prior beliefs rather than rely solely on naïve heuristics.

7. Significance, distinctions, and common misconceptions

The primary significance of LSP in the strict sense lies in how it reframes LLM assistance for sparse modeling. Rather than treating the LLM as an oracle whose weights should directly reshape the penalty landscape, LSP treats the LLM as a noisy source of prior information whose influence can be strengthened or suppressed by posterior learning (Skinner et al., 21 May 2026). This is the central technical distinction between LSP and LLM-Lasso.

A common misconception is to equate any LLM-guided sparsity mechanism with the Bayesian LSP framework. That conflation is not accurate. SparseInfer and WiSparse are related because they operationalize prior beliefs about sparsity structure, but they do so for inference-time computation rather than statistical variable selection. SparseInfer is training-free and uses sign bits to estimate which ReLU activations will be zero (Shin et al., 2024). WiSparse uses activation-weight saliency and heterogeneous block allocation to determine which channels to skip (Chen et al., 16 Feb 2026). Neither paper formulates a posterior over inclusion indicators analogous to the Bayesian LSP of (Skinner et al., 21 May 2026).

Another misconception is that robustness in LSP means insensitivity to all prompt or weight errors. The paper’s actual claim is narrower: hierarchical hyperpriors, particularly the zero-inflated prior on $\eta$2, allow the model to discount uninformative or misleading weights and, in the limit $\eta$3, revert to the baseline sparse model (Skinner et al., 21 May 2026). This is adaptive discounting, not immunity to poor elicitation.

Taken together, the literature suggests two coherent meanings of LLM sparsity prior. In Bayesian statistics, it denotes a formal prior inclusion mechanism for robust feature selection. In efficient inference, it denotes a structured inductive bias about which activations, channels, or blocks are expendable. The former is centered on posterior robustness to uncertain LLM guidance; the latter on runtime efficiency through structured sparsification.