Spike-and-Slab Group Lasso

• Spike-and-Slab Group Lasso is a Bayesian framework that uses a mixture of spike and slab priors to distinguish noisy groups from informative ones.
• It applies a nonconvex penalty that balances strong shrinkage for near-zero coefficients with reduced bias for large signals.
• Efficient inference is achieved via EM, MCMC, and Variational Bayes, ensuring scalability and optimal theoretical guarantees.

The Spike-and-Slab Group Lasso (SSGL) is a Bayesian methodology for group-wise variable selection and estimation in high-dimensional models, unifying shrinkage-inducing continuous group lasso priors with discrete latent selection indicators at the group level. By imposing a mixture prior—typically a combination of a "spike" (strong shrinkage / near-zero) and a "slab" (weak shrinkage / diffuse)—on the ℓ₂-norm of group coefficients, SSGL adaptively distinguishes between noisy or inactive groups and informative, signal-carrying groups. This construction extends the classical group lasso by providing exact sparsity, adaptive penalization across diverse signal magnitudes, formal uncertainty quantification, and optimal contraction and selection properties across a wide range of linear, generalized linear, graphical, and nonparametric settings.

1. Mathematical formulation and prior structure

Let $(x_i, y_i)$, $i=1,\dots,n$, denote independent samples, where $x_i \in \mathbb{R}^p$ is partitioned into $G$ groups: $x_i = (x_{i1}^T, \dots, x_{iG}^T)^T$, each $x_{ig} \in \mathbb{R}^{m_g}$ so that $\sum_{g=1}^G m_g = p$. The parameter vector is accordingly grouped: $\beta = (\beta_1^T, \dots, \beta_G^T)^T$, $\beta_g \in \mathbb{R}^{m_g}$.

The classical group lasso places a Laplace (multivariate) prior on each group: $$\Psi(\beta_g \mid \lambda) = C_g\, \lambda^{m_g} \exp(-\lambda \|\beta_g\|_2)$$ where $i=1,\dots,n$0 is a normalizing constant. The Spike-and-Slab Group Lasso instead specifies a mixture prior: $i=1,\dots,n$1 with $i=1,\dots,n$2 ("slab" and "spike" scales), and $i=1,\dots,n$3 mixing proportion. Through introduction of latent indicators $i=1,\dots,n$4, each group prior may be seen as

$i=1,\dots,n$5

This form enforces strong shrinkage near zero (via $i=1,\dots,n$6) but permits heavy tails elsewhere (via $i=1,\dots,n$7), creating sharp separation between inactive and active groups.

SSGL extends to a variety of likelihoods, including canonical and non-canonical generalized linear models, grouped regression, graphical models, additive models, and Bayesian neural networks (Bai, 2020, Xu et al., 2015, Lee et al., 2019, Bai et al., 2019, Jantre et al., 2023).

2. Penalty interpretation and MAP estimation

The SSGL prior induces a nonconvex penalty on grouped coefficients. The negative log-posterior (up to a constant) is

$i=1,\dots,n$8

where

$i=1,\dots,n$9

This groupwise penalty behaves as a group lasso ($x_i \in \mathbb{R}^p$0) for small $x_i \in \mathbb{R}^p$1 (spike contribution), but levels off for large $x_i \in \mathbb{R}^p$2 (slab contribution), mitigating shrinkage-induced bias for large signals.

The maximum a posteriori (MAP) estimator thus solves a nonconvex, sparsity-promoting objective: $x_i \in \mathbb{R}^p$3 Theoretical results guarantee the existence of a MAP solution under mild conditions (Bai, 2020, Bai et al., 2019).

3. Inference algorithms and computational strategies

Multiple computational paradigms support SSGL inference:

• Expectation-Maximization (EM): EM treats latent group indicators as missing data. In the E-step, group inclusion probabilities are updated:

$x_i \in \mathbb{R}^p$4

The M-step updates $x_i \in \mathbb{R}^p$5 by a weighted group lasso, with weights $x_i \in \mathbb{R}^p$6. These steps are typically implemented via block coordinate descent and IRLS for GLMs (Bai, 2020).

• MCMC: Posterior sampling alternates between updating latent indicators $x_i \in \mathbb{R}^p$7, group parameters $x_i \in \mathbb{R}^p$8, hyperparameters (e.g., mixing weight $x_i \in \mathbb{R}^p$9), and noise variance (e.g., via Gibbs steps or auxiliary-variable formulations). MCMC methods provide full posterior uncertainty quantification but can be computationally intensive for large $G$0 (Xu et al., 2015, Lee et al., 2019).
• Variational Bayes (VB): VB introduces a mean-field family assigning independent Bernoulli (inclusion), group Gaussian (active group coefficients), and inverse-gamma noise (with analogous slabs for neural networks), and maximizes the evidence lower bound via CAVI or stochastic-gradient methods. VB approximations attain nearly the same contraction rates as MCMC but with superior scalability (Komodromos et al., 2023, Jantre et al., 2023).
• Specialized Algorithms: For graphical models, EM with path-following over spike–slab ratio controls edge selection adaptively; for additive models, EM–Coordinate Descent algorithms combine functional and groupwise sparsity (Li et al., 2018, Guo et al., 2021).

4. Theoretical properties and contraction rates

SSGL exhibits optimal theoretical guarantees for estimation and variable selection:

• Near-minimax contraction: Both the MAP estimator and the full posterior contract at the near-minimax rate for recovery of the true sparse signal:

$G$1

where $G$2 is the number of signal groups (Bai, 2020, Bai et al., 2019).

• Posterior contraction: The posterior assigns vanishing probability to neighborhoods of the parameter space more distant than this rate, and the effective posterior dimension concentrates on $G$3 (Bai, 2020, Xu et al., 2015, Bai et al., 2019).
• Selection consistency: Under mild beta-min and design conditions, the posterior concentrates on the true group support with probability tending to one as $G$4 (Xu et al., 2015, Lee et al., 2019).
• Oracle and estimation properties: The SSGL posterior median yields consistent selection and asymptotically normal estimation under orthogonal or restricted eigenvalue-type conditions, unlike classical group lasso's suboptimal rate under consistent tuning (Xu et al., 2015).
• Extension to nonparametric and neural architectures: Instantiations for grouped basis expansion (sparse GAMs) and node-wise selection in BNNs demonstrate similar optimal contraction in function spaces, depending on group structure and network width/depth (Jantre et al., 2023, Bai et al., 2019, Guo et al., 2021).

5. Practical implementation and empirical performance

Implementations of SSGL involve careful specification of spike and slab scales (typically, $G$5 and $G$6), hierarchical priors (e.g., Beta prior on $G$7), and algorithmic choices (EM, MCMC, or VB as dictated by problem dimension and need for uncertainty quantification).

Empirically, SSGL outperforms:

• Classical group lasso and group MCP/SCAD in selection accuracy and estimation error due to its adaptive bias reduction for strong signals, exact group sparsity, and ability to quantify uncertainty.
• Posterior median thresholding identifies true effects with fewer false positives compared to cross-validated group lasso, with competitive or improved prediction error (Xu et al., 2015, Bai et al., 2019).
• MAP algorithms often achieve orders-of-magnitude speed-up over MCMC, while VB approaches achieve similar accuracy but enable credible sets and inclusion probabilities with runtime only slightly above group-lasso (Komodromos et al., 2023).

Applications include prediction of HIV drug resistance from protein sequences, sparse graphical model estimation for multiple networks, gene-expression association studies, brain MRI group selection for disease classification, and structured Bayesian neural network compression (Bai, 2020, Li et al., 2018, Lee et al., 2019, Jantre et al., 2023, Bai et al., 2019).

6. Extensions and diverse model classes

SSGL has been generalized to:

• Generalized Linear Models (GLMs): Analysis frameworks accommodate non-Gaussian models with both canonical and non-canonical links, allowing for group sparsity in high-dimensional regression, classification, and count models (Bai, 2020, Lee et al., 2019).
• Structured and functional sparsity: SSGL is embedded in models for sparse generalized additive models (GAMs), with block coordinate ascent estimation and de-biasing for uncertainty quantification (Bai et al., 2019, Guo et al., 2021).
• Graphical models: SSGL-based group and fused graphical lasso models use doubly spike-and-slab penalties for multi-network precision-matrix estimation and adaptive shrinkage with path-following (Li et al., 2018).
• Bayesian neural networks: Prior specifications adapt SSGL for structured sparsity in deep architectures, using group indicators to prune whole neurons or filters, with Gumbel-Softmax relaxations enabling scalable stochastic gradient VI and principled posterior contraction rates as a function of network topology (Jantre et al., 2023).
• Bi-level and hierarchical selection: Bi-level spike-and-slab group priors accommodate selection both at the group level and within groups, relevant for models with overcomplete or nested groupings (Xu et al., 2015).

7. Comparison with classical and alternative methods

Relative to classical group lasso and other convex penalization schemes, SSGL realizes several distinct advantages:

• Adaptive shrinkage differential: SSGL applies hard shrinkage to coefficients near zero, dramatically reducing bias for strong effects, in contrast to the uniform penalization of group lasso.
• Exact group sparsity: The mixture prior enforces coefficients to be exactly zero with positive posterior probability.
• Uncertainty quantification: Availability of posterior inclusion probabilities, credible intervals, and predictive distributions.
• Posterior contraction at optimal rates: Contrasts with the single-Laplace prior, for which the full posterior contracts slower than the MAP; with SSGL both contract at the minimax-optimal rate.
• Robustness to correlated predictors and weak signals: Selection consistency holds under weaker assumptions, and cross-validation-induced over-selection is mitigated.
• Computational options: Tractable via EM, coordinate ascent, stochastic-variational inference, blockwise MCMC, or scalable pathwise estimation, allowing use in high-dimensional settings.

Empirically, SSGL achieves lower false positive rates, improved estimation error, and more parsimonious support recovery than group lasso, SCAD, or MCP, in simulation and real-world studies (Xu et al., 2015, Bai et al., 2019, Lee et al., 2019, Komodromos et al., 2023). For large-scale or structured models (graphical, neural, GAMs), SSGL integrates functional or architectural sparsity with principled Bayesian inference (Li et al., 2018, Guo et al., 2021, Jantre et al., 2023).