---
title: Spike-and-Slab Priors in Structured Regression
url: https://www.emergentmind.com/topics/spike-and-slab-priors
type: topic
---

# Spike-and-Slab Priors in Structured Regression

Spike-and-slab priors are a class of hierarchical Bayesian mixture priors that enforce sparsity and selective shrinkage in high-dimensional models by mixing a “spike” component—highly concentrated near zero—with a diffuse “slab” component, usually assigned over blocks or groups of parameters or coefficients. In the context of structured additive regression and function selection, spike-and-slab priors are used to determine which model terms or function components are actively contributing to the predictor and which should be excluded by shrinking their coefficients strongly toward zero.

## 1. Structured Additive Regression and the Need for Function Selection

The structured additive regression framework generalizes both Gaussian and non-Gaussian regression models by allowing the linear predictor to consist of a sum of potentially complex functional terms, such as nonlinear covariate effects, spatial surfaces, varying coefficients, and random effects:
\[
\eta = \eta_0 + \sum_{j=1}^{p} X_j \beta_j
\]
where each term \( X_j \beta_j \) encodes a potentially multivariate or nonparametric function of one or more covariates. This flexibility necessitates a mechanism for function selection: (1) inclusion/exclusion of covariates, (2) selection between linear/nonlinear or different parameterizations, (3) detection of interactions and complex effects.

A spike-and-slab prior targets the selection of relevant functions/model terms by imposing strong shrinkage (the spike) on irrelevant components and allowing flexible estimation (the slab) of truly active components.

## 2. Normal Mixture-of-Inverse Gamma (NMIG) Prior: Block Spike-and-Slab Construction

For function selection where model terms are represented as blocks of basis coefficients (e.g., splines or spatial bases), the NMIG prior is constructed as follows:

- For block \( \beta_j \) (of dimension \( d_j \)), the conditional prior is
  \[
  \beta_j \mid v_j^2 \sim N(0, v_j^2 I)
  \]
- The prior variance is decomposed hierarchically:
  \[
  v_j^2 = \gamma_j \tau_j^2
  \]
  where \( \gamma_j \) is a binary indicator, and \( \tau_j^2 \sim \mathrm{Inv}\text{-}\Gamma(a_\tau, b_\tau) \). The indicator is distributed as
  \[
  \gamma_j \sim w\, \delta_1(\gamma_j) + (1-w)\, \delta_{v_0}(\gamma_j)
  \]
  with \( v_0 \ll 1 \), so when \( \gamma_j = 1 \) the “slab” is large, and for \( \gamma_j = v_0 \) the “spike” imposes near-zero variance.

When \( v_j^2 \approx 0 \), the corresponding function is excluded; when \( v_j^2 \) is large, it remains active.

## 3. Multiplicative Parameter Expansion (peNMIG): Improving MCMC Mixing

Standard Gibbs updates for coefficient blocks suffer from “stickiness” near the spike due to high posterior correlations. The paper introduces a novel multiplicative expansion:

\[
\beta_j = \alpha_j \xi_j
\]
where \( \alpha_j \in \mathbb{R} \) (scalar), \( \xi_j \in \mathbb{R}^{d_j} \). The prior is specified as:
- \( \alpha_j \sim N(0, v_j^2) \) with \( v_j^2 = \gamma_j \tau_j^2 \)
- \( \xi_{jk} | m_{jk} \sim N(m_{jk},1) \), with \( m_{jk} \sim \frac{1}{2}\delta_1(m_{jk}) + \frac{1}{2}\delta_{-1}(m_{jk}) \)

Advantages of this expansion:
- Only the scalar \( \alpha_j \) enters the block-wise spike-and-slab mechanism, eliminating poor MCMC mixing of high-dimensional blocks.
- The marginal prior on \( \beta_j \) has strong shrinkage at zero (spike) and heavy tails (slab), yielding \( \ell_q \)–like penalties for \( q<1 \), similar in behavior to horseshoe or normal–Jeffreys priors.
- The prior hierarchy is:
  \[
  \begin{aligned}
    \tau_j^2 &\sim \Gamma^{-1}(a_\tau, b_\tau) \\
    \gamma_j &\sim w \delta_1(\gamma_j) + (1-w) \delta_{v_0}(\gamma_j) \\
    \alpha_j | \gamma_j, \tau_j^2 &\sim N(0, \gamma_j \tau_j^2) \\
    \xi_{jk} | m_{jk} &\sim N(m_{jk}, 1),\quad m_{jk} \sim \frac{1}{2}\delta_1(m_{jk}) + \frac{1}{2}\delta_{-1}(m_{jk})
  \end{aligned}
  \]

## 4. Posterior Computation: Efficient MCMC via Blocked and Hierarchical Structure

- For Gaussian models, all conditional distributions have closed forms; thus, a Gibbs sampler can update all parameters efficiently:
  - Blocked updates for \( (\alpha_j, \xi_j) \) with \( \alpha_j \) controlling inclusion.
  - The indicator \( \gamma_j \) and variance \( \tau_j^2 \) are only updated based on \( \alpha_j \), not the full block \( \beta_j \), thus enabling rapid transitions between inclusion and exclusion states.
  - A rescaling step is used after updating \( \xi_j \) to keep its entries close to unit norm, maintaining identifiability.
- For non-Gaussian likelihoods, the conditional for \( (\alpha_j, \xi_j) \) is updated via Metropolis–Hastings with proposals based on iteratively weighted least squares.

## 5. Performance Characteristics, Sensitivity, and Comparative Results

- In simulation studies and classification benchmarks (e.g., UCI datasets), the peNMIG approach produces models with lower predictive deviance and higher sparsity compared to component-wise boosting, adaptive COSSO (ACOSSO), and frequentist penalization methods.
- The method is relatively robust to hyperparameter choices, especially the prior inclusion probability \( w \), except for the threshold parameter \( v_0 \), which strongly affects sparsity by controlling spike “sharpness.”
- Recovery of relevant functional terms—true model structure—is highly accurate even with complex high-dimensional predictors and when selecting among functional representations (e.g., linear vs. nonlinear, varying coefficients, spatial effects).
- Computation is feasible for hundreds of model terms due to the efficiency of the parameter-expanded sampler and is not limited by the curse of dimensionality in block sampling.

## 6. Applications: Survival, Spatial, and Mixed Model Scenarios

- **Additive piecewise exponential survival models**: The hazard function is modeled as
  \[
  \lambda(t, x_i) = \exp\left(g_0(j) + \sum_\ell g_\ell(j) v_{il}(j) + \sum_m f_m(u_{im}(j)) + z_i(j)' \gamma\right),\qquad t \in (\kappa_{j-1}, \kappa_j]
  \]
  The spike-and-slab approach allows model selection among time-varying covariate functions, testing linearity/nonlinearity, and inclusion/exclusion of functional and parametric effects.
- **Geoadditive regression**: For spatial analysis (e.g., Munich rent data), geoadditive smoothers modeled by Gaussian Markov random fields are included as blocks—spike-and-slab selection distinguishes spatial, smooth, and parametric components for interpretability.
- **Additive mixed logit models**: For categorical/discrete responses with random effects, the procedure supports selection among random, smooth, and spatial effects while maintaining a parsimonious model.

## 7. Summary and Theoretical Implications

- By combining blockwise spike-and-slab priors (with peNMIG construction) and a parameter expansion, efficient and robust function selection is feasible for structured additive regression with complex model spaces.
- The multiplicative expansion addresses critical MCMC mixing and convergence issues and induces strong shrinkage and heavy-tailed tolerance for large signals, paralleling properties of other state-of-the-art shrinkage priors.
- Spike-and-slab priors with this architecture provide a unified, computationally tractable framework for function/term selection in additive models with a broad class of effect types and interactions, and are validated by superior empirical model selection and predictive performance on benchmark data.

---

**Summary Table: Key Model Elements in Parameter-Expanded NMIG Spike-and-Slab Prior**

| Component         | Distribution               | Role in Model Selection            |
|-------------------|---------------------------|------------------------------------|
| \( \gamma_j \)    | \( w \delta_1 + (1-w)\delta_{v_0} \)  | Inclusion/exclusion of entire block|
| \( \tau_j^2 \)    | \( \Gamma^{-1}(a_\tau, b_\tau) \)      | Controls block-wise variance magnitude|
| \( \alpha_j \)    | \( N(0, \gamma_j\tau_j^2) \)           | Overall scale, drives block activity|
| \( \xi_{jk} \)    | \( N(m_{jk}, 1) \), \( m_{jk} = \pm 1 \) | Unit-norm constraint, identifies structure|

---

Source: https://www.emergentmind.com/topics/spike-and-slab-priors