BoATS: Adaptive Sparse Model Selection

• BoATS is a sparse regression method that first selects variables by thresholding coefficients against an empirically estimated null distribution.
• It refits the selected coefficients via ordinary least squares, ensuring minimal bias and accurate recovery of true signal magnitudes.
• The method has been benchmarked in simulations and neuroscience applications, showing improved model parsimony and interpretability compared to traditional regularizers.

Searching arXiv for the primary BoATS paper and closely related threshold-selection literature to ground the article in published work. Bootstrapped Adaptive Threshold Selection, or BoATS, is a sparse model selection and estimation procedure introduced in “Bootstrapped Adaptive Threshold Selection for Statistical Model Selection and Estimation” (Bouchard, 2015). It is designed for settings in which the underlying input–output relationship is believed to be sparse and where the objective is not only prediction but also recovery of interpretable coefficients with as little distortion as possible. The method combines adaptive thresholding relative to an empirically estimated null distribution of coefficient magnitudes with ordinary least-squares refitting, thereby providing what the authors describe as a minimally structured alternative to $L_1$, $L_2$, and Elastic Net regularization (Bouchard, 2015).

1. Definition and statistical objective

BoATS is a two-stage sparse estimation method. First, it performs variable selection by thresholding an initial coefficient estimate relative to an empirically estimated null distribution of coefficient magnitudes. Second, it refits the retained coefficients by ordinary least squares (OLS), without shrinkage (Bouchard, 2015). Its stated motivation is that standard structured regularizers improve stability and prediction by imposing explicit structural priors on coefficient values, but these priors can bias weights toward zero and complicate interpretation when the true nonzero coefficients do not resemble the assumed prior geometry (Bouchard, 2015).

The modeling framework in the paper is multilinear regression with i.i.d. inputs and additive Gaussian noise. The response is written as

$$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$

with coefficient vector $\beta \in \mathbb{R}^d$ and $m$ observations $Z_i=(x_i,y_i)$ (Bouchard, 2015). Estimation uses least-squares loss,

$$\hat{\beta} = \arg\min_{\beta \in \mathbb{R}^d} \sum_{i=1}^{m} (y_i - \beta^\top x_i)^2.$$

The sparse model selection problem is expressed through the support

$\operatorname{supp}(\beta)=\{j:\beta_j\neq 0\},$

with sparsity described by the $L_0$ “norm”

$$\|\beta\|_0 = \sum_{j=1}^{d}\mathbf{1}(|\beta_j|>0).$$

A model is sparse when only $L_2$0 out of $L_2$1 coefficients are nonzero, with $L_2$2 (Bouchard, 2015). In this setting, BoATS is intended to estimate coefficients unbiasedly or with minimal bias, set truly null coefficients exactly to zero, and yield reliable or low-variability results (Bouchard, 2015).

2. Core algorithmic procedure

The method is presented as adaptive threshold selection followed by OLS refitting, embedded in repeated train/validation/test splits and bootstrap resampling (Bouchard, 2015). Its first step is to estimate what coefficient values look like under no real relationship between inputs and outputs. This is done by randomizing or permuting the pairing between $L_2$3 and $L_2$4, forming randomized datasets $L_2$5, and refitting the regression repeatedly. The expected null estimate is

$L_2$6

The observed data are partitioned into three non-overlapping subsets: $L_2$7 for training, $L_2$8 for validation or selection, and $L_2$9 for final testing (Bouchard, 2015). An initial model is fit on the training set,

$$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$0

and then thresholded relative to the null magnitude. For threshold multiplier $$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$1, the paper uses the rule

$$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$2

Thus a feature is discarded if its estimated magnitude is less than a chosen multiple of the expected null magnitude for that coefficient (Bouchard, 2015).

After thresholding, the remaining predictors are refit by OLS on the training data: $$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$3 The threshold is chosen adaptively by minimizing out-of-sample loss on the selection set,

$$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$4

Final performance is then assessed on $$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$5, which is not used for fitting or threshold selection (Bouchard, 2015).

To estimate expected performance and variability, the full train/select/test procedure is embedded in a 100-iteration bootstrap procedure in which the data splits are rerandomized each time (Bouchard, 2015). The paper states that the same cross-validation and bootstrapping framework was used for all compared methods.

3. Relation to structured regularization

The paper situates BoATS against regularized estimators of the form

$$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$6

and specifically against the elastic-net-type objective

$$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$7

These regularizers are interpreted as imposing an approximate prior

$$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$8

with $$y = x^\top \beta + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0,\sigma^2),$$9 corresponding to a Laplace prior, $\beta \in \mathbb{R}^d$0 to a Gaussian prior, and Elastic Net to mixed structure (Bouchard, 2015).

BoATS is called minimally structured because it imposes only one substantive assumption, namely sparsity, while avoiding an additional requirement that nonzero coefficients conform to a particular parametric prior distribution (Bouchard, 2015). In contrast, Ridge shrinks all coefficients smoothly toward zero, Lasso encourages sparsity but also shrinks nonzero coefficients, and Elastic Net combines variable selection with grouped shrinkage (Bouchard, 2015). The paper explicitly frames this as a bias–variance tradeoff: hard-thresholding methods are more unbiased but can be more variable, whereas smooth regularizers are less variable but more biased (Bouchard, 2015).

This contrast is central to the interpretive role of BoATS. Because thresholding removes coefficients likely due to noise and OLS refits the surviving predictors without a penalty term, the retained coefficients are not shrunk after selection (Bouchard, 2015). The resulting models can therefore be sparser, set unsupported coefficients exactly to zero, and estimate surviving coefficients at more realistic magnitudes. This suggests that BoATS is especially attractive in scientific applications where support recovery and coefficient magnitude are themselves inferential targets, not merely intermediate quantities for prediction.

4. Performance in numerical experiments

The paper reports extensive simulation studies comparing Ridge Regression, Lasso, Elastic Net with 50/50 $\beta \in \mathbb{R}^d$1, and BoATS, which is referred to in figures and text as ATS-OLS or ATS-OLS Refit (Bouchard, 2015). The simulations vary sparsity, sample size relative to dimensionality $\beta \in \mathbb{R}^d$2, noise magnitude, and the distribution of the true nonzero coefficients (Bouchard, 2015). Sparsity is described as

$\beta \in \mathbb{R}^d$3

and sample size as

$\beta \in \mathbb{R}^d$4

The true nonzero coefficients are drawn from several distributions: Laplace, Uniform, Symmetric increasing exponential, and Asymmetric clustered distribution (Bouchard, 2015). One motivation is to test cases where the actual coefficient distribution diverges from the Laplace or Gaussian priors implicit in standard regularizers. In a highlighted example with 200 null dimensions, sparsity around $\beta \in \mathbb{R}^d$5, $\beta \in \mathbb{R}^d$6 with $\beta \in \mathbb{R}^d$7, and moderate noise, the paper reports that structured regularizers exhibited visible shrinkage bias and compressed predictive dynamic range, whereas BoATS recovered large and intermediate weights more accurately and set nearly all null dimensions exactly to zero (Bouchard, 2015).

The principal metrics are estimation error, variability, prediction accuracy, and model parsimony. Prediction accuracy is reported using

$\beta \in \mathbb{R}^d$8

and parsimony using test-set BIC,

$\beta \in \mathbb{R}^d$9

where $m$0 and $m$1 is the number of test samples (Bouchard, 2015). Estimation error is described as the RMS difference between true and estimated coefficients, and estimation variability as aggregate coefficient standard deviation across bootstrap samples (Bouchard, 2015).

A consistent regime dependence is emphasized. When $m$2 was small, especially near $m$3, all methods struggled, but BoATS did especially poorly relative to structured methods, showing higher variability and lower prediction accuracy (Bouchard, 2015). However, when the sample size was sufficiently large and the model sufficiently sparse—for example roughly $m$4 and $m$5—BoATS generally yielded smaller estimation error, comparable variability, and slightly better prediction accuracy (Bouchard, 2015). At $m$6, across the four coefficient distributions, BoATS had smaller estimation error, variability slightly greater or comparable to Lasso, and smaller selected models than all other methods; the paper notes that it often selected fewer parameters than were truly nonzero, especially for Laplace-distributed coefficients, while still achieving comparable or superior predictive performance and better BIC (Bouchard, 2015).

The noise experiments define noise as

$m$7

using the asymmetric clustered distribution and sparsity $m$8 (Bouchard, 2015). As noise increased, all methods showed increased estimation error and variability. Ridge and Elastic Net selected supports much larger than the true model, Lasso also tended to overselect, and BoATS selected supports smaller than the true support, with support size decreasing monotonically as noise increased (Bouchard, 2015). In the noiseless case, BoATS was reported to have estimation error more than 11 orders of magnitude smaller than the other methods (Bouchard, 2015). The paper characterizes BoATS as generally more accurate, similarly variable, and better at parameter selection except in the noisiest conditions.

5. Application to decoding speech from ECoG

The method is applied to a neuroscience decoding problem using previously collected human electrocorticography recordings from speech sensorimotor cortex (Bouchard, 2015). The task is to predict an acoustic feature of produced vowels from neural activity. Inputs are amplitudes of high-frequency signals extracted from ECoG field potentials on single trials, and outputs are the acoustic feature $m$9, specifically a log-transformed formant ratio, for the vowels /a/, /i/, and /u/ (Bouchard, 2015). The dataset contains $Z_i=(x_i,y_i)$0 total vocalizations.

Linear decoders were trained to map ECoG features to acoustic output values using the BoATS procedure for feature selection and weight estimation (Bouchard, 2015). Two analyses were performed: an across-vowel decoder trained jointly across vowel categories, and within-vowel decoders trained separately to predict trial-to-trial variability within each vowel (Bouchard, 2015).

The across-vowel BoATS-estimated linear model predicted about 80% of the variability, with

$Z_i=(x_i,y_i)$1

while the across-vowel decoder did not capture within-vowel variability well (Bouchard, 2015). When separate vowel-specific decoders were trained, the method also predicted within-vowel variability, with performance upwards of 30% explained variance (Bouchard, 2015). In the paper’s framing, this demonstrates that BoATS is not confined to simulation studies but can recover useful sparse linear decoders from high-dimensional neural recordings in a practically relevant setting (Bouchard, 2015).

A plausible implication is that the application exemplifies the methodological niche BoATS was designed to occupy: settings in which sparse support identification, coefficient interpretability, and predictive utility are all simultaneously relevant. The paper specifically associates this with neuroscience questions about which neural features contribute to behavior or stimulus representation (Bouchard, 2015).

6. Practical scope, limitations, and common confusions

The paper presents BoATS as most appropriate when the true model is believed to be sparse, coefficient interpretation matters, exact zeros are desired in the final model, the nonzero coefficient distribution may diverge from Laplace or Gaussian assumptions, and sample size is not too small relative to dimensionality (Bouchard, 2015). A practical advantage identified by the authors is that the method uses only one meta-parameter sweep, the threshold multiplier $Z_i=(x_i,y_i)$2, selected by minimizing validation loss (Bouchard, 2015). They contrast this with methods such as SCAD or relaxed Lasso, which require a two-dimensional hyperparameter search (Bouchard, 2015).

The procedure is conceptually simple but computationally nontrivial because it requires repeated null-distribution estimation by permuting input–output relationships, repeated model fitting across threshold values, and repeated bootstrap and cross-validation loops (Bouchard, 2015). The reported implementation used MATLAB and standard built-in regression tools (Bouchard, 2015). The simulations assume a linear input–output relationship, additive Gaussian noise, and i.i.d. inputs; the paper states that the method is easily extendable to other input/output distributions, mixed continuous/discrete data, and nonlinear models, but those extensions are not developed in detail (Bouchard, 2015).

The clearest failure mode is the low-sample regime in which $Z_i=(x_i,y_i)$3 is close to $Z_i=(x_i,y_i)$4. In that setting, thresholding without stronger structural assumptions is fragile, and BoATS performs substantially worse than structured regularizers in variability and prediction (Bouchard, 2015). Its advantages also diminish when the model is not strongly sparse, when noise is very large, or when support selection becomes unstable (Bouchard, 2015). The discussion further notes that non-orthogonal designs deserve further investigation, implying that the evidence in the paper is strongest for simpler designs and that correlated predictors remain a cautionary case (Bouchard, 2015). Another recurring feature is support underselection: BoATS often selects models smaller than the true support, particularly when some true coefficients are too small relative to the noise floor (Bouchard, 2015).

A common confusion arises from the phrase “adaptive threshold selection.” Not every adaptive-threshold method is BoATS. For example, “Event-based Feature Extraction Using Adaptive Selection Thresholds” introduces FEAST, whose thresholds control online event acceptance in neuromorphic unsupervised feature learning and are not estimated by resampling, not applied to regression coefficients, and not retained as a model-selection criterion at inference (Afshar et al., 2019). Likewise, “Data-Adaptive Automatic Threshold Calibration for Stability Selection” proposes ATS and EATS for calibrating stability-selection cutoffs from resampling-derived selection frequencies, using stability selection subsampling and, in EATS, a shuffled-data null frequency threshold; these methods are adjacent in spirit but operate on selection probabilities rather than coefficient magnitudes (Huang et al., 28 May 2025). Such distinctions matter because BoATS is specifically a sparse regression procedure built around a coefficient null distribution, adaptive thresholding, and post-selection OLS refitting (Bouchard, 2015).