Selection Confidence Set (SCS)

• Selection Confidence Set (SCS) is a set-valued inferential tool that replaces a single best model with all candidate models not statistically distinguishable from the optimum under likelihood-ratio screening.
• It addresses model-selection uncertainty by providing robust coverage guarantees across various applications, including variable selection, mixture order determination, and portfolio construction.
• Extensions such as adaptive stochastic search and sequential confidence sets enhance SCS methodology, allowing for effective inference even in complex or high-dimensional settings.

Searching arXiv for core and papers on Selection Confidence Sets and closely related model-selection confidence set formulations. Selection Confidence Set (SCS) denotes a set-valued inferential object that replaces a single selected model, order, subset, or assignment with a collection of statistically plausible selections at confidence level $1-\alpha$. Across the literature, the central idea is stable: rather than asking for one “best” choice, the procedure reports all candidates that are not statistically distinguishable from the true or optimal object under a specified screening rule. In this sense, SCS extends the familiar confidence-interval logic from parameters to discrete selection problems such as model choice, mixture order selection, variable subsets, equally weighted portfolios, and group membership assignments (Zheng et al., 2017). Recent work on Gaussian mixture order selection makes this interpretation explicit by defining a set of plausible orders around a reference order $\hat{k}$ and proving asymptotic coverage of the true order $k_0$ (Casa et al., 24 Mar 2025).

1. Conceptual foundation

The unifying motivation for SCS methods is model-selection uncertainty. In the formulations for general model spaces, finite mixture models, and ARMAX time-series models, the common objection to single-best selection is that multiple candidates can fit the data nearly equally well, especially when the sample is small or noisy, component separation is weak, criteria disagree, or forecasting performance is similar across several models (Zheng et al., 2017). A single selected model may therefore discard useful information and hide uncertainty that is consequential for interpretation and downstream decisions.

In the general likelihood-ratio formulation, the target is an unknown true model $\gamma^\ast$ inside a feasible model space $\Gamma$, with full model $\gamma_f$. The SCS, often called the Model Selection Confidence Set (MSCS), is the collection of candidate models that survive likelihood-ratio screening: $$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$ where

$$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$

Here $q(\alpha; p-p_\gamma)$ is the upper $\alpha$-quantile of $\hat{k}$0 (Zheng et al., 2017). In the variable-selection formulation for generalized linear models, the same logic appears as a Variable Selection Confidence Set (VSCS),

$\hat{k}$1

with $\hat{k}$2 (Zheng et al., 2015). In both formulations, the set contains all nested models that are not rejected at level $\hat{k}$3.

This framework is not limited to nested regression models. In Gaussian mixture order selection, the selected object is the number of components, not a variable subset. The paper defines

$\hat{k}$4

with $\hat{k}$5 as a reference order, typically selected by BIC, and interprets $\hat{k}$6 as the orders that are not significantly worse than the reference model $\hat{k}$7 (Casa et al., 24 Mar 2025). In the portfolio formulation, the selected object is a binary inclusion vector $\hat{k}$8, and the SCS targets the unknown optimal selection set

$\hat{k}$9

rather than a single parameter (Ferrari et al., 26 Sep 2025). This suggests that “SCS” is best understood as a general inferential schema for discrete argmin or selection problems, not a method tied to one model class.

2. Core constructions

The most common SCS construction uses screening by a test statistic relative to a benchmark or full model. In the general MSCS framework, each candidate model is tested through a likelihood ratio statistic against the full model, and the SCS retains every model that is not rejected (Zheng et al., 2017). The same mechanism underlies the time-series ARMAX construction, where a candidate model $k_0$0 is compared to a large benchmark/full model $k_0$1 via

$k_0$2

and the confidence set is

$k_0$3

The full model is always included by construction (Bortoli et al., 18 Feb 2026).

Mixture-order SCS modifies this screening principle because standard likelihood-ratio asymptotics do not apply in finite mixtures. There, each candidate order $k_0$4 is screened against the reference order $k_0$5 using a penalized likelihood ratio statistic

$k_0$6

with penalty adjustment $k_0$7 accounting for complexity differences. The paper gives AIC-type, BIC-type, and TIC corrections, with nominal parameter count $k_0$8 for the Gaussian mixture (Casa et al., 24 Mar 2025). The reference order $k_0$9 is always included by default, and other orders are screened relative to the null quantile $\gamma^\ast$0.

A distinct construction appears in equally weighted portfolio selection. The candidate object is a subset of assets encoded by $\gamma^\ast$1, and a portfolio is retained if its studentized loss differential relative to the empirical optimum $\gamma^\ast$2 is not too large: $\gamma^\ast$3 where

$\gamma^\ast$4

The target here is the optimal selection set $\gamma^\ast$5, not merely one empirical optimum (Ferrari et al., 26 Sep 2025).

A more specialized SCS construction arises in grouped panel models, where the selected object is the latent group-membership vector $\gamma^\ast$6. The confidence set is built by inverting simultaneous unit-specific one-sided tests of

$\gamma^\ast$7

using moment inequalities and a max-type statistic

$\gamma^\ast$8

The joint confidence set is formed as a Cartesian product of marginal unit-wise sets,

$\gamma^\ast$9

and covers the true membership vector jointly for all units with probability at least $\Gamma$0 asymptotically (Dzemski et al., 2017).

3. Coverage guarantees and asymptotic theory

The defining property of an SCS is coverage of the true selected object. In the general MSCS framework based on likelihood-ratio testing, the asymptotic guarantee is

$\Gamma$1

and if $\Gamma$2, then

$\Gamma$3

(Zheng et al., 2017). In the generalized-linear-model VSCS, the corresponding claim is

$\Gamma$4

with coverage tending to $\Gamma$5 when the full model equals the truth (Zheng et al., 2015).

Mixture-order selection requires different asymptotics because mixture models violate standard regularity conditions. The null distribution is derived from the asymptotic results of Vuong (1989) and Lo et al. (2001), leading to a weighted chi-square limit rather than ordinary chi-square: $\Gamma$6 Under assumptions including compact $\Gamma$7, positivity and continuity of $\Gamma$8, integrable envelope conditions for $\Gamma$9, uniqueness of the population maximizer $\gamma_f$0, and $\gamma_f$1, the mixture-order MSCS satisfies

$\gamma_f$2

(Casa et al., 24 Mar 2025).

The portfolio SCS is formulated as a confidence set for an argmin over a combinatorial space. Its main coverage statement is

$\gamma_f$3

The same paper also characterizes expected set size,

$\gamma_f$4

where $\gamma_f$5 is the standardized loss differential (Ferrari et al., 26 Sep 2025). This formalizes a recurrent qualitative conclusion in the SCS literature: the set enlarges when many candidates are nearly tied and shrinks as separation or sample size increases.

The high-dimensional theory in the general MSCS paper goes beyond coverage. For models missing at least one true variable, the likelihood-ratio statistic is asymptotically noncentral chi-square, with noncentrality parameter

$\gamma_f$6

A sufficient detectability condition is

$\gamma_f$7

which implies asymptotic rejection of misspecified models that omit important variables (Zheng et al., 2017). This suggests that SCS methods are not only coverage devices; under signal conditions they can asymptotically isolate the relevant structural information.

4. Boundary models, importance measures, and interpretation

Because an SCS may be large, several papers introduce summaries that preserve the set-valued perspective while making the result interpretable. A central device is the Lower Boundary Model (LBM), defined as a model in the confidence set with no nested submodel also in the confidence set. In the GLM variable-selection setting, LBMs are described as the most parsimonious models that still survive the screening, and in ARMAX time-series models the LBM set $\gamma_f$8 is the parsimonious subset of the MSCS (Zheng et al., 2015). Even when the full confidence set is large, the number of LBMs can remain small, which isolates a compact set of essential admissible structures (Bortoli et al., 18 Feb 2026).

Variable or term importance is then quantified by inclusion frequency across the SCS or across LBMs. In the general MSCS framework, inclusion importance for parameter element $\gamma_f$9 is

$$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$0

Under detectability, true variables satisfy $$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$1, while irrelevant variables are not expected to substantially exceed $$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$2 (Zheng et al., 2017). The AMD SNP paper refines this viewpoint by defining marginal, joint, and conditional inclusion importance over the LBM set: $$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$3

$$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$4

$$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$5

These are displayed as an inclusion-importance network, with node size given by marginal importance and directed edge thickness by conditional importance (Zheng et al., 2015).

The time-series MSCS paper distinguishes importance over the full MSCS from importance over LBMs. For parameter $$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$6, LBM inclusion importance is

$$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$7

and MSCS-wide inclusion frequency is

$$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$8

The normalization reflects the observation that irrelevant predictors may appear in about half of the models in the MSCS (Bortoli et al., 18 Feb 2026).

Portfolio SCS adopts analogous diagnostics. Inclusion importance for asset $$\widehat{\Gamma}_\alpha \equiv \left\{\gamma\in\Gamma:\Lambda_\gamma \le q(\alpha; p-p_\gamma)\right\},$$9 is

$$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$0

and co-inclusion importance for assets $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$1 is

$$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$2

These summaries identify assets that are structurally important across plausible portfolios even when they do not appear in the empirical optimum (Ferrari et al., 26 Sep 2025). A common misconception is therefore that an SCS is merely a larger version of a selected model. The literature instead treats the set’s size, its lower boundary, and inclusion frequencies as the primary diagnostics of selection uncertainty.

5. Major application domains

The SCS methodology has been instantiated in several distinct selection problems. In finite Gaussian mixtures, the object is the order $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$3, and the confidence set reports all plausible numbers of components around a reference choice. Simulation results for four Gaussian mixture densities, with sample sizes $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$4 and confidence levels $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$5, show that coverage is at or above the nominal level in essentially all settings, that the set tends toward a singleton $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$6 in easier settings, and that it remains wider in harder settings, reflecting genuine ambiguity in order selection (Casa et al., 24 Mar 2025).

In univariate time-series models involving autoregressive and moving average components, the MSCS identifies a set of ARMAX specifications that are statistically indistinguishable from the true data-generating process at a given confidence level. The size and composition of the set reveal model-selection uncertainty, and the LBM and inclusion-importance summaries isolate robust drivers such as lagged load, temperature, calendar effects, and solar generation in Italian hourly electricity load data (Bortoli et al., 18 Feb 2026). This application emphasizes that SCS is useful not only for inference but also for forecasting, because models inside the MSCS have much better and more stable out-of-sample RMSE/MAE, while LBMs remain parsimonious.

In financial selection problems, the selected object can be an equally weighted portfolio. For a universe of $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$7 assets, a selection is a binary vector $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$8, and the SCS collects all portfolios statistically indistinguishable from the unknown optimal portfolio(s) under a chosen loss function. The paper reports that, at the 95% confidence level, the French 17-Industry Portfolios application yields 408 portfolios in the SCS under mean-variance loss, while the Layer-1 cryptocurrencies application yields 122 portfolios under Sharpe ratio loss and 288 portfolios under mean-variance loss (Ferrari et al., 26 Sep 2025). The substantive point is that many allocations can be statistically indistinguishable from the empirical optimum.

In grouped panel models, SCS takes the form of a confidence set for the entire vector of latent group assignments. The procedure quantifies uncertainty about data-driven clustering by producing, for each unit, a set of groups that are not statistically ruled out, then aggregating these unit-wise sets into a joint confidence set over $$\Lambda_\gamma \equiv 2\left\{\ell_n(\hat{\boldsymbol\theta}_{\gamma_f})-\ell_n(\hat{\boldsymbol\theta}_\gamma)\right\}.$$9 (Dzemski et al., 2017). The resulting object is not a confidence set for coefficients but for the selected partition itself.

In genetic association studies, the VSCS was used to rank SNP combinations for age-related macular degeneration. At 90%, 95%, and 99% confidence, the VSCS contained 30,827, 57,273, and 156,459 models, while the corresponding LBM sets contained 43, 79, and 103 models (Zheng et al., 2015). This application illustrates the characteristic SCS pattern: very large overall ambiguity, but a much smaller boundary of parsimonious admissible models.

6. Extensions, computation, and related post-selection frameworks

When the feasible model space is large, exhaustive enumeration of all candidate models may be infeasible. The general MSCS paper therefore proposes MSCS-AS, an adaptive stochastic search algorithm inspired by cross-entropy methods. It samples model indicators $q(\alpha; p-p_\gamma)$0 from

$q(\alpha; p-p_\gamma)$1

updates the working level through the empirical p-value quantile,

$q(\alpha; p-p_\gamma)$2

and smooths the inclusion probabilities via

$q(\alpha; p-p_\gamma)$3

Reported practical choices are $q(\alpha; p-p_\gamma)$4, $q(\alpha; p-p_\gamma)$5, $q(\alpha; p-p_\gamma)$6, and $q(\alpha; p-p_\gamma)$7 (Zheng et al., 2017). The aim is to sample MSCS models with high probability rather than search the full $q(\alpha; p-p_\gamma)$8 space.

A different extension is sequential. Sequential model confidence sets replace fixed-sample inference with e-processes and confidence sequences, producing a time-indexed set $q(\alpha; p-p_\gamma)$9 that satisfies the time-uniform coverage property

$\alpha$0

The paper distinguishes strongly superior, uniformly weakly superior, and weakly superior target sets, and uses sequential testing or confidence sequences for pairwise loss gaps to maintain anytime-valid coverage (Arnold et al., 2024). This indicates that SCS ideas extend naturally to online monitoring and forecast evaluation.

The broader post-selection literature also clarifies when fixed-target confidence sets remain valid after selection. In the black-box selection setting, if $\alpha$1 is the selected object, $\alpha$2 is the corresponding target, and $\alpha$3 is a confidence set valid for each fixed $\alpha$4, then selected-target noncoverage obeys

$\alpha$5

and hence

$\alpha$6

Sample splitting is the zero-leakage case, with exact recovery of the fixed-target guarantee (Banerjee, 29 Apr 2026). This does not itself define a classical SCS, but it places SCS-style selected-target inference into a more general information-theoretic framework.

A separate but related line concerns post-selection confidence sets after lasso selection. There the confidence set is built conditionally on the observed lasso active set $\alpha$7 by sampling from $\alpha$8 using randomized estimator augmentation and Markov chain Monte Carlo (Min et al., 2019). This is a confidence-set construction after selection, but its target is a post-selection parameter such as $\alpha$9, not the selected model set itself. A plausible implication is that the SCS literature and the post-selection-parameter literature solve adjacent but distinct inferential problems: one covers discrete selected objects, while the other covers parameters conditional on a selected object.