Confidence Interval-Based Rejection

• Confidence interval-based rejection is a method that tests hypotheses by excluding the null value from a confidence set, linking classical test inversion with modern selective inference.
• It employs advanced techniques such as e-value inversion, sequential confidence sequences, and Monte Carlo methods to adjust for data-dependent selections and control error rates.
• Practical applications include A/B testing on social media and quality control in experiments, demonstrating robust post-selection error management under arbitrary dependence.

Searching arXiv for recent and foundational papers on confidence-interval-based rejection and related post-selection or sequential CI methods. {"query":"\"confidence interval\" rejection e-values post-selection inference arXiv", "max_results": 10} {"query":"(Xu et al., 2022)", "max_results": 5} Confidence interval-based rejection is an inferential scheme in which a null hypothesis is rejected by constructing a confidence interval or confidence set and checking whether the null value or null set is excluded. In its simplest form, the rule is: reject $H_0$ when $0 \notin C(X)$ or, more generally, when $C(X)\cap \Theta_0=\emptyset$. The method is the confidence-set side of test inversion, but its modern development is tied to selective inference, multiple testing, e-values, anytime-valid confidence sequences, and post-selection error criteria such as the false coverage rate (FCR). Across these settings, the central issue is no longer only marginal coverage of a single interval, but whether interval-based rejections remain valid after data-dependent selection, under arbitrary dependence, and under continuous monitoring (Xu et al., 2022).

1. Inferential principle and error criteria

The formal basis of confidence interval-based rejection is the duality between tests and confidence sets. If $A(\theta_0)$ is the acceptance region of a level-$\alpha$ test of $H_0:\theta=\theta_0$, then the inverted confidence set is

$C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$

and the rejection rule is $\theta_0\notin C_\alpha(x)$. In a broader formulation, one may choose $A(\theta_0)$ to maximize prior-weighted average power subject to the coverage constraint

$$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$

which leads to confidence sets obtained by inverting Neyman–Pearson tests against a prior-weighted mixture alternative (Bartels et al., 2019). The same inversion principle underlies conservative Monte Carlo confidence sets, where one tests each null $0 \notin C(X)$0 using a conservative Monte Carlo $0 \notin C(X)$1-value $0 \notin C(X)$2 and sets

$0 \notin C(X)$3

In that setting, coverage depends only on the level of the test of the true null, so the same Monte Carlo sample can be reused for all $0 \notin C(X)$4 (Glazer et al., 2024, Harrison, 2010).

In selective settings, marginal coverage of each base interval is insufficient. If $0 \notin C(X)$5 candidate parameters are screened and a data-dependent subset $0 \notin C(X)$6 is reported, then the relevant criterion is the false coverage proportion

$0 \notin C(X)$7

and its expectation, the false coverage rate,

$0 \notin C(X)$8

FCR controls the expected proportion of selected intervals that miss the truth. This differs from family-wise error rate, which controls the probability of any error, and from false discovery rate, which controls the expected proportion of false rejections among all rejections (Xu et al., 2022).

A persistent misconception is that using confidence intervals instead of $0 \notin C(X)$9-values automatically neutralizes selection effects. That is false. If intervals are reported only after seeing the data, their nominal post-selection guarantees generally fail. In particular, unadjusted CI-based rejection is invalid post-selection and can have FCR near $C(X)\cap \Theta_0=\emptyset$0, and the common practice of reporting intervals only when they exclude $C(X)\cap \Theta_0=\emptyset$1 yields filtered intervals that generally do not have their nominal confidence level (Xu et al., 2022, Owen, 2016).

2. E-value inversion and post-selection rejection

The most explicit modern formulation of confidence interval-based rejection under arbitrary selection is the e-value framework. A nonnegative random variable $C(X)\cap \Theta_0=\emptyset$2 is an e-value with respect to a null class $C(X)\cap \Theta_0=\emptyset$3 if

$C(X)\cap \Theta_0=\emptyset$4

Given a parameter $C(X)\cap \Theta_0=\emptyset$5 and the null class $C(X)\cap \Theta_0=\emptyset$6, a family $C(X)\cap \Theta_0=\emptyset$7 yields an $C(X)\cap \Theta_0=\emptyset$8-e-confidence interval by exact inversion,

$C(X)\cap \Theta_0=\emptyset$9

Validity follows from Markov’s inequality, since for the true $A(\theta_0)$0,

$A(\theta_0)$1

This class includes intervals constructed via supermartingale methods, universal inference, and Chernoff-style bounds (Xu et al., 2022).

For post-selection inference, the e-BY procedure adjusts the reported intervals only after the selected set $A(\theta_0)$2 is known. If $A(\theta_0)$3 is arbitrary and the base intervals are e-CIs, then for each selected $A(\theta_0)$4 one reports the adjusted interval at level $A(\theta_0)$5, that is,

$A(\theta_0)$6

The theorem is selection- and dependence-agnostic: if each $A(\theta_0)$7 is an $A(\theta_0)$8-e-CI, then the e-BY rule ensures $A(\theta_0)$9 for any selection rule $\alpha$0, including unknown and ad hoc selection, and for any dependence structure among the data or the intervals. The proof uses the e-value inequality $\alpha$1 together with $\alpha$2. The same framework yields a strengthened post-hoc guarantee,

$\alpha$3

and admits a weighted variant: if $\alpha$4 and $\alpha$5, then setting $\alpha$6 also guarantees $\alpha$7 (Xu et al., 2022).

Confidence interval-based rejection is immediate once the adjusted e-CIs are available. For each parameter $\alpha$8 with null set $\alpha$9, one rejects $H_0:\theta=\theta_0$0 if the adjusted interval excludes the null set,

$H_0:\theta=\theta_0$1

Under e-values this rejection rule is equivalent to a self-consistent e-BH rule, so the FCR guarantee for interval reporting yields false discovery rate control for the corresponding rejection set. This is the sense in which interval exclusion and multiple testing become two views of the same self-consistent thresholding mechanism (Xu et al., 2022).

A further structural result is that e-BY recovers the classical BY procedure through calibration. A calibrator is a nonincreasing, upper semicontinuous $H_0:\theta=\theta_0$2 with $H_0:\theta=\theta_0$3, and it transforms a CI family $H_0:\theta=\theta_0$4 into an e-CI through

$H_0:\theta=\theta_0$5

Using the BY calibrator $H_0:\theta=\theta_0$6, applying e-BY to the calibrated intervals yields exactly the BY output under arbitrary dependence. The distinction is that e-BY attains the tighter $H_0:\theta=\theta_0$7 without dependence assumptions when the intervals are e-CIs, whereas classical BY under arbitrary dependence pays the $H_0:\theta=\theta_0$8 penalty (Xu et al., 2022).

3. Sequential and anytime-valid confidence-sequence rejection

Sequential confidence interval-based rejection replaces a fixed-sample interval by a confidence sequence. A confidence sequence $H_0:\theta=\theta_0$9 satisfies time-uniform coverage,

$C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$0

so at any stopping time $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$1, $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$2 is a valid $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$3-CI. The rejection rule is the sequential analogue of fixed-sample inversion: test $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$4 by rejecting at time $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$5 when $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$6. Equivalently, one rejects when the associated test statistic crosses a calibrated boundary (Bibaut et al., 2022).

In the e-process formulation, if $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$7 is an e-process with respect to a null class $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$8, then for any stopping time $C_\alpha(x)=\{\theta_0:x\in A(\theta_0)\},$9,

$\theta_0\notin C_\alpha(x)$0

Nonnegative supermartingales therefore yield time-uniform e-CIs by the same inversion rule $\theta_0\notin C_\alpha(x)$1, and the e-BY post-selection guarantee continues to hold at stopping times, under continuous monitoring, and under bandit sampling (Xu et al., 2022).

A concrete construction is the delayed-start normal-mixture SPRT and the running-MLE SPRT. With partial sums $\theta_0\notin C_\alpha(x)$2, the nmSPRT statistic is

$\theta_0\notin C_\alpha(x)$3

and the rmlSPRT statistic is

$\theta_0\notin C_\alpha(x)$4

After a burn-in $\theta_0\notin C_\alpha(x)$5, one stops when the statistic crosses a boundary $\theta_0\notin C_\alpha(x)$6, and the implied confidence sequences are

$\theta_0\notin C_\alpha(x)$7

and

$\theta_0\notin C_\alpha(x)$8

Under Brownian data the coverage is exact, and under nonparametric, possibly dependent data with a strong invariance principle, type-I error is asymptotically equivalent to the target $\theta_0\notin C_\alpha(x)$9 as $A(\theta_0)$0 (Bibaut et al., 2022).

These sequential CIs support CI-based rejection with near-optimal expected stopping times. In the small-signal regime, the expected rejection time satisfies

$A(\theta_0)$1

and in the $A(\theta_0)$2 regime,

$A(\theta_0)$3

The paper’s simulations show that empirical-Bernstein concentration confidence sequences can reject nearly an order of magnitude later than mixture-SPRT confidence sequences at the same nominal level. In Netflix A/B testing, delayed-start nmSPRT and rmlSPRT rejected around just past burn-in, $A(\theta_0)$4 observations, while empirical-Bernstein concentration confidence sequences rejected at $A(\theta_0)$5 (Bibaut et al., 2022).

4. Directional, filtered, and post-selection interval rejection

Selective interval reporting has a second pathology beyond coverage loss: wrong-direction conclusions. If intervals are reported only when they exclude $A(\theta_0)$6, then in low-power settings the reported center points can be much farther from zero than the true parameter is and can frequently lie on the wrong side of zero. Under the asymptotically Gaussian model

$A(\theta_0)$7

the nominal two-sided interval excludes $A(\theta_0)$8 when $A(\theta_0)$9, but this filtered procedure does not preserve nominal coverage and can have a substantial conditional sign-error probability (Owen, 2016).

A simple refiltering correction uses a second threshold chosen to control sign error. Let $$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$0 denote initial selection and let $$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$1 be the stricter event. Then

$$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$2

where $$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$3 is the wrong-sign event. To ensure this conditional probability is at most $$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$4, it suffices to set

$$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$5

Equivalently, one can view the method as retesting at the stricter level $$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$6. The associated “sign $$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$7-value” is

$$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$8

This controls directional reliability rather than post-selection coverage (Owen, 2016).

A distinct development is direction-preferring confidence intervals, which deliberately use non-equivariant acceptance regions to improve sign exclusion in a preferred direction. In the normal model with known $$\inf_{\theta\in\Theta}\Pr_\theta(\theta\in C(X))\ge 1-\alpha,$$9, one may write

$0 \notin C(X)$00

with $0 \notin C(X)$01 and $0 \notin C(X)$02 when $0 \notin C(X)$03. Then the lower endpoint is pushed upward relative to the equal-tailed interval, which improves the probability that $0 \notin C(X)$04. Modified Pratt and Quasi-Conventional intervals use this asymmetry together with bounded-length design to obtain longer intervals in exchange for improved sign exclusions (Frostig et al., 2024).

After parameter selection, conditional direction-preferring intervals are constructed by inverting tests under the conditional distribution given the selection event. For example, under two-sided selection $0 \notin C(X)$05 in the normal model, the conditional law is truncated normal, and the endpoints are obtained by inverting the conditional CDF with an asymmetric tail split. The paper contrasts these conditional intervals with FCR-controlling intervals for selected parameters and reports that, for $0 \notin C(X)$06, CIs aimed at controlling the false coverage rate have higher power to determine sign than conditional CIs, while conditional confidence intervals control the marginal false coverage rate under any dependency (Frostig et al., 2024).

5. Distribution-free and computational constructions

Confidence interval-based rejection is not confined to classical parametric intervals. In local quantile regression, the central inferential object is a local marginal quantile $0 \notin C(X)$07 defined under a covariate-shifted distribution $0 \notin C(X)$08 that reweights $0 \notin C(X)$09 near $0 \notin C(X)$10 while keeping $0 \notin C(X)$11 unchanged. Two interval constructions are given. The Weighted Quantile method is asymptotically valid and asymptotically uniformly most accurate unbiased, while the Quantile Rejection method obtains finite-sample validity by rejection sampling from $0 \notin C(X)$12 and applying exact order-statistic quantile intervals to the accepted responses. In either case, CI-based rejection is implemented by rejecting $0 \notin C(X)$13 whenever

$0 \notin C(X)$14

For QR this controls type-I error at level $0 \notin C(X)$15 for all $0 \notin C(X)$16 and all $0 \notin C(X)$17; for WQ it is asymptotically correct. The paper emphasizes that WQ achieves nominal coverage once the effective sample size is roughly $0 \notin C(X)$18–$0 \notin C(X)$19 for moderate quantiles (Jang et al., 2023).

Monte Carlo inversion offers another general route. Conservative importance-sampling $0 \notin C(X)$20-values are obtained by incorporating the importance weight of the observed data,

$0 \notin C(X)$21

Inverting these corrected $0 \notin C(X)$22-values yields finite-sample valid Monte Carlo confidence intervals using a single importance sample (Harrison, 2010). More generally, conservative Monte Carlo confidence sets can be built from simulation, permutation, or randomization tests with any test statistic. If the map $0 \notin C(X)$23 is monotone or weakly unimodal for fixed data and fixed Monte Carlo draws, the endpoints can be computed by modified bisection while retaining conservativeness (Glazer et al., 2024).

A more classical optimality result is obtained by constructing acceptance regions directly from the ratio

$0 \notin C(X)$24

For each $0 \notin C(X)$25, the confidence set induced by

$0 \notin C(X)$26

maximizes prior-weighted average power subject to exact frequentist coverage. In the binomial example, the acceptance regions are calibrated by a Beta-binomial mixture and the resulting inverted intervals are exact in the discrete sense, conservative only insofar as equality is unattainable without randomization (Bartels et al., 2019).

6. Applications, scope, and limitations

The main empirical demonstration of robust post-selection CI-based rejection is the e-BY study of bounded means, stopped Brownian motions, and Twitter A/B testing. In the Twitter application there were $0 \notin C(X)$27 experiments over $0 \notin C(X)$28 years, with sequential user data and $0 \notin C(X)$29 metrics tracked daily, or $0 \notin C(X)$30 when treatment and control were counted separately. At $0 \notin C(X)$31, e-BY justified shipping in $0 \notin C(X)$32 experiments versus BY’s $0 \notin C(X)$33, and did so faster, with average $0 \notin C(X)$34 days versus $0 \notin C(X)$35 days (Xu et al., 2022). The sequential methods of delayed-start nmSPRT and rmlSPRT were also illustrated on Netflix client quality control A/B testing, where empirical type-I error approached the nominal level with burn-in and the rejection-time behavior matched the near-optimal theory (Bibaut et al., 2022).

A replication-oriented variant replaces direct rejection of $0 \notin C(X)$36 by inference on the exceedance probability

$0 \notin C(X)$37

for an exact replication study. If $0 \notin C(X)$38 is a confidence interval for $0 \notin C(X)$39, then under the normal model a confidence interval for the exceedance probability is

$0 \notin C(X)$40

The corresponding decision rule is to accept the claim that a replication will exceed $0 \notin C(X)$41 only if the lower endpoint $0 \notin C(X)$42 exceeds a prespecified threshold $0 \notin C(X)$43. This is not a substitute for ordinary hypothesis testing, but a different rejection criterion targeted at replicability (Segal, 2018).

The scope of the framework is therefore broad, but its guarantees are method-specific. The strongest post-selection guarantees currently require e-CIs: classical intervals that are not e-CIs do not automatically enjoy selection-agnostic and dependence-agnostic FCR control. They can, however, be calibrated into e-CIs, at the cost that calibration may widen intervals (Xu et al., 2022). Conditional confidence intervals after selection can be valid, but they require correct conditional modeling of the selection event and can be longer than FCR-controlling alternatives (Frostig et al., 2024). Filtered intervals that are reported only when they exclude $0 \notin C(X)$44 do not regain validity merely because they are intervals rather than tests (Owen, 2016).

The unifying lesson is that confidence interval-based rejection is not a single procedure but a design principle: construct or adjust a confidence set with the error criterion appropriate to the inferential regime, and reject only by exclusion from that set. In fixed-sample settings this recovers classical test inversion; in modern selective and sequential settings it becomes a framework for FCR control, FDR reduction, sign determination, and anytime-valid decision making under arbitrary dependence and optional stopping (Xu et al., 2022, Bibaut et al., 2022).