Self-Calibrated Empirical Null Samples (SENS)

• SENS is a conformalized empirical Bayes method that builds self-calibrated null samples using data-splitting to overcome deviations in theoretical null distributions.
• It relies on a symmetry assumption to generate exchangeable test and null samples, thereby reducing reliance on rigid parametric assumptions and ensuring finite-sample FDR control.
• Simulation and real-data analyses demonstrate that SENS outperforms traditional methods in power and robustness, particularly under heavy-tailed or non-Gaussian null conditions.

Searching arXiv for the SENS paper and closely related empirical-null work to ground the article in current literature. Self-Calibrated Empirical Null Samples (SENS) is a conformalized empirical Bayes method for large-scale multiple testing under an unknown null distribution with symmetric errors. It was introduced to address the concern, first raised by Efron (2004), that in modern data-intensive applications involving thousands or even millions of hypotheses, the theoretical null distribution of the test statistics often deviates from the true underlying null distribution, severely compromising false discovery rate (FDR) analysis. The framework sidesteps the use of potentially erroneous theoretical null distributions, mitigates the impact of estimation errors in the unknown null distribution on the validity of FDR control, imposes only a weak condition on the symmetry of the error distribution, and leverages conformal tools to achieve FDR control in finite samples while incorporating structural insights from empirical Bayes into inference (Tian et al., 4 Sep 2025).

1. Motivation and inferential setting

Large-scale multiple testing procedures in genomics and other high-dimensional studies typically rely on the theoretical null distribution for test statistics, for example a standard normal law. The motivating observation behind SENS is that the theoretical null often deviates from the actual empirical null due to correlation, hidden confounders, and related complexities. The consequences listed for this mismatch are inflated or misestimated false discovery rates, reduced statistical power, and the need for parametric, often Gaussian, assumptions when estimating the empirical null, which may themselves be wrong or hard to justify (Tian et al., 4 Sep 2025).

The basic model is formulated unitwise. For each unit $i \in [m]$,

$X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$

with hypotheses

$$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$

The goal is to control FDR while maximizing power. Traditional empirical Bayes FDR methods associated with Efron, Jin and Cai, and Sun and Cai attempt to estimate the empirical null, but the summary emphasizes that they still hinge on parametric assumptions and may be invalid under misspecification or estimation error. SENS is motivated by the need for FDR procedures that do not require correct specification or explicit estimation of the empirical or theoretical null, are robust to heavy-tailed and heterogeneous nulls, provide finite-sample FDR control rather than merely asymptotic guarantees, and remain power-competitive with procedures based on local false discovery rates (Tian et al., 4 Sep 2025).

A central point is therefore not simply replacing one null model by another. The method is designed for the setting in which the true null is unknown and cannot be directly estimated, and it constructs reference information from the observed data itself.

2. Symmetry assumption and self-calibrated null construction

SENS replaces parametric null modeling by a symmetry condition. For nulls $i \in \mathcal{H}_0$, the errors $\epsilon_{ij}$ are assumed independent and identically distributed within unit and to have a zero-symmetric conditional density, possibly unit-specific and dependent across units: $$f^{\epsilon}_{i}(-x\,|\,\text{others}) = f^{\epsilon}_{i}(x\,|\,\text{others}).$$ Under $H_0$, this implies that the distribution of test statistics is symmetric about zero, but not necessarily Gaussian. The summary explicitly notes that this accommodates heavy tails and heterogeneous nulls (Tian et al., 4 Sep 2025).

The defining construction of SENS is the generation of both “test” and “null” samples directly from the observed data by means of a data-splitting, or “plus-minus,” trick. In the one-sample case, for each unit $i$, the $n_i$ measurements are randomly split into two groups $(\mathcal{N}_{i1}, \mathcal{N}_{i2})$ of nearly equal size, and

$X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$0

Standardized statistics are then defined as

$X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$1

where $X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$2 is the CDF of Student’s $X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$3-distribution with $X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$4 degrees of freedom. The set $X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$5 is the test sample and $X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$6 is the self-calibrated null sample. A similar construction is used for two-sample problems, using the symmetry of errors in each group (Tian et al., 4 Sep 2025).

The fundamental theorem is pairwise exchangeability. Under the model and the symmetry assumption, for null indices,

$X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$7

This means that under the null, the test and calibration samples for each unit are interchangeable. The summary characterizes this as mirroring the property one would obtain if nulls could be sampled directly. In SENS, that surrogate sampling mechanism is achieved without direct knowledge of the null distribution.

3. Conformalized empirical Bayes scoring

The scoring stage combines conformal validity with empirical Bayes structure. Under the empirical Bayes two-group model, the optimal test for maximizing power at fixed FDR is described as a threshold on the local false discovery rate statistic

$X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$8

where $X_{ij} = \mu_i + \epsilon_{ij}, \quad j \in [n_i],$9 is the null density and $$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$0 is the mixture density (Tian et al., 4 Sep 2025).

SENS uses both $$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$1 and $$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$2 symmetrically to estimate these objects. The overall density is estimated by kernel smoothing over test and null samples,

$$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$3

and the working null density is estimated by a symmetrized estimator,

$$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$4

where $$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$5 is a filtered value ensuring symmetry and reducing the impact of alternatives. The resulting score function is

$$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$6

The paper states that this score mimics the local false discovery rate (Tian et al., 4 Sep 2025).

The significance of this construction lies in the fact that both samples are treated in an exchangeable fashion when constructing scores. The accompanying proposition states that these data-adapted scores are pairwise exchangeable under the null, which is the key ingredient for conformal FDR control. This gives SENS a hybrid character: it uses local-fdr-type scoring, but its validity is tied to exchangeability rather than to correct parametric specification of the null.

A common misunderstanding is that an empirical Bayes score necessarily reintroduces fragile parametric assumptions. In SENS, the score is empirical-Bayes-inspired, but the validity argument is not based on assuming a Gaussian null. The abstract explicitly contrasts SENS with empirical Bayes approaches that rely on Gaussian assumptions for the null models and states that SENS imposes only the weak symmetry condition on the errors (Tian et al., 4 Sep 2025).

4. Algorithmic form and finite-sample FDR control

The algorithm takes as input the data matrix $$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$7, a significance level $$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$8, and an option for null density estimation, either “JC” for Jin-Cai or “KN” for a kernel method. It outputs the set of discoveries. Its steps are:

1. Construct $$H_{0,i}: \mu_i = 0 \quad \text{vs.} \quad H_{1,i}: \mu_i \neq 0.$$9 and $i \in \mathcal{H}_0$0 for each unit via random splits.
2. Estimate $i \in \mathcal{H}_0$1 and $i \in \mathcal{H}_0$2.
3. Compute scores $i \in \mathcal{H}_0$3 and $i \in \mathcal{H}_0$4.
4. Form BC test statistics using an anti-symmetric function $i \in \mathcal{H}_0$5, with the recommended choice

$i \in \mathcal{H}_0$6

1. Apply a mirror FDR procedure with threshold

$i \in \mathcal{H}_0$7

and reject $i \in \mathcal{H}_0$8 if $i \in \mathcal{H}_0$9 (Tian et al., 4 Sep 2025).

The summary describes this as a conformalized and empirical Bayes-inspired version of the BC/knockoff filter for FDR control in a multiple testing context with unknown null and only requiring symmetry. The finite-sample FDR guarantee is derived through an e-value framework. For each hypothesis,

$\epsilon_{ij}$0

The e-BH procedure applied to these $\epsilon_{ij}$1 is stated to be equivalent to the SENS/BC algorithm. If pairwise exchangeability holds, the expected sum of e-values under the null is at most $\epsilon_{ij}$2, guaranteeing FDR control. The corollary reported in the summary states that under the symmetry assumption and the SENS construction, the procedure controls the FDR at level $\epsilon_{ij}$3 in finite samples, with no asymptotics or consistent estimation required (Tian et al., 4 Sep 2025).

The paper also includes a robustness result, Theorem~$\epsilon_{ij}$4, which quantifies the impact of departures from exchangeability. This places SENS within a setting in which exact exchangeability yields exact finite-sample control, while deviations from the condition can be explicitly analyzed rather than ignored.

5. Optimality under Efron’s two-group model

To study power, the authors consider Efron’s two-group model,

$\epsilon_{ij}$5

Within this model, SENS uses data-adaptive scores that mimic the oracle local false discovery rate. The summary states that thresholding on the lfdr scores, or on the SENS approximation to them, is the optimal FDR rule in the two-group model (Tian et al., 4 Sep 2025).

The asymptotic optimality theorem is stated for the two-group Gaussian mixture model under certain regularity conditions. There, SENS attains the same asymptotically optimal discovery boundary and power as the oracle local FDR procedure, identified in the summary with Sun and Cai’s AZ procedure. The stated novelty is that this is the first optimality theory for BC/mirror-based FDR methods in the literature, connecting SENS to the established benchmarks of empirical Bayes ideas (Tian et al., 4 Sep 2025).

This optimality result is important because model-free FDR procedures are often evaluated primarily on validity. SENS is positioned differently: it seeks to retain the robust, finite-sample guarantees associated with conformal or mirror procedures while also recovering the efficiency logic of local-fdr thresholding. A plausible implication is that SENS is intended as a bridge between procedures whose strength is validity under weak assumptions and procedures whose strength is power under a structured mixture model.

6. Empirical behavior and relation to other empirical-null methods

The simulation studies reported in the summary consider Gaussian, non-Gaussian, and heavy-tailed null distributions, together with varied sample sizes, sparsity levels, and effect sizes. The comparator set includes SENS with both “JC” and “KN” estimators, BH using theoretical and empirical nulls, and the model-free methods sfBH, stBC, RESS, and CLIPPER. The listed findings are that SENS always controls FDR at the nominal level, even when the null is misspecified or heavy-tailed; that it is as powerful as, or often more powerful than, empirical Bayes and model-free competitors because it uses lfdr-type scores; that when the null is non-Gaussian or poorly estimated it outperforms both BH$\epsilon_{ij}$6 and BH$\epsilon_{ij}$7; and that all valid model-free methods control FDR, but SENS is uniformly more powerful (Tian et al., 4 Sep 2025).

Two real-data analyses are highlighted. In insulin gene-expression data with paired samples, the theoretical null and empirical null are substantially different visually. SENS, especially the “JC” option, reports the highest number of discoveries at all FDR levels compared to all competitors, is robust to model misspecification, and maximizes the use of available data because the plus-minus split avoids efficiency loss. In bladder cancer two-sample data, the same pattern is reported: SENS consistently finds more discoveries and controls FDR (Tian et al., 4 Sep 2025).

A related but distinct line of work is "Bagged Empirical Null p-values: A Method to Account for Model Uncertainty in Large Scale Inference" (Mercaldo et al., 2017). That method combines empirical null standardization with model averaging via Bootstrap Aggregation, and it describes self-calibration in the sense that the empirical null is estimated for each bootstrap scenario, reflecting the actual structure and quirks of each resampled dataset under fitted models. BEN p-values therefore address model uncertainty and empirical-null calibration by averaging empirical null p-values across bootstrap samples and candidate models. SENS differs in construction and guarantee: it proposes a conformalized empirical Bayes method using self-calibrated empirical null samples, builds test and null samples directly from the observed data using symmetry, and bases its finite-sample FDR control on pairwise exchangeability rather than on bootstrap aggregation (Tian et al., 4 Sep 2025).

The comparison clarifies a frequent source of confusion. SENS is not simply another procedure for estimating an empirical null distribution. Its defining feature is the construction of self-calibrated empirical null samples that can stand in for direct null draws when the null is unknown and cannot be directly estimated. In that sense, its contribution lies simultaneously in calibration, conformal validity, and empirical-Bayes-style scoring.