Empirically Calibrated Conditional Independence Tests

Abstract: Conditional independence tests (CIT) are widely used for causal discovery and feature selection. Even with false discovery rate (FDR) control procedures, they often fail to provide frequentist guarantees in practice. We highlight two common failure modes: (i) in small samples, asymptotic guarantees for many CITs can be inaccurate and even correctly specified models fail to estimate the noise levels and control the error, and (ii) when sample sizes are large but models are misspecified, unaccounted dependencies skew the test's behavior and fail to return uniform p-values under the null. We propose Empirically Calibrated Conditional Independence Tests (ECCIT), a method that measures and corrects for miscalibration. For a chosen base CIT (e.g., GCM, HRT), ECCIT optimizes an adversary that selects features and response functions to maximize a miscalibration metric. ECCIT then fits a monotone calibration map that adjusts the base-test p-values in proportion to the observed miscalibration. Across empirical benchmarks on synthetic and real data, ECCIT achieves valid FDR with higher power than existing calibration strategies while remaining test agnostic.

• The paper introduces ECCIT, a test-agnostic wrapper that adversarially measures and corrects p-value miscalibration in conditional independence tests such as GCM and HRT.
• ECCIT’s Type-I and FDP calibrators restore error control under finite samples, heavy-tailed noise, and model misspecification, with FDP calibration generally preserving more statistical power.
• The paper proves conditional FDR validity when the true response mechanism belongs to the adversary class and reports stronger valid power than CONTRA on gene-expression and UCI benchmarks.

Conditional independence tests (CITs) are the workhorse of controlled variable selection and causal discovery, but their frequentist guarantees frequently break down in the regimes where they are most needed: small samples, high dimensionality, and model misspecification. This paper by Pan, de Mathelin, and Tansey (MSKCC) identifies two concrete failure modes—finite-sample miscalibration even under correctly specified models, and non-uniform null p-values under model misspecification—and proposes a wrapper procedure, Empirically Calibrated Conditional Independence Tests (ECCIT), that measures and corrects miscalibration adversarially. The method is test-agnostic, applies to both doubly robust asymptotic tests (GCM) and model-X tests (HRT), and is supported by a worst-case validity theorem and extensive synthetic and semi-synthetic benchmarks.

Motivation and failure modes

The target problem is controlled variable selection: given data on features $(X_1,\ldots,X_m)$ and a response $Y$, identify the Markov blanket of $Y$ by testing $X_j \perp Y \mid X_{-j}$ for each $j$. The paper situates CIT methodology within the three-category taxonomy of local permutation, model-X, and asymptotic methods, and observes that all three families can fail in practice. Local permutation requires subclassing that is infeasible for continuous covariates; kernel variants lose power in high dimensions. Asymptotic doubly robust methods such as the Generalized Covariance Measure (GCM) control type-I error only when the conditional-mean regressions converge sufficiently fast and at least one is correctly specified—conditions that are unverifiable when using black-box regressors. Model-X methods such as the Holdout Randomization Test (HRT) require accurate estimation of $P(X_j \mid X_{-j})$, which degrades in finite samples and high dimensions.

The paper's central diagnostic claim is that these failures manifest as non-uniform null p-values, and that existing robustification methods (CONTRA, Maxway CRT, covariate-shift-corrected CRT, generative doubly robust tests) reduce but do not eliminate type-I error inflation without providing a mechanism to calibrate to a target error rate. ECCIT fills precisely this gap: rather than making the base test more robust, it measures how badly the test miscalibrates and adjusts its output accordingly.

The ECCIT method

ECCIT takes as input a base test $T$ returning per-feature p-values, an adversary class $\mathcal{F}$ of response generators $Y = f(X,\epsilon)$, and a calibration metric $M(\cdot,\alpha)$. The procedure has three stages. First, an adversary is fit by solving

$Y$0

where the expectation over $Y$1 is approximated by bootstrap resampling of the observed covariates. The adversary parameterizes a binary feature-selection mask $Y$2 (learned via Gumbel-softmax reparameterization) and a smooth mean function $Y$3 fit jointly, generating adversarial responses $Y$4. Second, the base test is run on the adversarial responses, where the null set is known by construction ($Y$5). Third, a monotone calibrator is fit to the adversarial p-values, mapping nominal levels to the largest level $Y$6 whose worst-case realized metric does not exceed the target $Y$7; this map is then applied to the p-values from the real data.

The paper evaluates two calibration metrics. The Type-I metric measures the deviation of the empirical null p-value CDF from uniformity at level $Y$8, $Y$9, enforcing calibration per test and thereby tending toward conservatism. The FDP metric targets Benjamini–Hochberg (BH) selection directly, using the realized false discovery proportion at the BH threshold as the miscalibration score, which preserves more nominal power.

Validity guarantee

The paper's main theoretical result (Theorem 1) states that if the true conditional law $Y$0 lies in the adversary class $Y$1 used to compute the FDP metric function $Y$2, then running BH at the calibrated level $Y$3 satisfies $Y$4, and the adjustment is conservative whenever $Y$5. The supplementary proof proceeds by showing the expected FDP under any $Y$6 is bounded by $Y$7, where $Y$8 is the expected excess FDP loss of the fitted adversary and $Y$9 its optimization error. Two assumptions deserve emphasis: the guarantee requires the true response mechanism to lie in $X_j \perp Y \mid X_{-j}$0—a substantive modeling assumption, not a distribution-free result—and the proof assumes bounded optimization error of the adversary. The guarantee is therefore conditional on adversary class coverage, and the paper does not provide tools for verifying coverage in practice.

Experimental results

The benchmarks cover three sources of miscalibration—finite-sample noise under well-specified models, heavy-tailed exogenous noise, and model underspecification (linear test-side regressors fit to nonlinear ground-truth responses)—using GCM and HRT with ridge and single-hidden-layer MLP regressors, feature distributions including Gaussian, Laplace, Student-$X_j \perp Y \mid X_{-j}$1, and a one-factor correlated structure with pairwise correlation $X_j \perp Y \mid X_{-j}$2, and evaluation via valid power (power set to zero whenever the lower confidence bound on realized FDR exceeds the target) at $X_j \perp Y \mid X_{-j}$3 averaged over 100 runs.

Key findings:

• Well-specified models still miscalibrate in finite samples. Even when the test's regression class contains the true conditionals, both GCM and HRT show p-value non-uniformity at small $X_j \perp Y \mid X_{-j}$4, worsening as $X_j \perp Y \mid X_{-j}$5 grows; calibration corrects this, with miscalibration fading as $X_j \perp Y \mid X_{-j}$6 increases. This confirms that asymptotic guarantees offer little protection in the finite-sample regime.
• FDP calibration preserves more power than Type-I calibration. The Type-I metric enforces worst-case per-test calibration and is inherently more conservative; the FDP calibrator selects the smallest level whose realized FDP meets the target, recovering nominal discoveries. In the nonlinear ground-truth setting, the stronger Type-I correction did not substantially penalize power, but the authors concede that in other response regimes it could over-adjust small p-values and reduce discoveries.
• Calibration restores FDR control under underspecification and heavy tails. With linear test-side regressors against nonlinear responses and across all feature distributions and noise families, calibrated GCM and HRT recover valid FDR control and positive valid power where uncalibrated versions fail.
• Semi-synthetic GDSC gene expression benchmark. On $X_j \perp Y \mid X_{-j}$7 slices of the Genomics of Drug Sensitivity in Cancer data with synthesized nonlinear responses, ECCIT-calibrated GCM and HRT track nominal FDR across $X_j \perp Y \mid X_{-j}$8 and outperform CONTRA-calibrated variants in valid power. The paper reports that CONTRA-HRT's conditional sampler is misspecified and its predictor is not refit under null draws in this setting, and CONTRA-FASTCRT reuses a fixed statistic across null resamples; both achieve high nominal power but with false discovery proportions that yield near-zero valid power. On the supplementary UCI benchmarks the contrast is stark: on the breast cancer configuration, raw HRT realizes FDR of 0.44 at $X_j \perp Y \mid X_{-j}$9 (valid power 0.057), while calibrated HRT reduces FDR to 0.176 with valid power 0.150; CONTRA-HRT and CONTRA-FASTCRT realize FDR above 0.48–0.57 with valid power below 0.03. On the wine dataset, calibrated GCM achieves valid power 0.432 at FDR 0.043 versus 0.372/0.132 uncalibrated.

The single-hypothesis experiments extend the method beyond multiple testing, showing improved type-I error control for a single test statistic with a power tradeoff, in both independent and correlated designs.

Limitations and open questions

The paper is explicit that the central limitation is the robustness–power tradeoff governed by the adversary class. If $j$0 is too simple, the correction is insufficient; if too flexible, worst-case calibration becomes overly conservative and sacrifices power. The validity theorem's requirement that the true response mechanism lie in $j$1 means the method inherits a specification assumption comparable in spirit to those of the base tests it calibrates, albeit one that is now explicit and localized in the adversary class. The paper also notes the Type-I calibrator's global correction could over-adjust in untested response regimes, and leaves open the problem of improving power over a set of plausible response mechanisms rather than a single worst case. The bootstrap approximation to the population distribution of $j$2, and the normal approximation to the HRT null loss distribution, are additional approximation layers whose effect on the guarantee is not quantified.

Conclusion

ECCIT offers a test-agnostic, adversarially calibrated wrapper that converts any CIT returning p-values into one with empirically enforced error control, backed by a worst-case FDR validity theorem conditional on adversary-class coverage. Empirically, it restores FDR control under finite-sample noise, heavy tails, and model misspecification, and delivers higher valid power than CONTRA on gene expression and UCI benchmarks. Its practical applicability hinges on choosing an adversary class rich enough to cover the true response mechanism without becoming so flexible that worst-case calibration destroys power—a tradeoff the paper identifies but does not resolve.