The Generalised Kernel Covariance Measure

Abstract: We consider the problem of conditional independence (CI) testing and adopt a kernel-based approach. Kernel-based CI tests embed variables in reproducing kernel Hilbert spaces, regress their embeddings on the conditioning variables, and test the resulting residuals for marginal independence. This approach yields tests that are sensitive to a broad range of conditional dependencies. Existing methods, however, rely heavily on kernel ridge regression, which is computationally expensive when properly tuned and yields poorly calibrated tests when left untuned, which limits their practical usefulness. We propose the Generalised Kernel Covariance Measure (GKCM), a regression-model-agnostic kernel-based CI test that accommodates a broad class of regression estimators. Building on the Generalised Hilbertian Covariance Measure framework (Lundborg et al., 2022), we characterise conditions under which GKCM satisfies uniform asymptotic level guarantees. In simulations, GKCM paired with tree-based regression models frequently outperforms state-of-the-art CI tests across a diverse range of data-generating processes, achieving better type I error control and competitive or superior power.

• The paper presents the Generalised Kernel Covariance Measure (GKCM), a novel kernel-based test for conditional independence that generalizes beyond KRR.
• It decouples kernel embeddings from regression by allowing any RKHS-valued estimator, achieving robust error control and improved computational efficiency.
• Empirical results demonstrate that GKCM, particularly with random forests, achieves superior performance over standard methods in detecting nonlinear dependencies.

The Generalised Kernel Covariance Measure: Foundations and Implications

Introduction

The problem of conditional independence (CI) testing is central in statistics and causal discovery. Kernel-based CI tests, which embed variables into reproducing kernel Hilbert spaces (RKHSs), have become a de facto standard for assessing CI in high-dimensional and nonlinear settings. However, existing kernel-based tests overwhelmingly use kernel ridge regression (KRR) as their regression method, resulting in severe computational and calibration bottlenecks. "The Generalised Kernel Covariance Measure" (2604.03721) systematically addresses these issues by introducing the Generalized Kernel Covariance Measure (GKCM), a new kernel-based CI test that accommodates arbitrary Hilbert space-valued regression estimators, notably extending beyond KRR. This essay outlines the theoretical construction, compares GKCM to the state of the art, highlights empirical findings, and explores its broad theoretical and practical implications.

Background: Conditional Independence Testing and Kernel-Based Approaches

CI testing aims to evaluate whether $X \perp Y \mid Z$, meaning $X$ and $Y$ are independent conditional on $Z$. Residual-based tests—including partial covariances and the Generalised Covariance Measure (GCM)—typically estimate $E[X | Z]$ and $E[Y | Z]$, then test for vanishing covariance of residuals. While flexible in regression choices, these have fundamental limitations: they yield only mean-zero tests and can fail to detect dependence when only weak CI is violated. Further, many function classes considered by these tests do not have enough richness for universal detection.

Kernel-based CI tests overcome some of these issues by:

• Embedding $X$ and $Y$ (and sometimes $Z$) into RKHSs via characteristic or universal kernels, capturing a vast class of nonlinear relationships,
• Testing for conditional independence of embedded residuals, which is theoretically equivalent to testing for CI under broad kernel choices,
• Typically employing KRR to estimate conditional mean embeddings in the RKHS (see [zhang_kernel-based_2011], [fukumizu_kernel_2007]).

Despite their generality, kernel-based tests have practical drawbacks. KRR’s performance is highly sensitive to kernel hyperparameters and regularization, and exhaustive tuning is often infeasible in high dimensions. Additionally, theoretical guarantees for uniform type I error control are lacking—most earlier works offer only pointwise validity.

The GKCM Framework

GKCM generalizes kernel-based CI testing by decoupling the regression step from KRR, allowing for any regression approach that outputs RKHS-valued predictions. Conceptually, GKCM leverages the Generalised Hilbertian Covariance Measure (GHCM) framework [lundborg_conditional_2022], which characterizes broader classes of regression methods and establishes uniform asymptotic level guarantees if in-sample regression errors decrease sufficiently fast.

Formally, GKCM:

• Embeds $X \to F$ and $X$0 using continuous, bounded, injective kernels $X$1 and $X$2, mapping into separable RKHSs $X$3 and $X$4,
• Estimates conditional mean embeddings $X$5 and $X$6 via arbitrary regression models,
• Computes the mean conditional covariance operator

$X$7

• Constructs a test statistic $X$8, where the norm is taken over Hilbert–Schmidt operators,
• Approximates the null distribution via the nonzero eigenvalues of a matrix derived from inner products of residuals.

Uniform type I error control holds under achievable assumptions on in-sample regression accuracy, moment conditions, and distributional regularity, as established via the GHCM.

Comparison with Prior Methods

• KCIT/RCIT/RCoT: Always use KRR for conditional mean embedding; offer only pointwise error control and exhibit strong sensitivity to kernel hyperparameters.
• GCM/wGCM/PCM: Offer regression-model-agnostic alternatives but either remain mean-zero or are limited by the function class considered.
• GKCM: Allows for any regression method, offering both flexibility and uniform error control in mean-zero settings, while leveraging RKHS embeddings for nonlinearity.

Regression for RKHS-Valued Targets: KRR vs. Random Forests

A central practical challenge is RKHS-valued regression. KRR offers a closed-form solution but suffers in high-dimensional settings due to hyperparameter tuning cost and potential model misspecification. Random forests, by contrast, require little tuning and offer strong empirical performance, particularly when the conditional mean is not well captured by KRR or kernel parameters are hard to specify.

The core insight is that the regression step can benefit from leveraging methods not based on kernels over the conditioning set, such as random forests or even neural architectures, which can still output RKHS-valued predictions compatible with the kernel trick.

Empirical Evaluation

Simulation studies in the paper compare GKCM using random forests (GKCM RF) and KRR (GKCM KRR) against prominent CI tests (KCIT, RCIT, RCoT), as well as residual-based methods (GCM, wGCM, PCM), across various synthetic null and alternative scenarios.

Type I Error:

GKCM RF achieves approximately nominal type I error across most null settings and all sample sizes, outperforming other tests that exhibit inflated error under several scenarios.

Figure 1: Rejection rates in the null settings with rejection threshold $X$9 (100 iterations). Error bars indicate 95% Wilson confidence intervals and dashed lines the nominal level.

Power:

GKCM RF sustains power at or above the other kernel-based tests in complex nonlinear alternatives, while maintaining better error rates.

Figure 2: Rejection rates in the alternative settings with rejection threshold $Y$0 (100 iterations). Error bars indicate 95% Wilson confidence intervals.

Additional Comparisons

Figures adapted from prior works show that even in classical nonparametric regimes ([zhang_kernel-based_2011] scenarios), GKCM with random forests attains superior finite-sample rejection rates compared to KCIT and related kernel methods.

Figure 3: Rejection rates in the scenarios of [zhang_kernel-based_2011] highlighting the advantages of GKCM.

Theoretical Guarantees

The GHCM framework endows GKCM with uniform asymptotic level guarantees across broad model classes, contingent on regression consistency (not requiring out-of-sample generalization): for level-$Y$1 tests,

$Y$2

where $Y$3 indexes an appropriately regular subset of the null. This is a strong guarantee, not offered by earlier kernel-based CI tests, and is made possible by the modular design of GKCM.

Implications and Future Directions

Practical Implications

GKCM’s decoupling from KRR unlocks robust, computationally efficient CI testing in settings where existing kernel-based CI tests are ineffective—particularly, large-scale, high-dimensional, and structured data scenarios, and settings with complex or mixed-type covariates. The use of flexible regression methods (e.g., random forests, neural architectures) with kernel embeddings is expected to generalize well to practical causal discovery workflows and high-throughput multimodal data.

Theoretical Implications

GKCM clarifies that the difficulty of kernel-based CI testing is bottlenecked by conditional mean embedding estimation. Its regression-agnostic approach paves the way for further advances in RKHS-valued regression theory, including misspecification-robust error guarantees, minimax rates under nonparametric regression, and joint optimization of kernel and regression hyperparameters. The modularity of the framework can facilitate hybrid tests emulating doubly robust procedures (cf. recent work on double robustness [niu_reconciling_2023]).

GKCM’s formalization also motivates extensions to arbitrary function spaces, operator-valued kernels, and structured-output prediction [alvarez_kernels_nodate]. Links to universal kernel mean embeddings provide a pathway to fully nonparametric CI testing in general measurable spaces, yielding fertile ground for theory and algorithms that go beyond existing kernel or regression boundaries.

Open Problems and Extensions

• Robustness to Misspecified Regressors: Systematic evaluation of performance under misspecified regressors or function classes—especially for modern ML methods and in mixed-type data—is an ongoing challenge [polo_conditional_2023].
• Large-Scale Tuning: Scalable, non-exhaustive hyperparameter selection (e.g., using out-of-sample protocols [biza_out--sample_2024]) for kernel or forest-based regressors can further enhance power and error control.
• Heterogeneous and Mixed-Type Data: Extending empirical studies to settings with categorical, ordinal, or compositional variables, and to multimodal and multi-output contexts [xu_survey_2019].
• Theoretical Sharpness: Tighter minimax lower bounds for CI testing and optimality in specific regimes [neykov_minimax_2021].

Conclusion

The Generalised Kernel Covariance Measure advances the state of kernel-based conditional independence testing by introducing regression-model agnosticism, providing strong uniform error control, and enabling flexible, computationally efficient integration with modern ML regression techniques. It marks an important conceptual and practical development for both statistical CI testing and its numerous causal inference applications. Future work will further integrate these advances with automation, mixed-data support, and theoretically principled model selection.