Spectral Generalized Covariance Measure (SGCM)

• SGCM is a statistical framework that unifies high-dimensional dependence estimation, spectral theory, and nonparametric conditional independence testing.
• The methodology extends to both finite and infinite-dimensional settings using generalized covariance matrices and kernel-based operators.
• Empirical applications demonstrate robust size control and power in detecting dependencies and latent structures in complex, non-Euclidean data.

The Spectral Generalized Covariance Measure (SGCM) is a comprehensive statistical framework that unifies high-dimensional dependence estimation, spectral theory, and nonparametric conditional independence testing through the spectral properties of generalized covariance matrices and operators. SGCM generalizes classical correlation and covariance measures, allows for flexible non-Euclidean data representations, and establishes rigorous spectral and inferential results. Two foundational lines of research are central: the theory of spectral limits for φ-generalized covariance matrices in the high-dimensional regime (Benaych-Georges et al., 29 Sep 2025), and the development of scalable, doubly robust conditional independence tests in general Polish spaces (Miyazaki et al., 19 Nov 2025).

1. Formal Definition and Mathematical Construction

SGCM encompasses both finite-dimensional and infinite-dimensional settings. In the finite-dimensional, multivariate case, for $d \geq 2$, let $p$ vectors $X(1),\dots,X(p) \in \mathbb{R}^n$ be observed. For a fixed antisymmetric function $\varphi:\mathbb{R}^d \to \mathbb{R}$, the $\varphi$-covariance between vectors $u,v \in \mathbb{R}^n$ is defined as

$$(u, v)_\varphi := \frac{1}{\binom{n}{d}} \sum_{1\leq i_1 < \cdots < i_d \leq n} \varphi(u_{i_1},\ldots,u_{i_d})\varphi(v_{i_1},\ldots,v_{i_d}).$$

The $\varphi$-correlation, provided the variances are nonzero, is

$$\mathrm{corr}_\varphi(u,v) := \frac{(u,v)_\varphi}{\sqrt{(u,u)_\varphi (v,v)_\varphi}}.$$

The $\varphi$-covariance and $\varphi$-correlation matrices aggregate these measures over the $p$ vectors (Benaych-Georges et al., 29 Sep 2025).

In the infinite-dimensional (kernel) setting, for $(X,Y,Z)$ random variables valued in Polish spaces, with bounded, positive-definite kernels $k_X,k_Y,k_Z$, and associated RKHSs $H_X,H_Y,H_Z$, let the conditional mean embeddings be $\mu_{X|Z}(z) = \mathbb{E}[\phi_X(X)|Z=z]\in H_X$, and similarly for $Y$. The (conditional) cross-covariance operator (CCCO) is

$$\Sigma_{XYZ|Z} := \mathbb{E}\Big[ (\phi_X(X)-\mu_{X|Z}(Z)) \otimes (\phi_Y(Y)-\mu_{Y|Z}(Z)) \otimes \phi_Z(Z) \Big] \in H_X\otimes H_Y\otimes H_Z.$$

The SGCM for a joint law $P_{XYZ}$ is the squared Hilbert–Schmidt norm: $$\mathrm{SGCM}(P_{XYZ}) = \|\Sigma_{XYZ|Z}\|^2_{HS}.$$ It vanishes if and only if $X\perp Y|Z$ under mild characteristic-kernel conditions (Miyazaki et al., 19 Nov 2025).

2. Spectral Theory and Limiting Distributions

In the high-dimensional asymptotics ($p,n\to\infty$, $p/n\to q>0$), the empirical spectral distribution (ESD) of the $\varphi$-covariance and $\varphi$-correlation matrices admits a deterministic limit.

For the $\varphi$-covariance matrix, under independence and regularity (moment) conditions for $\varphi$, the ESD converges to an affine transform of the Marčenko–Pastur law: $\mathrm{MP}_q^{\beta-\alpha, -\alpha},$ where $\alpha = \lim_{n\to\infty} d\,\mathbb{E}[X_1^2]$ and $$\beta = \lim_{n\to\infty} \mathbb{E}[\varphi(X_1,\ldots,X_d)^2]$$ (Benaych-Georges et al., 29 Sep 2025). For the correlation case, the limiting law is

$$\mathrm{MP}_q^{t,\,1-t} \quad\text{where } t = \alpha/\beta.$$

The Marčenko–Pastur (MP) density for aspect ratio $q$ and affine parameters $(a, b)$ has support $[a(1-\sqrt{q})^2 + b,\, a(1+\sqrt{q})^2 + b]$ and

$$f_{\mathrm{MP}_q}(x) = \frac{1}{2\pi q} \sqrt{(\lambda_+ - x)(x - \lambda_-)} \, /\, x,\quad x\in[\lambda_-,\lambda_+],$$

with $\lambda_\pm = (1\pm\sqrt{q})^2$.

A central step is the Hoeffding-type decomposition of the entries: each $(X(k),X(\ell))_\varphi$ is approximated at the spectral level by a rank-one average of functions $U_i(k)$, resulting in convergence to the affine MP law after accounting for diagonal shifts and normalization.

Fluctuations about the limit are expected to satisfy a central limit theorem for linear spectral statistics, conditional on analogous conditions as in Bai and Silverstein's theory (Benaych-Georges et al., 29 Sep 2025).

3. Computation of SGCM in Finite and Kernelized Settings

For $X\in\mathbb{R}^{n\times p}$ and a function $\varphi$:

1. For each $i,k$, draw auxiliary random variables or use the empirical marginal of $X_{\cdot,k}$ to estimate conditional expectations.
2. Compute $$U_i(k) = \mathbb{E}[\varphi(X_i(k), X_2(k), ..., X_d(k)) \mid X_i(k)]$$ using closed-form expressions or Monte Carlo.
3. Form the matrix $U$, and compute $S = (1/n) UU^T$.
4. Add the appropriate diagonal shift or scaling, depending on whether covariance or correlation is sought.
5. Diagonalize the resulting matrix and compare the eigenvalue distribution to the Marčenko–Pastur prediction.

For kernelized, conditional independence scenarios (Miyazaki et al., 19 Nov 2025):

1. Split data into subsamples for spectral (basis) estimation and regression.
2. Compute empirical covariance operators and their leading eigenfunctions for $X$ and $Y$ on the first subsample.
3. Perform nonparametric regression of leading coordinate scores on $Z$ in the second subsample to obtain fitted conditional means; calculate residuals.
4. Form the SGCM statistic as a V-statistic over these residuals, weighted by the kernel on $Z$.

This regression-based dimension reduction eliminates the need for full RKHS regression and is effective even for high-dimensional or non-Euclidean data, subject to spectral gap and regularity constraints (Miyazaki et al., 19 Nov 2025).

4. Inference, Asymptotic Properties, and Wild Bootstrap

The limiting distribution of the kernelized SGCM statistic under the null hypothesis is a non-pivotal, weighted chi-squared mixture: $$n_1 \hat T_{n_1} \xrightarrow{d} \sum_{\ell=1}^\infty \xi_\ell G_\ell^2,$$ where $G_\ell \sim N(0,1)$ and $\xi_\ell$ are eigenvalues of the covariance operator $\mathbb{E}[\Psi\otimes\Psi]$. Calibration is performed via a wild-multiplier bootstrap, drawing i.i.d. multipliers with mean $0$ and variance $1$, yielding asymptotic control of test size (Miyazaki et al., 19 Nov 2025). Sufficient regularity conditions include bounded kernels, growing spectral gaps, vanishing regression and truncation biases, and operator nondegeneracy. Uniform asymptotic size control is established under double robustness: the test attains level $\alpha$ uniformly over a class of null distributions with vanishing estimation error.

5. SGCM with Non-Euclidean Data: Characteristic Kernels beyond $\mathbb{R}^d$

SGCM extends seamlessly to non-Euclidean sample spaces by employing characteristic kernels arising from negative-type semimetrics on Polish spaces. If $\rho:M\times M\to[0,\infty)$ is of negative type, then Laplacian-type kernels $k_{\exp,\rho}(u,v)=\exp(-\gamma\, \rho(u,v))$ for $\gamma>0$ are characteristic. More general completely monotone transforms $k_f(u,v)=f(\rho(u,v))$ retain this property if $f$ is non-constant, completely monotone, and $f(0)$ exists (Miyazaki et al., 19 Nov 2025). For product spaces, tensor products of characteristic kernels remain characteristic, supporting SGCM for structured or distributional data (e.g., Hilbert spheres, Wasserstein spaces, $L^p$-valued functions). Valid extension hinges on the identification and usage of such kernels, guaranteeing that SGCM retains its equivalence to conditional independence.

6. Applications: Independence Testing and High-Dimensional Dependency Estimation

The SGCM framework enables rigorous, scalable inference for independence and conditional independence:

• Under the null of independence, the empirical spectrum adheres to the predicted MP-law support, yielding a robust basis for hypothesis testing, including in heavy-tailed or outlier-rich settings when rank-based $\varphi$ functions (e.g., Kendall's $\tau$) are employed.
• For dependency estimation, deviations from the null manifest as outliers ("spikes") in the eigenvalue spectrum, allowing for detection of latent structure via principal component or spike-detection paradigms (e.g., Baik–Ben Arous–Péché transition) (Benaych-Georges et al., 29 Sep 2025).
• For conditional independence, the kernelized SGCM test exhibits robust size control and competitive power across various alternatives, including challenging even-moment or signed-latent scenarios. In high dimensions, it outperforms or matches state-of-the-art methods such as GCM, WGCM, KCI, and CDCOV in size and/or power, and maintains validity for complex objects such as distributions or curves (Miyazaki et al., 19 Nov 2025).

7. Illustrative Example and Practical Guidelines

For $d=2$, $\varphi(x,y)=\mathrm{sgn}(y-x)$ (Kendall's $\tau$), and $p/n\to q=0.5$, with Gaussian data, the limiting law for the (uncentered) SGCM is $\mathrm{MP}_{0.5}^{2,-1}$. Empirical spectra from large simulated matrices closely overlay the theoretical density (Benaych-Georges et al., 29 Sep 2025). Parameter selection typically fixes $d=2$ for standard correlation types; for robustness, truncation of $\varphi$ can ensure uniform moment conditions required for theory. Monte Carlo or closed-form computation is used for conditional expectations in complex settings.

SGCM thus provides a unified, flexible, spectral approach to high-dimensional dependence measurement and testing, rigorously grounded in random matrix theory and nonparametric kernel methods (Benaych-Georges et al., 29 Sep 2025, Miyazaki et al., 19 Nov 2025).