Double-Centered Kernel Matrices

• Double-centered kernel matrices are constructed by applying a centering operator to raw kernel data, ensuring that both row and column means are zero in the implicit feature space.
• Their spectral properties, including eigenvalue interlacing and introduction of a zero eigenvalue, facilitate rigorous theoretical analysis and improve the reliability of kernel-based algorithms.
• They are crucial in applications such as kernel alignment, manifold learning, and self-supervised methods, providing stable, unbiased representations for robust statistical and algorithmic performance.

Double-centered kernel matrices are foundational constructs in kernel methods, providing a mechanism for removing the influence of sample means from kernel-based representations. In both statistical learning and modern machine learning, double-centering is essential for centering the inner-product structure, enabling rigorous spectral analysis, enhanced interpretability, and improved algorithmic stability. The centering operator acts as an orthogonal projection on both rows and columns, ensuring that kernel evaluations directly reflect the dispersion or covariance about the mean in the implicit feature space. As such, double-centered kernel matrices are integral to the formulation, analysis, and performance guarantees of a wide range of kernel-based algorithms in classification, regression, clustering, completion, manifold learning, self-supervised learning, and random matrix theory.

1. Definition and Construction

Given a kernel function $k(x, x') = \langle \phi(x), \phi(x') \rangle$ defined on a dataset $\{x_1, ..., x_n\}$, the empirical kernel matrix $K$ is constructed as $K_{ij} = k(x_i, x_j)$. Double-centering is typically applied via the centering matrix $H = I_n - (1/n) \mathbf{1}_n \mathbf{1}_n^\top$, yielding the double-centered kernel matrix

$\widehat{K} = H K H.$

Alternatively, for Gram matrices derived from data representations $X \in \mathbb{R}^{d \times n}$, centering in feature space leads to the matrix $K = (I - P) X^\top X (I - P)$ with $P = (1/n)\mathbf{1}_n \mathbf{1}_n^\top$.

The explicit elementwise formula for double-centering a kernel matrix $K_{nc}$ (the "non-centered" kernel) is

$$K = K_{nc} - \frac{1}{n} \mathbf{1} \mathbf{1}^\top K_{nc} - \frac{1}{n} K_{nc} \mathbf{1} \mathbf{1}^\top + \frac{1}{n^2} \mathbf{1} \mathbf{1}^\top K_{nc} \mathbf{1} \mathbf{1}^\top.$$

This transformation ensures all row and column means are zero in the centered feature space representations (Honeine, 2014).

2. Spectral Properties and Theoretical Implications

Double-centering radically alters the spectral decomposition of kernel matrices. For centered matrices, all nonzero eigenvectors are orthogonal to the ones vector and correspond to directions of variance about the mean:

• All eigenvectors $\alpha_i$ associated with nonzero eigenvalues satisfy $\alpha_i^\top \mathbf{1}=0$.
• The spectrum $\{\mu_j\}$ of the centered kernel interlaces the original (non-centered) spectrum $\{\lambda_j\}$ as $\lambda_{j+1} \leq \mu_j \leq \lambda_j$ for $j=1,\ldots, n-1$, with $\mu_n=0$ (Honeine, 2014).
• Centering introduces a zero eigenvalue (with eigenvector $\mathbf{1}$), reflecting invariance to global shifts.

In random matrix theory and high-dimensional statistics, double-centering eliminates the dominant rank-one (mean) component in kernel matrices, exposing the informative higher-order structure. In regimes where $n,p\rightarrow\infty$ in fixed ratio, as in kernel ridge regression (KRR), centering ensures that convergence analysis and risk predictions (training and testing) capture only the informative component, leading to deterministic limiting distributions for both empirical and prediction risks (Elkhalil et al., 2019).

3. Double-Centered Kernels in Learning Algorithms

Double-centering is pivotal in the design and analysis of several kernel learning frameworks:

• Kernel Alignment: Alignment-based kernel learning measures the similarity between a data kernel and a target kernel (e.g., $yy^\top$ in classification) using the normalized centered inner product

$$\rho(K, K_Y) = \frac{\langle K_c, K_{Y,c}\rangle_F}{\|K_c\|_F \cdot \|K_{Y,c}\|_F},$$

where $K_c$ denotes double-centering. Maximizing this alignment is shown to correlate strongly with predictive performance, leading to theoretically justified kernel combination strategies and generalization bounds via stability analysis (Cortes et al., 2012).

• Multiple Kernel Learning and Approximation: Methods such as Mklaren apply double-centering to unnormalized Cholesky pivot columns before feature aggregation, enforcing that low-rank feature construction reflects the centered geometry of the data (Stražar et al., 2016).
• Self-Supervised Learning: In RKHS-based self-supervised objectives (e.g., Kernel VICReg), double-centered kernel matrices are used to compute RKHS variance, covariance, and invariance terms. The variance in feature space along principal directions is tied to the eigenvalues of the double-centered kernel, ensuring robust regularization and avoiding representation collapse (Sepanj et al., 8 Sep 2025).
• Manifold and Algebraic Structure Extraction: Algorithms rooted in duality theory, such as IPCA and AVICA, utilize SVD on externally constructed kernel matrices to extract both discriminative and generative structure, analogous to centering but in an algebraic context (Király et al., 2014).
• Sensor Fusion and High-Dimensional Inference: In sensor fusion via kernel methods, high-dimensional kernel affinity matrices are “double-centered” by removing not just mean effects but higher-order finite-rank corrections, aligning their spectrum with theoretical distributions (e.g., Marčenko–Pastur law, free multiplicative convolution) (Ding et al., 2019).

4. Extensions Beyond Conventional Centering

Double-centering has been generalized in several ways:

• Weighted Centering: Instead of uniform mean subtraction, double-centering may use a projection matrix $P_\omega = \mathbf{1} \omega^\top$ for a weight vector $\omega$ with $\mathbf{1}^\top \omega = 1$, supporting robustness or reflecting sample importance (Honeine, 2014).
• Adaptation to Incomplete Data: For incomplete or partially observed kernel matrices, as in multi-view completion, the reconstruction of double-centered kernels must maintain centering in the imputation, possibly via additional regularization terms (Bhadra et al., 2016, Rivero et al., 2018).
• Manifold Learning and Non-PSD Kernels: In classical multidimensional scaling and for kernels that are only conditionally positive definite, double-centering is used to recover meaningful inner-products from distance or dissimilarity data (Honeine, 2014).

5. Stability, Conditioning, and Spectral Alignment

The stability of double-centered kernel matrices is underpinned by their eigenvalue structure, which is influenced by kernel smoothness and data geometry:

• Stability estimates relate the minimum eigenvalue to the data separation and kernel smoothness (e.g., $\lambda_{\min}(A_X) \geq c_{\min} q_X^{2\tau-d}$), with refined multivariate Ingham-type theorems providing explicit constants (Wenzel et al., 6 Sep 2024).
• For kernels of different smoothness, Rayleigh quotients for the respective double-centered matrices satisfy squeezing inequalities, ensuring spectral alignment under controlled changes in smoothness and separation. This has direct implications for regularization and the selection of kernel hyperparameters in large-scale learning scenarios (Wenzel et al., 6 Sep 2024).

6. Applications, Empirical Performance, and Practical Considerations

Double-centered kernel matrices are utilized in diverse domains:

• Clustering: Multiple Kernel $k$-Means variants employ double-centered matrices to fuse similarity information, penalize redundancy, and enhance cluster extraction by integrating correlation and dissimilarity metrics—providing more robust and objective kernel fusion (Su et al., 6 Mar 2024).
• Kernel Completion: Approaches that impose parametric (e.g., low-rank plus noise) structure on double-centered kernels with LogDet divergence retain positive definiteness and filter informative directions post-centering, yielding improved performance in multi-source completion settings (Rivero et al., 2018).
• Spectral Clustering and Dimensionality Reduction: Double-centering ensures that extracted eigenvectors correspond to variance about the mean, a prerequisite for interpretable and mathematically consistent embeddings in kernel PCA and classical MDS (Honeine, 2014, Barthelmé et al., 2019).
• Random Matrix Theory and High-Dimensional Limit Theory: As dimension and sample size scale, double-centering removes mean effects, aligning the spectrum of the kernel matrix with universal laws (e.g., Marchenko–Pastur), exposing spiked and bulk components directly linked to function class complexity and interpretable "double descent" phenomena in regression risk (Misiakiewicz, 2022).

The ubiquitous requirement for double-centering in modern kernel methods validates its centrality for stability, interpretability, and transferability across regression, classification, clustering, and unsupervised tasks. Future research directions involve dynamic centering in adaptive or sequence-based learning, incorporation with multitask and multi-view frameworks, and further refinement of spectral alignment strategies vis-à-vis kernel selection and data geometry.