RCKA: Relation-based Centered Kernel Alignment

Updated 10 December 2025

RCKA is a metric that generalizes CKA by incorporating arbitrary, positive-definite relation kernels to capture pairwise relationships in data representations.
The framework provides theoretical guarantees such as invariance, normalization, and concentration bounds, facilitating convex optimization in kernel learning and model distillation.
Empirical studies demonstrate RCKA’s effectiveness in improving knowledge distillation across image classification, object detection, and NLP tasks.

Relation-based Centered Kernel Alignment (RCKA) generalizes Centered Kernel Alignment (CKA), a prominent metric for quantifying similarity between data representations in machine learning. RCKA extends the alignment concept to arbitrary relation kernels, supporting architectures, tasks, and learning objectives where capturing and transferring inter-instance or structural relations is essential, such as knowledge distillation, feature structure distillation, and kernel learning. This framework formalizes the statistical alignment of neural, relational, or structural representations, imbuing downstream optimization or learning algorithms with guarantees and interpretability previously rooted in theory for CKA. RCKA is empirically established in domains ranging from deep image models to LLMs and kernel-based systems (Zhou et al., 22 Jan 2024, Jung et al., 2022, Kornblith et al., 2019, Cortes et al., 2012).

1. Mathematical Foundations

Relation-based Centered Kernel Alignment builds on the classical CKA statistic, which measures the similarity between two Gram (kernel) matrices of example embeddings. For datasets with $n$ examples and two representations $X \in \mathbb{R}^{n \times p}$ , $Y \in \mathbb{R}^{n \times q}$ (often neural activations or feature matrices), CKA is defined as:

$\mathrm{CKA}(X, Y) = \frac{\|Y^\top X\|_F^2}{\|X^\top X\|_F \; \|Y^\top Y\|_F}$

where $\|\cdot\|_F$ denotes the Frobenius norm.

The "relation-based" extension explicitly considers an arbitrary positive-definite, symmetric relation kernel $R \in \mathbb{R}^{n \times n}$ , which can encode semantic, structural, or auxiliary information across example pairs (Cortes et al., 2012). The relation-based CKA between kernel $K$ and relation $R$ is:

$\hat\rho_S(K, R) = \frac{\langle K_c, R_c \rangle_F}{\|K_c\|_F \|R_c\|_F}$

where $K_c$ and $R_c$ are the centered versions (double-centered in sample space), and $\langle \cdot, \cdot \rangle_F$ denotes the Frobenius inner product.

CKA can also be interpreted as the normalized Hilbert–Schmidt Independence Criterion (HSIC) between two kernels (Kornblith et al., 2019, Jung et al., 2022). When $K=K_X=X X^\top$ and $R$ is a general relation-kernel, RCKA quantifies how well the representation $X$ captures the pairwise structure in $R$ .

2. Theoretical Properties and Guarantees

The theoretical analysis of RCKA extends classical CKA properties:

Invariance: RCKA is invariant to isotropic scaling and orthonormal transformations of features but sensitive to non-isotropic transformations, preserving the structure of principal axes (Kornblith et al., 2019).
Range and Normalization: RCKA is bounded within $[0, 1]$ when kernels are positive semidefinite and properly centered, facilitating stable optimization.
Alignment Maximization: Kernel learning objectives can maximize alignment with a relation kernel $R$ , yielding convex optimization problems (QPs) both in two-stage kernel selection and joint kernel-predictor learning (Cortes et al., 2012).
Generalization and Concentration: RCKA admits concentration bounds showing that empirical alignment converges to population alignment as sample size increases, and stability-based bounds tie high alignment to the existence of accurate predictors (Cortes et al., 2012).

RCKA also admits a decomposition relating it to Maximum Mean Discrepancy (MMD): the CKA loss between features can be rewritten as an upper bound on the squared MMD plus a constant shift, explaining its effectiveness as a distillation loss and introducing a natural regularization effect (Zhou et al., 22 Jan 2024).

3. Practical Algorithms and Implementation Details

RCKA admits multiple operationalizations, with the base algorithm applicable as follows (Zhou et al., 22 Jan 2024, Jung et al., 2022):

Feature and Logit Alignment (Knowledge Distillation): Simultaneously align instance-level feature maps, intra-class, and inter-class logit relations using CKA between teacher and student model representations. The general loss form:

$L_{RCKA} = L_{CE} + \alpha \, L_{FCKA} + \beta \left(L_{Intra-LCKA} + L_{Inter-LCKA}\right)$

Patch-Based CKA (PCKA): For small batch sizes or high-dimensional tasks (e.g., object detection), partition feature maps into patches and perform CKA over patch-wise relations, stabilizing training and reducing compute.
Feature Structure Distillation (NLP): Employ RCKA losses at three levels: intra-feature (within sample/token), local inter-feature (across sample-tokens), and global (clustered centroids), often using memory banks via k-means clustering for global structure transfer (Jung et al., 2022).
Kernel Alignment Learning: In classical kernel methods, identify optimal convex combinations of kernels by maximizing RCKA with a relation kernel, solved by non-negative QP (Cortes et al., 2012).

Below is canonical pseudocode for feature-level linear CKA:

def linear_CKA(X, Y):
    # Zero-center columns of X, Y (n x d, n x q)
    XtX = X.T @ X
    YtY = Y.T @ Y
    YtX = Y.T @ X
    num = (YtX * YtX).sum()  # ||YᵀX||_F^2
    denom = np.sqrt((XtX * XtX).sum()) * np.sqrt((YtY * YtY).sum())
    return num / denom

4. Task-Specific Customization and Applications

RCKA is widely applied in knowledge distillation for deep learning, where it transfers representational structure from teacher to student networks (Zhou et al., 22 Jan 2024, Jung et al., 2022):

Image Classification (large batch): Standard RCKA aligning feature and logit relations.
Object Detection (small batch): Patch-based RCKA (PCKA) to address limited instance diversity per batch, with channel-wise patch averaging for stability.
Transformer and LLMs: Distillation at the intra-feature (token), local inter-feature (across tokens), and global-feature (memory-augmented clustering) levels, leveraging relation kernels between tokens or memory centroids.

Criteria for choosing RCKA configuration include batch size, feature dimensionality, and the spatial or semantic alignment desiderata of the task. Hyperparameters (e.g., alignment weights $\alpha$ , $\beta$ ; patch count; memory sizes) are tuned per domain, with empirical guidance in the literature (Zhou et al., 22 Jan 2024, Jung et al., 2022).

5. Empirical Results and Evidence

Extensive experimental validation underlines RCKA's effectiveness:

CIFAR-100: RCKA outperforms or matches prior state-of-the-art distillation methods across 9 teacher-student pairs; e.g., for ResNet110 $\rightarrow$ ResNet20, baseline KD is 69.06%, RCKA achieves 72.26% (Zhou et al., 22 Jan 2024).
ImageNet-1k: For ResNet34 $\rightarrow$ ResNet18, KD baseline is 70.66%, RCKA improves to 72.34%. On MobileNet and ViT distillation, RCKA consistently delivers improvements.
MS-COCO Detection: PCKA yields large AP improvements, e.g., RetinaNet X101 $\rightarrow$ R50: baseline KD 37.2 AP, PCKA 40.3 AP (Zhou et al., 22 Jan 2024).
GLUE NLP Tasks: RCKA-based structure distillation in BERT transfer realized consistent gains across all nine GLUE tasks compared to prior distillation techniques, demonstrating robust preservation of feature structure (Jung et al., 2022).
Neural Representation Analysis: RCKA (CKA) correctly recovers layer correspondences across randomly initialized or differently structured CNNs, identifying both layerwise and cross-architecture similarity with high specificity (Kornblith et al., 2019).

6. Limitations, Variants, and Extensions

Several practical considerations and limitations apply:

Batch and Dimensionality Constraints: Standard (full) RCKA requires sufficient instance and feature diversity; otherwise, patch-based or global structure modifications are preferred.
Kernel Choice and Scaling: Most applications report linear kernel alignment. Nonlinear (e.g., RBF) variants yield similar results but can increase compute (Kornblith et al., 2019).
Sensitivity to Centering and Numerical Stability: Proper centering—and Gram matrix conditioning—are crucial for reliable RCKA loss operation and alignment interpretability.
Limitations in Distribution Shift Scenarios: When teacher and student architectures differ greatly, full-CKA alignment may degrade and require architectural or algorithmic adaptation (Zhou et al., 22 Jan 2024).
Parameter-Free by Design: Typical RCKA instantiations do not require extra parameterized transformations, reducing overhead.

Possible extensions include optimal kernel learning for alignments, modular subspace RCKA, and integration with clustering-based representation analysis (Cortes et al., 2012, Kornblith et al., 2019). A plausible implication is that future research may exploit side-relation kernels (e.g., from data augmentations or multi-view constraints) for more flexible or domain-specialized alignments.

7. Relation to Broader Kernel and Representation Learning

RCKA unifies kernel alignment-based approaches from classical kernel learning, neural representation similarity analysis, and deep model distillation. It enables learning representations attuned to specified relational structures (via $R$ ), supports structured prediction, multitask, and transfer learning, and facilitates interpretable analysis of model invariances and capacity. In kernel learning, RCKA solves the alignment maximization problem for arbitrary relation targets, providing convexity, stability, and statistical concentration guarantees (Cortes et al., 2012). In deep learning, RCKA operationalizes interpretable, regularized relational transfer across model hierarchies (Zhou et al., 22 Jan 2024, Jung et al., 2022). This suggests RCKA functions as a theoretical and algorithmic bridge connecting classical statistical alignment criteria with modern neural architecture optimization.