---
title: Centered Kernel Alignment (CKA) Similarity
url: https://www.emergentmind.com/topics/centered-kernel-alignment-cka-similarity
type: topic
---

# Centered Kernel Alignment (CKA) Similarity

Centered Kernel Alignment (CKA) similarity is a scale-invariant, kernel-based metric that quantifies the similarity between two sets of representations—such as neural activations across layers, models, or even species—by comparing the pairwise similarity structures induced over a shared set of observations. Widely adopted in neuroscience, deep learning, and multimodal AI, CKA underpins both empirical analysis and algorithmic regularization, notably in cross-system alignment, representation diagnostics, pruning, and transfer learning frameworks.

## 1. Mathematical Definition and Properties

Let $X \in \mathbb{R}^{n \times d_1}$ and $Y \in \mathbb{R}^{n \times d_2}$ be two sets of representations, each row corresponding to the same $n$ samples (stimuli, tokens, images, etc.), and (typically) column-centered. Define linear Gram (kernel) matrices
\[
K = XX^\top \qquad L = YY^\top
\]
and the centering matrix
\[
H = I_n - \frac{1}{n} \mathbf{1}_n\mathbf{1}_n^\top.
\]
Centered (feature) Gram matrices are
\[
K_c = HKH, \qquad L_c = HLH.
\]
The empirical Hilbert–Schmidt Independence Criterion (HSIC) is
\[
\mathrm{HSIC}(K,L) = \operatorname{tr}(K_c L_c)
\]
(usually divided by $(n{-}1)^2$, but this factor cancels in normalized CKA).

Centered Kernel Alignment (CKA) is then defined as
\[
\boxed{
\mathrm{CKA}(X, Y) = 
\frac{\operatorname{tr}(K_c L_c)}
     {\sqrt{\operatorname{tr}(K_c^2)\, \operatorname{tr}(L_c^2)}}.
}
\]
For linear kernels, if $X$ and $Y$ are zero-mean column-wise,
\[
\mathrm{CKA}_\text{Linear}(X,Y) = \frac{ \| X^\top Y \|_F^2 }
     { \| X^\top X \|_F \cdot \| Y^\top Y \|_F }.
\]
CKA yields a real scalar in $[0,1]$, with 1 if and only if the row spaces coincide up to orthogonal transformation and scaling.

### Invariance Structure

- **Orthogonal invariance:** $X \gets X Q$ or $Y \gets Y R$ ($Q$, $R$ orthonormal) leaves CKA unchanged.
- **Isotropic scaling invariance:** $X \gets \alpha X$, $Y \gets \beta Y$ leaves CKA unchanged.
- **Not invariant to arbitrary invertible transforms:** CKA is sensitive to non-orthogonal shape deformations.
- **Dependencies on mean removal:** Centering is essential; uncentered data yield misleadingly high alignment for trivial shifts.

These invariances distinguish CKA from CCA, Procrustes, and RV coefficients [1905.00414; 2411.08197; 2210.16156].

## 2. Statistical and Algorithmic Foundations

CKA is grounded in the Hilbert–Schmidt Independence Criterion [2411.08197]. For two neural populations or representation sets, the normalized trace in $\mathrm{CKA}$ computes the cosine of the “angle” between their centered similarity matrices in Frobenius space. This aligns their geometric structures rather than just individual responses.

Equivalently, CKA can be derived from the average alignment between $\ell_2$-regularized linear decoders (i.e., the mean normalized squared inner product between optimal readouts for random regression tasks over the population), establishing a tight link between geometry and functional alignability [2411.08197]. Linear CKA thus quantifies the normalized average agreement of optimal linear decoders across systems.

\textbf{Variants:}
CKA extends to non-linear kernels (e.g., Gaussian RBF), at the cost of higher $O(n^2)$ or $O(n^3)$ memory/time, but most modern deep learning applications adopt the linear variant [2409.18333].

Twelve major variants arise (kernel: linear or RBF) × (HSIC estimator: biased, unbiased, tril) × (scoring: “score” $[0,1]$, or “angular” $[0, \frac{\pi}{2}]$) [2409.18333]. These must not be conflated; different research communities have implemented various forms.

## 3. Bias and Estimation in High Dimensions

### Finite-Sample Bias

CKA’s popularity in high-dimensional, low-sample settings (e.g., neuroscience: $P \gg N$) exposes substantial finite-sample bias. The naive (biased) estimator tends to 1 even for completely random, unaligned representations as feature/sample ratio grows [2405.01012; 2502.15104]. This causes false discoveries of alignment when comparing, e.g., large fMRI ROIs to deep network layers or distinct networks on shared input [2405.01012].

#### Debiased and Bias-Corrected Estimators

Finite-sample correction is achieved via U-statistic-based unbiased centering [2405.01012]:
\[
\widetilde A_{ij} = a_{ij} - \frac{1}{N-2} \sum_{k} a_{ik} - \frac{1}{N-2} \sum_{k} a_{kj} +
       \frac{1}{(N-1)(N-2)} \sum_{k,\ell} a_{k\ell}, \quad (i \ne j)
\]
and final unbiased HSIC and CKA via
\[
\widehat{\mathrm{HSIC}}_\mathrm{unb}(K,L) = \frac{1}{N(N-3)} \sum_{i\ne j} \widetilde K_{ij} \, \widetilde L_{ij},
\]
\[
\mathrm{CKA}_\mathrm{debiased}(K,L) = \frac{\widehat{\mathrm{HSIC}}_\mathrm{unb}(K,L)}
              {\sqrt{ \widehat{\mathrm{HSIC}}_\mathrm{unb}(K,K) \, \widehat{\mathrm{HSIC}}_\mathrm{unb}(L,L) }}.
\]
Further generalization corrects both stimulus and feature sampling; this estimator retains near-unbiasedness down to very sparse neuronal sampling [2502.15104].

### Input-Driven Confounds

When the structure of the input data dominates, even networks initialized with random weights exhibit high CKA in shallow layers. Covariate-adjusted regression (dCKA) removes the influence of the input similarity structure, resolving spurious alignments [2202.00095].

## 4. Practical Algorithms and Usage

### Efficient Computation

For large-scale settings ($n$ up to $10^4$ and $d$ in $10^3$–$10^4$), linear CKA can be implemented without explicit $n \times n$ Gram matrices [1905.00414]. Central steps:

1. Center columns of $X, Y$.
2. Compute $C = X^\top Y$, $S_X = X^\top X$, $S_Y = Y^\top Y$.
3. Compute numerator: $\|C\|_F^2$, denominator: $\|S_X\|_F \cdot \|S_Y\|_F$
4. Return CKA: $\|C\|_F^2 / (\|S_X\|_F \cdot \|S_Y\|_F)$.

For non-linear kernels, matrix computations scale as $O(n^2d)$ to $O(n^3)$.

### Model Pruning and Regularization

CKA serves as an explicit criterion in pruning and training regularization:

- **Layer/Block Pruning:** Group layers with CKA $> \tau$ as “redundant”; prune all but one, retrain. For BERT and T5, $\tau = 0.98$–0.99 yields up to 50% reduction without accuracy loss [2408.13482; 2405.17081].
- **Sparse Training:** Minimizing interlayer CKA provably reduces mutual information and increases sparsity through the information bottleneck [2307.07389].
- **Cross-System Alignment:** In large language models for multilingual MT, layer-wise CKA alignment secures cross-lingual feature sharing; e.g., CKA-based terms at mid-layers yield $\sim$1 BLEU/chRF point gain in low-resource translation [2510.06249].

#### Example: CKA for Alignment Regularization in MT
For parallel sentence pairs $(x^{(A)},x^{(B)})$, extract $ H^{(A)}_\ell, H^{(B)}_\ell \in \mathbb{R}^{T \times d} $, flatten over tokens/batch, center, and compute
\[
\mathrm{CKA}(X, Y) = \frac{ \| X_c^\top Y_c \|_F^2 }
    { \sqrt{ \| X_c^\top X_c \|_F^2 } \sqrt{ \| Y_c^\top Y_c \|_F^2 } }.
\]
Apply a loss penalty $L_{\mathrm{CKA}} = 1 - \mathrm{CKA}(X,Y)$ at layer $\ell$ [2510.06249].

### Subspace-Level CKA

Global CKA can obscure fine-grained, trait-relevant leakage: subspace-level CKA restricts evaluation to task-discriminative directions (e.g., a single projection from a logistic regression classifier), revealing transferability not measured by global similarity [2511.01023].

\[
\mathrm{CKA}_\text{trait-subspace}(Z_T U, Z_S U)
\]
where $U$ spans the trait-relevant basis.

Thresholds on trait-subspace CKA and projection-penalty interventions can reduce leakage with no main-task loss.

## 5. Empirical Observations and Limitations

### Interpretation and Sensitivity

- **Dominance by Principal Components:** Linear CKA disproportionately emphasizes alignment of high-variance principal directions [2407.07059]. Misalignment of leading PCs leads to rapid score drop, while low-variance PCs contribute weakly; this differs from Procrustes or Bures measures which are linearly sensitive.
- **Functional Correspondence:** CKA and Procrustes best correlate with behaviorally meaningful distinctions in both neuroscience and vision models, outperforming predictivity or CCA in differentiating trained/untrained networks [2411.14633].
- **Lack of Universal Thresholds:** No “good” CKA value is universal; the threshold for functionally relevant transfer varies by data, task, and metric [2407.07059].
- **Manipulation and Cautions:** CKA is highly sensitive to outliers and can be manipulated independently of task performance; similar CKA scores may not imply functional equivalence [2210.16156].

### Summary Table: CKA Features and Caveats

| Aspect                 | Mathematical Property                   | Empirical Impact / Caveat                 |
|------------------------|-----------------------------------------|-------------------------------------------|
| Orthogonal/scaling     | Invariant                               | Captures subspace rather than basis       |
| Arbitrary transform    | Not invariant                           | Sensitive to shape, not mere isomorphism  |
| Principal components   | Quadratic sensitivity                   | Led by top-variance dimensions           |
| Outlier sensitivity    | Non-robust                              | Single-point shifts can suppress CKA     |
| Data bias              | Inflated under $P \gg N$                | Debias or covariate adjust for fairness   |
| Kernel choice          | Linear (fast), RBF (nonlinear)          | Linear default in deep learning           |
| Score range            | [0,1] (raw); [0, π/2] (angular)         | Multiple definitions in the literature    |

## 6. Cross-Domain, Multimodal, and Large-Scale Applications

CKA underlies modern approaches to:

- **Cross-modal alignment:** Relating vision and language encoders (including unaligned models) using global or localized CKA, with high CKA ($\sim 0.7$) between SS vision and language encoders, providing a foundation for zero-shot matching and retrieval via CKA-based quadratic assignment [2401.05224].
- **Brain-model alignment:** Applied to fMRI, MEG, and direct recordings; unbiased CKA recovers brain region–layer correspondences not visible with biased estimators or shuffled controls [2405.01012; 2502.15104].
- **Standardization and reproducibility:** Major repositories now catalog 100+ similarity measures, standardizing “linear–gretton–score,” “rbf–unbiased–angular,” etc. to resolve confusion in literature [2409.18333].

## 7. Recommendations and Best Practices

- Always center data or Gram matrices before computing CKA; do not rely on uncentered forms.
- For high-dimensional, low-$n$ regimes or mismatched feature counts, use unbiased corrections or bias-aware variants [2405.01012; 2502.15104].
- Avoid over-interpreting high CKA as functionally meaningful without verifying relevant subspaces or decoding accuracy [2210.16156; 2407.07059].
- Report multiple similarity metrics (CKA, Procrustes, CCA) to contextualize findings, especially in new neural or cross-modal applications [2411.14633; 2407.07059].
- When comparing representations for model selection, pruning, or transfer, use the same CKA variant and estimator across all comparisons for fair benchmarking [2409.18333].

CKA remains a principal—though not unproblematic—instrument for dissecting neural representation geometry and functional alignment in complex artificial and biological systems, especially when augmented with bias-correction and subspace diagnostics.

Source: https://www.emergentmind.com/topics/centered-kernel-alignment-cka-similarity