---
title: Local CKA-Based Retrieval
url: https://www.emergentmind.com/topics/local-cka-based-retrieval
type: topic
---

# Local CKA-Based Retrieval

Local CKA-based retrieval is a training-free method for matching or retrieving semantically paired data—such as images and captions—across modalities using only a small seed set of aligned examples. The approach leverages Centered Kernel Alignment (CKA) as a statistical metric to quantify, for each query pair, how much augmenting a base graph of aligned embeddings with that pair perturbs the overall modality-invariant geometry. Unlike global methods, which assess the alignment of entire datasets, local CKA evaluates fine-grained, instance-level consistency, enabling robust matching and retrieval across domains, languages, and even 0-shot classification without additional training or explicit alignment [2401.05224].

## 1. Foundations: Global CKA and its Limitations

Global CKA is a normalized similarity metric for assessing the agreement of pairwise geometries between two sets of representations $X \in \mathbb{R}^{d_1 \times N}$ and $Y \in \mathbb{R}^{d_2 \times N}$. Given kernels $K = k(X^\top, X) \in \mathbb{R}^{N \times N}$ and $L = \ell(Y^\top, Y)$ (with $k, \ell$ commonly linear or RBF), both kernels are centered with the centering matrix $H = I_N - (1/N) \mathbf{1}_N \mathbf{1}_N^\top$. The Hilbert–Schmidt Independence Criterion (HSIC) is
$$
\mathrm{HSIC}(K,L) = \frac{1}{(N-1)^2} \mathrm{Tr}(H K H L)
$$
CKA is then defined as
$$
\mathrm{CKA}(K,L) = \frac{\mathrm{HSIC}(K,L)}{\sqrt{\mathrm{HSIC}(K,K)\,\mathrm{HSIC}(L,L)}}
$$
yielding a value in $[0, 1]$. Global CKA, however, is limited: it provides only a global measure of dataset-level alignment. It cannot evaluate the quality of individual image–caption pairs, rendering it insufficient for fine-grained retrieval scenarios [2401.05224].

## 2. Localized CKA: Definition and Scoring Procedure

Localized CKA (localCKA) enables pairwise evaluation by measuring the compatibility of a single query image–caption pair with the geometry defined by a "base" set of $M$ aligned pairs:
- Base set: $B = \{(x_i^b, c_i^b)\}_{i=1}^M$, with $X^b \in \mathbb{R}^{d_1 \times M}$ and $C^b \in \mathbb{R}^{d_2 \times M}$.
- Query: given image $x_j^q$ and caption $c_k^q$.

The localCKA score is computed as:
$$
\mathrm{localCKA}\bigl(x_j^q, c_k^q\bigr) = \mathrm{CKA}\Big(k\left([X^b, x_j^q]^\top, [X^b, x_j^q]\right),\; \ell\left([C^b, c_k^q]^\top, [C^b, c_k^q]\right)\Big)
$$
This augments the base image and caption features with the candidate pair and computes the global CKA over the resulting $(M+1)$ points. A high score indicates the candidate pair maintains the semantic structure of the seed graph, suggesting a correct image–caption match [2401.05224].

## 3. Retrieval and Matching Algorithms

There are two key paradigms: global (seeded QAP) and local (localCKA) matching.

- **Seeded Quadratic Assignment Problem (QAP)**:
  1. Concatenate base and query sets: $X = [X^b, X^q]$, $C = [C^b, C^q]$.
  2. Compute centered, rescaled kernels $\bar K_X$, $\bar K_C$.
  3. Seek permutation $P$ on $N$ query elements that, in block-diagonal $I_M \oplus P$, maximizes $\mathrm{Tr}((I_M \oplus P)^\top \bar K_X (I_M \oplus P)\bar K_C )$.
  4. Fast seeded QAP solvers can be used for efficient approximate solution.

- **LocalCKA-based Retrieval or Matching**:
  - For each query image $x_i^q$ and caption $c_j^q$, compute
    $$
    K_i = k\bigl([X^b, x_i^q]^\top, [X^b, x_i^q]\bigr), \quad L_j = \ell\bigl([C^b, c_j^q]^\top, [C^b, c_j^q]\bigr)
    $$
    then compute $R_{i,j} = \mathrm{CKA}(K_i, L_j)$.
  - The retrieval for image $i$ is $\arg\max_j R_{i,j}$.
  - For full $N \times N$ matching, assemble $R$ and recover a global permutation via the Hungarian algorithm (linear-sum assignment) on $-R$.

- For both, a carefully chosen seed set is essential; k-means clustering on embeddings is an effective selection strategy. Feature-wise variance normalization ("stretching") is reported to enhance performance across approaches [2401.05224].

## 4. Graph-Theoretic Interpretation and Seed Usage

Data from each modality (images or captions) is represented as a graph with $M+N$ nodes corresponding to base and query embeddings. Edges are weighted by centered kernel values. The $M$ seeds (the base set) are fixed one-to-one matchings between modalities, serving as anchors in the matching process.

- For **seeded QAP**, the permutation aligns query nodes to maximize agreement of edge weights between graphs, subject to the seeds being fixed.
- For **localCKA**, the approach does not attempt a global matching but evaluates each candidate pairing by measuring how much its introduction into the seed graph distorts inter-node relations, as quantified by CKA [2401.05224].

## 5. Empirical Results and Benchmarking

Experiments span multiple vision and language encoders, modalities, and correspondence tasks:

| Task/Setting                        | Metric/Method                 | CLIP-ViT  | DINOv2  | ConvNeXt  | Baselines                |
|-------------------------------------|-------------------------------|-----------|---------|-----------|--------------------------|
| COCO→NoCaps cross-domain matching   | QAP@1                         | 67%       | 58%     | —         | Linear reg (∼50%), rel. (∼45%) |
| COCO→NoCaps cross-domain retrieval  | LocalCKA@5                    | 60.5%     | 61.8%   | —         | —                        |
| COCO val in-domain retrieval        | LocalCKA@5                    | ∼70%      | ∼70%    | —         | rel. (∼65%)              |
| ImageNet-100 zero-shot classification| LocalCKA top-1                | —         | 67.7%   | 83.3%     | CLIP (86.1%)             |
| XTD-10 cross-lingual                | LocalCKA@5                    | —         | —       | —         | CLIP-cos (0% non-Latin)  |

Additional qualitative observations show that even incorrect LocalCKA retrievals tend to be semantically similar to the target caption. Notably, LocalCKA outperforms CLIP-cosine baselines in cross-lingual settings, with CLIP-cosine collapsing (retrieval $\to0\%$) on non-Latin alphabets, while LocalCKA achieves $\sim58\%$ average, representing a $+17\%$ improvement over CLIP [2401.05224].

## 6. Complexity, Practicalities, and Limitations

- **Computational Complexity**:
  - Seeded QAP (fast approximate): $\mathcal{O}(N^3)$. On $N=500$ queries, $M=320$ seeds, computation is $\sim40$s on CPU.
  - Naive LocalCKA: $N^2$ pairings, each evaluating CKA on $(M+1)\times(M+1)$ kernels $\Rightarrow \mathcal{O}(N^2 M^3) \approx \mathcal{O}(N^4)$ if $M\sim N$; practical runtime $\sim5$min on GPU.
  - Baselines: Relative ($\mathcal{O}(N^2)$), Linear regression ($\mathcal{O}(N d)$).
- **Practical Considerations**:
  - Seed set size/control (e.g., $M\sim300$) and careful selection via k-means are critical for robust performance.
  - Feature normalization ("stretching") substantially improves LocalCKA and QAP.
  - LocalCKA’s computational overhead for large $N$ can be mitigated by reusing precomputed HSIC terms and employing incremental updates, yielding possible $\mathcal{O}(N^3)$ scaling.
- **Limitations**:
  - LocalCKA’s $\mathcal{O}(N^4)$ complexity is prohibitive for large-scale settings without optimization.
  - The choice of kernel (linear or RBF) and normalization strategy must be tuned per application.
  - Method requires a (possibly unlabeled) base set of parallel image–caption pairs.

## 7. Extensions and Future Directions

The framework is extensible along several axes:
- Employing approximate CKA computation (e.g., via random Fourier features) to reduce local computation cost.
- Introducing a learnable neural “prompt” layered atop the query augmentation to further enhance cross-modal alignment.
- Substituting seeded QAP with other graph-matching relaxations, such as Gromov–Wasserstein, to improve global scalability and potentially adapt to even broader domains [2401.05224].

In summary, local CKA-based retrieval augments a seed graph of aligned embeddings with candidate query pairs, scoring them by how little they distort the base structural geometry as measured by CKA. This training-free technique is robust across domains, languages, and image classification scenarios, and provides a modular alternative to supervised alignment for vision–language pairing and retrieval tasks [2401.05224].

Source: https://www.emergentmind.com/topics/local-cka-based-retrieval