---
title: Representational Similarity Matrices
url: https://www.emergentmind.com/topics/representational-similarity-matrices
type: topic
---

# Representational Similarity Matrices

Representational Similarity Matrices (RSMs) are central tools in the comparative analysis of neural representations across neuroscience and artificial intelligence. They are used to quantify, visualize, and statistically test the structural organization of activity patterns evoked by different stimuli or inputs in both biological neural systems and artificial neural networks. RSMs, and their corresponding Representational Dissimilarity Matrices (RDMs), abstract away from the idiosyncrasies of individual neurons or units, focusing instead on the pattern of pairwise similarities or dissimilarities between induced activations. This abstraction facilitates model-model, brain-model, and brain-brain comparisons, supporting inferences about underlying computational roles and invariances.

## 1. Foundations: Definition and Construction

RSMs encode the full pairwise similarity structure between activation patterns arising from a fixed set of inputs. Given $N$ stimuli, and for a particular layer or population response, let $R \in \mathbb{R}^{N \times D}$ denote the matrix of activations across $D$ features (neurons, units, or channels). The RSM $S \in \mathbb{R}^{N \times N}$ is computed by applying a chosen similarity function $s(\cdot, \cdot)$ to each pair of activation vectors:
\[
S_{i,j} = s(R_i, R_j),
\]
where $R_i$ and $R_j$ are the activation vectors for stimuli $i$ and $j$ respectively. Common choices include the inner product (linear kernel), cosine similarity, or the exponential of negative squared Euclidean distance (RBF kernel). The analogous Representational Dissimilarity Matrix is constructed via a (pseudo-)distance metric, e.g., $d_{i,j} = \|R_i - R_j\|_2$.

For neuroimaging data (fMRI, EEG), RDMs can be constructed from summary regression weights (e.g., GLM-derived betas) or directly from timeseries slices, using correlation distances or more elaborate noise-corrected metrics [1711.03058, 2102.08931]. In neural network analysis, RSMs and RDMs provide the basis for methods such as RSA, centered kernel alignment (CKA), Bures similarity, and a range of subspace alignment statistics [2305.06329, 2311.11436].

## 2. Properties and Invariance Structure

RSMs serve as summary statistics that are invariant to reorderings, translations, or rotations of the feature axes. This property makes them especially useful for comparing representations when there is no natural correspondence between neurons or network units across systems. For example, in the context of deep networks, two models trained independently on the same dataset may develop different weight configurations, but their RSMs can reveal whether the relational structure among inputs is matched up to an orthogonal transformation [1811.11684]. In fMRI analysis, this invariance allows for cross-subject or cross-session analysis without explicitly aligning voxels or sensors.

These invariance properties are further formalized in work contrasting alignment-based measures (which seek explicit mappings such as Procrustes or CCA) and kernel-based or geometric measures (CKA, RSA) that compare RSMs directly, agnostic to neuron identity [2311.11436, 2411.14633]. The choice of inner similarity function $s(\cdot,\cdot)$ directly determines which transformations the RSM is invariant to: for instance, cosine similarity yields invariance under rotation and isotropic scaling, while Euclidean distances are preserved under orthogonal transformations.

## 3. Applications Across Neuroscience and AI

RSMs are foundational to Representational Similarity Analysis (RSA), which tests computational models by comparing predicted and observed representational geometries. In neuroscience, RSA and its variants have been used to quantify similarity structures in fMRI, EEG/MEG, and invasive recordings, enabling tests for the presence of specific computational motifs and the comparison of semantic or perceptual category structures [1906.09264, 2110.03529, 2309.11028]. In AI, RSM-based metrics provide core comparators for layer-wise analysis of neural networks, cross-architecture evaluations, and are increasingly employed in transfer learning and domain adaptation [2202.00095, 2410.23107].

Notably, methods such as GRSA and SSL-GRSA exploit RSMs in scalable and efficient ways for large fMRI datasets, using gradient-based optimization and searchlight techniques to enable whole-brain analyses [1809.04429]. In computational pathology, hierarchical clustering over RSMs facilitates identification of the sources of representational specificity—such as slide, disease, or training paradigm—in foundation models [2509.15482].

RSMs additionally underlie Turing-RSA, which extends similarity analysis to behavioral and artificial cognition spaces by using pairwise human (and model) similarity judgments to benchmark and interpret model alignment with human semantics [2412.00577].

## 4. From Geometry to Topology: Extensions of RSMs

Traditional RSM-based analysis focuses on geometric information—i.e., the exact pattern of distances among representations. Recent work has proposed extensions that abstract from geometry to topology, introducing geo-topological or topological RSA (tRSA) [2309.11028, 2408.11948]. In tRSA, nonlinear monotonic transformations compress small and large distance values, emphasizing the underlying neighborhood relationships rather than sensitivity to all metric detail. For example, a piecewise linear map $GT_{l,u}(d)$ thresholds distances near zero and one but stretches intermediate values, making the resultant descriptors robust to noise and inter-individual idiosyncrasies.

Such approaches have demonstrated that population codes can be compared on computationally salient features while suppressing irrelevant idiosyncratic detail, leading to increased robustness in model selection and region/layer identification. Time-extended versions, e.g., as in Procrustes-aligned MDS, further facilitate dynamic representation analysis in both neurophysiological and single-cell transcriptomic data [1906.09264, 2408.11948].

## 5. Methodological Considerations: Bias, Confounds, and Scalability

While RSMs offer powerful abstraction, a number of methodological challenges must be addressed to ensure valid inference:

- **Noise-induced bias**: In the presence of measurement noise, estimators for distance-based RDMs are positively biased. Cross-validated “unbiased” estimators eliminate this bias at the cost of higher variance [2007.02789]. Analytical derivations for both bias and covariance structure allow explicit quantification (see equations for $\mathrm{E}(\hat d_k)$ and $\mathrm{var}(\tilde d_k)$ in the source).
- **Statistical dependence**: The entries in an RSM or RDM are not statistically independent due to shared conditions or stimuli. The covariance among distances can be non-trivial (e.g., $\approx 0.25$ when pairs share a condition). Methods such as whitening the error structure—embodied in the Whitened Unbiased RDM Cosine Similarity (WUC)—improve model selection power by incorporating these dependencies [2007.02789].
- **Confounds from input structure**: Input similarity can artifactually inflate apparent similarity between models or layers. Covariate adjustment, as implemented by regressing out input structure from RSMs prior to comparison (dCKA, dRSA), restores the semantic resolution of similarity metrics [2202.00095].
- **Spatial alignment and its irrelevance**: In computer vision, conventional RSMs conflate semantic similarity and spatial alignment. Semantic RSMs, optimized over spatial permutations (set matching), decouple content from location, yielding similarity measures that better reflect functional/semantic equivalence [2410.23107].
- **Scalability and Efficient Estimation**: For large-scale applications, gradient-based approaches (GRSA, DRSL) and local patch-based strategies (searchlight) enable RSM-based analyses to operate efficiently on high-dimensional datasets, including whole-brain fMRI and large neural architectures [1809.04429, 2010.02012].

## 6. Metric Selection, Interpretation, and Benchmarking

The choice of RSM-based similarity metric directly governs the sensitivity and specificity of comparative analysis. Recent systematic benchmarks have evaluated over twenty similarity metrics on large, cross-domain suites of models and datasets, using tests grounded in functional outcomes (accuracy, prediction divergence) and designed perturbations (label randomization, shortcut manipulation) [2408.00531, 2509.04622].

Empirical findings indicate that methods imposing stronger geometric alignment constraints—such as RSA with rigid geometric preservation, Soft Matching, or Procrustes alignment—achieve higher discriminative power compared to more flexible linear predictivity. RSM-centric metrics excel at separating architectural families and training paradigms, and decisively outperform looser alignment methods with respect to d-prime, silhouette, and ROC-AUC separability statistics.

Furthermore, functional benchmarks in NeuroAI indicate that RSM-based methods emphasizing global geometry—specifically linear CKA and Procrustes distance—align best with behavioral outcomes and effectively distinguish trained from untrained models [2411.14633]. These insights reinforce the centrality of RSMs in meaningful, behaviorally-linked comparative research, and caution against over-reliance on flexible but potentially non-specific mapping-based approaches.

| Metric Category      | Alignment Constraint         | Typical Use/Strengths                                              |
|---------------------|-----------------------------|--------------------------------------------------------------------|
| RSM-based (e.g. RSA, CKA)      | None (geometry preserved)   | High separability, robust to neuron relabeling, used for structure |
| Alignment-based (e.g. Procrustes) | Orthogonal or soft mapping   | Enforces match in global shape/geometry, high discriminability     |
| Linear Predictivity | Arbitrary linear map         | Captures maximal overlap, less sensitive to structure              |
| Soft Matching       | Optimal transport/permutation | Sensitive to fine neuron tuning, high separability                 |

## 7. Extensions and Future Directions

Continued advances in RSM methodology and application are anticipated in several directions:

- **Topology-based and adaptive similarity analysis:** tRSA, AGTDM, and related methods will likely see broader use in scenarios requiring robustness to noise, individual heterogeneity, and developmental changes, as well as in bridging modalities (neural data, single-cell, AI models) [2309.11028, 2408.11948].
- **Bayesian and uncertainty-aware metrics:** Bayesian comparisons based on predictive distributions introduce pseudo-metrics (e.g., Jensen-Shannon or total variation of predictive outputs) linked directly to the utility of a representation under linear probes, thereby incorporating both noise and inductive bias [2411.08739].
- **Human-AI alignment:** Hybrid behavioral/similarity frameworks (Turing RSA) will supplement existing accuracy-based and neuro-comparative analyses by quantifying not only structural alignment but individual- and group-level correspondence in semantic spaces [2412.00577].
- **Benchmark-driven development:** The use of comprehensive, multi-metric benchmark suites (ReSi) ensures that new representational similarity measures are assessed against functional, structural, and perturbation-based groundings, favoring approaches that are both theoretically principled and empirically validated [2408.00531].
- **Dynamic, temporal, and multimodal analysis:** Time-resolved and cross-modal generalizations of RSMs (movies of dynamic RDMs, temporal alignment) advance the analysis of representations in both biological and artificial agents, including their evolution over experience and learning stages [1906.09264, 2408.11948].

A plausible implication is that the integration of geometric, topological, and distributional perspectives—properly harnessed through RSMs and their modern extensions—will drive further unification of methodologies across neuroscience, cognitive science, and AI, promoting new discoveries of representational code structure, robustness, and transferability.

Source: https://www.emergentmind.com/topics/representational-similarity-matrices