Representational Similarity Matrices

• Representational Similarity Matrices (RSMs) are statistical constructs that quantify pairwise similarities across neural activations, enabling direct comparisons between biological and artificial systems.
• They are computed using various similarity functions like cosine similarity or inner product, and their invariance to feature reordering supports robust model and brain alignment.
• Applied in neuroscience and AI, RSMs facilitate rigorous analysis of representational geometry, supporting methodologies like RSA and advancing our understanding of neural coding.

Representational Similarity Matrices (RSMs) are central tools in the comparative analysis of neural representations across neuroscience and artificial intelligence. They are used to quantify, visualize, and statistically test the structural organization of activity patterns evoked by different stimuli or inputs in both biological neural systems and artificial neural networks. RSMs, and their corresponding Representational Dissimilarity Matrices (RDMs), abstract away from the idiosyncrasies of individual neurons or units, focusing instead on the pattern of pairwise similarities or dissimilarities between induced activations. This abstraction facilitates model-model, brain-model, and brain-brain comparisons, supporting inferences about underlying computational roles and invariances.

1. Foundations: Definition and Construction

RSMs encode the full pairwise similarity structure between activation patterns arising from a fixed set of inputs. Given $N$ stimuli, and for a particular layer or population response, let $R \in \mathbb{R}^{N \times D}$ denote the matrix of activations across $D$ features (neurons, units, or channels). The RSM $S \in \mathbb{R}^{N \times N}$ is computed by applying a chosen similarity function $s(\cdot, \cdot)$ to each pair of activation vectors: $S_{i,j} = s(R_i, R_j),$ where $R_i$ and $R_j$ are the activation vectors for stimuli $i$ and $j$ respectively. Common choices include the inner product (linear kernel), cosine similarity, or the exponential of negative squared Euclidean distance (RBF kernel). The analogous Representational Dissimilarity Matrix is constructed via a (pseudo-)distance metric, e.g., $d_{i,j} = \|R_i - R_j\|_2$.

For neuroimaging data (fMRI, EEG), RDMs can be constructed from summary regression weights (e.g., GLM-derived betas) or directly from timeseries slices, using correlation distances or more elaborate noise-corrected metrics (Shvartsman et al., 2017, Viviani, 2021). In neural network analysis, RSMs and RDMs provide the basis for methods such as RSA, centered kernel alignment (CKA), Bures similarity, and a range of subspace alignment statistics (Klabunde et al., 2023, Harvey et al., 2023).

2. Properties and Invariance Structure

RSMs serve as summary statistics that are invariant to reorderings, translations, or rotations of the feature axes. This property makes them especially useful for comparing representations when there is no natural correspondence between neurons or network units across systems. For example, in the context of deep networks, two models trained independently on the same dataset may develop different weight configurations, but their RSMs can reveal whether the relational structure among inputs is matched up to an orthogonal transformation (Lu et al., 2018). In fMRI analysis, this invariance allows for cross-subject or cross-session analysis without explicitly aligning voxels or sensors.

These invariance properties are further formalized in work contrasting alignment-based measures (which seek explicit mappings such as Procrustes or CCA) and kernel-based or geometric measures (CKA, RSA) that compare RSMs directly, agnostic to neuron identity (Harvey et al., 2023, Bo et al., 21 Nov 2024). The choice of inner similarity function $s(\cdot,\cdot)$ directly determines which transformations the RSM is invariant to: for instance, cosine similarity yields invariance under rotation and isotropic scaling, while Euclidean distances are preserved under orthogonal transformations.

3. Applications Across Neuroscience and AI

RSMs are foundational to Representational Similarity Analysis (RSA), which tests computational models by comparing predicted and observed representational geometries. In neuroscience, RSA and its variants have been used to quantify similarity structures in fMRI, EEG/MEG, and invasive recordings, enabling tests for the presence of specific computational motifs and the comparison of semantic or perceptual category structures (Lin et al., 2019, Cheng, 2021, Lin et al., 2023). In AI, RSM-based metrics provide core comparators for layer-wise analysis of neural networks, cross-architecture evaluations, and are increasingly employed in transfer learning and domain adaptation (Cui et al., 2022, Wald et al., 30 Oct 2024).

Notably, methods such as GRSA and SSL-GRSA exploit RSMs in scalable and efficient ways for large fMRI datasets, using gradient-based optimization and searchlight techniques to enable whole-brain analyses (Sheng et al., 2018). In computational pathology, hierarchical clustering over RSMs facilitates identification of the sources of representational specificity—such as slide, disease, or training paradigm—in foundation models (Mishra et al., 18 Sep 2025).

RSMs additionally underlie Turing-RSA, which extends similarity analysis to behavioral and artificial cognition spaces by using pairwise human (and model) similarity judgments to benchmark and interpret model alignment with human semantics (Ogg et al., 30 Nov 2024).

4. From Geometry to Topology: Extensions of RSMs

Traditional RSM-based analysis focuses on geometric information—i.e., the exact pattern of distances among representations. Recent work has proposed extensions that abstract from geometry to topology, introducing geo-topological or topological RSA (tRSA) (Lin et al., 2023, Lin, 21 Aug 2024). In tRSA, nonlinear monotonic transformations compress small and large distance values, emphasizing the underlying neighborhood relationships rather than sensitivity to all metric detail. For example, a piecewise linear map $GT_{l,u}(d)$ thresholds distances near zero and one but stretches intermediate values, making the resultant descriptors robust to noise and inter-individual idiosyncrasies.

Such approaches have demonstrated that population codes can be compared on computationally salient features while suppressing irrelevant idiosyncratic detail, leading to increased robustness in model selection and region/layer identification. Time-extended versions, e.g., as in Procrustes-aligned MDS, further facilitate dynamic representation analysis in both neurophysiological and single-cell transcriptomic data (Lin et al., 2019, Lin, 21 Aug 2024).

5. Methodological Considerations: Bias, Confounds, and Scalability

While RSMs offer powerful abstraction, a number of methodological challenges must be addressed to ensure valid inference:

• Noise-induced bias: In the presence of measurement noise, estimators for distance-based RDMs are positively biased. Cross-validated “unbiased” estimators eliminate this bias at the cost of higher variance (Diedrichsen et al., 2020). Analytical derivations for both bias and covariance structure allow explicit quantification (see equations for $\mathrm{E}(\hat d_k)$ and $\mathrm{var}(\tilde d_k)$ in the source).
• Statistical dependence: The entries in an RSM or RDM are not statistically independent due to shared conditions or stimuli. The covariance among distances can be non-trivial (e.g., $\approx 0.25$ when pairs share a condition). Methods such as whitening the error structure—embodied in the Whitened Unbiased RDM Cosine Similarity (WUC)—improve model selection power by incorporating these dependencies (Diedrichsen et al., 2020).
• Confounds from input structure: Input similarity can artifactually inflate apparent similarity between models or layers. Covariate adjustment, as implemented by regressing out input structure from RSMs prior to comparison (dCKA, dRSA), restores the semantic resolution of similarity metrics (Cui et al., 2022).
• Spatial alignment and its irrelevance: In computer vision, conventional RSMs conflate semantic similarity and spatial alignment. Semantic RSMs, optimized over spatial permutations (set matching), decouple content from location, yielding similarity measures that better reflect functional/semantic equivalence (Wald et al., 30 Oct 2024).
• Scalability and Efficient Estimation: For large-scale applications, gradient-based approaches (GRSA, DRSL) and local patch-based strategies (searchlight) enable RSM-based analyses to operate efficiently on high-dimensional datasets, including whole-brain fMRI and large neural architectures (Sheng et al., 2018, Yousefnezhad et al., 2020).

6. Metric Selection, Interpretation, and Benchmarking

The choice of RSM-based similarity metric directly governs the sensitivity and specificity of comparative analysis. Recent systematic benchmarks have evaluated over twenty similarity metrics on large, cross-domain suites of models and datasets, using tests grounded in functional outcomes (accuracy, prediction divergence) and designed perturbations (label randomization, shortcut manipulation) (Klabunde et al., 1 Aug 2024, Wu et al., 4 Sep 2025).

Empirical findings indicate that methods imposing stronger geometric alignment constraints—such as RSA with rigid geometric preservation, Soft Matching, or Procrustes alignment—achieve higher discriminative power compared to more flexible linear predictivity. RSM-centric metrics excel at separating architectural families and training paradigms, and decisively outperform looser alignment methods with respect to d-prime, silhouette, and ROC-AUC separability statistics.

Furthermore, functional benchmarks in NeuroAI indicate that RSM-based methods emphasizing global geometry—specifically linear CKA and Procrustes distance—align best with behavioral outcomes and effectively distinguish trained from untrained models (Bo et al., 21 Nov 2024). These insights reinforce the centrality of RSMs in meaningful, behaviorally-linked comparative research, and caution against over-reliance on flexible but potentially non-specific mapping-based approaches.

Metric Category	Alignment Constraint	Typical Use/Strengths
RSM-based (e.g. RSA, CKA)	None (geometry preserved)	High separability, robust to neuron relabeling, used for structure
Alignment-based (e.g. Procrustes)	Orthogonal or soft mapping	Enforces match in global shape/geometry, high discriminability
Linear Predictivity	Arbitrary linear map	Captures maximal overlap, less sensitive to structure
Soft Matching	Optimal transport/permutation	Sensitive to fine neuron tuning, high separability

7. Extensions and Future Directions

Continued advances in RSM methodology and application are anticipated in several directions:

• Topology-based and adaptive similarity analysis: tRSA, AGTDM, and related methods will likely see broader use in scenarios requiring robustness to noise, individual heterogeneity, and developmental changes, as well as in bridging modalities (neural data, single-cell, AI models) (Lin et al., 2023, Lin, 21 Aug 2024).
• Bayesian and uncertainty-aware metrics: Bayesian comparisons based on predictive distributions introduce pseudo-metrics (e.g., Jensen-Shannon or total variation of predictive outputs) linked directly to the utility of a representation under linear probes, thereby incorporating both noise and inductive bias (Schütt, 13 Nov 2024).
• Human-AI alignment: Hybrid behavioral/similarity frameworks (Turing RSA) will supplement existing accuracy-based and neuro-comparative analyses by quantifying not only structural alignment but individual- and group-level correspondence in semantic spaces (Ogg et al., 30 Nov 2024).
• Benchmark-driven development: The use of comprehensive, multi-metric benchmark suites (ReSi) ensures that new representational similarity measures are assessed against functional, structural, and perturbation-based groundings, favoring approaches that are both theoretically principled and empirically validated (Klabunde et al., 1 Aug 2024).
• Dynamic, temporal, and multimodal analysis: Time-resolved and cross-modal generalizations of RSMs (movies of dynamic RDMs, temporal alignment) advance the analysis of representations in both biological and artificial agents, including their evolution over experience and learning stages (Lin et al., 2019, Lin, 21 Aug 2024).

A plausible implication is that the integration of geometric, topological, and distributional perspectives—properly harnessed through RSMs and their modern extensions—will drive further unification of methodologies across neuroscience, cognitive science, and AI, promoting new discoveries of representational code structure, robustness, and transferability.