Representational Similarity Analysis (RSA)

• Representational Similarity Analysis (RSA) is a statistical method that transforms high-dimensional neural or model activity into representational dissimilarity matrices (RDMs) for cross-system comparison.
• It leverages varied metrics such as Euclidean distance and Pearson’s correlation along with techniques like GLM, gradient descent, and deep learning to robustly compare complex data.
• RSA is widely applied in neuroscience, AI, and behavioral research to enhance model interpretability, align human and machine representations, and support innovative cross-modal analyses.

Representational Similarity Analysis (RSA) is a statistical and computational framework for quantifying and comparing the internal representational geometries of neural, behavioral, and artificial systems. By abstracting complex, high-dimensional activity patterns into similarity structures—commonly represented as representational dissimilarity matrices (RDMs)—RSA enables rigorous comparisons across measurement modalities, subjects, species, computational models, and stimuli. It is heavily utilized in cognitive neuroscience, systems neuroscience, and increasingly in artificial intelligence and computational linguistics for model comparison, interpretability, and alignment assessment.

1. Theoretical Principles and Core Definitions

At its foundation, RSA operates by transforming multivariate response patterns (e.g., fMRI voxel time series, neural network activations, behavioral similarity judgments) into a pairwise (dis)similarity matrix. For a set of N stimuli or experimental conditions, responses are collected as vectors $r_i$ for each stimulus $i$. The RDM is then constructed via a dissimilarity function $d$:

$RDM_{ij} = d(r_i, r_j)$

The choice of metric (e.g., Euclidean distance, correlation distance, Mahalanobis distance) is dictated by data type and scientific question. Unlike raw activity patterns, the RDM is invariant to orthogonal transformations and provides an abstract “geometry” of representational space. When applied across systems (e.g., brain regions, neural network layers, behavioral modalities), RSA summarizes by correlating the upper-triangular entries of the respective RDMs:

$$\text{RSA score} = \mathrm{corr}(\text{vec}(RDM_A), \text{vec}(RDM_B))$$

This correlation (often Spearman’s $\rho$ or Pearson’s $r$) reflects the alignment of relational geometry, enabling direct comparison even when representational bases differ completely (Chrupała et al., 2019, Abnar et al., 2019, Bersch et al., 2022).

2. Methodological Implementations and Extensions

Classical workflow. Standard RSA leverages a general linear model (GLM) for neural data, estimating response coefficients $B$ from the relation

$Y = X \cdot B + \epsilon$

where $Y$ is the time series data, $X$ is the design matrix, and $B$ contains neural signatures per condition or category (Sheng et al., 2018). Dissimilarities are then computed between coefficient vectors $B$ for all stimulus pairs.

Regularized and scalable algorithms. Classical approaches depend on inversion of large covariance matrices, which is problematic for high-dimensional data. This computational bottleneck is addressed by Gradient-based RSA (GRSA) (Sheng et al., 2018), which:

• Reformulates the estimation as an optimization problem with L1 (LASSO) regularization:

$J(B) = \|Y - X B\|^2_F + \alpha \|B\|_1$

• Solves via mini-batch stochastic gradient descent (SGD), sidestepping matrix inversion and enabling scalability to full-brain and multi-subject settings.

Searchlight RSA. For spatial mapping, the searchlight approach slides a small, local region (e.g., 3×3×3 voxels) across the brain. RSA is computed within each local cube, producing a detailed spatial map of representational similarity (Sheng et al., 2018, Bersch et al., 2022).

Deep extensions and nonlinearity. Deep Representational Similarity Learning (DRSL) replaces the linear transformation with subject-specific neural networks, enabling complex nonlinear mapping from raw fMRI signals to compact, information-rich signatures (Yousefnezhad et al., 2020).

Partial correlation and whitening. When the design matrix $X$ is not orthogonal, classical RSA can be confounded. Corrective frameworks include:

• Partialing out the bias by controlling for the covariance matrix in the GLM (i.e., (X′X)$^{-1}$), removing spurious correlations (Viviani, 2021).
• Whitened unbiased RDM cosine similarity (WUC) combines cross-validated (unbiased) estimators of dissimilarity with whitening by the full (co-)variance of estimates, enabling statistically robust model selection in the presence of correlated and heteroscedastic noise (Diedrichsen et al., 2020).

Deconfounded similarity. In network comparison contexts, confounding from input population structure is removed by regressing out baseline input similarity from the representational similarity matrices before final correlation (the “deconfounded RSA”) (Cui et al., 2022).

Topological extensions. Recent proposals generalize the RDM using nonlinear, monotonic (piecewise linear) transforms, emphasizing discrete topological structure (e.g., neighborhood relations) rather than fine-grained metric geometry. This yields geo-topological matrices and “topological RSA” (tRSA), which can be “tuned” from pure geometry to pure topology via threshold parameters (Lin et al., 2023, Lin, 21 Aug 2024).

3. Application Domains

Neuroscience and systems biology. RSA bridges data from fMRI, EEG, single-unit recordings, or other modalities to compare representations across brain regions, species, or levels of analysis. For example:

• Searchlight and spatiotemporal GRSA enable tractable, robust comparison of cognitive task representations across the whole brain (Sheng et al., 2018).
• Single-trial RSA extends the approach to time-resolved EEG, revealing dynamic encoding of semantic features in emotion processing (Cheng, 2021).
• Topological RSA and allied methods are now used to identify computational signatures resistant to individual variability and measurement noise (Lin et al., 2023, Lin, 21 Aug 2024).

Artificial intelligence model comparison. RSA is widely deployed to interpret, compare, and audit neural network representations:

• Linguistic models: RSA detects encoding of syntactic and semantic features in BERT, ELMo, and other encoders, including layerwise tracking of linguistic phenomena (Chrupała et al., 2019, Abdou et al., 2019, Lepori et al., 2020).
• Foundation models for vision and computational pathology: RSA reveals how architectural family (e.g., CNN vs. Transformer), training paradigm (self-supervised vs. contrastive), and even stain normalization affect internal representation geometry (Wu et al., 4 Sep 2025, Mishra et al., 18 Sep 2025).
• Cross-lingual speech: RSA using Centered Kernel Alignment (CKA) quantifies the preservation of phonological and acoustic structure across languages and encoder architectures (Abdullah et al., 2021).

Human-model alignment. Turing RSA uses group and individual pairwise similarity ratings to assess semantic alignment between human representations and LLMs/VLMs, revealing model strengths and limitations in reproducing the structure and variability of human cognition across modalities (Ogg et al., 30 Nov 2024).

4. Statistical, Computational, and Interpretational Considerations

Discriminability and separability. RSA is among the highest-performing methods (d′ ≈ 3.8, ROC-AUC > 0.91) for separating model families when compared to other similarity metrics such as linear predictivity, Procrustes alignment, or soft-matching, due to its strict preservation of relative geometric structure (Wu et al., 4 Sep 2025).

Sampling constraints and denoising. Limited neuron sampling systematically underestimates representational similarity due to eigenvector delocalization. Analytical correction using random matrix theory and spectral denoising allows recovery of population-level similarity from under-sampled data (Kang et al., 27 Feb 2025).

Bias and confounding. Non-orthogonality in experimental design or stimulus dependencies can bias classical RSA scores. Approaches such as partial correlation correction (controlling for off-diagonal design matrix structure), cross-validated distance estimation, and whitening achieve near-unbiased inference (Viviani, 2021, Diedrichsen et al., 2020, Cui et al., 2022).

Model flexibility and regularization. Advanced implementations combine L1 (LASSO) and L2 (ridge) regularization in the regression model, and deep learning-based pipelines (e.g., DRSL) for nonparametric adaptability to complex, high-dimensional fMRI or multi-subject data (Sheng et al., 2018, Yousefnezhad et al., 2020).

Interpretability. RSA uniquely enables higher-order and cross-modal comparison (e.g., model–neural–behavioral), direct model selection, and elucidation of when and where cognitive or computational models capture functionally relevant stimulus structure (Chrupała et al., 2019, Abdou et al., 2019, Ogg et al., 30 Nov 2024).

5. Practical Guideline Table: RSA Implementation and Model Comparison

RSA Variant	Matrix Input	Key Alignment Metric	Domain Suitability
Classical/GLM-based	GLM $\to$ RDM	Pearson/Spearman correlation	fMRI; small/medium voxels
Gradient-based (GRSA)	Data, Mini-batches	SGD, L1/L2 loss	Whole-brain, large N
Deep (DRSL)	Neural net $\to$ RDM	Deep learn. + regression	fMRI, multi-subject, nonlin.
Partialled RSA	BB', Bcov	Partial correlation	Searchlight, bias-prone
Whitened Unbiased (WUC)	Cross-validated RDM	Cosine/whitened similarity	All, correlated noise/data
Deconfounded	RSMs, input simil.	Residual RSA/CKA	Model/model, OOD, transfer
Topological (tRSA)	RGTM, RDM	Varying topology/geometry	Robust/variant-invariant

Key: RDM = Representational Dissimilarity Matrix, RSM = Representational Similarity Matrix, RGTM = Geo-Topological Matrix, OOD = Out-of-distribution.

6. Impact, Contemporary Directions, and Future Prospects

RSA is a central tool for interrogating how brains, models, or behavioral systems encode, transform, and structure information. Its flexibility (abstracting away from basis, scaling, or modality), coupled with evolving methodological extensions, underpins its capacity for neuroscientific, cognitive, and AI research. Recent advances emphasize several trends:

• Integration of topological methods (tRSA, geo-topological transforms, persistent homology) for robust, noise-resistant model/brain comparisons (Lin et al., 2023, Lin, 21 Aug 2024).
• Scaling to high-dimensional, massive, or temporally-resolved data, using computationally efficient algorithms (mini-batch SGD, deep architectures, temporal persistence methods) (Yousefnezhad et al., 2020, Lin et al., 2019).
• Bias reduction, statistical optimality, and model selection robustness via whitening, crossvalidation, and spectral techniques (Diedrichsen et al., 2020, Kang et al., 27 Feb 2025).
• Application to human–machine alignment on both group and individual levels, revealing the transfer, gaps, and variability in semantic and perceptual geometry (Ogg et al., 30 Nov 2024).

Future directions include linking RSA-derived topological invariants to information-theoretic coding principles, designing new alignment metrics for richer or more complex data structures, and further extending RSA frameworks to unsupervised and time-resolved analyses across neuroscience and machine learning domains (Lin, 21 Aug 2024).

RSA continues to serve as a cornerstone in quantitative model–brain, brain–behavior, and model–model comparison, with its methodological flexibility and theoretical soundness ensuring continued impact in systems, cognitive neuroscience, and AI research.