Metric Similarity Analysis (MSA)

• MSA is a method that compares neural representations based on intrinsic geometry, using pullback Riemannian metrics induced on a common input manifold.
• It leverages a spectral ratio pseudo-distance to assess local metric tensors, ignoring uniform scaling to focus on shape and anisotropy differences.
• Applications span comparing network regimes, nonlinear dynamics in RNNs, and diffusion processes in generative models, highlighting its versatility in latent geometry analysis.

Metric Similarity Analysis (MSA) is a geometry-aware similarity measure for neural representations that compares intrinsic geometry rather than the usual extrinsic geometry of activations in state space. Under the manifold hypothesis, it is defined as a distance between the pullback Riemannian metrics induced by two neural representations on a common input manifold, and it was introduced as a framework for comparing how neural systems warp the same underlying manifold across static representations, nonlinear dynamics, and statistical manifolds (Gajic et al., 30 Mar 2026).

1. Conceptual basis

MSA is motivated by the claim that many standard representation-similarity methods compare the geometry of activations as embedded point clouds in ambient state space, whereas mechanistically relevant differences may lie in how a network transforms the underlying input manifold itself. The paper’s motivating contrast is the difference between a flat $2$-dimensional sheet and a Swiss roll: in activation space these embeddings look very different, so extrinsic methods can report low similarity, even though the manifolds have identical intrinsic distance/angle structure. MSA therefore asks whether two systems endow the same latent variables with the same local geometry, rather than whether their activation clouds align in ambient coordinates (Gajic et al., 30 Mar 2026).

This formulation depends on the manifold hypothesis. The input data are assumed to lie on an intrinsically low-dimensional manifold $\mathcal M$, and the representation map is assumed smooth enough to define Jacobians and pullback metrics. In the core Riemannian setting, the pushforward is assumed injective locally so that the pullback metric is symmetric positive definite. A common misconception is that MSA is simply another activation-space similarity score. In fact, the compared object is the metric structure induced on the underlying manifold, not the raw activation configuration in $\mathbb{R}^n$ (Gajic et al., 30 Mar 2026).

2. Mathematical formulation on representation manifolds

The paper models a neural system as

$\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$

where $\mathcal M$ is an $m$-dimensional input manifold, $\psi$ embeds $\mathcal M$ into input space, $\varphi$ maps inputs to the hidden representation of interest, and $\zeta$ maps onward to outputs. The representation under study is the image $\mathcal M$0, but MSA compares the induced metric on $\mathcal M$1, not that image directly (Gajic et al., 30 Mar 2026).

Let $\mathcal M$2 be the Jacobian of $\mathcal M$3 at $\mathcal M$4. For tangent vectors $\mathcal M$5,

$\mathcal M$6

The local metric tensor is therefore

$\mathcal M$7

This pullback metric encodes local lengths, angles, anisotropy, and more generally local distance structure on the manifold as represented by the network. The paper explicitly notes that MSA does not directly compare curvature tensors, geodesic distances, or higher-order invariants; its operative object is the pointwise pullback metric $\mathcal M$8, making the method a first-order intrinsic comparison (Gajic et al., 30 Mar 2026).

The global MSA distance between two representations $\mathcal M$9 and $\mathbb{R}^n$0 is

$\mathbb{R}^n$1

with associated bounded similarity

$\mathbb{R}^n$2

Thus MSA compares two metric fields on the same manifold and averages the local discrepancy over $\mathbb{R}^n$3 (Gajic et al., 30 Mar 2026).

3. Spectral ratio and invariance structure

The local discrepancy $\mathbb{R}^n$4 is defined on symmetric positive definite matrices by a generalized eigenvalue problem. Given

$\mathbb{R}^n$5

with eigenvalues ordered so that $\mathbb{R}^n$6, the spectral ratio is

$\mathbb{R}^n$7

The paper states that $\mathbb{R}^n$8 and proves that it is a pseudo-distance on SPD matrices, satisfying separation, symmetry, and triangle inequality (Gajic et al., 30 Mar 2026).

A crucial consequence of the spectral-ratio design is that MSA compares shape/anisotropy rather than absolute overall scale. If two local metrics differ only by a scalar factor, then all generalized eigenvalues are equal and $\mathbb{R}^n$9. This means that MSA intentionally ignores uniform local scaling when local shape is preserved. The paper explicitly notes this as a subtle but important design choice (Gajic et al., 30 Mar 2026).

MSA is also characterized by strong invariance properties. It is invariant to independent orthogonal transformations of the two activation spaces: $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$0 because orthogonal transformations leave $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$1 unchanged. It is also invariant under smooth changes of local coordinates on the manifold, since coordinate changes transform both metric tensors by the same congruence and preserve the generalized eigenvalues. The appendix further notes invariance to state-space transformations that are pointwise Euclidean isometries (Gajic et al., 30 Mar 2026).

4. Computational procedure and statistical-manifold extension

The deterministic MSA pipeline is Jacobian-based rather than neighborhood-based. One identifies the latent manifold $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$2, parameterizes it, chooses the representation layer or system state to compare, computes Jacobians $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$3, forms pullback metrics $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$4, evaluates $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$5 pointwise, and then integrates over $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$6. The paper emphasizes that this avoids neighborhood graphs, tangent PCA, or point-cloud tangent estimation; under the assumed manifold parameterization, the comparison is exact at the level of the pullback metric field (Gajic et al., 30 Mar 2026).

The same framework extends to statistical manifolds. If $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$7 parameterizes a density $\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$8, then averaging outer products of score vectors yields the Fisher information matrix

$\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},$9

This $\mathcal M$0 is the local coordinate form of the Fisher–Rao metric, and MSA compares such Fisher metrics exactly as it compares pullback metrics in deterministic settings. The paper uses this extension for diffusion models, where the relevant geometry is information-geometric rather than purely representation-geometric (Gajic et al., 30 Mar 2026).

Practically, the paper does not give a formal complexity analysis, but it states that Jacobian computation dominates for high-dimensional networks, whereas the generalized eigenproblem is cheap when manifold dimension is low. This suggests that MSA is most natural in the low-dimensional manifold regime posited by the manifold hypothesis. The paper also notes that if the manifold is not explicitly known, one may need to learn it first with a parametric model, provided data are sampled densely enough (Gajic et al., 30 Mar 2026).

5. Empirical applications

The first application concerns one-hidden-layer networks trained in rich and lazy regimes on a $\mathcal M$1-dimensional manifold $\mathcal M$2. Here the paper reports a central qualitative result: PCA of activations suggested rich and lazy representations looked similar extrinsically, and Procrustes reported near-perfect similarity, but MSA reported low similarity between rich and lazy networks, while giving higher similarity for same-regime different-seed comparisons. In a hierarchy varying regime, task, and seed, MSA recovered the intended hierarchy of representational difference (Gajic et al., 30 Mar 2026).

The second application compares nonlinear dynamics in vanilla RNNs and structured state-space models on a working-memory task with input manifold $\mathcal M$3. Time is added as an extra coordinate, producing a $\mathcal M$4-dimensional manifold and $\mathcal M$5 pullback metrics. The paper reports that pairwise MSA distances, visualized by multidimensional scaling, separate trained from untrained models and cluster architectures by type across seeds. It also reports distinct temporal behavior: RNNs maintained high similarity through the delay, whereas structured state-space models showed different temporal geometric changes (Gajic et al., 30 Mar 2026).

The third application studies StableDiffusionXL through the geometry of a $\mathcal M$6-dimensional manifold of text embeddings created by bilinear interpolation. In this setting the paper compares Fisher–Rao geometries over diffusion time and across guidance strengths. It reports that the statistical manifold geometry changes during denoising, that MDS of pairwise MSA distances visualizes a trajectory of information geometry through the diffusion process, and that beyond a certain guidance strength the manifold becomes more similar to the guidance-free model, especially near $\mathcal M$7. This extends MSA from deterministic representation analysis to generative-model information geometry (Gajic et al., 30 Mar 2026).

6. Relation to broader metric and similarity learning

MSA is not a metric-learning method in the standard supervised sense. In "Fast Metric Learning For Deep Neural Networks" the learned object is an embedding space constructed from pairwise similarity constraints, followed by a regression model from raw features into that target space; depending on the loss, similarity is encoded by Euclidean distance or dot product (Gouk et al., 2015). By contrast, MSA assumes that representations already exist and compares the pullback metrics they induce on a shared manifold. This suggests that MSA is best understood as a method for analyzing learned geometries rather than for fitting a task-specific similarity function.

The distinction is also visible at the level of theory. "Generalization Bounds for Metric and Similarity Learning" studies pairwise hinge-risk minimization with matrix-norm regularization and derives bounds through Rademacher averages over pairwise sample blocks (Cao et al., 2012). That line of work addresses the learnability of pairwise scores such as Mahalanobis-type distances or bilinear similarities. MSA instead compares metric tensors $\mathcal M$8 or Fisher information matrices $\mathcal M$9, so its central object is a field of local geometric operators rather than a supervised pairwise predictor. This suggests an adjacent but distinct theoretical domain.

A broader geometric context is provided by work on transforming similarities into valid metric distances. "Metric distances derived from cosine similarity and Pearson and Spearman correlations" shows that angular geometry and metric-preserving transforms provide principled routes from cosine or correlation values to true metric distances (Dongen et al., 2012). MSA belongs to the same broad effort to make similarity comparison geometrically well-posed, but it does so by comparing SPD metric tensors through a spectral-ratio pseudo-distance rather than by transforming pairwise scalar similarities.

7. Limitations, scope, and terminology

The paper is explicit that MSA has practical and conceptual limits. It requires a characterization of the underlying manifold; when the manifold is unknown, this may be difficult, and the method is less naturally suited to sparse-data settings. It compares the geometry of a hidden layer or statistical manifold, not how later layers exploit that geometry. The paper also notes that if a decoder is rank-deficient or ignores some subspace, MSA may register geometric differences that are functionally irrelevant downstream. More generally, similarity measures, including MSA, are interpretive rather than causal: they can reveal structural distinctions, but they do not by themselves explain why those distinctions cause performance differences (Gajic et al., 30 Mar 2026).

The statistical-manifold version introduces an additional limitation: Fisher information estimation can be expensive and noisy. Even in the deterministic setting, MSA depends on exact or numerically stable Jacobian computation. These constraints make it particularly natural for controlled settings in which the manifold variables are known, such as synthetic tasks, explicitly parameterized latent manifolds, or designed interpolation experiments, and less straightforward for unconstrained observational data (Gajic et al., 30 Mar 2026).

A final terminological point is that the acronym MSA is overloaded. In adjacent literatures it also denotes multiple sequence alignment, as in work on short-sequence alignment and covariation analysis in protein MSAs (Takács, 2015, Clark et al., 2014). This suggests that explicit expansion to Metric Similarity Analysis is useful whenever the term appears in cross-disciplinary contexts. Within its own domain, however, the term now names a specific framework: comparison of neural systems through the intrinsic geometry they induce on a shared manifold, using pullback Riemannian or Fisher–Rao metrics and the spectral-ratio pseudo-distance (Gajic et al., 30 Mar 2026).