---
title: Metric Similarity Analysis (MSA)
url: https://www.emergentmind.com/topics/metric-similarity-analysis-msa
type: topic
---

# Metric Similarity Analysis (MSA)

Metric Similarity Analysis (MSA) is a geometry-aware similarity measure for neural representations that compares **intrinsic geometry** rather than the usual **extrinsic geometry** of activations in state space. Under the manifold hypothesis, it is defined as a distance between the **pullback Riemannian metrics** induced by two neural representations on a common input manifold, and it was introduced as a framework for comparing how neural systems warp the same underlying manifold across static representations, nonlinear dynamics, and statistical manifolds [2603.28764].

## 1. Conceptual basis

MSA is motivated by the claim that many standard representation-similarity methods compare the geometry of activations as embedded point clouds in ambient state space, whereas mechanistically relevant differences may lie in how a network transforms the underlying input manifold itself. The paper’s motivating contrast is the difference between a flat \(2\)-dimensional sheet and a Swiss roll: in activation space these embeddings look very different, so extrinsic methods can report low similarity, even though the manifolds have identical intrinsic distance/angle structure. MSA therefore asks whether two systems endow the same latent variables with the same local geometry, rather than whether their activation clouds align in ambient coordinates [2603.28764].

This formulation depends on the manifold hypothesis. The input data are assumed to lie on an intrinsically low-dimensional manifold \(\mathcal M\), and the representation map is assumed smooth enough to define Jacobians and pullback metrics. In the core Riemannian setting, the pushforward is assumed injective locally so that the pullback metric is symmetric positive definite. A common misconception is that MSA is simply another activation-space similarity score. In fact, the compared object is the metric structure induced on the underlying manifold, not the raw activation configuration in \(\mathbb{R}^n\) [2603.28764].

## 2. Mathematical formulation on representation manifolds

The paper models a neural system as
\[
\mathcal{M} \xlongrightarrow{\psi} \mathbb{R}^{n_{\text{in}}} \xlongrightarrow{\varphi} \mathbb{R}^{n} \xlongrightarrow{\zeta} \mathbb{R}^{n_{\text{out}}},
\]
where \(\mathcal M\) is an \(m\)-dimensional input manifold, \(\psi\) embeds \(\mathcal M\) into input space, \(\varphi\) maps inputs to the hidden representation of interest, and \(\zeta\) maps onward to outputs. The representation under study is the image \((\varphi\circ\psi)(\mathcal M)\subset \mathbb R^n\), but MSA compares the induced metric on \(\mathcal M\), not that image directly [2603.28764].

Let \(J(p)\) be the Jacobian of \(\varphi\circ\psi\) at \(p\in\mathcal M\). For tangent vectors \(\mathbf v_i,\mathbf v_j\in T_p\mathcal M\),
\[
(J(p)\mathbf v_i)\cdot (J(p)\mathbf v_j)=\mathbf v_i^\top J(p)^\top J(p)\mathbf v_j.
\]
The local metric tensor is therefore
\[
G(p)=J(p)^\top J(p)\in\mathbb{R}^{m\times m}.
\]
This pullback metric encodes local lengths, angles, anisotropy, and more generally local distance structure on the manifold as represented by the network. The paper explicitly notes that MSA does **not** directly compare curvature tensors, geodesic distances, or higher-order invariants; its operative object is the pointwise pullback metric \(G(p)\), making the method a first-order intrinsic comparison [2603.28764].

The global MSA distance between two representations \(\varphi^1\) and \(\varphi^2\) is
\[
d_{\mathrm{MSA}}(\varphi^{1}, \varphi^{2}) =
\frac{1}{\mathrm{Vol}_g(\mathcal{M})}\int_{\mathcal{M}}
d_\mathrm{SR}(G_p^{\varphi^1}, G_p^{\varphi^2}) \,\mathrm{dvol}_g(p),
\]
with associated bounded similarity
\[
1-d_\mathrm{MSA}(\varphi^1,\varphi^2)\in[0,1].
\]
Thus MSA compares two **metric fields** on the same manifold and averages the local discrepancy over \(\mathcal M\) [2603.28764].

## 3. Spectral ratio and invariance structure

The local discrepancy \(d_{\mathrm{SR}}\) is defined on symmetric positive definite matrices by a generalized eigenvalue problem. Given
\[
G\mathbf v_i=\lambda_i G'\mathbf v_i,
\]
with eigenvalues ordered so that \(\lambda_{i+1}\le \lambda_i\), the **spectral ratio** is
\[
d_\mathrm{SR}(G,G')=1-\sqrt{\frac{\lambda_m}{\lambda_1}}.
\]
The paper states that \(d_\mathrm{SR}\in[0,1]\) and proves that it is a **pseudo-distance** on SPD matrices, satisfying separation, symmetry, and triangle inequality [2603.28764].

A crucial consequence of the spectral-ratio design is that MSA compares **shape/anisotropy** rather than absolute overall scale. If two local metrics differ only by a scalar factor, then all generalized eigenvalues are equal and \(d_\mathrm{SR}=0\). This means that MSA intentionally ignores uniform local scaling when local shape is preserved. The paper explicitly notes this as a subtle but important design choice [2603.28764].

MSA is also characterized by strong invariance properties. It is invariant to independent orthogonal transformations of the two activation spaces:
\[
d_\mathrm{MSA}(Q_1\circ\varphi^1,Q_2\circ\varphi^2)=d_\mathrm{MSA}(\varphi^1,\varphi^2),
\]
because orthogonal transformations leave \(J^\top J\) unchanged. It is also invariant under smooth changes of local coordinates on the manifold, since coordinate changes transform both metric tensors by the same congruence and preserve the generalized eigenvalues. The appendix further notes invariance to state-space transformations that are pointwise Euclidean isometries [2603.28764].

## 4. Computational procedure and statistical-manifold extension

The deterministic MSA pipeline is Jacobian-based rather than neighborhood-based. One identifies the latent manifold \(\mathcal M\), parameterizes it, chooses the representation layer or system state to compare, computes Jacobians \(J(p)\), forms pullback metrics \(G(p)=J(p)^\top J(p)\), evaluates \(d_\mathrm{SR}\) pointwise, and then integrates over \(\mathcal M\). The paper emphasizes that this avoids neighborhood graphs, tangent PCA, or point-cloud tangent estimation; under the assumed manifold parameterization, the comparison is exact at the level of the pullback metric field [2603.28764].

The same framework extends to statistical manifolds. If \(\boldsymbol\pi\in\mathcal M\) parameterizes a density \(p_{\boldsymbol\pi}(\mathbf x)\), then averaging outer products of score vectors yields the Fisher information matrix
\[
F(\boldsymbol{\pi})=
\mathbb E_{\mathbf x\sim p_{\boldsymbol\pi}}
\left[
\nabla_{\boldsymbol\pi}\log p_{\boldsymbol\pi}(\mathbf x)\otimes
\nabla_{\boldsymbol\pi}\log p_{\boldsymbol\pi}(\mathbf x)
\right].
\]
This \(F(\boldsymbol\pi)\) is the local coordinate form of the Fisher–Rao metric, and MSA compares such Fisher metrics exactly as it compares pullback metrics in deterministic settings. The paper uses this extension for diffusion models, where the relevant geometry is information-geometric rather than purely representation-geometric [2603.28764].

Practically, the paper does not give a formal complexity analysis, but it states that Jacobian computation dominates for high-dimensional networks, whereas the generalized eigenproblem is cheap when manifold dimension is low. This suggests that MSA is most natural in the low-dimensional manifold regime posited by the manifold hypothesis. The paper also notes that if the manifold is not explicitly known, one may need to learn it first with a parametric model, provided data are sampled densely enough [2603.28764].

## 5. Empirical applications

The first application concerns one-hidden-layer networks trained in **rich** and **lazy** regimes on a \(2\)-dimensional manifold \(\mathcal M=[-1,1]^2\). Here the paper reports a central qualitative result: PCA of activations suggested rich and lazy representations looked similar extrinsically, and Procrustes reported near-perfect similarity, but MSA reported **low similarity** between rich and lazy networks, while giving higher similarity for same-regime different-seed comparisons. In a hierarchy varying regime, task, and seed, MSA recovered the intended hierarchy of representational difference [2603.28764].

The second application compares nonlinear dynamics in vanilla RNNs and structured state-space models on a working-memory task with input manifold \(\mathcal M=\mathbb T^2\). Time is added as an extra coordinate, producing a \(3\)-dimensional manifold and \(3\times 3\) pullback metrics. The paper reports that pairwise MSA distances, visualized by multidimensional scaling, separate trained from untrained models and cluster architectures by type across seeds. It also reports distinct temporal behavior: RNNs maintained high similarity through the delay, whereas structured state-space models showed different temporal geometric changes [2603.28764].

The third application studies StableDiffusionXL through the geometry of a \(2\)-dimensional manifold of text embeddings created by bilinear interpolation. In this setting the paper compares Fisher–Rao geometries over diffusion time and across guidance strengths. It reports that the statistical manifold geometry changes during denoising, that MDS of pairwise MSA distances visualizes a trajectory of information geometry through the diffusion process, and that beyond a certain guidance strength the manifold becomes more similar to the guidance-free model, especially near \(t\approx 0\). This extends MSA from deterministic representation analysis to generative-model information geometry [2603.28764].

## 6. Relation to broader metric and similarity learning

MSA is not a metric-learning method in the standard supervised sense. In "Fast Metric Learning For Deep Neural Networks" the learned object is an embedding space constructed from pairwise similarity constraints, followed by a regression model from raw features into that target space; depending on the loss, similarity is encoded by Euclidean distance or dot product [1511.06442]. By contrast, MSA assumes that representations already exist and compares the **pullback metrics they induce** on a shared manifold. This suggests that MSA is best understood as a method for analyzing learned geometries rather than for fitting a task-specific similarity function.

The distinction is also visible at the level of theory. "Generalization Bounds for Metric and Similarity Learning" studies pairwise hinge-risk minimization with matrix-norm regularization and derives bounds through Rademacher averages over pairwise sample blocks [1207.5437]. That line of work addresses the learnability of pairwise scores such as Mahalanobis-type distances or bilinear similarities. MSA instead compares metric tensors \(G(p)=J(p)^\top J(p)\) or Fisher information matrices \(F(\boldsymbol\pi)\), so its central object is a field of local geometric operators rather than a supervised pairwise predictor. This suggests an adjacent but distinct theoretical domain.

A broader geometric context is provided by work on transforming similarities into valid metric distances. "Metric distances derived from cosine similarity and Pearson and Spearman correlations" shows that angular geometry and metric-preserving transforms provide principled routes from cosine or correlation values to true metric distances [1208.3145]. MSA belongs to the same broad effort to make similarity comparison geometrically well-posed, but it does so by comparing SPD metric tensors through a spectral-ratio pseudo-distance rather than by transforming pairwise scalar similarities.

## 7. Limitations, scope, and terminology

The paper is explicit that MSA has practical and conceptual limits. It requires a characterization of the underlying manifold; when the manifold is unknown, this may be difficult, and the method is less naturally suited to sparse-data settings. It compares the geometry of a hidden layer or statistical manifold, not how later layers exploit that geometry. The paper also notes that if a decoder is rank-deficient or ignores some subspace, MSA may register geometric differences that are functionally irrelevant downstream. More generally, similarity measures, including MSA, are interpretive rather than causal: they can reveal structural distinctions, but they do not by themselves explain why those distinctions cause performance differences [2603.28764].

The statistical-manifold version introduces an additional limitation: Fisher information estimation can be expensive and noisy. Even in the deterministic setting, MSA depends on exact or numerically stable Jacobian computation. These constraints make it particularly natural for controlled settings in which the manifold variables are known, such as synthetic tasks, explicitly parameterized latent manifolds, or designed interpolation experiments, and less straightforward for unconstrained observational data [2603.28764].

A final terminological point is that the acronym **MSA** is overloaded. In adjacent literatures it also denotes **multiple sequence alignment**, as in work on short-sequence alignment and covariation analysis in protein MSAs [1510.09037, 1404.6684]. This suggests that explicit expansion to **Metric Similarity Analysis** is useful whenever the term appears in cross-disciplinary contexts. Within its own domain, however, the term now names a specific framework: comparison of neural systems through the intrinsic geometry they induce on a shared manifold, using pullback Riemannian or Fisher–Rao metrics and the spectral-ratio pseudo-distance [2603.28764].

Source: https://www.emergentmind.com/topics/metric-similarity-analysis-msa