Decoding Perspective on Similarity Measures

• Decoding perspective on similarity measures is a framework that reinterprets similarity as the alignment between optimal linear decoders, unifying geometric invariances with functional readouts.
• The approach highlights the sensitivity of traditional measures like cosine and Pearson to outliers and non-normal distributions, advocating rank-based metrics to better capture decodability.
• It offers actionable guidelines for selecting similarity metrics by calibrating task-specific decoding performance against different variance regimes in neural, linguistic, and network applications.

A decoding perspective on similarity measures recasts the problem of comparing mathematical, neural, or linguistic objects in terms of reconstructing or aligning decodable information. This approach seeks to characterize not only the geometric or algebraic invariances underlying similarity measures but also their implications for the recoverable information, statistical robustness, and downstream applications. By focusing on the mapping between representations, regularization, and the structure of decoding tasks, this perspective exposes both the strengths and potential failures of traditional similarity indices, motivates the adoption of more robust or task-aligned alternatives, and provides a principled diagnostic toolkit for scientific and engineering applications.

1. Formal Equivalences and Statistical Foundations

Several ubiquitous vector-based similarity measures, including cosine similarity and Pearson correlation, are formally equivalent under mean centering. For word or sentence embeddings $u,v\in\mathbb{R}^D$: $\cos(u,v) = \frac{u\cdot v}{\|u\|\|v\|}\,,$

$$\rho_{\mathrm{Pearson}}(u,v) = \frac{\sum_{i=1}^D (u_i-\bar{u})(v_i-\bar{v})}{\sqrt{\sum_{i=1}^D (u_i-\bar{u})^2} \sqrt{\sum_{i=1}^D (v_i-\bar{v})^2}}\,.$$

Mean-centering transforms cosine similarity into Pearson's $r$. Empirical studies confirm that for standard embeddings (GloVe, word2vec, fastText), the difference between cosine and Pearson is negligible ($\bar u\approx0$), justifying the widespread use of cosine as a default (Zhelezniak et al., 2019).

However, Pearson correlation (hence cosine) is maximally sensitive to outliers and assumes approximate normality of marginals. Non-normal or heavy-tailed dimensions can cause large distortions: a single extreme coordinate or a small fraction of coordinates can dominate, invalidating the implicit linear-association assumption.

Alternative, rank-based correlation coefficients (Spearman’s $\rho$, Kendall’s $\tau$) sidestep this sensitivity by depending only on the order of coordinates: $$\rho_\mathrm{Spearman}(u,v) = \rho_{\mathrm{Pearson}}(R(u), R(v))$$

$$\tau(u,v) = \frac{2}{D(D-1)} \sum_{i<j} \operatorname{sgn}(u_i-u_j)\,\operatorname{sgn}(v_i-v_j)$$

Because ranks are uniformly bounded, a small number of outliers changes only a few ranks, not the overall similarity. Empirically, rank-based measures can improve correlation with human judgments by up to 10 points on word-level tasks and up to 15 points on sentence-level tasks for embeddings with strongly non-Gaussian marginals (e.g., GloVe) (Zhelezniak et al., 2019).

2. Decoding Alignment and Representation Comparisons

A core insight is that many similarity measures can be recast as quantifying the alignment between optimal linear decoders trained on two systems. This re-interpretation unifies geometric invariances with functional alignment. Let $X,Y$ be data matrices encoding neural populations (or model layers) to $M$ items. For (possibly regularized) linear decoders

$$w^* = (1/M) G(X)^{-1} X^T z,\qquad v^* = (1/M) G(Y)^{-1} Y^T z\,,$$

the average alignment across decoding tasks $z$ is

$\langle X w^*, Y v^* \rangle = z^T K_X K_Y z$

with $K_X = (1/M) X G(X)^{-1} X^T$. Centered Kernel Alignment (CKA) and canonical correlation analysis (CCA) arise as normalized traces over such alignments, while the Procrustes distance provides sharp upper and lower bounds (controlled by the participation ratio of $K_X-K_Y$), forming a tight link between similarity geometry and decodability (Harvey et al., 12 Nov 2024).

Key implications:

• CKA, CCA, and Procrustes quantify how well information can be linearly extracted from paired representations.
• The unification explains the invariance properties: CKA is invariant to isotropic scaling and rotation, CCA to arbitrary invertible affine transforms, Procrustes to orthogonal equivalence (Harvey et al., 12 Nov 2024).
• The participation ratio quantifies how closely average decoding similarity predicts Procrustes (shape) similarity.

3. Principal Component Sensitivity and Metric Selection

Similarity measures differ fundamentally in the variance regime they prioritize:

• CKA and ordinary $R^2$ regression disproportionately emphasize high-variance principal components. Under perturbations that zero out a low-variance PC $k$,

$$\text{CKA}(X,\tilde X_k) \approx \frac{\sum_{i\neq k} \lambda_i^2}{\sum_i \lambda_i^2}$$

where $\lambda_i$ are variances. Low-variance PCs must be accurately captured for CKA to approach unity; otherwise, CKA remains near 1 even with catastrophic low-mode failures (Cloos et al., 9 Jul 2024).

• Procrustes-derived angular similarity (APS) or normalized Bures similarity (NBS) respond more sensitively to errors in mid-to-low-variance dimensions. APS, in particular, shows earlier capture of such dimensions when maximizing similarity via gradient-based optimization.
• No universal “good” threshold exists: for the same decoding task, the CKA, APS, and $R^2$ scores marking acceptable solution can vary by dataset and metric. Even high CKA/R^2 is not sufficient to guarantee task-relevant decodability (Cloos et al., 9 Jul 2024).

Best practices:

• Choose a similarity metric matched to the critical variance regime for downstream tasks.
• Calibrate “good” similarity by evaluating task-relevant decoding accuracy, not arbitrary metric cutoffs.
• Report and interpret multiple complementary measures to fully constrain representation similarity (Cloos et al., 9 Jul 2024).

4. Robustness, Context-Dependence, and Metric Failures

Decoding-oriented diagnostics make explicit when standard similarity measures fail:

• For embeddings or representations with non-normal or heavy-tailed marginal or joint statistics, Pearson/cosine similarity can be arbitrarily reduced by outliers, misleadingly understating true association (Zhelezniak et al., 2019).
• In semantic similarity, edge-counting or path-based structural measures (e.g., Jaccard, Wu–Palmer) are fast but insensitive to concept specificity or depth, while information-content (IC) and hybrid measures better align with human judgments, especially in deep or complex ontologies (Slimani, 2013).
• In social networks, classic symmetric measures (Jaccard, Adamic–Adar) ignore edge directionality and multi-step neighborhood density. The Neighborhood Density-based Edge Similarity (NDES) introduces asymmetry and normalization by a node-specific maximum, strongly improving accuracy and modularity in community detection tasks (Das et al., 2022).

Context-specific adaptations can include:

• Adopt rank-based measures (Spearman, Kendall) where distributional assumptions break (Zhelezniak et al., 2019).
• Jointly optimize multiple similarity objectives to map out attainable similarity regions and understand measure constraints (Cloos et al., 9 Jul 2024).
• Task-prioritize similarity by suitably weighting decoding tasks (e.g., stimulus dimensions of interest) in similarity computation (Harvey et al., 12 Nov 2024).

5. Information-Theoretic and Geometric Generalizations

A decoding perspective also clarifies the transition from strict identity to soft similarity, underlying generalized and robust metrics:

• Kronecker’s delta is successively “relaxed” into min/max-based, sum-based, and product-based similarity indices—recovering Jaccard and the inner product as special cases. These indices extend to vectors, multisets, and real functions, providing unified formulas for both hard and soft similarity, with min/max indices being more robust near-identity and inner products suited for global comparison (Costa, 2021).
• In geometry-based analysis, only ordinal (rank-order) comparisons are needed to infer the embedding dimension and underlying space form (Euclidean, spherical, hyperbolic) of a representation. The ordinal spread and capacity provide tight lower bounds on required embedding dimensions and enable detection of curvature sign from similarity graphs, supporting robust inference under noise and partial data (Tabaghi et al., 2020).
• In neuro-perceptual similarity, the human brain’s comparison of stimuli operates as Riemannian distance computation, with the synaptic Jacobian encoding a position-dependent metric tensor $G(x)$. This model captures context-dependence, triangle-inequality violations, and cross-population variability in similarity judgments, outperforming classical Euclidean or Mahalanobis approaches in practical applications such as perceptually optimized image compression (Rodriguez et al., 2017).

6. Applications and Empirical Outcomes

The decoding perspective yields concrete improvements and insights across diverse domains:

• In semantic textual similarity, mean-centering and rank-based correlation yield large empirical gains, with averaged fastText embeddings and Spearman correlation rivaling or exceeding deep sentence encoder benchmarks (Zhelezniak et al., 2019).
• In Chinese spelling correction, decoding-time aggregation of phonetic and glyph-based character similarity via DISC increases F1 scores and reduces over-correction, outperforming hard confusion-set constraints and eliminating the need for model retraining. The method demonstrates robust precision across multiple datasets (Qiao et al., 17 Dec 2024).
• In online social networks, integrating neighborhood density and directionality via NDES leads to consistently better community detection quality (higher modularity, lower conductance, cut ratio, and expansion) compared to Jaccard or cosine (Das et al., 2022).
• For biological and artificial neural systems, representational similarity measures anchored in decoding alignment enable principled evaluation and cross-model comparison, but require metric selection aligned with the intended inference regime and task variance structure (Harvey et al., 12 Nov 2024, Cloos et al., 9 Jul 2024).

7. Best-Practice Guidelines

A decoding-based analytic framework prescribes the following diagnostic and methodological steps:

• Always assess the statistical properties of coordinates in the target representations (e.g., normality, presence of outliers, empirical mean).
• For non-normal or heavy-tailed distributions, employ rank-based or order-invariant metrics.
• Calibrate similarity scores via task-matched decodability, not external metric cutoffs.
• Cross-validate not only the measure but also its applicability to the variance regime and downstream decoding requirements.
• Where relevant, exploit the tight links between similarity geometry and decoding cost, selecting regularization and normalization consistent with the invariances or robustness required by the application (Zhelezniak et al., 2019, Harvey et al., 12 Nov 2024, Cloos et al., 9 Jul 2024).