Symmetric Divergence and Normalized Similarity: A Unified Topological Framework for Representation Analysis

Abstract: Topological Data Analysis (TDA) offers a principled, intrinsic lens for comparing neural representations. However, existing paired topological divergences (e.g., RTD) are limited by heuristic asymmetry and, more critically, unbounded scores that depend on sample size, hindering reliable cross-scenario benchmarking. To address these challenges, we develop a unified topological toolkit serving two complementary needs: fine-grained structural diagnosis and robust, standardized evaluation. First, we complete the RTD framework by introducing Symmetric Representation Topology Divergence (SRTD) and its efficient variant SRTD-lite. Beyond resolving the theoretical asymmetry of prior variants, SRTD consolidates diagnostic information into a single, comprehensive cross-barcode signature. This allows for precise localization of structural discrepancies and serves as an effective optimization objective without the overhead of dual directional computations. Second, to enable reliable benchmarking across heterogeneous settings, we propose Normalized Topological Similarity (NTS). By measuring the rank correlation of hierarchical merge orders, NTS yields a scale-invariant metric bounded between -1 and 1, effectively overcoming the scale and sample-dependence of unnormalized divergences. Experiments across synthetic and real-world deep learning settings demonstrate that our toolkit captures functional shifts in CNNs missed by geometric measures and robustly maps LLM genealogy even under distance saturation, offering a rigorous, topology-aware perspective that complements measures like CKA.

• The paper introduces SRTD as a symmetric topological divergence that resolves asymmetry issues in traditional RTD methods.
• It proposes NTS for scale-invariant, bounded similarity measures, enhancing standard geometric approaches like CKA.
• Extensive evaluations on synthetic data, CNNs, and LLMs validate improved structural diagnostics and computational scalability.

Unified Topological Framework for Representation Analysis

Motivation and Context

The paper addresses a significant gap in neural representation analysis by proposing a unified topological toolkit that enables both fine-grained structural diagnosis and robust, standardized evaluation. Existing geometric methods, such as CCA and CKA, capture similarity through linear or kernel-based subspace alignments but fail to probe the intrinsic structural organization of data. Topological Data Analysis (TDA), particularly persistent homology, is leveraged to capture higher-order connectivity and shape features of representations. Prior divergence-style topological measures (e.g., RTD and variants) suffer from theoretical asymmetry and unbounded, sample-dependent scores, limiting interpretability and cross-scenario benchmarking. The authors introduce Symmetric Representation Topology Divergence (SRTD) and Normalized Topological Similarity (NTS) to resolve these deficiencies.

Theoretical Contributions

Completion and Generalization of RTD

RTD quantifies topological divergence by comparing barcodes derived from persistent homology on pairwise dissimilarity matrices for corresponded samples. Its conventional symmetric variant relies on the arithmetic mean of directional divergences, which can exhibit marked asymmetry, leading to interpretability issues.

SRTD formalizes a symmetric barcode construction based on a specially constructed auxiliary matrix, $M_{\text{sym}}$, linking RTD and Max-RTD via mapping cone constructions and homological algebra. This yields a deep mathematical relationship:

$$\text{Max-RTD}_i(w,\tilde{w}) + \text{RTD}_i(w,\tilde{w}) - \text{SRTD}_i(w,\tilde{w}) = \int_{0}^{\infty} \left( \dim(\ker(\gamma_i)) + \dim(\ker(\gamma_{i-1})) \right) d\alpha$$

SRTD provides a single, comprehensive cross-barcode, offering efficient computation and interpretable localization of structural discrepancies.

SRTD-lite, an MST-based analogue, is introduced for scalable computation on large datasets, maintaining the theoretical relationships and strict hierarchical ordering observed empirically:

Figure 1: The RTD family: theoretical and empirical relationships among RTD, Max-RTD, and SRTD.

Normalized Topological Similarity (NTS)

NTS is proposed to address the need for scale-invariant, bounded similarity scores for benchmarking across layers, models, or datasets. By leveraging Spearman’s rank correlation on hierarchical merge orders (as obtained from MST merge times), NTS is resilient to sample size and scale effects, supporting robust comparison in heterogeneous settings. Two variants are defined: NTS-E (edge distance-based) and NTS-M (merge time-based), both computed on the MST core edges derived from representations.

Key properties include tightness under identical ranking and strictness of NTS-E relative to NTS-M:

• $NTS-M(w, \tilde{w}) = 1$ iff merge-time orders coincide
• $$NTS-E(w, \tilde{w}) = 1 \implies NTS-M(w, \tilde{w}) = 1$$

This provides a principled, normalized topological similarity complementary to geometric methods.

Figure 2: Smooth topological shift across UMAP embeddings with increasing n_neighbors; NTS tracks structural evolution where CKA fails.

Empirical Evaluation

Synthetic Hierarchical Shifts

The "Clusters" experiment demonstrates the sensitivity of SRTD and NTS to increasing structural divergence in a controlled setting, outperforming CKA and rectifying the anomalous behavior of RTD-lite without $\max(w,\tilde{w})$.

Figure 3: Clusters experiment—topological measures capture increasing structural divergence, while CKA remains insensitive.

CNN Layer-Wise Analysis

Layer-wise analysis of TinyCNNs on CIFAR-10 reveals that NTS and CKA exhibit coherent diagonal similarity decay, reflecting hierarchical feature development. Topological measures additionally highlight abrupt transitions at pooling layers, revealing functional shifts not captured by geometric similarity.

Figure 4: CKA and NTS show near-diagonal similarity in CNN layers; only topological measures capture structural breaks at pooling.

LLMs

In intra-model and inter-model comparisons, NTS uncovers family-specific hierarchical fingerprints and unsaturated similarity landscapes, contrasting with CKA's saturation and lack of lineage sensitivity, particularly in distillation scenarios—DeepSeek-R1-Ds vs Qwen2.5-Math-7B.

Figure 5: NTS reveals structured hierarchical consistency in LLM families, while CKA exhibits saturation and inconsistent heatmaps.

Figure 6: Inter-model similarity: NTS-E distinguishes lineage and genealogy more effectively than CKA, especially in saturated regimes.

Practical Implications and Computational Efficiency

The toolkit is efficient. SRTD-lite and NTS-E operate at $O(n^2\alpha_{\text{uf}}(n)+d)$, suitable for large-scale benchmarks and representation diagnostics. NTS-E demands minimal normalization and memory, outperforming RTD-based methods in runtime and scalability.

Figure 7: Runtime comparison validates the superior scalability of NTS-E and SRTD-lite over divergence-based baselines.

Limitations and Future Directions

While SRTD and SRTD-lite offer comprehensive diagnostic information and serve as viable optimization objectives, NTS is inherently non-differentiable and thus restricted to analysis-only settings. Addressing differentiability for NTS to enable topology-aware regularization is a critical future avenue. The focus on 0-dimensional features in lightweight variants precludes finer unit-level topological information, which could be explored by integrating higher-dimensional persistent features.

Conclusion

This work advances the theoretical and practical foundations of representation analysis by unifying divergence and normalized similarity within a topological framework. SRTD addresses the inefficiency and asymmetry of prior methods, while NTS brings bounded, robust benchmarking. Their complementary roles enable scalable diagnostics and genealogy mapping in high-dimensional models, including deep CNNs and LLMs. The methods are poised for broad adoption in interpretability and model auditing, contingent upon future extensions for differentiation and higher-order topological analysis.