The topology of synergy: linking topological and information-theoretic approaches to higher-order interactions in complex systems (2504.10140v1)

Abstract: The study of irreducible higher-order interactions has become a core topic of study in complex systems. Two of the most well-developed frameworks, topological data analysis and multivariate information theory, aim to provide formal tools for identifying higher-order interactions in empirical data. Despite similar aims, however, these two approaches are built on markedly different mathematical foundations and have been developed largely in parallel. In this study, we present a head-to-head comparison of topological data analysis and information-theoretic approaches to describing higher-order interactions in multivariate data; with the aim of assessing the similarities and differences between how the frameworks define ``higher-order structures." We begin with toy examples with known topologies, before turning to naturalistic data: fMRI signals collected from the human brain. We find that intrinsic, higher-order synergistic information is associated with three-dimensional cavities in a point cloud: shapes such as spheres are synergy-dominated. In fMRI data, we find strong correlations between synergistic information and both the number and size of three-dimensional cavities. Furthermore, we find that dimensionality reduction techniques such as PCA preferentially represent higher-order redundancies, and largely fail to preserve both higher-order information and topological structure, suggesting that common manifold-based approaches to studying high-dimensional data are systematically failing to identify important features of the data. These results point towards the possibility of developing a rich theory of higher-order interactions that spans topological and information-theoretic approaches while simultaneously highlighting the profound limitations of more conventional methods.

• The paper demonstrates that information-theoretic synergy is strongly associated with 3D topological voids, revealing intrinsic higher-order interactions in both synthetic and fMRI data.
• It employs multivariate measures such as normalized O-information with KNN estimators alongside Vietoris-Rips persistence to quantify complex dependencies.
• Results suggest that standard analyses like PCA and functional connectivity may overlook crucial synergistic interactions, advocating for integrated TDA and MIT approaches.

This paper explores the relationship between two frameworks for identifying higher-order interactions in complex systems: multivariate information theory (MIT) and topological data analysis (TDA) (2504.10140). It aims to understand how these approaches, developed largely in parallel, conceptualize "higher-order structure," particularly focusing on information synergy. Traditional network analysis often relies on pairwise interactions (like functional connectivity), potentially missing irreducible interactions involving three or more elements.

The paper compares these frameworks using both synthetic data with known topological structures and real-world fMRI data.

Methodologies:

1. Multivariate Information Theory (MIT):
   • Utilizes measures like Total Correlation (TC), Dual Total Correlation (DTC), O-information (O), and S-information (S) to quantify dependencies among multiple variables.
   • The O-information ($O = TC - DTC$) is central, measuring the balance between redundancy ($O>0$, information duplicated across subsets) and synergy ($O<0$, information only present in the whole).
   • Normalized O-information ($\bar{O} = O/S$) is used for better comparison across systems, bounded between [-1, 1].
   • Estimators: For continuous data (like point clouds or fMRI time series), K-Nearest Neighbors (KNN) based estimators are employed. These non-parametric methods estimate entropy and mutual information based on distances to the $k^{th}$ nearest neighbors in the data space, avoiding assumptions like multivariate normality.
     • Specifically, a KNN-based O-information estimator derived from Gomez-Herrero et al. (Miller et al., 2015) is used, designed to minimize bias by performing only one neighbor search in the joint space. This estimator is implemented in the JIDT toolbox.

       \hat{O}(X) = (2-N)\left[F(k) + \sum_i^N\frac{1}{N-2}\left\langle F(k_i(t)) - F(k^{-i}(t))\right\rangle_t \right]

       where $F(k) = \psi(k) - \psi(N)$, $\psi$ is the digamma function, $k_i(t)$ and $k^{-i}(t)$ are counts from range searches based on the $k^{th}$ neighbor distance $\epsilon(t)$ in the joint space.
2. Topological Data Analysis (TDA):
   • Focuses on identifying the shape and structure of data, particularly "voids" or "cavities" in high-dimensional point clouds.
   • The Vietoris-Rips filtration is used: simplicial complexes are built by connecting points within a certain distance ($\epsilon$). As $\epsilon$ increases, topological features (connected components, loops, voids) appear (birth) and disappear (death).
   • Persistence: The lifetime ($\text{death} - \text{birth}$) of these features quantifies their significance. The paper focuses on 3D voids (cavities).
   • Metrics: Average persistence lifetime and the total number of unique 3D voids are calculated using the Ripser package. Chebyshev distance is used for consistency with MIT estimators.
3. Data & Analysis:
   • Synthetic Data: Point clouds (10,000 points) sampled from simple 3D manifolds: line, plane, hollow sphere, solid ball, hollow torus, solid torus, trefoil knot, (5,3)-knot. MIT and TDA measures were calculated.
   • fMRI Data: Resting-state fMRI time series from the Human Connectome Project (HCP) for one subject (4 concatenated scans, 4400 frames total, 200 brain regions). Triads of regional time series were treated as 3D point clouds (see Figure 1). O-information, TC, DTC, and TDA measures were calculated for significantly redundant or synergistic triads (determined via permutation testing using circular shifts).
   • Intrinsic vs. Contextual Information: Principal Component Analysis (PCA) was applied to rotate point clouds. Higher-order information persisting after PCA was termed "intrinsic," while information dependent on the specific embedding orientation was termed "contextual."

Key Findings:

1. Synthetic Manifolds:
   • Simple low-dimensional structures (line, plane) embedded in 3D exhibit contextual redundancy or synergy that vanishes after PCA aligns them with principal axes.
   • Structures with 3D cavities (hollow sphere, hollow torus) exhibit intrinsic synergy, meaning $O<0$ even after PCA rotation. Solid versions (ball, filled torus) lack this intrinsic synergy. Spheres, being rotationally symmetric, show perfectly intrinsic synergy.
   • Knots (locally 1D curves irreducibly embedded in 3D) exhibit intrinsic redundancy ($O>0$). This suggests redundancy arises from strong local constraints within a globally complex structure.
   • Main Insight: Information-theoretic synergy appears strongly linked to the presence of 3D topological cavities (voids) in the data manifold. Intrinsic redundancy is linked to low-dimensional structures forced into higher dimensions (like knots).
2. fMRI Data:
   • A strong negative correlation exists between normalized O-information ($\bar{O}$) and both the number and average persistence of 3D TDA voids. More synergy (more negative $\bar{O}$) corresponds to more numerous and larger/longer-lived voids. This relationship holds, albeit weaker, after PCA rotation, indicating a link between intrinsic synergy and topological voids.
   • The relationship between TC/DTC and TDA features is complex: negative correlation before PCA, but positive after PCA. This highlights the difference between contextual and intrinsic dependencies.
   • PCA-based dimensionality reduction preferentially captures redundancy. The variance explained by the first PC correlates positively with $\bar{O}$ (redundant triads are more compressible) and negatively with TDA void features (topologically complex triads are less compressible).
   • Main Insight: Synergistic interactions in the brain are associated with topologically complex (cavity-rich) data structures. Standard methods like PCA and likely functional connectivity (which primarily reflects redundancy) may systematically miss these synergistic and topological features, overlooking a potential "shadow structure" of brain organization.

Discussion & Conclusion:

• The paper provides strong correlational evidence linking information-theoretic synergy (especially intrinsic synergy) with 3D topological voids identified by TDA. This suggests these two distinct mathematical frameworks capture related aspects of higher-order structure.
• The concept of intrinsic vs. contextual higher-order information, distinguished using PCA, is introduced as a crucial consideration for interpreting results.
• The finding that knots are intrinsically redundant offers a new perspective on redundancy arising from geometric constraints.
• A key practical implication is that common analysis techniques like PCA and manifold learning, as well as standard functional connectivity, appear biased towards redundant information (like synchrony) and may fail to capture the brain's synergistic interactions and topological complexity.
• Limitations include the correlational nature of the findings (no formal proofs) and the computational cost of KNN/TDA methods, which restricted the fMRI analysis sample size per triad.

Practical Implementation Considerations:

• To capture higher-order interactions beyond pairwise dependencies, researchers should consider using both MIT measures (like O-information with KNN estimators for continuous data) and TDA methods (like Vietoris-Rips persistence homology).
• When using dimensionality reduction (e.g., PCA) or functional connectivity, be aware of the potential bias towards redundant information and the possibility of missing synergistic effects.
• Use PCA rotation to probe whether observed higher-order information is intrinsic to the data's structure or dependent on its specific embedding.
• The provided KNN O-information estimator offers a less biased way to compute this measure from continuous data compared to naive combinations of estimators.
• Software packages like JIDT (for information theory) and Ripser (for TDA) can facilitate these analyses.
• Be mindful of the computational demands, especially for large datasets or high-dimensional TDA. Downsampling (as done for fMRI TDA) might be necessary.