Information Theory for Complex Systems Scientists (2304.12482v3)

Abstract: In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modelling, and ultimately controlling complex systems: systems comprised of a large number of non-trivially interacting components whose collective behaviour can be difficult to predict. Information theory, a branch of mathematics historically associated with questions about encoding and decoding messages, has emerged as something of a lingua franca for those studying complex systems, far exceeding its original narrow domain of communication systems engineering. In the context of complexity science, information theory provides a set of tools which allow researchers to uncover the statistical and effective dependencies between interacting components; relationships between systems and their environment; mereological whole-part relationships; and is sensitive to non-linearities missed by commonly parametric statistical models. In this review, we aim to provide an accessible introduction to the core of modern information theory, aimed specifically at aspiring (and established) complex systems scientists. This includes standard measures, such as Shannon entropy, relative entropy, and mutual information, before building to more advanced topics, including: information dynamics, measures of statistical complexity, information decomposition, and effective network inference. In addition to detailing the formal definitions, in this review we make an effort to discuss how information theory can be interpreted and develop the intuition behind abstract concepts like "entropy," in the hope that this will enable interested readers to understand what information is, and how it is used, at a more fundamental level.

• The paper defines key information-theoretic measures such as entropy, mutual information, and Kullback-Leibler divergence to analyze interdependencies in complex systems.
• The paper details information dynamics by explaining how systems store, transfer, and modify information to generate emergent behaviors.
• The paper explores network inference and novel complexity metrics, enabling a granular evaluation of integration, segregation, and overall system performance.

Information Theory for Complex Systems Scientists: What, Why, How?

The paper by Thomas F. Varley provides an extensive review of modern information theory with a focus on complex systems. It situates information theory as a versatile toolkit for modeling, understanding, and controlling systems with numerous interacting parts. This review aims at both introducing the fundamental concepts of information theory to complex systems scientists and providing advanced methodologies relevant to the field.

Core Contributions

The paper begins by defining complex systems, highlighting their interconnectedness, non-linearity, multiscale dynamics, and time dependencies. It notes the inadequacy of classical statistical models in capturing the full complexity of these systems, thereby advocating for information theory as a more suitable approach.

Varley outlines several key measures of information theory and their applicability in complex systems research:

1. Entropy: Serves as a measure of uncertainty within a system.
2. Mutual Information (MI): Quantifies dependency between variables, highlighting how much information one variable reveals about another.
3. Conditional and Joint Entropies: Offer insights into how system components influence each other.
4. Kullback-Leibler Divergence: Measures the difference between two probability distributions, aiding in iterative inference.

These measures provide researchers the means to analyze dependencies, redundancies, and higher-order interactions within complex systems.

Information Dynamics

A significant section is devoted to information dynamics that describe how information is stored, transferred, and modified within a system. It discusses:

• Information Storage: Captures how past states influence future states.
• Information Transfer: Captures directional influence between system elements.
• Information Modification: Involves interactions among different sources leading to emergent properties.

These dynamics are crucial for understanding the computational processes within complex systems.

Partial Information Decomposition (PID)

The paper introduces advanced concepts like PID, which decomposes the mutual information into unique, redundant, and synergistic components. This decomposition allows for a more granular analysis of information flow in multi-element systems and aids in modeling interactions beyond pairwise dependencies.

Network Inference

Varley explores the construction of network models from empirical data. Functional and effective connectivity networks are discussed, each providing different insights into the interdependencies and directional flows within systems. The limitations of these approaches in capturing synergies are addressed, and alternative models like hypergraphs and simplicial complexes are suggested.

Complexity and Measures

The TSE complexity, O-information, and S-information are presented as metrics to quantify the balance of integration and segregation within a system. These metrics help evaluate the overall complexity of a system and identify its potential for processing and exchanging information effectively.

Practical Implications and Future Directions

The paper posits that information theory, with its model-free nature, can reveal insights into complex systems that traditional methods may overlook. This potential, however, comes with challenges such as data demands and computational complexity. It encourages the continued development of information-theoretic tools and their application across various scientific domains.

Looking forward, the integration of information theory with causal inference methods may offer deeper insights into system dynamics, enabling more precise modeling and control of complex systems.

Conclusion

Varley's paper stands as a comprehensive guide for researchers seeking to leverage information theory in the paper of complex systems. By encompassing both foundational concepts and cutting-edge methodologies, it serves as a valuable resource for scientists aiming to tackle the intricate dynamics of modern scientific challenges.