A Complexity-Based Theory of Compositionality (2410.14817v5)

Abstract: Compositionality is believed to be fundamental to intelligence. In humans, it underlies the structure of thought, language, and higher-level reasoning. In AI, compositional representations can enable a powerful form of out-of-distribution generalization, in which a model systematically adapts to novel combinations of known concepts. However, while we have strong intuitions about what compositionality is, we lack satisfying formal definitions for it that are measurable and mathematical. Here, we propose such a definition, which we call representational compositionality, that accounts for and extends our intuitions about compositionality. The definition is conceptually simple, quantitative, grounded in algorithmic information theory, and applicable to any representation. Intuitively, representational compositionality states that a compositional representation satisfies three properties. First, it must be expressive. Second, it must be possible to re-describe the representation as a function of discrete symbolic sequences with re-combinable parts, analogous to sentences in natural language. Third, the function that relates these symbolic sequences to the representation, analogous to semantics in natural language, must be simple. Through experiments on both synthetic and real world data, we validate our definition of compositionality and show how it unifies disparate intuitions from across the literature in both AI and cognitive science. We also show that representational compositionality, while theoretically intractable, can be readily estimated using standard deep learning tools. We hope that our definition can inspire the design of novel, theoretically-driven models that better capture the mechanisms of compositional thought. We make our code available at https://github.com/EricElmoznino/complexity_compositionality.

• The paper's main contribution is a novel, quantitative definition of compositionality based on algorithmic information theory.
• It employs discrete auto-encoders with learned priors to validate the approach on synthetic context-free grammar data and natural language scenarios.
• The findings imply that simple semantic mappings and modular functions boost compositionality, paving the way for improved neurosymbolic systems.

A Complexity-Based Theory of Compositionality

The paper "A Complexity-Based Theory of Compositionality" proposes a novel and quantitative definition of compositionality, termed representational compositionality. The authors postulate that compositionality is essential to both human cognition and artificial intelligence, facilitating advanced out-of-distribution generalization. Despite its foundational status in intelligence theories, a formal, measurable definition has been elusive. This work seeks to provide such a definition within the framework of algorithmic information theory.

Definition of Representational Compositionality

The authors introduce a definition based on the concept of representational compositionality, describing it as a measure of how well a representation can be described as a simple function of a set of discrete symbolic parts. This is grounded in the principles of algorithmic information theory, specifically Kolmogorov complexity. The proposed definition involves a representation that is (1) expressive, (2) describable as discrete symbolic sequences, and (3) simple in the function that maps these sequences to representations.

Formal and Empirical Validation

The paper's authors derive a mathematical expression for representational compositionality that balances the expressivity of a representation against its compressibility using simple compositional rules. They further substantiate their definition through empirical experiments on both synthetic and real datasets, establishing its consistency with intuitive understandings of compositionality from both AI and cognitive science.

Implementation and Results

The paper discusses a methodology for estimating compositionality using discrete auto-encoders with learned priors. By training these models, the research demonstrates the estimation of representational compositionality across various datasets, including synthetic ones generated using context-free grammars and real-world scenarios such as emergent languages and natural languages. Notably, they reveal that higher compositionality arises from using simple semantic mappings and modular functions, aligning with linguistic insights.

Implications and Future Directions

The definition presented offers significant implications for both theoretical and practical applications. Theoretically, this framework provides a robust tool for evaluating the compositionality of neural and cognitive representations. Practically, it suggests directions for designing models with enhanced compositionality through principled approaches. The concept has potential applications in improving neurosymbolic systems and developing better algorithms for tokenization in LLMs.

The paper invites future research to explore further optimization methods for estimating compositionality and to test its application across diverse AI architectures and learning paradigms. By doing so, it holds promise for elucidating the underpinnings of compositional generalization mechanisms in both artificial and natural systems.

Conclusion

The authors present a comprehensive and formalized definition of compositionality that addresses previous conceptual ambiguities. By embedding this definition in the framework of algorithmic information theory, the paper establishes a solid foundation for future research in compositional representations, potentially leading to advancements in AI and cognitive science applications.