Topos and Stacks of Deep Neural Networks (2106.14587v3)

Abstract: Every known artificial deep neural network (DNN) corresponds to an object in a canonical Grothendieck's topos; its learning dynamic corresponds to a flow of morphisms in this topos. Invariance structures in the layers (like CNNs or LSTMs) correspond to Giraud's stacks. This invariance is supposed to be responsible of the generalization property, that is extrapolation from learning data under constraints. The fibers represent pre-semantic categories (Culioli, Thom), over which artificial languages are defined, with internal logics, intuitionist, classical or linear (Girard). Semantic functioning of a network is its ability to express theories in such a language for answering questions in output about input data. Quantities and spaces of semantic information are defined by analogy with the homological interpretation of Shannon's entropy of P.Baudot and D.Bennequin in 2015). They generalize the measures found by Carnap and Bar-Hillel (1952). Amazingly, the above semantical structures are classified by geometric fibrant objects in a closed model category of Quillen, then they give rise to homotopical invariants of DNNs and of their semantic functioning. Intentional type theories (Martin-Loef) organize these objects and fibrations between them. Information contents and exchanges are analyzed by Grothendieck's derivators.

• The paper introduces a topos-theoretic model that recasts DNNs as objects in Grothendieck toposes to capture the network architecture.
• It formalizes learning dynamics and semantic information by interpreting backpropagation as a flow of natural transformations and using homotopical invariants.
• The framework offers a novel route for modular network design and interpretability by integrating type theory with stacks and categorical invariants.

Topos Theoretic and Homotopical Foundations for Deep Neural Networks

The paper "Topos and Stacks of Deep Neural Networks" (2106.14587) systematically develops a categorical and homotopical framework for the semantic analysis of deep neural networks (DNNs). By rigorously recasting the architecture, learning dynamics, and invariance properties of DNNs in terms of Grothendieck toposes, stacks, and closed model categories, the authors introduce a formalism that enables the study of deep learning as a phenomenon in modern category theory, with far-reaching implications for understanding semantics, generalization, and information in neural computation.

Formalization of DNNs as Topos Objects

The paper establishes that every (known) artificial DNN can be canonically associated with an object in a Grothendieck topos. The underlying directed acyclic graph (DAG) representing layers and their connections is modeled as a finite poset. Functorial constructions assign to each layer (vertex in the graph) and connection (edge) appropriate sets (e.g., neuron activities, weights) and structure-preserving maps, resulting in a presheaf (contravariant functor) on the category associated with the network DAG.

Key aspects:

• Objects: Layers, weights, and their activities are represented as (pre)sheaves over the network's organizational category.
• Arrows: Mappings between adjacent or related layers form the natural transformations "flowing" information and parameters through the network.
• Topos: The category of these presheaves constitutes the topos associated with the network.

The construction generalizes beyond simple chain DNNs to arbitrarily complex architectures (e.g., CNNs, RNNs, LSTMs). It is shown that, up to categorical equivalence, the associated topos is always coherent, localic, and generated as a presheaf topos over a sub-poset corresponding to relevant network vertices (input/output layers and forks introduced to resolve multi-input connections).

Learning Dynamics as Flows in Topos Morphisms

The learning process—specifically, backpropagation—is formalized as a flow of natural transformations in the topos. For supervised learning, the backpropagation procedure is shown to correspond to a parametrized flow on the functor of weights, $\mathbb{W}$, with the update rule represented as a gradient vector field on the parameter manifold, inducing a flow in the space of natural transformations.

Strong claims include:

• Backpropagation is not merely a compositional modular process, but rather a dynamical flow of transformations internal to the topos of the network's architecture.
• The class of toposes arising from DNN architectures always corresponds to sheaf categories over Alexandrov topologies on finite posets, allowing for the transfer of structural stability and coherence properties directly from topos theory.

Stacks, Invariance, and Modularity

The theory is extended to incorporate invariance and equivariance properties, critical for the generalization performance of structured networks such as CNNs. By enriching the category of presheaves to stacks (i.e., fibered categories over the site), the authors formalize how group symmetries and semantic invariances induce higher structure:

• Stacks over the network architecture site naturally encode local group actions (e.g., translation symmetry in convolution), logic, and invariance structures.
• The internal logic of DNNs, including their ability to "carry" types and theories through their layers, is described via the classifying topos and the propagation of logical structures through stacks.
• The paper introduces sufficient geometric conditions (e.g., openness and fibration properties of the functors between fibers) for seamless propagation of semantic theories through the network, ensuring "fluid circulation of semantics".

Homotopical and Type-Theoretic Classification

A central contribution is the classification of the semantic structure of DNNs via homotopy-theoretic and type-theoretic methods:

• The category of stacks over a DNN architecture is shown to possess the structure of a closed model category in the sense of Quillen, with explicit identification of fibrant, cofibrant, and weak-equivalence classes. Fibrant objects correspond to networks admitting fluid semantic propagation throughout their architecture.
• Every DNN naturally gives rise to an associated Martin-Löf type theory, structuring contexts and dependent types as fibrations and morphisms in the model category. The associated "semantic functioning" of the network—its ability to express (and interpret) theories and semantic content—is determined by these homotopical and type-theoretic invariants.

Semantic Information in Networks: Homology and Homotopy

Generalizing the notion of entropy and information from probabilistic graphical models, the authors introduce:

• Semantic information quantities and spaces: These are defined via (co)homological constructions on the categories of theories and propositions propagated through the network, with the bar complex and simplicial methods employed to compute mutual information, semantic ambiguity, and semantic Kullback-Leibler divergence.
• Homotopy types of semantic information: The flow of theories and propositions through a stack-structured network is shown to define bi-simplicial sets, whose homotopy colimits encapsulate the higher-order relationships and ambiguities inherent in semantic reasoning by the network.
• Implications for information flow and abstraction: These homological and homotopical invariants are conjectured to quantitatively and qualitatively capture the generalization and abstraction abilities of DNNs—critical properties that lie beyond the scope of mere statistical learning.

Memory Cells, Natural Language, and Higher Groupoid Structures

Recurrent architectures (e.g., LSTMs, GRUs) are analyzed categorically, with the stack structure elucidating their modularity and role in capturing linguistic or cognitive invariants. Notably:

• The groupoids arising in the stack structure of LSTMs correspond to braid groups (e.g., Artin's $B_3$ for three-strand braids), reflecting the algebraic relations between memory update pathways and the semantics of temporal and relational reasoning.
• The categorical semantics naturally interface with intentional type theory and developments such as homotopy type theory (HoTT), further reinforcing the view that semantic reasoning in DNNs can be studied within the modern paradigm of types-as-propositions and higher categories.

Multi-Network Assemblies and $3$-Category Structure

The framework is extended to modular and multi-network settings, introducing a $3$-category of networks, stacks, and their morphisms and derivators. This apparatus allows for the formal analysis of modularity, composability, and obstructions to semantic integration across different components and tasks. The notion of semantic communication between networks is proposed, grounded in the categorical invariants developed.

Theoretical and Practical Implications

Theoretical

• Unified semantic theory for DNNs: By embedding DNNs in the categorical landscape of topos, stacks, and homotopical algebra, the field gains a pathway for rigorous reasoning about compositionality, modularity, and semantics in AI systems.
• Novel invariants and classification tools: The introduction of (co)homological and homotopical invariants for semantic information in DNNs promises an intermediate abstraction between detailed stochastic modeling and logical reasoning.
• Bridging logic and learning: The interaction between logical type-theoretic structures and statistical learning dynamics is rendered precise, enabling analysis that incorporates both formal semantics and statistical generalization.

Practical and Implementation Considerations

• Modularity and architectural design: The categorical approach informs the design of modular, invariant-preserving network architectures, promoting reusability and composability.
• Semantic generalization and transfer: By capturing invariances and logic in the stack structure, networks can be more readily equipped for extrapolation, transfer, and semantic reasoning—a key shortcoming of conventional architectures.
• Information-theoretic analysis: The homological semantic information framework paves the way for new measures of representation quality, mutual information across layers, and diagnostic/probing techniques beyond traditional statistical metrics.
• Programming and specification: The connection to type theory opens new avenues for programming frameworks, formal verification, and automated reasoning about the semantic properties of DNNs.
• Limitations and computational overhead: Practical implementation of these mathematical structures for large-scale networks may require development of efficient algorithms for handling presheaf and homotopy computations, and possibly their integration into existing deep learning frameworks.

Future Directions

• Empirical computation of semantic information spaces: Development of concrete algorithms to compute homological and homotopical invariants from network states and activities, including leveraging persistent homology, will be critical for deploying these concepts in mainstream AI research and practice.
• Higher categorical architectures: Deployment of $n$-category structures to model ever more complex network modularity and interaction schemes.
• Semantic communication protocols: Formalization and testing of semantic communication between artificial agents and between artificial and biological systems.
• Operationalization for interpretability and debugging: Adapting the categorical semantics for the purposes of interpretability, model auditing, and safety in AI.

Conclusion

"Topos and Stacks of Deep Neural Networks" (2106.14587) provides a rigorous categorical and homotopical framework for the semantic analysis and abstract study of deep neural networks. By systematically reconstructing architectures, learning, invariance, and information flow within the machinery of topos and homotopical algebra, the work establishes foundational links between compositional logic, type theory, and statistical learning. This framework lays the groundwork for a new paradigm in the theory of AI systems, where modularity, semantic generalization, and information flow are formalized and studied with the full power of contemporary categorical mathematics, with substantial implications for both theoretical development and practical advances in the design and understanding of intelligent systems.