Spatial embedding promotes a specific form of modularity with low entropy and heterogeneous spectral dynamics (2409.17693v1)

Abstract: Understanding how biological constraints shape neural computation is a central goal of computational neuroscience. Spatially embedded recurrent neural networks provide a promising avenue to study how modelled constraints shape the combined structural and functional organisation of networks over learning. Prior work has shown that spatially embedded systems like this can combine structure and function into single artificial models during learning. But it remains unclear precisely how, in general, structural constraints bound the range of attainable configurations. In this work, we show that it is possible to study these restrictions through entropic measures of the neural weights and eigenspectrum, across both rate and spiking neural networks. Spatial embedding, in contrast to baseline models, leads to networks with a highly specific low entropy modularity where connectivity is readily interpretable given the known spatial and communication constraints acting on them. Crucially, these networks also demonstrate systematically modulated spectral dynamics, revealing how they exploit heterogeneity in their function to overcome the constraints imposed on their structure. This work deepens our understanding of constrained learning in neural networks, across coding schemes and tasks, where solutions to simultaneous structural and functional objectives must be accomplished in tandem.

• The paper demonstrates that spatial embedding actively generates low entropy modularity in RNNs during training.
• It reveals that spatial constraints focus connections on shorter distances, resulting in a predictable, tightly clustered network topology.
• The study finds that embedding alters eigenspectral dynamics, yielding varied and interpretable activation patterns in both rate and spiking RNNs.

Spatial Embedding Promotes a Specific Form of Modularity with Low Entropy and Heterogeneous Spectral Dynamics

In the paper "Spatial embedding promotes a specific form of modularity with low entropy and heterogeneous spectral dynamics," Sheeran et al. examine how spatial embedding influences structural and functional organization in recurrent neural networks (RNNs). This work adds depth to our understanding of how biological neural circuits, governed by multiple constraints such as geometry, development, and energy budgets, attain specific configurations during learning.

Methodological Overview

The authors employ spatially embedded recurrent neural networks (seRNNs) with neurons positioned within a discrete three-dimensional Euclidean space. This embedding introduces constraints reflecting spatial distance costs of wiring and communication.

Four groups of networks were tested:

• Baseline (L1): Promotes sparsity without spatial constraints.
• Baseline + Space + Communicability (seRNN): Considers both spatial distance and topological distance.
• Baseline + Space Only: Considers only spatial distance.
• Baseline + Communicability Only: Considers only topological distance.

Rate and spiking implementations were pursued. Rate RNNs were tested using a one-choice inference task, while spiking RNNs were evaluated using the Spiking Heidelberg Digits (SHD) task. Network performance and configurations were analyzed using modularity, Shannon entropy, and spectral entropy derived from their eigenspectra.

Key Findings

1. Low Entropy Modularity in Spatially Embedded Networks

Both rate and spiking seRNNs develop low entropy modular networks, a property significantly pronounced relative to baseline models. The Shannon entropy of seRNN weight matrices decreases more markedly over the training period, indicating a concentration of weights. The paper showed a linear relationship in seRNNs between high modularity and low Shannon entropy, suggesting that spatial constraints induce a more defined modular structure.

2. Interpretability Through Distance-Dependent Connectivity

The paper reveals that the seRNNs' lower Shannon entropy results, in part, from spatial constraints concentrating weights among shorter connections. The connection probability within seRNNs correlates negatively with connection length, and these connections form a regular spatial structure, confirming that spatially constrained networks form a specific low-entropy modularity.

3. Low Entropy Communicability and Regular Network Topologies

Additionally, seRNNs demonstrate decreased Shannon entropy in communicability matrices, indicating more concentrated communicative paths. This supports that spatial and communication constraints create a regular, low-entropy topology of weights, leading to more predictable communication pathways.

4. Eigenspectral Dynamics

Constraints also markedly influence the eigenspectrum of RNNs. SeRNNs exhibit smaller leading eigenvalues and higher spectral entropy compared to L1 networks, suggesting more varied dynamic behavior. The eigenvalues in seRNNs tend to be more real and significantly spread along the real axis due to increased symmetry in weight matrices imparted by spatial constraints.

Implications and Future Directions

This research signifies that spatial constraints guide neural networks toward low entropy solutions with specific modular configurations. These findings are instrumental for both theoretical and practical aspects of computational neuroscience, as they emphasize the influence of biologically inspired constraints on learning outcomes.

Future research could explore the scalability of seRNNs to larger, more complex systems and other forms of biologically realistic constraints. Investigating how these structural constraints interact with different neural coding schemes, such as mixed selectivity and diverse neuronal properties, could provide further insights.

Conclusion

Sheeran et al.'s work illustrates that spatial embedding and communication constraints lead to neural networks with specific low entropy modular configurations. Their findings bridge the gap between structural constraints and functional dynamics. This understanding opens new research pathways focusing on constrained learning in neural networks, emphasizing that both structural and functional objectives can be jointly optimized during network training.