Graph Structure of Neural Networks (2007.06559v2)

Abstract: Neural networks are often represented as graphs of connections between neurons. However, despite their wide use, there is currently little understanding of the relationship between the graph structure of the neural network and its predictive performance. Here we systematically investigate how does the graph structure of neural networks affect their predictive performance. To this end, we develop a novel graph-based representation of neural networks called relational graph, where layers of neural network computation correspond to rounds of message exchange along the graph structure. Using this representation we show that: (1) a "sweet spot" of relational graphs leads to neural networks with significantly improved predictive performance; (2) neural network's performance is approximately a smooth function of the clustering coefficient and average path length of its relational graph; (3) our findings are consistent across many different tasks and datasets; (4) the sweet spot can be identified efficiently; (5) top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks. Our work opens new directions for the design of neural architectures and the understanding on neural networks in general.

• The paper introduces relational graphs as a novel framework that reinterprets neural network layers as rounds of message exchanges.
• The paper demonstrates how optimal graph measures, including clustering coefficient and average path length, significantly improve performance across models such as MLPs, CNNs, and ResNets.
• The paper establishes that the WS-flex graph generator efficiently identifies optimal network topologies that mirror biological neural networks, linking computational and neuroscience insights.

The paper "Graph Structure of Neural Networks" explores the underlying relationship between the graph structure of neural networks and their predictive performance. The authors introduce a novel concept in the representation of neural networks as relational graphs, where the layers in a neural network are perceived as rounds in which messages are exchanged along these graph structures.

Key Contributions:

1. Relational Graphs for Neural Networks: The paper introduces relational graphs as a tool to represent neural networks, diverging from the conventional computational graph approach. This representation focuses on message exchanges rather than directed information flows, allowing for a more general set of graph structures including those that are not directed or acyclic.
2. Graph Measures and Neural Network Performance:
   • The paper systematically examines how certain graph measures, namely the clustering coefficient and average path length, can influence the predictive performance of neural networks.
   • Detailed observation reveals that networks with a certain range of graph measures—termed the "sweet spot"—demonstrate significantly enhanced performance under controlled computational budgets.
3. Consistency Across Architectures and Tasks: The findings about relational graphs and the "sweet spot" for optimal performance hold consistently across various neural network architectures, such as MLPs, CNNs, and ResNets, tested on different datasets including CIFAR-10 and ImageNet.
4. WS-flex Graph Generator: The authors design a graph generator named WS-flex, an adaptation of the Watt-Strogatz model which can generate graphs that span a wide array of measure spaces, encompassing a broader range of potential neural network architectures.
5. Efficient Search for Optimal Graph Structures:
   • A proposed methodology allows for quick identification of optimal graph structures, reducing computational costs significantly as it requires sampling fewer graphs and using fewer training epochs.
   • Even a small number of samples, like 52 graphs, can yield a high correlation with results from exhaustive search, pointing toward computational efficiency in exploring network design spaces.
6. Biological Neural Networks: Remarkably, the neural networks represented by the optimal graph structures bear resemblance to biological neural networks, such as those found in the macaque and cat cortex, in terms of clustering and path length measures. This opens avenues for interdisciplinary research connecting network science, neuroscience, and machine learning.

Implications and Future Directions:

• The relational graph framework aligns with principles from neuroscience and network science, suggesting that deep learning architectures can benefit from insights used in these fields.
• Understanding graph structures more intricately can directly impact neural architecture search (NAS), potentially leading to more efficient and targeted searches through the space of network topologies.
• There is potential for evolving this foundational insight into hierarchical graph structures and investigating how this might interplay with other architectural levels, such as blocks and modules in state-of-the-art networks.

The paper's approach calls for an interdisciplinary dialogue that not only enhances the design of neural networks but also furthers the theoretical understanding of deep learning mechanisms. The proposed relational graph model and tools could serve as pivotal building blocks for future neural network architecture research.