Exploring Randomly Wired Neural Networks for Image Recognition (1904.01569v2)

Abstract: Neural networks for image recognition have evolved through extensive manual design from simple chain-like models to structures with multiple wiring paths. The success of ResNets and DenseNets is due in large part to their innovative wiring plans. Now, neural architecture search (NAS) studies are exploring the joint optimization of wiring and operation types, however, the space of possible wirings is constrained and still driven by manual design despite being searched. In this paper, we explore a more diverse set of connectivity patterns through the lens of randomly wired neural networks. To do this, we first define the concept of a stochastic network generator that encapsulates the entire network generation process. Encapsulation provides a unified view of NAS and randomly wired networks. Then, we use three classical random graph models to generate randomly wired graphs for networks. The results are surprising: several variants of these random generators yield network instances that have competitive accuracy on the ImageNet benchmark. These results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design.

• The paper introduces a stochastic network generator that leverages random graph models to automate neural architecture design.
• The paper shows that randomly wired networks can match or outperform traditional models like ResNet-50 on ImageNet.
• The paper highlights the potential of reducing manual design in neural networks, thereby fostering more unbiased and efficient model architectures.

Exploring Randomly Wired Neural Networks for Image Recognition

The paper presented investigates an innovative approach in neural network design by exploring randomly wired neural networks for the task of image recognition. Traditional neural network architectures have relied heavily on manually designed connectivity, such as the influential designs of ResNet and DenseNet, where specific wiring patterns have been engineered to improve performance. However, the authors propose a novel method that utilizes random graph models to generate neural network architectures, thus aiming to remove the constraints imposed by manual design.

Methodological Advances

The core technical contribution of the paper is the concept of a stochastic network generator. This generator uses classical random graph models—specifically, Erdős-Rényi (ER), Barabási-Albert (BA), and Watts-Strogatz (WS)—to produce network architectures. These graph-based approaches help define a diverse set of connectivity patterns allowing automatic generation of the networks. By converting these random graphs into directed acyclic graphs (DAGs) suited for image recognition, the authors encapsulate the network generation process under a unified framework.

The paper emphasizes the importance of exploring new methodologies for network generation, exceeding the capabilities of neural architecture search (NAS) while bypassing its constraints. Unlike NAS, which still involves a significant amount of manual design, the proposed stochastic generation framework drastically expands the search space by randomizing the wiring in neural architectures.

Empirical Evaluation

The experimentation focuses on the application of these randomly generated network architectures to the ImageNet dataset. Remarkably, several instances of randomly wired networks matched or outperformed the accuracy of traditionally designed networks such as ResNet-50, highlighting the potential benefits of less constrained search spaces. The best observed results with WS-generated networks achieved classification accuracies that were superior to ResNet-50 under similar computational budgets.

The paper claims low variance in the accuracy results across different random samples of network architectures produced by the same network generator, while variations in accuracy were noticeable across different random generator designs. This observation underscores the critical need for an effective generator design rather than excessive fine-tuning of hyperparameters through exhaustive searches.

Implications and Future Directions

The results imply that significant performance gains in deep learning could be achieved by rethinking the fundamentals of network architecture design. The promising outcomes indicate that future work might focus on refining network generators to produce even more effective structures autonomously. The implications extend beyond practical efficiency improvements to a theoretical re-evaluation on how architecture design impacts deep learning performance.

Moving forward, it would be pertinent to explore other graph models and fine-tune network generator designs further to distribute neural network research advancements towards more automated, less bias-prone methods. This approach could also spur future development of network generators as a parameterized prior in network design, fundamentally shifting the way in which architectures are crafted and optimized.

Ultimately, the paper's exploration of the role of architecture design within randomly wired networks contributes significantly to the field, providing evidence that such an automated approach to network wiring might gain traction as a mainstream methodology in the continued evolution of artificial intelligence.