GloNets: Globally Connected Neural Networks (2311.15947v1)

Abstract: Deep learning architectures suffer from depth-related performance degradation, limiting the effective depth of neural networks. Approaches like ResNet are able to mitigate this, but they do not completely eliminate the problem. We introduce Globally Connected Neural Networks (GloNet), a novel architecture overcoming depth-related issues, designed to be superimposed on any model, enhancing its depth without increasing complexity or reducing performance. With GloNet, the network's head uniformly receives information from all parts of the network, regardless of their level of abstraction. This enables GloNet to self-regulate information flow during training, reducing the influence of less effective deeper layers, and allowing for stable training irrespective of network depth. This paper details GloNet's design, its theoretical basis, and a comparison with existing similar architectures. Experiments show GloNet's self-regulation ability and resilience to depth-related learning challenges, like performance degradation. Our findings suggest GloNet as a strong alternative to traditional architectures like ResNets.

• The paper presents a novel approach by superimposing a global aggregation layer on existing networks to balance depth and reduce training time.
• It integrates direct connections from every layer to a global feature aggregator that simplifies architecture and bypasses batch normalization.
• Empirical tests on benchmarks like MNIST and CIFAR-10 show that GloNet maintains or boosts performance, eliminating the need for network architecture search.

Introduction

The perennial challenge in deep learning is managing the depth of neural networks—deeper architectures theoretically have the potential to learn more intricate features but often face performance issues. Traditional solutions like ResNet have attempted to address these issues, though not entirely eliminating them. This paper introduces a new architecture extension, termed Globally Connected Neural Networks (GloNet), which aims to solve these depth-related issues by enhancing neural network models without increasing their complexity or decreasing performance. GloNet acts as a regulation layer that balances the influence of network layers, promoting stable training irrespective of the depth of the network.

Model Description

GloNet functions by superimposing on an existing network and creating direct connections between each layer of the network and a global feature aggregator, right before the model's output layer. Traditional network blocks consist of nonlinear transformations that can hinder the learning process by obscuring simpler features learned in earlier layers. GloNet, on the other hand, maintains access to features across all levels of abstraction, allowing these to be summed in a new layer—named the GloNet layer—prior to the final predictive output. Unlike other network architectures that interconnect blocks in intricate ways or require normalization techniques such as batch normalization, GloNet simplifies the network structure while preserving or even enhancing performance.

Empirical Validation

The efficacy of the GloNet architecture was rigorously tested across various tasks, including SGEMM GPU kernel performance prediction, image classification on the MNIST and CIFAR-10 datasets, and integration with a Vision Transformer model. A standout result is GloNet's ability to reduce training time by approximately half when compared to equivalent ResNet architectures, with comparable or better performance. The paper also shows that GloNet is capable of self-regulating its depth during training, which means it can avoid the diminishing returns on network performance typically associated with increasing network depth. This innate ability of the network to find an effective depth during the training process also renders Network Architecture Search methods unnecessary for determining the optimal depth, further reducing computational requirements.

Advantages and Practicality

What distinguishes GloNet from other architectures like ResNet and DenseNet are several practical advantages:

1. It facilitates faster training without the need for batch normalization.
2. It serves as an effective alternative to ResNet, especially for very deep architecture requirements.
3. By self-regulating its depth, GloNet effectively reduces the model's complexity, avoiding the need for a separate network search process.
4. Lastly, GloNet provides a straightforward method for balancing efficiency against performance by discarding selective layers, thus optimizing the network to suit specific computational constraints or performance targets.

These features make GloNet a promising tool for future deep learning architecture design, potentially leading to more efficient and powerful AI systems.