Adaptively Connected Neural Networks

Abstract: This paper presents a novel adaptively connected neural network (ACNet) to improve the traditional convolutional neural networks (CNNs) {in} two aspects. First, ACNet employs a flexible way to switch global and local inference in processing the internal feature representations by adaptively determining the connection status among the feature nodes (e.g., pixels of the feature maps) \footnote{In a computer vision domain, a node refers to a pixel of a feature map{, while} in {the} graph domain, a node denotes a graph node.}. We can show that existing CNNs, the classical multilayer perceptron (MLP), and the recently proposed non-local network (NLN) \cite{nonlocalnn17} are all special cases of ACNet. Second, ACNet is also capable of handling non-Euclidean data. Extensive experimental analyses on {a variety of benchmarks (i.e.,} ImageNet-1k classification, COCO 2017 detection and segmentation, CUHK03 person re-identification, CIFAR analysis, and Cora document categorization) demonstrate that {ACNet} cannot only achieve state-of-the-art performance but also overcome the limitation of the conventional MLP and CNN \footnote{Corresponding author: Liang Lin ([email protected])}. The code is available at \url{https://github.com/wanggrun/Adaptively-Connected-Neural-Networks}.

• The paper introduces ACNet, which dynamically selects between global and local feature inferences through an adaptive connectivity framework.
• The paper extends traditional neural networks by handling non-Euclidean data via learned position encoding, enhancing versatility.
• The paper demonstrates state-of-the-art performance on tasks like ImageNet classification, COCO detection, and CUHK03 re-identification while remaining computationally efficient.

Adaptively Connected Neural Networks: Bridging CNNs and MLPs with Adaptive Global and Local Inference

The paper "Adaptively Connected Neural Networks" by Guangrun Wang et al. introduces a novel neural network architecture termed the Adaptively Connected Neural Network (ACNet), which dynamically balances global and local feature inferences. This innovation aims to address some of the inherent limitations observed in traditional Convolutional Neural Networks (CNNs) and offers robust performance across a range of benchmarks and tasks.

The key insight of ACNet is its ability to adaptively select, during runtime, between global and local inferences for different nodes within a network. This is achieved via a flexible mechanism that optimizes the connectivity among feature nodes and adjusts the mode of inference based on the significance degrees which are learned from the data itself. This approach positions ACNets as a generalization of existing network architectures, notably CNNs, Multi-layer Perceptrons (MLPs), and Non-local Networks (NLNs), by encompassing them as special instances within its broader adaptive framework.

Principal Contributions and Findings

1. Adaptive Connectivity Framework: ACNet features an adaptive connectivity mechanism that combines the local versatility of CNNs with the global context awareness of MLPs. Importantly, the network determines the most relevant connections between nodes on a per-task and data-specific basis, without the need for exhaustive manual tuning.
2. Support for Non-Euclidean Data: Unlike conventional CNNs that are limited to Euclidean data, ACNet extends its capability to handle non-Euclidean data structures, such as graphs and manifold data. This extension is realized by leveraging position encoding functions that compensate for the intrinsic structural variability of non-Euclidean spaces.
3. Empirical Performance Gains: The paper provides comprehensive experimental evaluations on various datasets, showing that ACNet achieves state-of-the-art performance. For instance, on the ImageNet-1k classification task, ACNet demonstrated an improvement in top-1 accuracy compared to ResNet models. Similarly, performance gains are also observed in object detection and segmentation on the COCO 2017 dataset and person re-identification tasks on the CUHK03 dataset.
4. Computational Feasibility: Despite its adaptive nature, ACNet maintains computational efficiency comparable to traditional CNN architectures, ensuring its practical applicability. This is achieved through the design of the weights and connections in the network, which are efficiently learned and optimized via standard back-propagation.
5. Visualization of Adaptive Weights: The research provides visualizations of the adaptive weights, illustrating their variability across different layers and highlighting ACNet’s ability to integrate global context at shallow layers while focusing on localized features deeper in the network.

Implications and Future Directions

The development of ACNet suggests broader implications for neural network design, promoting a view where adaptability and context sensitivity are integral to improving performance across diverse datasets. It challenges the paradigm of fixed model architectures and offers a compelling direction for constructing models that dynamically adjust their structural parameters in response to input data characteristics.

Moving forward, the ideas encapsulated in ACNet could spur advancements in several areas of artificial intelligence and machine learning:

• Graph-based Machine Learning: With its capability to manage non-Euclidean data, ACNet's adaptive approach could significantly advance graph neural networks (GNNs), commonly used in social network analysis and bioinformatics.
• Resource-Efficient Models: By optimizing connections adaptively, future iterations of ACNet could focus on reducing redundancy and enhancing computational efficiency, crucial for deployment in resource-constrained environments like mobile devices or edge computing.
• Cross-Domain Applications: The flexibility in handling diverse data types and structures renders ACNet suitable for applications beyond traditional computer vision, including natural language processing and multi-modal data integration.

Overall, the dynamic architecture of the Adaptively Connected Neural Network represents a significant evolution in neural network design, encouraging a shift towards more adaptable, data-driven model configurations capable of handling complexity and variability in modern AI tasks.