Deep Networks with Internal Selective Attention through Feedback Connections (1407.3068v2)

Abstract: Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNets feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model.

• The paper introduces dasNet, a network that incorporates iterative feedback connections to implement dynamic internal selective attention.
• It leverages reinforcement learning via scalable natural evolution strategies to optimize the focus of convolutional filters.
• Testing on CIFAR-10 and CIFAR-100 demonstrates enhanced performance, reducing test error rates compared to traditional feedforward CNNs.

Deep Networks with Internal Selective Attention through Feedback Connections

The research reported in "Deep Networks with Internal Selective Attention through Feedback Connections" explores the integration of feedback mechanisms into convolutional neural networks (CNNs), emulating the feedback process present in human neural architectures. Traditional CNNs, comprising a series of feedforward layers, have set a benchmark in various computer vision tasks. However, their architecture mirrors only the bottom-up processing observed in biological visual systems, lacking dynamic adaptability during evaluation. This paper introduces the Deep Attention Selective Network (dasNet), which employs feedback pathways to dynamically adjust convolutional filter sensitivities, thus enabling an internal attentional mechanism.

The architecture of dasNet deviates from traditional CNNs by implementing iterative feedback connections, allowing the network to focus selectively on convolutional filters during the classification process. This feedback is trained using Scalable Natural Evolution Strategies (SNES) in a multidimensional parameter space, capitalizing on sequential processing improvements. The researchers conducted experiments on CIFAR-10 and CIFAR-100 datasets, where dasNet achieved superior performance compared to previous state-of-the-art models, highlighting its efficacy in handling difficult classification instances.

The cornerstone of dasNet's architecture is rooted in reinforcement learning (RL) applied to control selective attention. The policy, trained via SNES, determines the focus of internal attention dynamically, thereby optimizing the use of this non-stationary CNN structure. The large parameter space, which exceeds one million dimensions, renders conventional RL methods inefficient, justifying the selection of scalable evolutionary strategies for training the policy governing feedback connections. SNES optimizes a distribution over network parameters, iteratively improving the focus mechanism.

Experimental results demonstrate dasNet's robust performance enhancements over typical feedforward CNNs, indicating a relative improvement in classification accuracy due to its attention model. For example, when trained on the CIFAR-10 dataset, dasNet achieved a test error rate of 9.22%, surpassing the 9.61% of a strong Maxout network baseline. This result signifies not only a theoretical enhancement but presents a practical improvement with potential real-world applications. The implications extend beyond mere accuracy improvements; the dynamic updating capability offered by attention mechanisms introduces a form of plasticity in neural networks, offering insights into the adaptability and efficiency required for complex pattern recognition.

Theoretically, this work suggests a paradigm shift in neural network design, advocating for architectures that not only mimic the structure but also the dynamic processes of biological systems. This shift could influence further developments in deep learning models, potentially enhancing their capacity to process complex stimuli using a smaller architectural footprint by focusing computational resources on critical features iteratively.

Future directions could involve incorporating more sophisticated feedback actions and exploring different RL techniques to fine-tune these attentional processes. There is also potential to expand the use of feedback networks for real-time applications where dynamic adaptability is crucial, or in scenarios involving varying task difficulty and ambiguity.

To conclude, the integration of internal selective attention through feedback connections represents a noteworthy progression in the field of neural networks, demonstrating both theoretical insight and practical enhancement in image classification endeavors. The research lays the groundwork for further exploration into feedback-connected networks, encouraging adaptation in current neural architectures that reflect more closely the intricate operations of biological systems.