Exploring Self-attention for Image Recognition (2004.13621v1)

Abstract: Recent work has shown that self-attention can serve as a basic building block for image recognition models. We explore variations of self-attention and assess their effectiveness for image recognition. We consider two forms of self-attention. One is pairwise self-attention, which generalizes standard dot-product attention and is fundamentally a set operator. The other is patchwise self-attention, which is strictly more powerful than convolution. Our pairwise self-attention networks match or outperform their convolutional counterparts, and the patchwise models substantially outperform the convolutional baselines. We also conduct experiments that probe the robustness of learned representations and conclude that self-attention networks may have significant benefits in terms of robustness and generalization.

• The paper demonstrates that self-attention networks outperform CNNs by achieving up to 78% top-1 accuracy with lower parameters and FLOPs.
• The paper introduces pairwise and patchwise self-attention mechanisms, identifying a 7x7 footprint as optimal for contextual efficiency.
• The paper shows that vector attention enhances robustness against rotations and adversarial attacks, improving stability in image recognition tasks.

Exploring Self-attention for Image Recognition

The paper "Exploring Self-attention for Image Recognition" explores the capabilities of self-attention mechanisms as fundamental operators for image recognition models. This work is situated within the broader trend in deep learning where convolutional neural networks (CNNs) have dominated computer vision over the past decade. However, this paper explores whether self-attention, an operator borrowed from NLP, can serve as a more efficient and effective alternative to convolution for image recognition tasks.

Overview of Self-attention Mechanisms

The authors examined two types of self-attention mechanisms: pairwise and patchwise. Pairwise self-attention generalizes the standard dot-product attention used in NLP and functions as a set operator, making it invariant to permutations and the number of elements. Patchwise self-attention, on the other hand, is strictly more powerful than convolution as it considers specific locations within its footprint and adapts its weights accordingly.

Importantly, the paper introduces vector attention, which allows the weights in the self-attention mechanism to vary both spatially and across feature channels. This is in contrast to scalar attention, which typically employs a single weight across all channels, thus limiting its expressiveness.

Experimental Setup and Results

The researchers constructed multiple self-attention networks (SANs), specifically SAN10, SAN15, and SAN19, to benchmark against various ResNet models (ResNet26, ResNet38, and ResNet50). These models were trained and evaluated on the ImageNet dataset.

Key Findings:

1. Performance: Pairwise self-attention networks either matched or outperformed their convolutional counterparts. For instance, SAN10 achieved 74.9% top-1 accuracy compared to 73.6% by ResNet26. Patchwise self-attention models demonstrated even higher performance gains. SAN15 attained 78% top-1 accuracy, outperforming ResNet50's 76.9% with a 37% lower parameter count and FLOP count.
2. Footprint Size: The optimal footprint size for both pairwise and patchwise self-attention was found to be 7x7. This size strikes a balance between capturing sufficient context and computational efficiency.
3. Robustness: Self-attention networks exhibited enhanced robustness to image manipulations and adversarial attacks. For example, the pairwise SAN19 model showed a top-1 accuracy drop of only 18.9 percentage points under a 180-degree rotation attack, while the ResNet50's accuracy fell by 24 percentage points. Furthermore, under adversarial attacks, self-attention models consistently performed better than convolution-based networks.

Implications

Practical Implications

The findings suggest that self-attention can serve as a viable alternative to convolution, particularly in tasks where robustness and generalization are critical. The lower parameter counts and FLOPs associated with self-attention networks also translate to more efficient models, beneficial for deployment in resource-constrained environments.

Theoretical Implications

The success of vector attention in outperforming scalar attention underscores the importance of considering channel-wise adaptivity in self-attention mechanisms. The superior performance of patchwise attention models indicates potential for new architectural innovations that leverage the generalization capabilities of self-attention beyond traditional convolutional operations.

Future Directions

The emergence of self-attention as a robust operator for image recognition opens several avenues for future research:

1. Hybrid Models: Exploring hybrid models that integrate both convolutional and self-attention mechanisms may harness the strengths of both worlds.
2. Generalization to Other Tasks: Extending self-attention architectures to other computer vision tasks, such as object detection and segmentation, to evaluate their generalizability.
3. Efficiency Improvements: Further optimizing self-attention mechanisms to reduce computational overhead while maintaining or enhancing performance.

Conclusion

This paper demonstrates the effectiveness of self-attention as a fundamental operator for image recognition. By extensively evaluating various self-attention forms and comparing them against convolutional networks, the paper provides substantial evidence supporting the shift towards self-attention mechanisms in deep learning for computer vision. The robustness and efficiency of self-attention networks position them as a compelling alternative to convolutional networks, potentially leading to more resilient and adaptable vision systems.