Rotate to Attend: Convolutional Triplet Attention Module (2010.03045v2)

Abstract: Benefiting from the capability of building inter-dependencies among channels or spatial locations, attention mechanisms have been extensively studied and broadly used in a variety of computer vision tasks recently. In this paper, we investigate light-weight but effective attention mechanisms and present triplet attention, a novel method for computing attention weights by capturing cross-dimension interaction using a three-branch structure. For an input tensor, triplet attention builds inter-dimensional dependencies by the rotation operation followed by residual transformations and encodes inter-channel and spatial information with negligible computational overhead. Our method is simple as well as efficient and can be easily plugged into classic backbone networks as an add-on module. We demonstrate the effectiveness of our method on various challenging tasks including image classification on ImageNet-1k and object detection on MSCOCO and PASCAL VOC datasets. Furthermore, we provide extensive in-sight into the performance of triplet attention by visually inspecting the GradCAM and GradCAM++ results. The empirical evaluation of our method supports our intuition on the importance of capturing dependencies across dimensions when computing attention weights. Code for this paper can be publicly accessed at https://github.com/LandskapeAI/triplet-attention

• The paper proposes a three-branch Triplet Attention mechanism that captures cross-dimensional interactions in CNNs.
• It utilizes tensor rotation and Z-pooling to efficiently integrate spatial and channel dependencies.
• Empirical results show up to 2.28% Top-1 accuracy gains on ImageNet-1k with minimal computational overhead.

Convolutional Triplet Attention Module: An Overview

The paper "Rotate to Attend: Convolutional Triplet Attention Module" introduces an innovative attention mechanism tailored for convolutional neural networks (CNNs). This mechanism, dubbed Triplet Attention, optimizes computational efficiency while maintaining efficacy, making it suitable for diverse computer vision tasks such as image classification and object detection.

Key Contributions

This research aims to enhance feature representation by leveraging cross-dimensional interactions within input tensors. Traditional attention methods often compute channel or spatial dependencies separately, potentially missing out on inter-dimensional relationships. Triplet Attention addresses this by employing a three-branch system that captures dependencies across spatial and channel dimensions effectively.

Methodology

The Triplet Attention mechanism consists of three branches, each dedicated to capturing distinct inter-dimensional interactions:

1. Cross-Dimensional Interaction: Two branches focus on interactions between the channel dimension and one of the spatial dimensions (height or width), achieved via tensor rotation and Z-pooling. The third branch captures interactions between the two spatial dimensions.
2. Rotation and Z-Pool: By rotating input tensors and utilizing Z-pooling, this mechanism preserves rich feature representations while minimizing computational complexity.
3. Element-Wise Operations: Attention weights are derived using sigmoid activations, ensuring seamless integration into existing CNN architectures. These weights are then applied to the rotated tensors, which are reverted to their original orientation before aggregation.

Empirical Evaluation

The authors employ several datasets, including ImageNet-1k and MSCOCO, to validate the Triplet Attention module across various models like ResNet and MobileNetV2. Significant performance improvements are recorded, with up to 2.28% gains in Top-1 accuracy on ResNet-50 for ImageNet-1k, while introducing minimal additional parameters and computational overhead. Notably, a ResNet-50 model incorporating Triplet Attention achieves competitive results in object detection tasks, surpassing several established attention mechanisms.

Implications and Future Directions

The research establishes the practicality of exploiting inter-dimensional dependencies for effective attention computation. By maintaining minimal additional computational demands, Triplet Attention is positioned as a versatile module for enhancing both lightweight and heavyweight CNN architectures.

Future developments may include exploring alternative methods for capturing cross-dimensional interactions that further improve performance efficiency. Additionally, the integration of Triplet Attention into more advanced architectures like EfficientNets could yield promising outcomes in further minimizing computational demands without sacrificing accuracy.

In conclusion, the paper presents a compelling case for the integration of Triplet Attention in CNNs, underscoring the significance of capturing inter-dimensional dependencies with negligible computational overhead. This advancement not only contributes to theoretical understanding but also offers practical benefits for a wide range of computer vision applications.