PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture (2201.00978v1)

Abstract: Transformer networks have achieved great progress for computer vision tasks. Transformer-in-Transformer (TNT) architecture utilizes inner transformer and outer transformer to extract both local and global representations. In this work, we present new TNT baselines by introducing two advanced designs: 1) pyramid architecture, and 2) convolutional stem. The new "PyramidTNT" significantly improves the original TNT by establishing hierarchical representations. PyramidTNT achieves better performances than the previous state-of-the-art vision transformers such as Swin Transformer. We hope this new baseline will be helpful to the further research and application of vision transformer. Code will be available at https://github.com/huawei-noah/CV-Backbones/tree/master/tnt_pytorch.

• The paper introduces PyramidTNT, combining a pyramid architecture and convolutional stem to enhance vision transformer baselines.
• It achieves 82% top-1 accuracy on ImageNet-1K and 42 AP on COCO, outperforming previous TNT models with reduced computational cost.
• The hierarchical multi-scale feature extraction enables robust local and global representation learning, paving the way for future hybrid transformer designs.

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

The paper "PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture" presents an advancement in vision transformers by introducing PyramidTNT, an architecture that integrates a pyramid design and convolutional stem to enhance the existing Transformer-in-Transformer (TNT) frameworks. This work is situated within the landscape of evolving transformer architectures aimed at improving computer vision tasks, notably surpassing the performance of prominent models such as the Swin Transformer.

Methodological Advancements

Two primary modifications characterize PyramidTNT:

1. Pyramid Architecture: This design utilizes a hierarchical approach, decreasing resolutions at each stage to extract multi-scale representations. This architectural choice is aligned with previous approaches like PVT, leveraging pyramid network principles to efficiently capture diverse spatial features across different scales. The pyramid strategy enhances the modeling of both local and global visual representations, contributing to improved large-object detection.
2. Convolutional Stem: By incorporating convolutional layers early in the model, the authors address optimization stability issues and enhance the overall performance. The convolutional stem operates on the initial patches, transforming them into visual word and sentence representations, thereby strengthening the hierarchical feature extraction process.

Additionally, the implementation incorporates several advanced techniques such as relative position encoding and Linear Spatial Reduction Attention (LSRA), further optimizing computational efficiency and performance.

Experimental Evaluation

The efficacy of PyramidTNT is demonstrated through extensive experiments on the ImageNet-1K dataset and the COCO object detection benchmark. Notable results include:

• ImageNet-1K Classification: PyramidTNT-S achieved a top-1 accuracy of 82.0% with only 3.3B FLOPs, outperforming the original TNT-S by 0.5% while employing fewer computational resources. This positions PyramidTNT as a state-of-the-art vision transformer.
• COCO Detection: PyramidTNT-S attained an AP of 42.0 with lower computational costs in comparison to other models like Swin-T. The hierarchical architecture facilitated improved performance in detecting large objects, as evidenced by the AP$_L$ scores.

Implications and Future Prospects

The innovations in PyramidTNT contribute theoretically and practically to the domain of vision transformers. The pyramid architecture could inspire further research into multi-scale representation learning, particularly in applications requiring nuanced recognition over varied spatial scales. Moreover, the successful integration of convolutional elements with transformers may catalyze additional explorations into hybrid architectures that combine the strengths of CNNs and transformers.

Looking forward, future developments in AI could capitalize on the foundational work of PyramidTNT to enhance real-time object detection, improve model efficiency, and expand into other complex vision tasks. The adoption of such architectures could potentially extend beyond traditional computer vision to interdisciplinary applications by leveraging the robust multi-scale feature extraction capabilities demonstrated in this paper.

In summary, PyramidTNT represents a strategic enhancement in the field of vision transformers, offering a robust baseline for future explorations and applications. The paper provides compelling evidence of the advantages of pyramid architectures in improving neural network performance, setting a precedent for subsequent research in this rapidly advancing field.