Patches Are All You Need? (2201.09792v1)

Abstract: Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/locuslab/convmixer.

Patches Are All You Need?

The paper "Patches Are All You Need?" by Asher Trockman and J. Zico Kolter explores the performance implications of Vision Transformers (ViTs) compared to traditional convolutional architectures in computer vision. With the conjecture that patch embeddings might be as crucial to the performance of ViTs as their architectural innovations, the paper introduces a new model, ConvMixer.

Overview of the Paper

Introduction

For years, convolutional neural networks (CNNs) dominated computer vision tasks due to their ability to efficiently manage spatial hierarchies via convolutions and pooling operations. However, Vision Transformers (ViTs) have recently emerged with a compelling performance advantage, leveraging self-attention mechanisms traditionally used in NLP. The distinct architecture of ViTs necessitates breaking down images into patches to manage the quadratic complexity of self-attention layers. This modification raises the question: Do these performance gains derive from the inherently more powerful architecture of Transformers, or are patches themselves a significant contributing factor?

ConvMixer Model

To investigate this, the paper introduces the ConvMixer model, which maintains the patch-based input structure while utilizing standard convolution operations. ConvMixer operates on patches, preserves resolution throughout the network layers, and separates the mixing of spatial and channel dimensions. This architecture diverges from the ViT and MLP-Mixer by using simple convolutions rather than self-attention or MLP layers.

1. Patch Embeddings: Each image is divided into patches which are linearly embedded. This process is similar to a convolution with a kernel size and stride equal to the patch size.
2. ConvMixer Blocks: Each block comprises a depthwise convolution followed by a pointwise convolution, interspersed with activation functions and BatchNorm layers. Depthwise convolutions have large kernel sizes to emulate the receptive field of self-attention mechanisms.
3. Classifier: The final feature map is globally pooled and passed through a softmax classifier.

This design exemplifies an extremely lightweight model, implementable in just a few lines of dense PyTorch code, that contrasts standard architectures like ResNet and advanced ones such as ViT.

Experimental Results

The experiments largely focus on ImageNet-1k, adopting standard and widely used augmentation and optimization techniques to ensure comparability. Some key findings include:

• Performance: ConvMixer-1536/20 achieves 81.37% top-1 accuracy, competitive with sophisticated models like DeiT-B and ResNet-152, despite its simplicity.
• Efficiency: ConvMixer models display less inference throughput due to smaller patch sizes relative to Transformer-based models. Conversely, models with larger patches underperform in accuracy yet show increased throughput, suggesting a tradeoff.
• Scalability: The depth and width of ConvMixers can be adjusted for smaller datasets such as CIFAR-10, demonstrating high accuracy (over 96%) with a minimal parameter count (approximately 700K for kernel size 13).

Implications

Practical Implications: The ConvMixer's simplicity offers potential efficiency benefits, particularly in settings where quick deployment and interpretability are critical. Its flexible architecture can be adapted for use in resource-constrained environments.

Theoretical Implications: The results imply that the advantages of ViTs could indeed be partially attributed to the use of patches rather than the Transformer architecture itself. This calls for a deeper investigation into the inherent properties of patch-based architectures across different layers of neural networks.

Future Directions

1. Hyperparameter Tuning: Given the non-exhaustive hyperparameter optimization, fine-tuning could further elevate the performance of ConvMixers, potentially closing the gap with state-of-the-art models.
2. Task Generalization: Applying ConvMixers to tasks like semantic segmentation and object detection could validate their versatility and uncover additional use cases.
3. Architectural Innovations: Introducing minor enhancements such as bottlenecks or hybrid layers could amplify performance while retaining simplicity.
4. Optimization: Low-level optimizations for large-kernel depthwise convolutions might bolster inference speed, especially in large-scale applications.

Conclusion

"Patches Are All You Need?" challenges prevalent assumptions in computer vision by demonstrating that patches alone, when combined with convolutional operations, can rival top-tier models. And while ConvMixers are inherently simpler, they highlight the potential of patch-based approaches, questioning whether the architectural complexity of modern Transformer-based models is always necessary. The paper opens up pathways for more streamlined, effective model architectures that continue to leverage the ubiquity of convolutional operations with the novel insight of patch embeddings.