Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding

Abstract: Hierarchical structures are popular in recent vision transformers, however, they require sophisticated designs and massive datasets to work well. In this paper, we explore the idea of nesting basic local transformers on non-overlapping image blocks and aggregating them in a hierarchical way. We find that the block aggregation function plays a critical role in enabling cross-block non-local information communication. This observation leads us to design a simplified architecture that requires minor code changes upon the original vision transformer. The benefits of the proposed judiciously-selected design are threefold: (1) NesT converges faster and requires much less training data to achieve good generalization on both ImageNet and small datasets like CIFAR; (2) when extending our key ideas to image generation, NesT leads to a strong decoder that is 8$\times$ faster than previous transformer-based generators; and (3) we show that decoupling the feature learning and abstraction processes via this nested hierarchy in our design enables constructing a novel method (named GradCAT) for visually interpreting the learned model. Source code is available https://github.com/google-research/nested-transformer.

• The paper introduces NesT, a nested hierarchical transformer that leverages localized self-attention on non-overlapping image blocks to improve accuracy and efficiency.
• It demonstrates fast convergence, reaching 82.3% accuracy in just 100 epochs on ImageNet and up to 97.2% on CIFAR10 in base configurations.
• The model’s design enhances interpretability via GradCAT, offering clear hierarchical visual decision pathways and faster generative performance than comparable transformers.

Evaluation of the Nested Hierarchical Transformer for Visual Understanding

The paper "Nested Hierarchical Transformer: Towards Accurate, Data-Efficient, and Interpretable Visual Understanding" presents a novel approach to vision transformers, aiming to address the complexities and data inefficiencies characterizing prior transformer models used in visual tasks. This approach, termed NesT (Nested Hierarchical Transformer), leverages hierarchical nesting of transformers on non-overlapping image blocks, significantly differing from methods requiring global self-attention layers.

Key Contributions and Architectural Design

The NesT model departs from the conventional full-attention mechanisms in transformers, proposing a localized form of self-attention that is hierarchically nested. This design utilizes:

1. Hierarchical Nesting with Block Aggregation:
   • Transformers within NesT are applied to non-overlapping image blocks independently and aggregated hierarchically. This block aggregation allows for significant reduction in data and computational demands, demonstrating enhanced efficiency for image classification tasks, notably achieving competitive ImageNet accuracy. For instance, a NesT model with 68M parameters yields an 83.8% accuracy rate.
2. Data Efficiency and Fast Convergence:
   • NesT models exhibit fast convergence rates and require significantly less training data compared to previous designs. The paper reports that a NesT model can rapidly attain 82.3% accuracy after just 100 training epochs, highlighting its potential for efficient visual feature learning.
3. Extension to Generative Modeling:
   • By transposing the architectural principles from classification to image generation, NesT is adapted as a decoder for generative tasks. It reportedly achieves better performance metrics than analogous convolutional frameworks and outperforms other transformer-based generators like TransGAN in terms of throughput (8× faster), thereby underscoring its utility in generative modeling.
4. Visual Interpretability Through GradCAT:
   • A salient feature of NesT is its interpretability enabled by GradCAT, a novel gradient-based class-aware traversal method. This facilitates interpretable visual pathways akin to decision trees, offering insights into model decisions at different hierarchical levels. The method is noted to enhance understanding of how models prioritize and process visual information hierarchically.

Comparative Analysis and Performance Results

Comparative evaluations place NesT favorably against both convolutional architectures and other transformer-based models across datasets like CIFAR and ImageNet. Its hierarchical configuration not only leads to improved data efficiency on smaller datasets but also matches, or in some cases, surpasses the performance of complex architectures like Swin Transformer on larger datasets.

For CIFAR datsets, NesT consistently exhibits higher accuracy, reaching up to 97.2% with base configurations on CIFAR10. On ImageNet, NesT models achieve robust performance with minimal architectural changes, meeting the benchmarks set by models that invest in intricate attention mechanisms.

Theoretical and Practical Implications

The proposed NesT model advances the theoretical understanding of attention mechanisms in transformers by simplifying the architecture without compromising performance. Its practical implications are profound, suggesting pathways for deploying efficient, scalable, and interpretable transformer models across varied visual tasks, from classification to generation.

NesT's efficacy further implies potential in semi-supervised learning contexts and other data-constrained environments, where efficient learning and interpretability are critical. This study presents an instructive step toward optimizing transformers for visual tasks without the traditional necessity for extensive computational resources or complex modifications.

Conclusion and Future Directions

The paper offers significant advancements in the domain of vision transformers by introducing a simplified, efficient, and interpretable architecture. As researchers pursue further optimization of visual models, the concepts underpinning NesT provide a robust framework adaptable to various applications beyond image analysis, potentially influencing future design in domains where data efficiency and interpretability are paramount requisites. Future work could extend these insights to non-visual domains, testing the versatility and robustness of hierarchical nesting in transformers across different datasets and application contexts.