HRFormer: High-Resolution Transformer for Dense Prediction (2110.09408v3)

Abstract: We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer by $1.3$ AP on COCO pose estimation with $50\%$ fewer parameters and $30\%$ fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer.

• The paper presents HRFormer, a model that fuses multi-resolution architecture with transformer methodologies to tackle dense prediction challenges.
• It integrates a 3×3 depth-wise convolution within its feed-forward network to boost contextual feature exchange while reducing computational costs.
• Empirical results demonstrate HRFormer's superior performance on tasks like pose estimation and segmentation, using fewer parameters and FLOPs compared to existing models.

Overview of HRFormer: High-Resolution Transformer for Dense Prediction

The paper "HRFormer: High-Resolution Transformer for Dense Prediction" introduces an innovative framework aimed at enhancing dense prediction tasks such as human pose estimation and semantic segmentation. The research presents the High-Resolution Transformer (HRFormer), an architecture designed to produce high-resolution representations while addressing the inefficiencies observed in conventional Vision Transformers (ViTs).

Key Architectural Insights

HRFormer leverages a multi-resolution parallel design, a concept borrowed from the High-Resolution Convolutional Network (HRNet), and integrates it with transformer-based methodologies. The architecture employs local-window self-attention, which restricts self-attention operations to small, non-overlapping sections of the input, optimizing memory and computational costs.

Additionally, a distinctive feature of HRFormer is the introduction of a convolutional component within the Feed-Forward Network (FFN). This convolution, particularly a $3 \times 3$ depth-wise operation, facilitates information exchange across the otherwise isolated image windows, enhancing the contextual understanding necessary for dense prediction tasks.

Empirical Results and Performance

HRFormer demonstrates substantial improvements over existing models in both efficiency and performance metrics:

• Pose Estimation: On the COCO dataset, HRFormer surpassed the Swin transformer by 1.3 AP while utilizing 50% fewer parameters and 30% fewer FLOPs. For instance, HRFormer-B achieved a 77.2% AP on COCO's validation set.
• Semantic Segmentation: The model attained significant mIoU improvements on benchmarks such as PASCAL-Context and COCO-Stuff. Notably, HRFormer-B combined with OCR showed notable efficiency in parameter usage compared to SETR-PUP, achieving competitive performance with a reduction in resource demands.
• Image Classification: On the ImageNet-1K dataset, HRFormer-B achieved a $+1.0\%$ top-1 accuracy improvement over DeiT-B, operating with 40% fewer parameters and 20% fewer FLOPs.

Components and Design Considerations

The HRFormer employs a combination of strategies to enhance its effectiveness:

• Multi-Resolution Design: The architecture maintains multiple resolution streams in parallel, similar to HRNet, allowing the model to effectively tackle variations in scale.
• Local-Window Attention: This strategy reduces complexity from quadratic to linear concerning spatial size, significantly enhancing computational efficiency.
• Depth-Wise Convolution in FFN: This design choice facilitates interaction across windows, broadening the receptive field and enriching feature representation space.

Broader Implications and Future Directions

The emergence of HRFormer indicates a promising direction in bridging convolutional and transformer architectures, particularly for tasks that demand high spatial detail. This approach may inspire further exploration into transformer designs optimized for vision applications, with a focus on balancing the trade-offs between detail preservation and computational resource consumption.

Looking forward, potential areas for extension include refining window-attention mechanisms and exploring cross-domain applications of HRFormer. Its adaptability to various tasks suggests scope for optimization in real-time systems and other resource-constrained environments.

In summary, HRFormer illustrates a thoughtful synthesis of high-resolution network design and transformer efficiency, setting a foundation for future advancements in dense prediction tasks across computer vision domains.