Hierarchical Multi-Scale Attention for Semantic Segmentation (2005.10821v1)

Abstract: Multi-scale inference is commonly used to improve the results of semantic segmentation. Multiple images scales are passed through a network and then the results are combined with averaging or max pooling. In this work, we present an attention-based approach to combining multi-scale predictions. We show that predictions at certain scales are better at resolving particular failures modes, and that the network learns to favor those scales for such cases in order to generate better predictions. Our attention mechanism is hierarchical, which enables it to be roughly 4x more memory efficient to train than other recent approaches. In addition to enabling faster training, this allows us to train with larger crop sizes which leads to greater model accuracy. We demonstrate the result of our method on two datasets: Cityscapes and Mapillary Vistas. For Cityscapes, which has a large number of weakly labelled images, we also leverage auto-labelling to improve generalization. Using our approach we achieve a new state-of-the-art results in both Mapillary (61.1 IOU val) and Cityscapes (85.1 IOU test).

• The paper introduces a novel hierarchical multi-scale attention mechanism that dynamically aggregates pixel-level semantic predictions.
• It enhances segmentation accuracy while reducing memory consumption by approximately fourfold, enabling larger crop sizes.
• The study demonstrates state-of-the-art performance on Cityscapes and Mapillary Vistas and proposes an auto-labelling strategy to enrich training data.

Hierarchical Multi-Scale Attention for Semantic Segmentation

The paper "Hierarchical Multi-Scale Attention for Semantic Segmentation," authored by Andrew Tao, Karan Sapra, and Bryan Catanzaro from Nvidia, addresses the challenge of semantic segmentation through a novel approach utilizing hierarchical multi-scale attention. This paper presents a promising methodological advancement by combining the advantages of attention mechanisms with multi-scale inference to improve the semantic segmentation process in a resource-efficient manner.

The main contribution of this research is the introduction of a hierarchical attention mechanism, a structured approach that allows for the efficient aggregation of semantic predictions from multiple image scales. Unlike traditional methods which rely on averaging or max pooling to integrate predictions across different scales, this approach leverages attention to determine the optimal combination of these predictions on a pixel level. This transition from fixed-scale pooling methods to a dynamic, attention-driven methodology represents a significant refinement, as the content-dependent nature of the attention mechanism accommodates the diverse feature requirements inherent in semantic segmentation tasks.

Key results from the paper demonstrate that this hierarchical attention mechanism not only enhances the overall prediction accuracy but also achieves a reduction in memory consumption during training by approximately fourfold compared to other state-of-the-art approaches. This efficiency enables the use of larger crop sizes, thereby improving model performance without a concomitant increase in computational demand. Notably, the method achieves state-of-the-art results on the Cityscapes and Mapillary Vistas datasets, with Intersection over Union (IoU) metrics of 85.1 and 61.1, respectively.

The authors additionally propose an auto-labelling strategy for the Cityscapes dataset, which utilizes coarse-labelled images to enrich the training data pool. This method of generating pseudo-labels effectively expands the volume of training data, thereby enhancing generalization. The hard-threshold based labeling reduces computational resources by providing non-soft labels, thus optimizing disk I/O operations. This aspect of the research aligns with trends towards semi-supervised learning strategies, enabling the exploitation of large unlabelled datasets in an efficient manner.

The implications of this research are manifold. Practically, adopting a hierarchical attention model facilitates semantic segmentation tasks where scale variation significantly impacts performance, such as autonomous driving and drone imagery analysis. Theoretically, this approach could spark further research into adaptive scale aggregation techniques using attention mechanisms, potentially influencing future developments in neural architecture design where multi-scale features are critical.

Despite the robustness of the proposed method, there remain avenues for future exploration. Further work could explore the integration of hierarchical attention mechanisms with other advanced forms of neural networks beyond HRNet-OCR, refining this approach for wider applicability. Additionally, the exploration of more complex auto-labelling techniques or iterative refinement algorithms might augment the benefits observed from leveraging coarsely labeled data.

Overall, the introduction of hierarchical multi-scale attention presents a noteworthy contribution to semantic segmentation, balancing accuracy, and computational feasibility, potentially setting a framework for future innovations in machine vision tasks.