LawinASPP: Multi-scale Attention for Segmentation
- LawinASPP architecture is a multi-scale decoder for semantic segmentation that combines large window attention with spatial pyramid pooling to capture both local and global context.
- It fuses multi-resolution features from hierarchical Vision Transformers to enhance segmentation accuracy while reducing computational cost.
- The design achieves state-of-the-art mIoU results (e.g., 84.4% on Cityscapes) through adaptive pooling and efficient multi-head attention mechanisms.
LawinASPP architecture is a multi-scale decoder head for semantic segmentation Vision Transformers (ViTs) that integrates a novel large window attention mechanism with spatial pyramid pooling. Developed to efficiently capture multi-scale contextual information while maintaining computational economy, LawinASPP serves as the decoder in the Lawin Transformer framework, which establishes new state-of-the-art performance across standard semantic segmentation benchmarks (Yan et al., 2022).
1. Architectural Overview
LawinASPP operates as the decoder within a semantic segmentation pipeline whose encoder is a hierarchical ViT (e.g., MiT or Swin). The encoder provides four multi-resolution feature maps: where each has shape , with , .
The features are upsampled to the spatial resolution of , concatenated along the channel axis, and projected via a convolution to produce: This tensor is processed by LawinASPP, yielding 0. Subsequently, 1 is upsampled to match 2's resolution, concatenated with 3, passed through another 4 projection, and then through the final classifier.
2. Large Window Attention Mechanism
A key innovation in LawinASPP is large window attention, which enables each local patch in 5 to attend to a substantially broader contextual window with low overhead. The input 6 is partitioned into non-overlapping patches (“query patches”) of size 7, with 8.
For each patch:
- The query 9 is defined as 0.
- The context 1 is a surrounding region of spatial size 2, 3, with ratio 4.
To manage computational growth, context is pooled by an average pooling (downsampling factor 5), producing 6. Multi-head attention is employed with 7 heads. Each head applies a position-mixing MLP across the spatial patch dimension: 8
The processed features are re-concatenated and projected. Per-head queries, keys, and values are computed, and multi-head attention per patch is assembled as: 9 Aggregated output is: 0
3. Multi-Scale Contextual Representation
Multi-scale context extraction is achieved by repeating large window attention for 1. This produces feature maps with effective receptive fields of 2, 3, and 4. This multi-scale approach allows the representation of both local and global context crucial for precise semantic segmentation.
4. Spatial Pyramid Pooling Integration
LawinASPP integrates the multi-scale large window attention outputs via a spatial pyramid pooling design. Its branches are:
- Shortcut: identity mapping of 5.
- LWA with 6.
- LWA with 7.
- LWA with 8.
- Global pooling: global average pooling of 9, followed by 0 convolution and upsampling to match spatial size.
Each produces a feature map of shape 1; these are concatenated to form a 2 tensor. A 3 convolution reduces channel dimension to 4, followed by BatchNorm and activation (GELU) produces output 5: 6 where 7 is the activation function.
5. Hyperparameters and Implementation
Selected hyperparameters are:
- Patch size 8.
- Ratios 9.
- Number of heads 0.
- Channels: for MiT‑B3, 1; for MiT‑B5 or Swin‑L, 2.
Pseudocode for LawinASPP and large window attention is:
9
6. Computational Complexity and Empirical Results
For feature maps 3 of size 4, standard local window attention incurs: 5 Lawin attention modifies this to: 6 Since 7, the increase is marginal.
Empirically, on 8 inputs with MiT‑B3 backbone:
- SegFormer‑B3 w/ ASPP: ~79.0 GFlops
- Lawin‑B3: 61.7 GFlops This yields ~22% FLOP reduction with mIoU on ADE20K improving from 48.7% to 49.9%.
7. Benchmark Performance and Applications
LawinASPP as part of Lawin Transformer achieves state-of-the-art mIoU:
- 84.4% on Cityscapes
- 56.2% on ADE20K
- Competitive results on COCO-Stuff
This demonstrates the architecture’s capacity to enhance both segmentation accuracy and efficiency by leveraging large window attention within a multi-scale spatial pyramid pooling framework (Yan et al., 2022).