Height-Driven Attention Networks (HANet)

• HANet are lightweight attention modules that exploit height-aware spatial priors to enhance semantic segmentation and 3D object detection.
• They dynamically weight features using dual 1D convolutions and vertical pooling, aligning network bias with structured scene statistics.
• Empirical studies on benchmarks like Cityscapes and BEV show consistent mIoU gains and improved detection accuracy with minimal overhead.

Height-Driven Attention Networks (HANet) are lightweight attention modules that exploit height- (or vertical-) aware spatial priors in structured environments, with notable applications in semantic segmentation of urban scenes and 3D object detection from multi-view inputs. HANet variants have been empirically validated to yield consistent accuracy gains with minimal computational overhead across diverse neural segmentation and detection backbones, by learning to selectively emphasize features based on row (in 2D) or slice (in 3D) location. Two main operational paradigms exist: (1) 2D image-based HANet, which dynamically predicts per-channel, per-height attention for urban-scene segmentation by pooling vertical context and applying learned 1D convolutions, and (2) 3D slice-based mechanisms, exemplified by the BEV-SAN extension for Bird’s-Eye-View (BEV) object detection, which aggregates multi-view features within adaptively sampled height slices. Both incarnations demonstrate that height, often overlooked by isotropic convolutional attention, is a dominant source of semantic structure in visual perception tasks.

1. Motivation and Background

Urban-scene images and 3D BEV reconstructions exhibit strong vertical organization: semantic categories (e.g., road, building, sky) or object types (e.g., barrier, truck) are concentrated at characteristic vertical locations. Empirical analysis on Cityscapes demonstrates that, for the lower third of an image, 87.9% of pixels correspond to road, whereas the upper third is composed of 47.8% building and 35.4% vegetation, with an order-of-magnitude reduction in class entropy compared to full-image context (Choi et al., 2020). This decreases per-pixel label uncertainty given vertical position, a property not exploited by conventional channel/spatial attention (e.g., SENet, CBAM), which ignore directionality in pooling and gating. Likewise, in BEV detection, objects of interest occupy distinct physical height layers, but flattening the BEV space along height loses this stratification (Chi et al., 2022). HANet and its slice-attention variants address this by introducing explicit height-driven, row-dependent attention, thereby aligning network inductive bias with environmental statistics.

2. Architectural Frameworks

2.1 2D Image HANet for Semantic Segmentation

The canonical HANet module operates as a plug-in parallel branch in encoder–decoder segmentation architectures. For two feature maps, a lower-level $X_\ell \in \mathbb{R}^{C \times H_\ell \times W}$ and a higher-level $X_h \in \mathbb{R}^{C \times H_h \times W_h}$, HANet computes a per-row, per-channel attention matrix $A \in [0,1]^{C \times H_h}$. The architectural steps are:

1. Width-wise pooling: Collapse $X_\ell$ along width to form $$Z(c,h) = \frac{1}{W} \sum_{w=1}^{W} X_\ell(c, h, w)$$.
2. Positional Encoding: Add sinusoidal vertical encodings

$$\mathrm{PE}(h,2i) = \sin(h/10000^{2i/C}),\quad \mathrm{PE}(h,2i+1) = \cos(h/10000^{2i/C})$$

to $Z$, yielding $\widetilde Z$.

1. Dual 1D convolutions: Apply two small 1D convolutions (downstream and upstream) to $\widetilde Z$, fused and projected to a coarse attention map via sigmoid:

$$A_{\mathrm{coarse}}(c, h) = \sigma(\mathrm{Conv}^{1 \times 1}(G_{\downarrow}(c,h)+G_{\uparrow}(c,h)))$$

2. Upsampling and application: Bilinearly interpolate $X_h \in \mathbb{R}^{C \times H_h \times W_h}$0 to $X_h \in \mathbb{R}^{C \times H_h \times W_h}$1, and apply as multiplicative mask to $X_h \in \mathbb{R}^{C \times H_h \times W_h}$2:

$X_h \in \mathbb{R}^{C \times H_h \times W_h}$3

This mechanism allows the model to directly learn and apply vertical spatial priors (Choi et al., 2020, Sharma, 2021).

2.2 Height-Slice Attention in 3D BEV Feature Construction

In BEV-SAN, height-driven attention is realized by explicitly constructing “slices” along the height dimension (Chi et al., 2022):

1. Slice definition: Partition the BEV volume ($X_h \in \mathbb{R}^{C \times H_h \times W_h}$4) into overlapping global slices $X_h \in \mathbb{R}^{C \times H_h \times W_h}$5 and narrower local slices $X_h \in \mathbb{R}^{C \times H_h \times W_h}$6.
2. LiDAR-guided sampling: For local slices, compute histogram of LiDAR return heights, normalize to a pmf $X_h \in \mathbb{R}^{C \times H_h \times W_h}$7, form cumulative distribution $X_h \in \mathbb{R}^{C \times H_h \times W_h}$8, and segment at quantiles such that each local slice contains an equal fraction of total LiDAR points.
3. Feature aggregation: Use standard lift–splat–shoot or depth-distribution pipeline to project multi-view image features and depth distributions into view-frustum volumes; then back-project into BEV and sum within each slice:

$X_h \in \mathbb{R}^{C \times H_h \times W_h}$9

1. Attention fusion: Feed global and local BEV slice features into a channel-wise SE-attention module and dual-branch transformer, then synthesize for downstream detection.

3. Mathematical Formulation

3.1 2D Image-based HANet

Formalizing the semantic segmentation HANet block (Choi et al., 2020, Sharma, 2021):

• Width pooling:

$A \in [0,1]^{C \times H_h}$0

• Add positional encoding:

$A \in [0,1]^{C \times H_h}$1

• Apply dual 1D convs with nonlinearity:

$A \in [0,1]^{C \times H_h}$2

• Fuse and sigmoid:

$A \in [0,1]^{C \times H_h}$3

• Upsample and rescale:

$A \in [0,1]^{C \times H_h}$4

3.2 BEV-Slice Attention

Key formulas for BEV slice aggregation (Chi et al., 2022):

• LiDAR-guided slice boundary for local slice $A \in [0,1]^{C \times H_h}$5:

$A \in [0,1]^{C \times H_h}$6

• Aggregated BEV slice features (per-slice, per-channel, per-XY grid):

$A \in [0,1]^{C \times H_h}$7

4. Practical Integration and Computational Cost

HANet is architecturally agnostic and minimally invasive. For 2D segmentation, HANet modules are “hooked” after any desired feature map with no modification to the backbone or decode head. For instance, in DeepLabv3+ (ResNet-101), HANet is inserted in parallel with skip connections; the WASP module (replacing ASPP) does not change HANet integration (Sharma, 2021). Empirical results show that HANet adds at most 1–2M parameters (<3% of total) and <0.05 TFLOPs per forward pass on Cityscapes, with essentially unchanged training time per epoch (Choi et al., 2020, Sharma, 2021). The BEV-SAN instance computes slice features via efficient batched operations and attention fusion, with the LiDAR-guided quantile selection requiring a one-time dataset scan (Chi et al., 2022).

5. Empirical Results and Ablations

5.1 Segmentation Performance

Adding HANet to DeepLabv3+ (ResNet-101, Cityscapes val) yields consistent ~1–3% absolute mIoU improvements. Typical quantitative results (Sharma, 2021): | Model | mIoU (%) | Road | Wall | Fence | Bus | |------------------------------|---------:|-----:|-----:|------:|----:| | DeepLabv3+ (baseline) | 77.8 | 98.4 | 54.8 | 58.8 |77.9 | | DeepLabv3+ + HANet | 80.9 | 98.6 | 60.8 | 65.7 |91.6 | | DeepLabv3+ + HANet + WASP | 81.0 | 98.4 | 63.4 | 63.9 |92.4 |

Key per-class gains are observed for small/mid-frequency classes (fence: +6.9%, bus: +13.7%, wall: +6.0%) (Sharma, 2021).

On the Cityscapes test set, single-model ResNeXt-101+HANet (pretrained) achieves 83.2% mIoU, leading against prior art (Choi et al., 2020). All major backbones (ShuffleNetV2, MobileNetV2, ResNet-50, ResNet-101) benefit, with ΔmIoU ≈ +0.7% to +1.3%.

5.2 BEV Object Detection

BEV-SAN demonstrates improved exploitation of height structure, as local slices, selected with LiDAR-guided quantile boundaries, concentrate on informative height bands—barriers at low heights, trucks at high. LiDAR-guided sampling outperforms uniform slicing in detection accuracy (Chi et al., 2022).

5.3 Ablation Studies

Systematic ablations corroborate the height-driven hypothesis:

• Largest segmentation gains occur in upper/lower vertical regions with lowest class entropy (Choi et al., 2020).
• Sinusoidal positional encoding at the second 1D conv is optimal.
• Adding HANet at 4 backbone stages is preferable to additional ASPP-level gating.
• HANet’s row-channel attention maps cluster by semantic strata: bottom for “road,” top for “sky,” etc.

6. Applications and Design Recommendations

HANet is designed as a general, plug-and-play module for tasks where vertical (or height) priors correlate with semantics. Primary applications are:

• Urban-scene 2D semantic segmentation (Cityscapes, BDD100K), yielding improvement for both mobile and large backbones (Sharma, 2021, Choi et al., 2020).
• Multi-view 3D BEV detection, focusing attention on stratified object distributions by height (Chi et al., 2022).

Recommended design choices include:

• Height bins/coarse rows: $A \in [0,1]^{C \times H_h}$8
• Reduction ratio: $A \in [0,1]^{C \times H_h}$9
• Pooling: width-wise average pooling preferred
• Three 1D convs (kernel size = 3)
• Sinusoidal positional encoding at conv2 (Choi et al., 2020)

HANet can be integrated with any encoder-decoder network; no changes to backbone weights are necessary. For BEV tasks, union of global and informed local slices maximizes vertical discriminability.

7. Context, Impact, and Future Directions

Height-Driven Attention Networks validate that exploiting environmental vertical regularities yields measurable improvements in structured scene understanding. The approach complements task heads and contextual modules by providing an orthogonal learning signal: spatial attention conditioned on absolute row or height, a property that standard attention does not natively incorporate. HANet’s empirical robustness, low computational burden, and agnostic “add-on” design position it as a practical baseline for spatially structured vision problems (Choi et al., 2020, Sharma, 2021, Chi et al., 2022).

A plausible implication is that directions for future work include generalizing height-driven attention to other structured priors (e.g., radial for panoramic images), learning height stratification end-to-end (vs. analytic or LiDAR-guided quantiles), or fusing height-driven attention with more sophisticated transformers. Research has consistently shown that vertical position is a strong prior in visual reasoning, and height-driven attention modules operationalize this prior effectively.