SE-ASPP: Efficient Multiscale Feature Modeling

• SE-ASPP is an architectural technique that integrates squeeze-and-excitation with atrous spatial pyramid pooling to efficiently capture multiscale contextual features.
• It employs parallel or sequential atrous convolutions with varying dilation rates for spatial context aggregation, followed by SE blocks for adaptive channel recalibration.
• This module enhances performance in segmentation tasks like real-time scene parsing, medical imaging, and remote sensing by balancing accuracy with computational efficiency.

Squeeze-and-Excitation Atrous Spatial Pyramid Pooling (SE-ASPP) is an architectural technique designed for efficient multiscale feature representation and adaptive channel attention in deep convolutional networks, primarily for dense prediction tasks such as semantic segmentation. By fusing Squeeze-and-Excitation (SE) attention with variants of Atrous Spatial Pyramid Pooling (ASPP), SE-ASPP modules adaptively recalibrate multi-dilation feature maps and deliver a variable receptive field with constrained computational overhead. SE-ASPP has been instantiated in various forms to meet the efficiency and accuracy demands in real-time scene parsing, medical image segmentation, and large-scale remote sensing pipelines (Zhang et al., 2023, Wang et al., 2021, Song, 2023).

1. Architectural Foundations and SE-ASPP Formulation

SE-ASPP modules systematically integrate two core operations: parallel or sequential atrous (dilated) convolutions for spatial context aggregation, and squeeze-and-excitation blocks for channel-wise recalibration.

A canonical SE-ASPP (as in Cross-CBAM (Zhang et al., 2023)) comprises:

• Atrous branches: Two branches operate in parallel over an input tensor $X$; the first applies a $1\times1$ convolution with dilation $r_1=1$ (local context), the second a $3\times3$ convolution with dilation $r_2=3$ (larger field). Both yield output tensors $Y_1$, $Y_2$, each with $C$ channels (typically $C=256$).
• Fusion via addition: The two outputs are summed: $Y_{\text{sum}} = Y_1 + Y_2$.
• SE attention: $1\times1$0 is passed through a squeeze step (global average pooling across $1\times1$1), followed by excitation (two-layer fully-connected bottleneck with reduction $1\times1$2, typically 16), yielding channel weights $1\times1$3:

$1\times1$4

The final reweighted feature is $1\times1$5, where $1\times1$6.

• Output fusion: $1\times1$7 and $1\times1$8 are concatenated, then projected via $1\times1$9 convolution back to $r_1=1$0 channels.

In alternative instantiations (PLU-Net (Song, 2023)), the ASPP submodule is realized as four parallel depthwise separable atrous convolutions with increasing rates (e.g., $r_1=1$1), their outputs concatenated and then passed through pointwise convolution and an SE block.

2. Mechanisms for Multiscale Context Aggregation

SE-ASPP advances conventional ASPP by:

• Reduced branch count: Real-time variants minimize the number of parallel atrous convolutions (e.g., two vs. four in standard ASPP) to control inference cost (Zhang et al., 2023).
• Receptive field diversity: Branches with differing dilation rates capture both fine-grained and broad contextual features. For example, $r_1=1$2, $r_1=1$3 achieves a 7$r_1=1$47 effective receptive field, while $r_1=1$5, $r_1=1$6 is fully local (Zhang et al., 2023).
• Dense spatial aggregation (alternative SE-ASPP forms): In PLU-Net, four branches, each with a 3$r_1=1$73 depthwise separable atrous convolution, deliver fields $r_1=1$8 (Song, 2023). A dense cascade of dilated convolutions (DenseDDSSPP) further amplifies coverage (Mahara et al., 2024).

This module design allows the network to process visual patterns at multiple scales simultaneously, which is indispensable for tasks with high spatial variability, such as biomedical or remote sensing datasets.

3. Squeeze-and-Excitation for Channel Recalibration

The SE block implements global context-sensitive channel gating via two key stages:

• Squeeze: Global average pooling compresses each channel to a scalar, yielding $r_1=1$9
• Excitation: A two-layer MLP (bottlenecked by $3\times3$0) applies non-linear channel mixing; the sigmoid-activated output produces attention weights $3\times3$1.
• Scaling: Channel-wise multiplication gates $3\times3$2 by $3\times3$3.

When appending SE to the ASPP pathway, the recalibration acts on the aggregated multi-dilation features, yielding feature maps that dynamically prioritize semantically salient channels and suppress redundant activations. The effect is empirically to amplify the contribution of scale-relevant features and mitigate over-representation of particular receptive fields (Zhang et al., 2023, Song, 2023).

4. Comparative Module Schematics

Source	Atrous Paths	SE Position	Feature Fusion	Typical Dilation Rates
Cross-CBAM	2 (par., 1×1/3×3)	After sum	$3\times3$4 concat, $3\times3$5 conv	1, 3
PLU-Net	4 (par., 3×3 sep.)	After concat & p.w.	SE on total merged features	1, 6, 12, 18
SAR-U-Net	4 (par., classic)	Pre-ASPP (encoder)	ASPP separate from SE in pipeline	1, 6, 12, 18
Mahara et al.	Dense cascade	After concat	Dense stacked dilation, then SE	6, 12, 18, 24

This table highlights architectural differences in SE-ASPP instantiations across key publications (Zhang et al., 2023, Song, 2023, Wang et al., 2021, Mahara et al., 2024).

5. Computational Efficiency and Parameterization

SE-ASPP targets real-time and lightweight regimes by controlling FLOPs and parameters through structural simplifications:

• Branch count: Two-branch SE-ASPP (Cross-CBAM) incurs ≈11.3G FLOPs, 12.2M params (input $3\times3$6, $3\times3$7). Increasing $3\times3$8 to 512 raises cost, but accuracy gains saturate (Zhang et al., 2023).
• Depthwise separable convolutions: Used in PLU-Net and remote-sensing variants to minimize computation while maintaining multiscale representational power (Song, 2023, Mahara et al., 2024).
• SE block overhead: Controlled by reduction ratio $3\times3$9 (commonly 16), bottlenecking per-channel attention cost.

Relative to standard ASPP, SE-ASPP configurations can achieve $r_2=3$0 - $r_2=3$1 points mIoU gain or $r_2=3$2–$r_2=3$3 points F1/IOU gain at sub-15G FLOPs overheads (Zhang et al., 2023, Song, 2023, Mahara et al., 2024).

6. Empirical Performance and Application Domains

SE-ASPP has been evaluated extensively in both scene parsing and medical imaging:

• Scene segmentation: In Cross-CBAM on Cityscapes, SE-ASPP enables $r_2=3$4 mIoU at $r_2=3$5 FPS and $r_2=3$6 mIoU at $r_2=3$7 FPS on GTX 1080Ti (Zhang et al., 2023).
• Medical image segmentation: PLU-Net’s SE-ASPP achieves up to $r_2=3$8 Dice improvement over vanilla U-Net, with only $r_2=3$9 more parameters (Song, 2023). SAR-U-Net integrates both SE and ASPP at multiple U-Net blocks, delivering state-of-the-art Dice on liver segmentation benchmarks ($Y_1$0–$Y_1$1) (Wang et al., 2021).
• Remote sensing: DeepLabV3+ augmented with a dense SE-ASPP (DenseDDSSPP + SE) improves IOU by $Y_1$2–$Y_1$3 points relative to the DeepLabV3+ ASPP baseline, also outperforming several segmentation networks on Massachusetts and DeepGlobe datasets (Mahara et al., 2024).

7. Variants and Generalizations

Several design generalizations arise in the literature:

• Dense Cascading (DenseDDSSPP): Instead of parallel dilations, a stacked, densely connected chain with progressive dilation and SE reweighting further amplifies multiscale expressiveness (Mahara et al., 2024).
• Flexible insertion point: SE blocks may operate after ASPP (PLU-Net, DeepLab+) or pre-ASPP/encoder (SAR-U-Net), with distinct architectural and performance tradeoffs.
• Depthwise separable convolution: Favored for extremely lightweight networks (PLU-Net), enabling multiscale aggregation with minimal arithmetic penalty (Song, 2023).

A plausible implication is that future SE-ASPP architectures may hybridize dense and parallel dilation strategies, optimize SE bottleneck ratios, or incorporate cross-attention to further elevate real-time segmentation accuracy under resource constraints.

---

Squeeze-and-Excitation Atrous Spatial Pyramid Pooling thus forms a core building block for modern, resource-efficient, and adaptive multiscale feature extractors in both 2D natural image and volumetric medical or satellite segmentation pipelines (Zhang et al., 2023, Wang et al., 2021, Song, 2023, Mahara et al., 2024).