FAU-Net: Feature Pyramid Attention

• The paper demonstrates FAU-Net’s integration of additive attention gates and feature pyramid attention, significantly enhancing boundary delineation and robust feature fusion.
• The architecture leverages multi-scale convolutions and gating mechanisms to effectively combine local and global contextual information for superior segmentation performance.
• Empirical results on prostate MRI segmentation show that FAU-Net outperforms traditional U-Net variants with improved IoU and DSC metrics while maintaining parameter efficiency.

Feature Pyramid Attention (FAU-Net) refers to a family of encoder–decoder convolutional neural network architectures that incorporate feature pyramid-based attention mechanisms into the U-Net design for improved dense prediction tasks. This approach targets the simultaneous leveraging of local and global contextual information across multiple spatial scales and channels, with specific instantiations demonstrating state-of-the-art performance in medical image segmentation and related domains (Quihui-Rubio et al., 2023, Xu et al., 2024, Zhang et al., 2020).

1. Architectural Principles and Definitions

FAU-Net augments the canonical U-Net by integrating two principal attention mechanisms into the skip-connection pathways between the encoder and decoder. Specifically:

• Additive Attention Gates (AG) operate on lower-resolution skip connections (levels 2–4 in a four-level hierarchy). These gates, following the formulation of Oktay et al., compute a spatial attention map that filters encoder activations by context provided from the decoder, suppressing irrelevant background responses prior to skip-concatenation.
• Feature Pyramid Attention (FPA) is applied to the highest-resolution (level 1) skip connection. The FPA module fuses multi-scale convolutions (3×3, 5×5, 7×7) to assemble an attention map sensitive to both fine-scale and global context, promoting accurate boundary detection and feature refinement.

Mathematically, at encoder level $i$, the AG module computes: $$\begin{aligned} \theta_x(x^i) &= W_x x^i + b_x, \ \theta_g(g) &= W_g g + b_g, \ q &= \text{ReLU}(\theta_x(x^i) + \theta_g(g)), \ \psi &= w^T q + b_\psi, \ \alpha &= \sigma(\psi), \ \tilde{x}^i &= \alpha \odot x^i, \end{aligned}$$ where $g$ is a gating signal from the decoder, and $\odot$ denotes elementwise multiplication.

For the FPA block, high-resolution features $s$ are processed by parallel convolutions and progressively upsampled-additive integration: $$\begin{aligned} p_1 &= \mathrm{Conv}_{3\times3}(s), \ p_2 &= \mathrm{Conv}_{5\times5}(s), \ p_3 &= \mathrm{Conv}_{7\times7}(s), \ m_3 &= p_3, \ m_2 &= p_2 + \mathrm{Up}_2(m_3), \ m_1 &= p_1 + \mathrm{Up}_2(m_2), \ A &= \sigma(\mathrm{Conv}_{1\times1}(m_1)), \ \tilde{s} &= A \odot s. \end{aligned}$$ (Quihui-Rubio et al., 2023)

2. Layerwise Network Anatomy and Workflow

The typical FAU-Net instantiation for 2D dense prediction (prostate MRI segmentation) employs the following block-wise progression (input $I\in\mathbb{R}^{1\times256\times256}$, $C=5$ classes, $F_0=16$ base filters):

Encoder:

• Level 1: Two 3×3 Conv+ReLU → FPA block → $\tilde{s}_1$ skip.
• Level 2: Two 3×3 Conv+ReLU → AG₂.
• Level 3: Two 3×3 Conv+ReLU → AG₃.
• Level 4: Two 3×3 Conv+ReLU → AG₄.
• Bottleneck: Two 3×3 Conv+ReLU.

Decoder:

• Each decoder block upsamples via nearest neighbor, concatenates with gated skip, applies two 3×3 Conv+ReLU layers.

Final head: $$\begin{aligned} \theta_x(x^i) &= W_x x^i + b_x, \ \theta_g(g) &= W_g g + b_g, \ q &= \text{ReLU}(\theta_x(x^i) + \theta_g(g)), \ \psi &= w^T q + b_\psi, \ \alpha &= \sigma(\psi), \ \tilde{x}^i &= \alpha \odot x^i, \end{aligned}$$0 convolution yields per-class logits, followed by channelwise softmax.

The model uses ≈2.16M parameters and omits explicit data augmentation in the cited implementation.

3. Empirical Performance and Benchmarking

On a prostate MRI segmentation benchmark (205 slices, 19 patients), FAU-Net demonstrates the following test set results:

Model	IoU (%)	DSC (%)	Loss
U-Net	70.76	80.00	0.0138
Dense U-Net	74.53	83.65	0.0225
Swin U-Net	75.24	83.91	0.0124
Attention U-Net	74.92	84.01	0.0114
Attention Dense U-Net	75.12	84.01	0.0211
FAU-Net (proposed)	75.49	84.15	0.0107
R2U-Net	76.60	85.30	0.0131
Att R2U-Net	76.89	85.42	0.0120

The FPA mechanism at level 1 yields improved delineation of difficult anatomic zones, with +0.5% IoU and +0.1% DSC over Attention U-Net (Quihui-Rubio et al., 2023).

FAU-Net-like strategies have also been validated for monocular depth estimation (Xu et al., 2024) and pneumonia detection (Zhang et al., 2020), with consistent evidence that pyramid attention enhances both global context modeling and local edge preservation.

4. Generalization to Broader Attention-Pyramid Schemes

Variants on the FAU-Net principle introduce more elaborate feature pyramid attention strategies:

• Dual-scale Channel Attention (DCAM) and Spatial Pyramid Attention Module (SPAM), as in PFANet, allow separate, multi-scale aggregation along channel and spatial dimensions respectively. DCAM couples global channel attention (GAP/GMP fusion) and local channel recalibration (pointwise convolutions), then multiplies and projects to recalibrate features. SPAM applies a spatial pyramid of 3×3 Conv+sigmoid attention blocks at multiple downsampling scales and merges multi-resolution attention maps.
• In FPAENet for pneumonia detection, Feature Enhancement Modules (FEM) combine 1×1, 3×3, and 5×5 Conv mixing and apply an attention map normalized via channel-spatial Softmax (Zhang et al., 2020).

A plausible implication is that the FAU-Net architecture is extensible to other dense prediction domains (semantic segmentation, surface normal regression, object detection) by suitable placement of channel and spatial pyramid attention modules.

5. Training Regimens and Objective Functions

In canonical FAU-Net, training employs categorical cross-entropy loss with Adam optimizer (lr=1e-4, batch size=6, no explicit augmentation, 145 epochs) (Quihui-Rubio et al., 2023). Key evaluation metrics are Dice Score Coefficient (DSC) and mean Intersection-over-Union (IoU).

Extensions under the pyramid attention framework incorporate domain-specific objective functions. For instance, scale-invariant gradient loss terms—penalizing multi-scale edge misalignment—are added in monocular depth settings (Xu et al., 2024). In object detection, focal loss and smooth-$$\begin{aligned} \theta_x(x^i) &= W_x x^i + b_x, \ \theta_g(g) &= W_g g + b_g, \ q &= \text{ReLU}(\theta_x(x^i) + \theta_g(g)), \ \psi &= w^T q + b_\psi, \ \alpha &= \sigma(\psi), \ \tilde{x}^i &= \alpha \odot x^i, \end{aligned}$$1 regression losses are typically combined (Zhang et al., 2020).

6. Limitations and Future Directions

Identified limitations include:

• Single-level (shallow) FPA deployment; deeper, multi-scale stacking of FPA blocks could yield additional gains, particularly for small object or fine structure segmentation.
• Dataset size constraints; network capacity (parameter count) relative to data volume can induce overfitting, especially on high-capacity transformer-based alternatives.
• Absence of explicit data augmentation or uncertainty-aware losses, which could further improve generalization and segmentation confidence estimation (Quihui-Rubio et al., 2023).

Future research avenues propose:

• Integrating FAU-Net architectures with Recurrent Residual U-Nets (R2U-Net), transformer-based backbones, or boundary-aware loss functions.
• Expanding cohort sizes, introducing robust augmentation regimes such as elastic deformations, and incorporating uncertainty estimation for clinical deployment.
• For semantic segmentation, adapting attention normalization (e.g., group convs per class) or combining DCAM/SPAM with edge-aware objectives to further sharpen class boundaries (Xu et al., 2024).

7. Relation to Broader Feature Pyramid and Attention Frameworks

FAU-Net family models are positioned alongside other pyramid-based attention frameworks (e.g., PFANet, FPAENet) that share the goal of adaptive multi-scale feature fusion. Key distinctions include the localization of attention modules (skip vs. bottleneck vs. head), the dimensionality of attention (spatial vs. channel vs. combined), and application-specific module designs (e.g., spatial pyramid in SPAM, FEM in FPAENet).

A distinguishing factor is that FAU-Net and its close relatives systematically embed multi-scale attention blocks at critical structural junctures (skip connections, top-down fusion, decoder concatenation) rather than confining attention to the encoder head or a single intermediate feature map. This architectural property is fundamental to their observed gains in localization and recall, especially where boundaries and fine structures are critical (Quihui-Rubio et al., 2023, Xu et al., 2024, Zhang et al., 2020).