---
title: Frequency-attentive Feature Pyramid Prediction Network
url: https://www.emergentmind.com/topics/frequency-attentive-feature-pyramid-prediction-network-ffpn
type: topic
---

# Frequency-attentive Feature Pyramid Prediction Network

The Frequency-attentive Feature Pyramid Prediction Network (FFPN) is a detect-to-segment (DTS) framework that integrates frequency analysis of object contours via Fourier descriptors with a multi-level feature pyramid network, specifically designed for real-time and highly accurate ultrasound (US) image segmentation. FFPN introduces a frequency-aware mechanism for encoding contours and distributing their representation across feature pyramid levels, combined with a dedicated contour refinement module. This architecture addresses issues in contour encoding and leverages the strengths of modern FPN-based detection and segmentation backbones, achieving state-of-the-art accuracy and efficiency across several US benchmarks [2308.13790].

## 1. Fourier Descriptor Contour Encoding

Object contours are parameterized as ordered sequences of points $\{(x_k, y_k)\}_{k=0}^{T-1}$, which are combined into complex-valued sequences $Z_k = x_k + i y_k$. The contour is encoded via discrete Fourier descriptors:

\[
A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N
\]

The real and imaginary parts of $A_n$ are separated as $a_n$ and $b_n$ for $x$-coordinates, and $c_n$, $d_n$ for $y$-coordinates. Further, the contour is expressed in real-valued "ellipse" form:

\[
x_N(t) = L_x + \sum_{n=1}^N \bigl( a_n\sin\frac{2\pi n t}{T} + b_n\cos\frac{2\pi n t}{T} \bigr)
\]
\[
y_N(t) = L_y + \sum_{n=1}^N \bigl( c_n\sin\frac{2\pi n t}{T} + d_n\cos\frac{2\pi n t}{T} \bigr)
\]
where $(L_x, L_y)$ is the contour center, $N$ is the maximum Fourier order (set to 7), and $t=0,\ldots,T-1$.

A key design is the frequency-aware grouping of Fourier coefficients: lower-order (low-frequency, global contour) coefficients are assigned to semantically rich, coarse FPN levels; higher-order (high-frequency, fine contour) coefficients are routed to high-resolution spatial levels. Specifically, coefficients are partitioned among $L=3$ FPN levels using

\[
K = \left\lceil \frac{N}{L} \right\rceil, \qquad \ell(n) = L - \left\lfloor\frac{n-1}{K}\right\rfloor
\]
so that each Fourier order $n$ is predicted at its assigned level $\ell(n)$, and all four coefficient streams ($a_n, b_n, c_n, d_n$) are concatenated per level.

## 2. Feature Pyramid Network and Prediction Heads

The backbone (ResNet-50 or similar) generates feature maps $\{C_3, C_4, C_5\}$ with strides $\{8, 16, 32\}$. Feature pyramid levels $P_3$, $P_4$, and $P_5$ are constructed using $1\times1$ convolutions, up-sampling, and $3\times3$ smoothing convolutions, each with 256 channels. To match resolutions, $P_1$ and $P_2$ are upsampled versions of $P_3$ and $P_4$, all aligned to stride 8.

On each pyramid level $P_\ell$, a dedicated detection head composed of four 3×3 Conv–BN–ReLU layers and three 3×3 Conv output layers predicts:

- Fourier offsets $\Delta F_\ell \in \mathbb{R}^{(4 n_\ell) \times H \times W}$ for $n_\ell$ coefficients at level $\ell$
- Center location offsets $\Delta L_\ell \in \mathbb{R}^{2 \times H \times W}$
- Object classification logits $S_\ell \in \mathbb{R}^{1 \times H \times W}$

Per-level predictions are merged by concatenating the Fourier offsets and averaging the center-location and classification logits:

\[
\Delta F = \Vert_{\ell=1}^3 \Delta F_\ell,\quad 
\Delta L = \frac{1}{3}\sum_{\ell=1}^3 \Delta L_\ell, \quad
S = \frac{1}{3}\sum_{\ell=1}^3 S_\ell
\]

Overall, this facilitates distributed learning of global and local contour features while maintaining spatial-semantic trade-offs characteristic of FPNs.

## 3. Contour Sampling Refinement (CSR) Module

The Contour Sampling Refinement (CSR) module refines coarse contour predictions using local feature aggregation and regression:

1. **Contour Proposal Aggregation**: The top $n$ scoring contours from FFPN, all corresponding to the same object, are grouped into a cluster $C_c$ if their pairwise IoU $>0.7$. Clusters are averaged in Fourier space to yield merged contours $M_c$.
2. **Local Feature Sampling**: For each $M_c$, $k+1$ ($k=16$) points along the contour and the object center are uniformly sampled. At each $(x_t, y_t)$, small neighborhood features are bilinearly interpolated from the fused FPN features ($F_{\text{csr}} = \mathrm{Concat}(P_1,P_2,P_3)$) using RoI-Align.
3. **Refinement Head and Regression**: The $k+1$ feature vectors are concatenated and input to a three-layer MLP, which regresses residual offsets for the Fourier coefficients and center. These are added to the merged contour coefficients, yielding the final refined coefficients.

This one-shot refinement explicitly gathers local boundary information, enhancing the accuracy of predicted object outlines.

## 4. Loss Functions and Training Regimen

FFPN employs a compound loss integrating four objectives:

1. **Localization Loss**: SmoothL1 loss on center offsets $\mathcal{L}_\mathrm{Loc}$.
2. **Fourier Regression Loss**: SmoothL1 loss for each Fourier coefficient $\mathcal{L}_\mathrm{Fou}$.
3. **Contour IoU Loss**: Defined as $1 - (\mathrm{PolarIoU} \times \mathrm{BoxIoU})$, blending ray-wise contour match (PolarMask) and standard bounding-box IoU $\mathcal{L}_\mathrm{Con}$.
4. **Classification Loss**: $0.25\times$ binary cross-entropy plus $0.75\times$ focal loss $\mathcal{L}_\mathrm{Cls}$:
   \[
   \mathcal{L}_\mathrm{Cls} = \alpha\,\mathrm{BCE}(S,S^{\mathrm{gt}})+\beta\,\mathrm{Focal}(S,S^{\mathrm{gt}}),\ \ \alpha=0.25,\,\beta=0.75
   \]

The total objective is:
\[
\mathcal{L} = \lambda_1\,\mathcal{L}_\mathrm{Cls} + \lambda_2\,\mathcal{L}_\mathrm{Loc} + \lambda_3\,\mathcal{L}_\mathrm{Fou} + \lambda_4\,\mathcal{L}_\mathrm{Con}
\]
with $\lambda_i=1$ for all components. The same regime is applied to both the main FFPN outputs and the CSR head refinements.

The network is trained end-to-end for 200 epochs on $416 \times 416$ inputs with Adam optimizer ($\mathrm{lr}=10^{-3}$), on a single NVIDIA 2080Ti GPU.

## 5. Comparative Experimental Evaluation

FFPN and its refinement variant FFPN+CSR (FFPN-R) are benchmarked on three datasets: 2CH (LV), FH (head), and Camus (multi-class). Reported metrics include Dice, IoU, Hausdorff Distance (HD), Conformity, GPU memory, and FPS. A representative selection is given below:

| Method           | Dice % (2CH) | HD px (2CH) | Mem (GB) | FPS  |
|------------------|-------------|-------------|----------|------|
| U-Net            | 88.95 ± 4.95| 22.41 ± 18.8| 1.33     | 19.7 |
| DeepLabV3        | 88.92 ± 5.06| 19.98 ± 12.1| 0.49     | 15.8 |
| PolySnake        | 86.50 ± 6.05| 22.75 ± 13.1| 0.17     | 14.7 |
| CPN              | 87.62 ± 6.23| 23.81 ± 15.8| 0.20     | 27.2 |
| FFPN             | 88.16 ± 5.97| 21.10 ± 14.3| 0.20     | 41.5 |
| FFPN-R           | 89.08 ± 5.24| 19.76 ± 12.5| 0.23     | 33.5 |

FFPN improves Dice over PolySnake by $+1.66\%$ and over CPN by $+0.54\%$, while reducing HD by $1.65$ px and $2.71$ px, respectively. Inference throughput is the highest among all tested DTS and U-shape models (41.5 FPS, 0.2 GB RAM usage). On the Camus dataset, FFPN-R yields the best mean Dice (88.72%) and mean HD (19.44 px) among DTS methods, competitive with DeepLabV3.

Qualitative assessment demonstrates that FFPN and FFPN-R produce crisper, more accurate contours, particularly in regions with weak ultrasound boundaries.

## 6. Impact and Methodological Significance

FFPN introduces three core methodological advances:

- Frequency-aware assignment of Fourier descriptors to appropriate FPN levels based on contour granularity.
- Joint prediction of grouped Fourier coefficients via level-specific regression heads.
- A one-shot contour refinement procedure (CSR) that incorporates local RoI-aligned features explicitly tailored for boundary precision.

This design achieves an advantageous trade-off between segmentation accuracy, computational speed (over 40 FPS), and memory usage (under 0.25 GB). Experimental results indicate that FFPN outperforms prior DTS approaches and several pixel-based segmentation baselines on challenging ultrasound benchmarks.

The architecture, although developed for ultrasound segmentation, can generalize to other object detection or instance segmentation tasks where real-time, high-precision contour encoding is required [2308.13790].

Source: https://www.emergentmind.com/topics/frequency-attentive-feature-pyramid-prediction-network-ffpn