Frequency-attentive Feature Pyramid Prediction Network

• The paper introduces a detect-to-segment framework that integrates Fourier descriptor-based contour encoding with a multi-level feature pyramid for precise ultrasound segmentation.
• It employs a frequency-aware mechanism to assign lower and higher frequency Fourier coefficients to coarse and fine FPN levels, enhancing contour representation.
• Comprehensive evaluations demonstrate that FFPN achieves superior Dice scores and inference speed, outperforming existing DTS and U-shape models on ultrasound benchmarks.

The Frequency-attentive Feature Pyramid Prediction Network (FFPN) is a detect-to-segment (DTS) framework that integrates frequency analysis of object contours via Fourier descriptors with a multi-level feature pyramid network, specifically designed for real-time and highly accurate ultrasound (US) image segmentation. FFPN introduces a frequency-aware mechanism for encoding contours and distributing their representation across feature pyramid levels, combined with a dedicated contour refinement module. This architecture addresses issues in contour encoding and leverages the strengths of modern FPN-based detection and segmentation backbones, achieving state-of-the-art accuracy and efficiency across several US benchmarks (Chen et al., 2023).

1. Fourier Descriptor Contour Encoding

Object contours are parameterized as ordered sequences of points $\{(x_k, y_k)\}_{k=0}^{T-1}$, which are combined into complex-valued sequences $Z_k = x_k + i y_k$. The contour is encoded via discrete Fourier descriptors:

$$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$

The real and imaginary parts of $A_n$ are separated as $a_n$ and $b_n$ for $x$-coordinates, and $c_n$, $d_n$ for $y$-coordinates. Further, the contour is expressed in real-valued "ellipse" form:

$Z_k = x_k + i y_k$0

$Z_k = x_k + i y_k$1

where $Z_k = x_k + i y_k$2 is the contour center, $Z_k = x_k + i y_k$3 is the maximum Fourier order (set to 7), and $Z_k = x_k + i y_k$4.

A key design is the frequency-aware grouping of Fourier coefficients: lower-order (low-frequency, global contour) coefficients are assigned to semantically rich, coarse FPN levels; higher-order (high-frequency, fine contour) coefficients are routed to high-resolution spatial levels. Specifically, coefficients are partitioned among $Z_k = x_k + i y_k$5 FPN levels using

$Z_k = x_k + i y_k$6

so that each Fourier order $Z_k = x_k + i y_k$7 is predicted at its assigned level $Z_k = x_k + i y_k$8, and all four coefficient streams ($Z_k = x_k + i y_k$9) are concatenated per level.

2. Feature Pyramid Network and Prediction Heads

The backbone (ResNet-50 or similar) generates feature maps $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$0 with strides $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$1. Feature pyramid levels $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$2, $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$3, and $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$4 are constructed using $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$5 convolutions, up-sampling, and $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$6 smoothing convolutions, each with 256 channels. To match resolutions, $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$7 and $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$8 are upsampled versions of $$A_n = \frac{1}{T} \sum_{k=0}^{T-1} Z_k\, e^{-i 2\pi n k/T}, \quad n = -N,\ldots,N$$9 and $A_n$0, all aligned to stride 8.

On each pyramid level $A_n$1, a dedicated detection head composed of four 3×3 Conv–BN–ReLU layers and three 3×3 Conv output layers predicts:

• Fourier offsets $A_n$2 for $A_n$3 coefficients at level $A_n$4
• Center location offsets $A_n$5
• Object classification logits $A_n$6

Per-level predictions are merged by concatenating the Fourier offsets and averaging the center-location and classification logits:

$A_n$7

Overall, this facilitates distributed learning of global and local contour features while maintaining spatial-semantic trade-offs characteristic of FPNs.

3. Contour Sampling Refinement (CSR) Module

The Contour Sampling Refinement (CSR) module refines coarse contour predictions using local feature aggregation and regression:

1. Contour Proposal Aggregation: The top $A_n$8 scoring contours from FFPN, all corresponding to the same object, are grouped into a cluster $A_n$9 if their pairwise IoU $a_n$0. Clusters are averaged in Fourier space to yield merged contours $a_n$1.
2. Local Feature Sampling: For each $a_n$2, $a_n$3 ($a_n$4) points along the contour and the object center are uniformly sampled. At each $a_n$5, small neighborhood features are bilinearly interpolated from the fused FPN features ($a_n$6) using RoI-Align.
3. Refinement Head and Regression: The $a_n$7 feature vectors are concatenated and input to a three-layer MLP, which regresses residual offsets for the Fourier coefficients and center. These are added to the merged contour coefficients, yielding the final refined coefficients.

This one-shot refinement explicitly gathers local boundary information, enhancing the accuracy of predicted object outlines.

4. Loss Functions and Training Regimen

FFPN employs a compound loss integrating four objectives:

1. Localization Loss: SmoothL1 loss on center offsets $a_n$8.
2. Fourier Regression Loss: SmoothL1 loss for each Fourier coefficient $a_n$9.
3. Contour IoU Loss: Defined as $b_n$0, blending ray-wise contour match (PolarMask) and standard bounding-box IoU $b_n$1.
4. Classification Loss: $b_n$2 binary cross-entropy plus $b_n$3 focal loss $b_n$4:

$b_n$5

The total objective is: $b_n$6 with $b_n$7 for all components. The same regime is applied to both the main FFPN outputs and the CSR head refinements.

The network is trained end-to-end for 200 epochs on $b_n$8 inputs with Adam optimizer ($b_n$9), on a single NVIDIA 2080Ti GPU.

5. Comparative Experimental Evaluation

FFPN and its refinement variant FFPN+CSR (FFPN-R) are benchmarked on three datasets: 2CH (LV), FH (head), and Camus (multi-class). Reported metrics include Dice, IoU, Hausdorff Distance (HD), Conformity, GPU memory, and FPS. A representative selection is given below:

Method	Dice % (2CH)	HD px (2CH)	Mem (GB)	FPS
U-Net	88.95 ± 4.95	22.41 ± 18.8	1.33	19.7
DeepLabV3	88.92 ± 5.06	19.98 ± 12.1	0.49	15.8
PolySnake	86.50 ± 6.05	22.75 ± 13.1	0.17	14.7
CPN	87.62 ± 6.23	23.81 ± 15.8	0.20	27.2
FFPN	88.16 ± 5.97	21.10 ± 14.3	0.20	41.5
FFPN-R	89.08 ± 5.24	19.76 ± 12.5	0.23	33.5

FFPN improves Dice over PolySnake by $x$0 and over CPN by $x$1, while reducing HD by $x$2 px and $x$3 px, respectively. Inference throughput is the highest among all tested DTS and U-shape models (41.5 FPS, 0.2 GB RAM usage). On the Camus dataset, FFPN-R yields the best mean Dice (88.72%) and mean HD (19.44 px) among DTS methods, competitive with DeepLabV3.

Qualitative assessment demonstrates that FFPN and FFPN-R produce crisper, more accurate contours, particularly in regions with weak ultrasound boundaries.

6. Impact and Methodological Significance

FFPN introduces three core methodological advances:

• Frequency-aware assignment of Fourier descriptors to appropriate FPN levels based on contour granularity.
• Joint prediction of grouped Fourier coefficients via level-specific regression heads.
• A one-shot contour refinement procedure (CSR) that incorporates local RoI-aligned features explicitly tailored for boundary precision.

This design achieves an advantageous trade-off between segmentation accuracy, computational speed (over 40 FPS), and memory usage (under 0.25 GB). Experimental results indicate that FFPN outperforms prior DTS approaches and several pixel-based segmentation baselines on challenging ultrasound benchmarks.

The architecture, although developed for ultrasound segmentation, can generalize to other object detection or instance segmentation tasks where real-time, high-precision contour encoding is required (Chen et al., 2023).