NT-FPN: Noise-Resilient Feature Pyramid Network

• The paper demonstrates that NT-FPN reduces noise in multi-scale feature fusion, yielding a +3.1 AP gain over the baseline in small object detection.
• NT-FPN explicitly separates spatial and semantic preservation using InfoNCE-style contrastive losses to align lateral and fused features.
• Empirical results show that preserving detailed lateral features and stable representations significantly enhances small object detection performance.

Searching arXiv for the cited NT-FPN paper and closely related FPN/noise-resilience work. Noise-Tolerance Feature Pyramid Network (NT-FPN) is a feature-pyramid module introduced in “Revisiting DETR for Small Object Detection via Noise-Resilient Query Optimization” as part of the broader Noise-Resilient Query Optimization (NRQO) framework for small object detection (SOD) (Fang et al., 25 Jul 2025). It is proposed as a replacement or augmentation for a standard feature pyramid network (FPN), with the specific purpose of reducing the noise introduced during multi-scale feature fusion. In the formulation given for NRQO, NT-FPN is responsible for improving feature quality before query generation and downstream detection, with the stated goal of preserving spatial information from lateral features and semantic information from upper-level features while discouraging mismatched noisy fusion.

1. Problem setting and motivation

The motivation for NT-FPN is tied to the failure mode that the paper identifies in standard FPN-style fusion for small object detection (Fang et al., 25 Jul 2025). In a conventional FPN, top-down features are upsampled and fused with lateral features from lower levels. The paper argues that this process introduces two distinct problems: redundancy or noise from upsampling high-level features, and information loss from channel reduction in lateral features. For small objects, these effects are especially harmful because such objects already have limited pixels, weak semantics, low resolution, and high sensitivity to distortion.

Within this account, the central claim is that some small objects are missed because of noise introduced during feature fusion. NT-FPN is therefore motivated not by scale aggregation alone, but by the need to preserve information integrity during fusion. The paper states that the module is designed to preserve spatial information from lateral features and semantic information from upper-level features. This framing distinguishes NT-FPN from pyramid variants that treat fusion as intrinsically beneficial and optimize only the topology of multi-scale aggregation.

NT-FPN occupies the feature-extraction side of NRQO. The same framework also includes Pairwise-Similarity Region Proposal Network (PS-RPN), which improves query or anchor assignment quality using position and shape similarity and does so without the need for additional hyperparameters (Fang et al., 25 Jul 2025). In that division of labor, NT-FPN makes the pyramid features more reliable before query generation or detection heads use them.

2. Architectural formulation

The paper describes NT-FPN as a contrastive learning–style feature pyramid module (Fang et al., 25 Jul 2025). For each FPN level $i$ and minibatch sample $j$, the lateral feature $C_{i,j}$ is projected into spatial representations $$\{s_{i,j}^c, \hat{s}_{i,j}^c \in \mathbb{R}^{256}\}$$, while the upper-level feature $P_{i,j}$ is projected into spatial or semantic representations $$\{s_{i,j}^p, \hat{s}_{i,j}^p \in \mathbb{R}^{256}\}$$.

The stated training goal is twofold. First, it seeks to minimize the distance between the spatial information of fused and lateral features. Second, it seeks to align the semantic information of fused features with upper-level features. NT-FPN therefore does not merely perform top-down fusion followed by ordinary detection supervision; it imposes additional representation constraints on what the fused features should retain.

This arrangement implies a deliberate decomposition of the fusion problem into spatial preservation and semantic preservation. Spatial preservation is anchored in the relation between fused features and lateral inputs, while semantic preservation is anchored in consistency across pyramid levels. A plausible implication is that NT-FPN treats fusion noise as a representational mismatch problem rather than only as an architectural deficiency in interpolation or channel adaptation.

3. Spatial contrastive mechanism

NT-FPN defines a spatial contrastive loss to preserve spatial integrity during feature fusion (Fang et al., 25 Jul 2025). For the $x$-th FPN level and $y$-th image, the spatial query representation $q^s$ is taken from $s_{x,y}^p$, and the positive key $j$0 is taken from $j$1. These form a positive pair because they are expected to be spatially consistent.

Negative samples are drawn from other levels or images in the minibatch and are defined as

$j$2

The paper’s interpretation is that features from different images in the minibatch contain different spatial information and are therefore suitable negatives.

The spatial loss is

$j$3

where $j$4 is the temperature parameter. The overall spatial loss averages this quantity over levels and minibatch samples: $j$5

The paper characterizes this as an InfoNCE-style contrastive loss. Its function is to pull together spatially consistent lateral and fused features while pushing apart features from other images or levels. In the context of SOD, that objective is meant to preserve precise spatial localization, which the paper treats as critical for small objects.

4. Semantic contrastive mechanism and joint optimization

NT-FPN also defines a semantic contrastive loss to preserve semantic consistency across pyramid levels (Fang et al., 25 Jul 2025). For the $j$6-th level and $j$7-th image, the query $j$8 corresponds to the semantic representation $j$9, and the positive key $C_{i,j}$0 corresponds to $C_{i,j}$1. These form a positive pair because adjacent pyramid levels are expected to carry the same semantics.

Negative samples are semantic representations from other images: $C_{i,j}$2 The paper notes that the printed semantic-loss formula has formatting issues, but its intended meaning is again InfoNCE-like: semantic features from neighboring pyramid levels are encouraged to match, while semantic features from other images act as negatives. The overall semantic loss averages semantic alignment over adjacent pyramid levels and minibatch samples.

NT-FPN contributes to the overall NRQO objective through

$C_{i,j}$3

where $C_{i,j}$4 balances the NT-FPN losses against PS-RPN (Fang et al., 25 Jul 2025). The paper reports that $C_{i,j}$5 is sensitive enough that too small or too large values hurt performance, but that overall performance is not overly sensitive, and it sets $C_{i,j}$6 as best.

The same source does not describe a special inference procedure for NT-FPN. The stated implication is that NT-FPN shapes the feature pyramid during training, while at inference the detector uses the learned pyramid features normally. NT-FPN is therefore presented as a jointly trained feature-quality module rather than as a separate inference-time branch.

5. Empirical evidence

The paper’s ablation study isolates the contribution of NT-FPN within NRQO (Fang et al., 25 Jul 2025). The reported results are as follows.

Configuration	AP
Baseline without NT-FPN or PS-RPN	28.5
With NT-FPN only	31.6
With PS-RPN only	30.9
With both	32.1

From these numbers, NT-FPN alone improves AP by $C_{i,j}$7 over the baseline. The paper also states that NT-FPN improves $C_{i,j}$8, $C_{i,j}$9, and AP across object sizes. Its explicit conclusion is that NT-FPN reduces noise during multi-scale fusion, enhances small-object representation, and improves detection ability for small objects.

The qualitative evidence cited in the same study consists of heatmaps showing that NT-FPN helps better preserve object-focused activations, reduce missed small targets, and maintain spatial and semantic details. This suggests that the module’s effect is not limited to a scalar AP gain; it also changes the concentration and stability of the fused representation.

6. Distinction from standard FPN and relation to adjacent literature

A standard FPN typically performs lateral $$\{s_{i,j}^c, \hat{s}_{i,j}^c \in \mathbb{R}^{256}\}$$0 projection, top-down upsampling, elementwise addition, and sometimes smoothing (Fang et al., 25 Jul 2025). In that formulation, fusion is generally assumed to be beneficial, and the network is not explicitly optimized for noise robustness. NT-FPN differs in four ways stated by the paper: it explicitly models noise sensitivity, separates spatial and semantic preservation, uses contrastive objectives in addition to structural fusion, and targets small objects specifically. The paper therefore characterizes it as a noise-aware, representation-constrained FPN.

A common misconception is to treat NT-FPN as simply another fusion topology. The formulation in NRQO is narrower and more explicit than that: the module is defined by representational constraints on fused features, not only by the mechanics of upsampling and addition. It is also not described as an image denoising stage and does not introduce a special test-time path (Fang et al., 25 Jul 2025).

Related work shows that “noise-resilient” pyramid design has been pursued in several distinct senses. “Object Detection based on OcSaFPN in Aerial Images with Noise” proposes OcSaFPN, an octave convolution-based semantic attention feature pyramid network tailored to frequency-divided feature maps and noisy aerial imagery, with emphasis on Gaussian noise or multiplicative noise and on bridging semantic gaps between multi-level, multi-frequency features (Li et al., 2020). “You Should Look at All Objects” revisits FPN from an optimization perspective, attributing part of FPN’s failure mode to improper back-propagation paths and describing a remedy in terms of robust gradient propagation rather than literal sensor noise (Jin et al., 2022). “NS-FPN: Improving Infrared Small Target Detection and Segmentation from Noise Suppression Perspective” addresses infrared small target detection and segmentation by integrating low-frequency guided feature purification and spiral-aware feature sampling into an FPN structure (Yuan et al., 9 Aug 2025).

Taken together, these works indicate that pyramid-level robustness has multiple interpretations across the literature: resistance to sensor noise, resistance to fusion-induced feature corruption, resistance to optimization pathologies, and suppression of false alarms in cluttered infrared backgrounds. NT-FPN belongs specifically to the second category. Its defining claim is that small-object performance can be improved by explicitly constraining feature fusion to preserve spatial integrity and semantic consistency within the pyramid (Fang et al., 25 Jul 2025).