Residual Bi-Fusion FPN Architecture

• The paper proposes a bidirectional fusion strategy with scale-specialized parallel paths and residual connections, achieving up to +6.8% AP improvement on benchmarks.
• The architecture employs modular CORE and BFM blocks with residual links for stable deep feature refinement and efficient integration with various CNN backbones.
• The method extends to vision-language pipelines through semantic gating, using cosine similarity to filter detections with matching text embeddings.

A Residual Bi-Fusion Feature Pyramid Network (Residual Bi-Fusion FPN, PRB-FPN) is a hierarchical, multi-path feature fusion module designed to address key shortcomings in conventional Feature Pyramid Networks (FPNs) for dense object detection. The approach jointly leverages bidirectional—top-down and bottom-up—fusion, architectural parallelism across anchor scales, and residual connections to enhance multi-scale semantic representation, particularly for small and complex objects. The PRB-FPN supports efficient integration with deep CNN backbones, modular extension to vision-language pipelines, and yields state-of-the-art performance under real-time and resource-constrained settings (Huang et al., 7 Nov 2025, Chen et al., 2020, Chen et al., 2019).

1. Architectural Foundations and Motivation

Classic FPNs aggregate features in a purely top-down manner, propagating high-level semantics to shallow layers via upsampling and lateral fusion (typically element-wise addition). However, this directionality has several critical limitations:

• Loss of spatial precision: Non-shift-invariant pooling erodes localization cues, impeding detection of small objects.
• Diminishing returns with deeper pyramids: Stacking more layers without bottom-up information recovery causes performance to plateau or degrade (Chen et al., 2019).
• Single-path bottleneck: Sequentially fusing all scales constrains specialization for objects of different sizes (Chen et al., 2020).

PRB-FPN addresses these issues via three design innovations:

1. Bidirectional Fusion: Integrates top-down semantic propagation with a bottom-up pathway that re-injects fine-grained localization signals from shallow backbone outputs.
2. Parallelism Across Anchor Regimes: Constructs independent fusion pipelines, each optimizing features for a specific anchor or object scale (small, medium, large), and merges only at final prediction heads.
3. Residual Connections: Residual links wrap all fusion blocks, facilitating stable deep feature refinement (“purification”) and supporting arbitrarily deep pyramids (Huang et al., 7 Nov 2025).

2. Mathematical Formulation of Bi-Fusion Blocks

The fusion structure of PRB-FPN employs two core modules: the bottom-up CORE and the top-down BFM, each with residual pathways.

Let $X_i$ denote the $i$-th backbone feature map, ordered from highest (deepest, lowest resolution) to lowest (shallowest, highest resolution). For each parallel path $j$ and pyramid level $k$:

• Bottom-Up CORE Module:

$$\mathrm{CORE}_k^{(j)} = \operatorname{ReLU}(W_{1\times 1} \cdot [U(\mathrm{CORE}_{k-1}^{(j)}), X_{4-k}, X_{3-k}]) + U(\mathrm{CORE}_{k-1}^{(j)})$$

where $U(\cdot)$ is $2\times$ upsampling ($\mathrm{CORE}_0^{(j)}$ initialized as zero), $[\,\cdot\,]$ denotes channel-wise concatenation, and $W_{1\times 1}$ is a learned $i$0 convolution tensor.

• Top-Down BFM Module:

$i$1

with $i$2 as $i$3 downsampling (as necessary), and $i$4 a learned $i$5 convolution.

Each path, indexed by $i$6, iterates through all $i$7 pyramid levels. Outputs are concatenated across paths at the prediction stage:

• Lead head: $i$8
• Auxiliary head: $i$9

These residual operations generalize to deeper or modular purification—multi-stage refinement—improving feature expressiveness and training robustness (Huang et al., 7 Nov 2025, Chen et al., 2020, Chen et al., 2019).

3. Parallel, Scale-Specialized Fusion Paths

PRB-FPN establishes three parallel, identical fusion pathways (for $j$0). Each pathway maintains independent CORE$j$1BFM stacks responsible for a fixed anchor size (i.e., small, medium, or large objects):

• Path independence: No early sharing or mixing of features occurs between paths, allowing scale-specialized representations to develop.
• Late fusion: Only after both CORE and BFM modules have completed do the outputs from all paths get concatenated and delivered to the appropriate prediction head.
• Task separation: Auxiliary heads operate on bottom-up CORE features (emphasizing localization), while lead heads use top-down BFM features (capturing context necessary for category-level discrimination and box regression).

This suggests the architecture balances specialization and representational power across object scales, aligning with empirical advantages shown on datasets with broad size variation (Huang et al., 7 Nov 2025, Chen et al., 2020).

4. Semantic Gating and Vision-Language Integration

In multi-modal detection pipelines (e.g., PRB-FPN-Net), language-derived semantic cues are incorporated in a strictly late-fusion manner:

• BERT and FastText models, with lemmatization, generate category embeddings $j$2 from text queries.
• After lead head predictions (candidate boxes and class logits), each visual-detected category embedding $j$3 from bounding box $j$4 is compared to $j$5 via cosine similarity.
• Only detections with $j$6 (threshold $j$7) are retained.

This semantic gating step prevents mismatched detections (e.g., spurious classes unrelated to the language context) without polluting the convolutional fusion pathways. The approach outperforms joint mid-level cross-modal feature fusion in efficiency and accuracy for the targeted task of tiny object detection (Huang et al., 7 Nov 2025).

5. Integration with CNN Backbones and Modularity

PRB-FPN is backbone-agnostic and can be attached to various CNN feature extractors (ELAN, MSP, CSP, ResNet, VGG, DenseNet):

• Backbone produces multi-level feature maps $j$8 or more, each adapted to uniform channel width via $j$9 convolutions as necessary.
• Each path uses these backbone outputs as CORE/BFM inputs, allowing plug-and-play integration.
• Extension to deeper pyramids, anchor-free detection, or additional tasks (segmentation, depth estimation) is direct by modifying the prediction heads (Chen et al., 2020, Chen et al., 2019).

Empirical results demonstrate consistent AP gains across CNN backbones, without hyper-specialization or parameter inflation (Huang et al., 7 Nov 2025, Chen et al., 2020).

6. Quantitative Performance and Empirical Analysis

The PRB-FPN architecture yields substantial improvements in average precision (AP), particularly for small objects, with competitive or superior efficiency:

Backbone	Params (M)	GFLOPs	COCO AP (%)	Small Object AP (%)	Notes
CSP + PRB-FPN	58.8	153.6	47.2	≈35	COCO2017 val, PRB-FPN-Net (Huang et al., 7 Nov 2025)
ELAN + PRB-FPN	96.4	252.8	48.4
MSP + PRB-FPN	101.1	368.1	52.6
YOLO-World v2	44.6	203.9	45.8	≈30	Baseline, no PRB-FPN
GLIP-T	232	-	55.4		Transformer-based vision-language detector

• PRB-FPN achieves $k$0 AP improvement over YOLO-World, $k$1 higher small-object AP, and operates with approximately half the number of parameters of GLIP-T, remaining within $k$2 in overall AP relative to that large Transformer model.
• On Objects365, PRB-FPN matches large-model accuracy ($k$3 AP vs $k$4) with $k$5 fewer parameters (Huang et al., 7 Nov 2025).
• Ablation studies confirm the individual and combined necessity of: (a) parallelism ($k$6 AP when reduced to single path), (b) residual connections ($k$7), and (c) bidirectional fusion ($k$8) (Huang et al., 7 Nov 2025).

Further supporting evidence from UAVDT17 and MS COCO benchmarks demonstrates consistent, significant accuracy gains at modest parameter and runtime cost relative to standard FPNs (Chen et al., 2020, Chen et al., 2019).

7. Limitations, Extensions, and Future Directions

The increased model capacity and memory overhead of PRB-FPN relative to standard single-path FPNs results in modest slowdown (5–10%) and augmented parameter count, though these are constrained by design choices such as depthwise separable convolution and shallow purification iteration ($k$9 typically sufficient) (Chen et al., 2020, Chen et al., 2019).

Potential future enhancements include:

• Exploration of depthwise separable or attention-gated fusion to further reduce compute costs (Chen et al., 2019).
• Applying parallel residual bi-fusion principles to broader dense prediction tasks (segmentation, depth estimation).
• Systematic study of the architecture's impact in transformer-based vision backbones and anchor-free frameworks.

Such directions reflect the demonstrated portability and scalability of the PRB-FPN design, with clear empirical support for cross-domain utility and continued relevance in resource-constrained and high-accuracy scenarios (Huang et al., 7 Nov 2025, Chen et al., 2020, Chen et al., 2019).