BiFPN: Bidirectional Feature Pyramid Network

• BiFPN is a bidirectional feature pyramid network that fuses multiscale features using learnable, normalized weighted fusion for improved accuracy and efficiency.
• It employs interleaved top-down and bottom-up pathways to propagate both semantic and spatial cues using depthwise-separable convolutions and optimized connectivity.
• Empirical evaluations show that BiFPN enhances detection performance (e.g., higher mAP) while reducing parameters and computational overhead compared to traditional FPNs.

A Bi-Directional Feature Pyramid Network (BiFPN) is a neural architecture for multiscale feature fusion, designed to enhance feature representations for tasks such as object detection and structured prediction. BiFPNs combine bottom-up and top-down pathways, enabling efficient bidirectional information flow across spatial resolutions and semantic hierarchies. Core principles include fast, normalized, learnable weighted fusion and connectivity optimizations that enable improved accuracy and computational efficiency compared to standard Feature Pyramid Networks (FPNs) and similar designs.

1. Historical Development and Motivation

The motivation for BiFPN stems from the limitations of conventional one-way FPNs, which perform single-direction top-down fusion to transfer semantic information from deeper (coarser) layers to shallower (finer) ones. This design omits a bottom-up path, failing to propagate fine-grained spatial details back to coarser scales. Early bidirectional approaches, such as the Bidirectional Pyramid in BPN ["Single-Shot Bidirectional Pyramid Networks for High-Quality Object Detection" (Wu et al., 2018)], introduced explicit bottom-up and top-down passes to enrich all pyramid levels with both semantic and spatial cues. Subsequent work, including EfficientDet ["EfficientDet: Scalable and Efficient Object Detection" (Tan et al., 2019)], generalized bidirectional fusion, introduced weighted fast-normalized fusion, and iterated the process across multiple stacked blocks. Variants such as the Residual Bi-Fusion FPN ["Residual Bi-Fusion Feature Pyramid Network for Accurate Single-shot Object Detection" (Chen et al., 2019)], MF-PAM’s BiFPN ["MF-PAM: Accurate Pitch Estimation through Periodicity Analysis and Multi-level Feature Fusion" (Chung et al., 2023)], and the fully reversible RevBiFPN ["RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network" (Chiley et al., 2022)] further diversified the architectural landscape.

2. Core Architectural Principles

BiFPNs operate over a set of multiscale feature maps produced by a CNN backbone, typically denoted as $\{P_i\}$. They organize feature flow into interleaved top-down and bottom-up fusion passes, connecting all relevant pyramid levels bidirectionally. The central components are:

• Top-down pathway: Propagates high-level semantics from deeper/coarser feature maps towards higher-resolution layers via upsampling and fusion.
• Bottom-up pathway: Routes fine spatial details from shallow/finer maps to deeper ones via downsampling and fusion.
• Bidirectional cross-scale connectivity: Each level is fused using information from adjacent (and sometimes skip-connected or lateral) nodes in both directions.
• Learnable fast-normalized weighted fusion: Feature fusion is parametrized by per-input positive scalars and normalized with a small $\epsilon$ to ensure numerical stability:

$$\tilde{P}_i = \frac{\sum_{j \in \mathcal{I}(i)} w_j X_j}{\sum_{j \in \mathcal{I}(i)} w_j + \epsilon}$$

where $w_j$ is a learnable scalar (enforced non-negative via ReLU), and $\mathcal{I}(i)$ indexes the inputs to fusion node $i$ (Tan et al., 2019, Chung et al., 2023).

• Depthwise-separable convolutions and block repeats: Fusion outputs are processed by depthwise-separable convolutions (often 3×3, sometimes kernel=5 in variants) and activation functions (e.g., Swish/SiLU). The BiFPN module may be stacked multiple times to deepen fusion.

3. Mathematical Formulation and Fusion Strategies

Fusion operations are explicitly defined:

• Weighted sum normalization: Both EfficientDet (Tan et al., 2019) and MF-PAM (Chung et al., 2023) utilize normalized, non-negative learnable scalar weights to fuse each input node at a fusion point. Weights are initialized to 1.0 and learned via backpropagation, with normalization to avoid degenerate solutions and facilitate stable training.
• Fusion node computation (EfficientDet):

At input feature set $\{X_j\}$ and learned weights $\{w_j\}$:

$P_i = \frac{\sum_j w_j X_j}{\sum_j w_j + \epsilon}$

• Fusion node computation (MF-PAM):

Top-down pass (for $i = 1\ldots4$):

$\epsilon$0

Bottom-up pass (for $\epsilon$1):

$\epsilon$2

where $\epsilon$3 is a depthwise-separable convolution with activation (Chung et al., 2023).

• Residual variant: The Residual Bi-Fusion FPN performs fusion via concatenation followed by 1×1 conv "gating" and residual addition without explicit sigmoid gating or channel-wise masking (Chen et al., 2019).

4. Implementation Details and Variants

Key implementation parameters and notable BiFPN variants include:

• EfficientDet BiFPN: Five pyramid levels ($\epsilon$4–$\epsilon$5), fusion at each level, pruned single-input nodes, repeated BiFPN blocks ($\epsilon$6 stack depth), all fusion post-processed by 3×3 ConvDW + BatchNorm + Swish. Upsampling mainly by nearest-neighbor, downsampling typically by max pooling or strided depthwise conv (Tan et al., 2019).
• Bidirectional Pyramid (BPN): Four pyramid stages ($\epsilon$7–$\epsilon$8), with three quality stages: (1) direct use, (2) bottom-up FP (deconv upsampling, lateral sum, 3×3 conv), (3) top-down "reverse FPN" (stride=2 conv downsampling, fusion with previous, lateral input, 3×3 conv), all at 256 channels (Wu et al., 2018).
• MF-PAM BiFPN: Modified for 1-D feature fusion for pitch estimation. Five levels, pre-size block ensures same temporal stride, single BiFPN block (no repeat), channels = 48, uniform 1-D depthwise-separable conv (kernel=5), no extra skip connections, strictly follows fast-normalized weighted fusion (Chung et al., 2023).
• Residual Bi-Fusion (ReBiF Net): Employs a three-way fusion (upsampled, native, reorganized) via standard 1×1 conv acting as implicit gating followed by residual aggregation. A final learned 1×1 conv module (“BiFusion Module”) fuses reorg'ed multi-scale information to improve large-object localization and alleviate shift-variance (Chen et al., 2019).
• RevBiFPN: Stacks invertible "RevSilo" building blocks for bidirectional multi-scale fusion, enabling exact recomputation of feature activations during backpropagation and reducing memory requirements from $\epsilon$9 to $$\tilde{P}_i = \frac{\sum_{j \in \mathcal{I}(i)} w_j X_j}{\sum_{j \in \mathcal{I}(i)} w_j + \epsilon}$$0 in network depth, with minimal empirical accuracy loss (Chiley et al., 2022).

Variant	Input Pyramid	Fusion Strategy	Key Params	Unique Aspects
EfficientDet	5 levels	Weighted, normalized	Channels 64–160, repeats	Pruned nodes, skip edges
BPN	4 levels	Sum + convolution	3×3 conv, 256 channels	3 quality stages, deconv
MF-PAM	5 levels	Fast-normalized, 1D	Channels 48, kernel=5	Audio, pre-sizing block
ReBiF Net	3–6 levels	Residual, concat+conv	1×1 and 3×3 conv	Reorg, residual fusion
RevBiFPN	N-levels	Reversible coupling	MBConv/F blocks	$$\tilde{P}_i = \frac{\sum_{j \in \mathcal{I}(i)} w_j X_j}{\sum_{j \in \mathcal{I}(i)} w_j + \epsilon}$$1 activation memory

5. Empirical Impact and Ablation Results

Empirical studies across multiple domains confirm the performance and efficiency benefits of BiFPN-based fusion:

• Object detection (EfficientDet): Replacing FPN with BiFPN on EfficientNet-B3 improves mAP on COCO from 40.3 to 44.4 (single scale/val), with parameter count dropping from 21 M to 12 M and FLOPs from 75B to 24B. Weighted fusion confers an additional +0.5 AP (Tan et al., 2019).
• High-quality detection (BPN): The Bidirectional Pyramid in BPN yields substantial gains at higher IoU thresholds (e.g., [email protected]), both on VOC and COCO, due to improved localization from bidirectional feature flow (Wu et al., 2018).
• Pitch estimation (MF-PAM): In extreme noise, inclusion of a light BiFPN improves raw pitch accuracy by ≈4 % versus a single-level LSTM baseline. Full multi-level fusion gains an additional ≈1 %, both with minimal parameter overhead (≤0.4 M) (Chung et al., 2023).
• Residual Bi-Fusion Net: Introduces consistent gains for small and large objects in detection, with up to ∼7 points of AP increase for small objects and robust improvements even as pyramid depth increases (Chen et al., 2019).
• RevBiFPN: Achieves nearly 20× reduction in train-time memory (e.g., RevBiFPN-S6: 0.25 GB/sample vs. EfficientNet-B7: 5.05 GB) with similar or improved accuracy (84.2% vs. 84.3% ImageNet top-1), with compute overhead being amortized by larger feasible batch sizes (Chiley et al., 2022).

6. Theoretical Considerations and Computational Efficiency

• Parameter and compute scaling: BiFPN architectures, by employing depthwise-separable convolutions and pruning redundant fusion nodes, reduce parameter count and computational FLOPs by up to ∼2× relative to fully connected or standard FPNs, with ablated EfficientDet-B3 results showing a 0.88× parameter ratio and 0.68× FLOPs versus repeated FPN (Tan et al., 2019).
• Convex normalization of fusion weights: Enforcing $$\tilde{P}_i = \frac{\sum_{j \in \mathcal{I}(i)} w_j X_j}{\sum_{j \in \mathcal{I}(i)} w_j + \epsilon}$$2 with normalization ensures that fusion remains a convex linear combination, stabilizing learning dynamics and making fusion coefficients interpretable (Tan et al., 2019, Chung et al., 2023).
• Reversible architectures: RevBiFPN architectures achieve constant activation memory with only a modest computational penalty (≈2× MACs in worst case for full recomputation), unlocking the ability to train deeper or wider networks under fixed hardware constraints (Chiley et al., 2022).

7. Application Domains and Extensions

BiFPN has been widely applied across domains:

• Object Detection and Segmentation: Original context (BPN, EfficientDet, ReBiF Net, RevBiFPN) for improved multiscale detection, especially high-precision and small object localization.
• Audio/Speech Structured Prediction: MF-PAM adapts BiFPN for 1-D temporal fusion in robust pitch estimation, demonstrating the architecture’s applicability beyond traditional 2-D raster domains (Chung et al., 2023).
• Reversible and Memory-Efficient Backbones: RevBiFPN and related reversible pyramid networks extend bidirectional pyramid ideas to ultra-large scale training (Chiley et al., 2022).

A plausible implication is that any structured prediction or dense prediction task requiring rich cross-scale feature synthesis can benefit from BiFPN principles, whether using full spatial pyramids or specialized streaming architectures.

---

BiFPN constitutes a class of architectures exploiting bidirectional, learnable, normalized multiscale feature fusion, yielding improved trade-offs in accuracy, efficiency, and scalability compared to classic FPN paradigms. Its variants have demonstrated consistent empirical improvements in detection, segmentation, regression, and structured prediction under diverse constraints and modalities.