Triple-Stage Feature Pyramid (TSFP)

• Triple-Stage Feature Pyramid (TSFP) is a hierarchical multi-scale fusion architecture that integrates coarse, intermediate, and fine stages to capture both global context and fine details.
• It employs top-down and bottom-up pathways with convolutions, attention mechanisms, and lateral connections to systematically fuse spatial and temporal features.
• TSFP has demonstrated significant performance gains in applications like facial avatar reconstruction, video saliency detection, and autonomous driving object detection.

A Triple-Stage Feature Pyramid (TSFP) is a hierarchical multi-scale feature fusion architecture designed to address scale variation, fine-detail preservation, and contextual reasoning in dense visual recognition and generation. TSFP modules appear across multiple modalities and application domains, including facial avatar reconstruction, video saliency detection, and object detection for autonomous driving. They systematically integrate information across spatial and (where applicable) temporal scales, employing multi-stage or cross-scale interactions to yield representations that are both semantically rich and spatially/temporally precise.

1. Fundamental Concepts and Architectural Principles

TSFP architectures construct hierarchical feature pyramids comprising three (or more) levels, with each stage designed to cover a distinct scale:

• Coarse/global: captures overall context or low-frequency structure.
• Intermediate: aggregates mid-scale patterns or regional detail.
• Fine/local: preserves high-frequency information and small-object sensitivity.

TSFP employs explicit multi-stage feature fusion. Most architectures use both top-down and bottom-up (sometimes also lateral) pathways, implemented by upsampling, downsampling, convolutions, and attention mechanisms, to propagate and refine information across stages. Each pyramid level generates features either for direct decoding or as input to further task-specific heads.

2. Methodological Instantiations

2.1 Tri²-plane for Volumetric Head Avatar Reconstruction

The "Tri²-plane" model (Song et al., 17 Jan 2024) introduces a three-tri-plane decomposition for dynamic facial neural rendering. Three tri-planes, each at 128² resolution but covering progressively smaller regions—entire face (coarse, Φ₁₂₈), ¼ patch (mid, Φ₂₅₆), and 1/16 patch (fine, Φ₅₁₂)—are synthesized via a cascade of StyleGAN-based generators with top-down lateral connections. The generation flow is:

$$\begin{aligned} \Phi_{128} &= G_1(I_t, \beta_t, \gamma_t) \ \Phi_{256} &= G_2(\uparrow\!\Phi_{128}, I_t, \beta_t, \gamma_t) \ \Phi_{512} &= G_3(\uparrow\!\Phi_{128} + \uparrow\!\Phi_{256}, I_t, \beta_t, \gamma_t) \end{aligned}$$

At inference, feature sampling for each ray-marched 3D point involves tri-plane lookups at all scales, fused as a convex combination before MLP-based prediction of density and color for differentiable volume rendering:

$$\mathbf f(\mathbf x_k) =\sum_{i=1}^3 w_i(\mathbf x_k)\,\mathbf f^{(i)}(\mathbf x_k)$$

A geometry-aware sliding window augmentation during training improves robustness and cross-identity generalization.

2.2 Temporal-Spatial Feature Pyramid for Video Saliency

In video saliency detection (Chang et al., 2021), TSFP utilizes a 3D fully convolutional encoder-decoder. The encoder produces a hierarchy of spatio-temporal feature maps; TSFP applies lateral 1×1×1 convolutions for channel unification, followed by a top-down pathway that sequentially upsamples deep maps and adds them to shallower maps, forming a temporal-spatial pyramid:

$$P_5 = L_5,\,\,\,P_\ell = L_\ell + U(P_{\ell+1}),\,\,\ell=4,3,2,1$$

Each decoded pyramid level is independently upsampled to full resolution and summed, with a final 3D conv to obtain the output saliency map:

$$S_t = \sigma\left(\rho\left(\sum_{\ell=1}^5 D_\ell\right)[T_1]\right)$$

The hierarchical design allows explicit disentanglement of scale, spatial, and temporal context.

2.3 Feature Pyramid Transformer for Visual Recognition

The Feature Pyramid Transformer (FPT) (Zhang et al., 2020) enriches CNN feature pyramids via three transformers per scale:

• Self-Transformer (ST): captures non-local spatial interactions within each scale using a Mixture-of-Softmax attention.
• Grounding Transformer (GT): top-down, projecting global concepts to local positions.
• Rendering Transformer (RT): bottom-up, propagating fine detail to coarser features.

Each output per stage is concatenated and projected to the original channel number, retaining the spatial resolution at each level. Explicit pseudocode details iterative application across L scales, maintaining modularity with the head network.

2.4 TSFP in YOLOv8n-SPTS for Small Target Detection

In autonomous driving object detection (Wu, 10 Dec 2025), the TSFP module augments the model’s neck with a fourth, high-resolution (160×160) shallow fusion stage and three bidirectional fusion stages, each concatenating upsampled and downsampled feature pairs. The revised detection head architecture eliminates the 20×20 large-object head and instead allocates three refined heads (160, 80, 40) for enhanced small- and medium-object performance:

$$\begin{aligned} \hat F^{(l)} &= \text{Conv}_{3\times3}(\text{Concat}[\,\text{Up}(F^{(l+1)}),\, F^{(l)}\,]) \ \hat F^{(l+1)} &= \text{Conv}_{3\times3}(\text{Concat}[\,\text{Down}(F^{(l)}),\, F^{(l+1)}\,]) \end{aligned}$$

3. Training Objectives, Losses, and Evaluation Metrics

TSFP-based systems typically deploy composite, scale- and stage-wise losses tailored to task objectives.

Tri²-plane supervises each tri-plane individually, aggregating multi-level RGB L₁ losses, LPIPS, segmentation mask errors, and regularization; weights of 10 (LPIPS), 0.1 (mask), and 0.01 (reg) are employed (Song et al., 17 Jan 2024).

Video saliency TSFP uses a weighted sum of Kullback-Leibler divergence (KL), correlation coefficient (CC), and normalized scanpath saliency (NSS):

$$L = L_{\rm KL}(S, G) + 0.5\,L_{\rm CC}(S, G) + 0.1\,L_{\rm NSS}(S, F)$$

Standard object detection TSFP implementations leverage conventional anchor-based and anchor-free detection losses depending on the detector head, while precision, recall, AP, and mAP are used for assessment.

4. Performance Characteristics and Empirical Results

Tri²-plane attains state-of-the-art metrics in monocular head avatar self-reconstruction (lower F-LMD, SD, LPIPS; higher PSNR, SSIM), for instance:

Method	F-LMD ↓	PSNR ↑	SD ↓	LPIPS ↓
IMAvatar	2.90	19.39	15.2	0.220
INSTA	2.81	25.97	7.73	0.094
Style-Avatar	2.64	27.83	4.78	0.075
Tri²-plane	2.48	27.75	3.50	0.058

(Song et al., 17 Jan 2024)

Feature Pyramid Transformer yields significant gains in detection/segmentation: e.g., with ResNet-101, MS-COCO box-AP improves from 33.1 (FPN) to 41.6, and mask-AP from 32.6 to 38.6; Cityscapes mIoU increases from 80.6 to 82.2 (Zhang et al., 2020).

YOLOv8n-SPTS + TSFP demonstrates improved small-target detection (VisDrone2019-DET): precision 61.9% (vs. 58.5%), recall 48.3% (vs. 45.0%), [email protected] 52.6% (vs. 48.1%), with pronounced accuracy gains on occluded and tiny objects (Wu, 10 Dec 2025).

5. Design Rationale, Optimization, and Trade-Offs

TSFP architectures optimize for the following:

• Hierarchical self-supervision: Each scale is trained on differently sized regions, forcing features to encode scale-appropriate information and enabling high-frequency gradients to be injected at fine scales (Song et al., 17 Jan 2024).
• Memory and computational efficiency: Most architectural variants demonstrate that the benefit for deeper pyramids levels off at three stages; beyond this, memory/GPU use increases superlinearly with sublinear accuracy returns.
• Task-adaptive scale focus: In detection, pruning large-object heads reallocates FLOPs and feature capacity towards improved sensitivity for small objects (Wu, 10 Dec 2025).

Pyramid sizes (resolution doubling/halving per stage), number of levels, channel widths, fusion operations (additive/attention/concat), and task-specific decoder/heads are empirically tuned per application.

6. Extensions, Comparative Analysis, and Application Domains

TSFP variants demonstrate flexibility across domains:

• Volumetric neural rendering: Tri²-plane pyramid accommodates fine-grained dynamic surface phenomena such as wrinkles, tooth edges, and reflection by hierarchically decomposing facial patches (Song et al., 17 Jan 2024).
• Spatio-temporal saliency: Multi-stage temporal-spatial fusion allows for effective modeling of motion-driven saliency, outperforming single-scale or vanilla encoder-decoder 3D CNN architectures (Chang et al., 2021).
• Object detection and segmentation: Cross-scale transformers (FPT) enable context propagation across disparate scales, giving rise to enhanced recognition of small/thin structures and dense spatial relationships (Zhang et al., 2020).
• Autonomous driving perception: TSFP’s high-resolution heads improve detection reliability under occlusion, small object size, and cluttered backgrounds, trade compute for pronounced gains in small target mAP (Wu, 10 Dec 2025).

A plausible implication is that TSFP principles—multi-scale fusion, explicit per-stage decoding, and scale-adaptive supervision—are beneficial when object scale variation and spatial/temporal detail are critical for task success. However, this comes with increased parameter count and FLOPs, motivating careful design and ablation.

7. Scaling Laws, Limitations, and Future Directions

Analyses show sublinear reduction of total MSE as the number of pyramid levels L increases, formalized as:

$$\mathrm{MSE}_{\rm tot}(L)\approx \sum_{i=1}^L \frac{1}{R_i^2}\,\mathrm{MSE}_i\longrightarrow0\text{ as }R_i\to\infty$$

This suggests diminishing returns as scales proliferate. In practical deployment, most architectures limit to three stages for optimal efficiency/accuracy trade-off (Song et al., 17 Jan 2024). Further extensions may focus on adaptive level allocation, dynamic scale selection, or hybrid attention-convolutional fusion blocks.

TSFP modules are modular and typically “plug-and-play” between backbone and head networks; no changes to detection/segmentation heads are required (Zhang et al., 2020). This modularity ensures broad applicability wherever pyramid-based representations are beneficial.

---

In summary, the Triple-Stage Feature Pyramid is a scalable, multi-domain architectural principle enabling robust multi-scale feature learning, with empirical evidence of substantial gains in fine-detail reconstruction, object recognition, and video saliency—principally by explicitly structuring cross-scale information flow within deep neural models.