Multi-Scale Feature Fusion
- Multi-Scale Feature Fusion (MSFF) is a design principle that extracts and fuses features from multiple scales to capture both fine details and global context.
- It employs diverse mechanisms such as top-down pathways, lateral connections, and multi-branch architectures to refine and combine features across resolutions.
- Empirical studies demonstrate that well-aligned MSFF significantly boosts performance in tasks like semantic segmentation, object detection, and person re-identification.
Taken together, the cited literature presents Multi-Scale Feature Fusion (MSFF) as a family of mechanisms that extract, propagate, aggregate, or fuse representations from multiple scales so that a model can jointly exploit fine-grained detail and high-level context. In the surveyed works, MSFF is used to “dynamically fuse features cross different scales” for person re-identification, to “capture both fine-grained details and high-level semantics” for multi-frame pose estimation, and to preserve “feature topology during cross-scale integration” for wound tissue segmentation (Wang, 2020, Pace et al., 14 Jan 2025, Kabir et al., 19 May 2026). The term is therefore not a single algorithm but a recurring design principle spanning CNNs, Transformers, Vision Mamba, encoder-decoder segmentation models, object detectors, self-supervised depth systems, neural video representations, and object-centric learning.
1. Problem setting and functional objectives
The motivating problem is consistent across domains: features at different depths or resolutions encode different kinds of information, and single-scale processing is often insufficient. In person re-identification, low-level features are described as rich in “local/color/texture info” but lacking “discriminative abstraction,” whereas high-level features are “more semantic” but lack “finer details”; the stated role of MSFF is to refine each scale before fusion so that the final embedding contains both global and local cues (Wang, 2020). In semantic segmentation, the argument is that the semantics of each pixel often depend on both “nearby neighbors and far-away context,” so “a more powerful multi-scale feature fusion network plays a critical role” (Meng et al., 2022).
Domain-specific formulations sharpen this general objective. XOCT targets vessels with “a wide range of spatial scales and orientations—thin, elongated, and sometimes low-contrast,” and its MSFF therefore combines isotropic, anisotropic, and large-kernel operators (Khosravi et al., 9 Sep 2025). UAV detection emphasizes that small objects occupy small portions of images and that existing fusion methods often fail to balance “classification and localization performance for small objects,” so the fusion framework explicitly separates high-resolution localization cues from deeper contextual cues (Wang et al., 29 Jan 2025). This suggests that MSFF is best viewed as a scale-management strategy rather than merely a decoder convenience.
A second objective is reliability under task-specific nuisance factors. In dynamic-scene depth estimation, MSFF is inserted before cost-volume construction to improve feature matching “especially between frames with large camera motion” (Zhong et al., 2023). In online knowledge distillation, MFEF uses multi-scale extraction and fusion to generate “more informative feature maps for distillation” than logit-only transfer (Zou et al., 2022). In remote sensing segmentation, the MFMSBlock is explicitly designed because direct skip connections have “limited discriminative power” and difficulty “achieving detailed fusion” (Cao et al., 2024).
2. Recurrent architectural patterns
A dominant pattern is the feature-pyramid formulation. In person re-identification, features are extracted from a ResNet backbone, passed through a top-down pathway with lateral connections, and transformed into before MSFF refines and concatenates them (Wang, 2020). In seismic noise localization, MobileNet features at strides $8$, $16$, and $32$ are fused by an FPN with lateral projections, nearest-neighbor upsampling, and element-wise summation (Busson et al., 2020). In ConvNeXt-based image tampering localization, the decoder again follows an FPN-like top-down pathway in which higher-level features are progressively upsampled and added to lower-level ones before multi-scale concatenation (Zhu et al., 2022).
A second pattern is progressive coarse-to-fine aggregation with explicit topology preservation. WoundFormer replaces SegFormer’s “All-MLP” decoder with a “spatially-preserving multi-scale aggregation head.” Features at resolutions are channel-aligned, and fusion proceeds from downward by bilinear upsampling, channel concatenation, projection, and a final 0 convolutional refinement (Kabir et al., 19 May 2026). This architecture is designed to avoid flattening-induced loss of spatial relationships.
A third pattern is parallel multi-branch processing with heterogeneous receptive fields. XOCT uses 1 isotropic convolutions, anisotropic 2, 3, and 4 convolutions, plus a 5 depthwise convolution, followed by 6 fusion and channel reweighting (Khosravi et al., 9 Sep 2025). The Fluff block for object detection combines multi-level and multi-branch fusion in a “latticed” architecture based on 7 dilated convolutions with different dilation rates, concatenation of all branch-level outputs, and a shortcut connection (Shi et al., 2020). DILRAN for multimodal medical image fusion similarly combines dilated convolutions with dilation rates 8 and a pyramid attention block that approximates 9, $8$0, and $8$1 receptive fields (Zhou et al., 2022).
A fourth pattern is staged or serial fusion. In multimodal gait recognition, MSFFS performs fusion at the frame level, the spatial-temporal level, and the global level, with MSSTFE extracting part-, local-, and global-scale spatial-temporal features and AFFM aligning silhouette regions with skeleton joints (Zou et al., 2023). In anatomy-oriented hand joint detection, MSFF modules are cascaded serially, and each module contains three parallel branches operating at $8$2, $8$3, and $8$4 resolutions before heatmap refinement through anatomy-oriented mutual reinforcement (Li et al., 2021).
A fifth pattern extends scale handling beyond spatial feature maps. MSNeRV uses a multi-scale spatial decoder, a multi-scale feature block with hybrid upsampling and cross-depth fusion, and supervision at every decoder resolution (Zhu et al., 18 Jun 2025). Object-centric learning with Multi-Scale Fusion uses an image pyramid and inter-/intra-scale fusion of quantized super-pixels so that objects of different sizes fall within the VAE’s “comfort zone” at at least one scale (Zhao et al., 2024).
3. Fusion operators and mathematical forms
MSFF is implemented through several recurring operators: concatenation, addition, learned weighted summation, attention-based gating, and multistep integration.
A canonical concatenative form appears in person re-identification, where scale-specific post-CSIP features are further abstracted by bottleneck stacks and then fused as
$8$5
$8$6
The purpose is to avoid directly concatenating shallow features before they acquire sufficient abstraction (Wang, 2020).
A progressive hierarchical formulation appears in WoundFormer:
$8$7
$8$8
followed by
$8$9
Here the fusion operator is concatenation plus projection, and the explicit goal is coarse-to-fine contextual conditioning without destroying topology (Kabir et al., 19 May 2026).
Weighted pyramid aggregation appears in ESeg, where all fused pyramid levels are upsampled and combined by a learned softmax-weighted sum:
$16$0
This formulation is central to the paper’s claim that neither high internal resolution nor atrous convolutions are necessary if the multi-scale fusion network is sufficiently strong (Meng et al., 2022).
Soft attention-gated fusion appears in the MFMSBlock for remote sensing segmentation. Encoder and decoder features are first added, a multi-frequency and local attention mask is generated,
$16$1
and the final fusion is
$16$2
The same section introduces 2D DCT-based multi-frequency pooling and Adaptive 1D Convolution for channel weighting (Cao et al., 2024).
A mathematically distinct formulation appears in FuseUNet, which reconceptualizes skip fusion as solving an initial value problem through a linear multistep method:
$16$3
With nmODE-inspired dynamics,
$16$4
the decoded state at a stage is updated by predictor-corrector multistep rules rather than simple addition or concatenation (He et al., 6 Jun 2025). This extends MSFF from an architectural heuristic to an explicitly numerical formulation.
4. Representative task instantiations
The surveyed literature shows that MSFF is not tied to a single application family.
| Task and paper | Fusion structure | Reported result |
|---|---|---|
| Person Re-identification (Wang, 2020) | CSIP top-down/lateral pathway, bottleneck refinement, concat of $16$5 | Rank-1 improves to 92.3 and mAP to 81.9 |
| OCT to OCTA translation (Khosravi et al., 9 Sep 2025) | Isotropic, anisotropic, and $16$6 depthwise branches, $16$7 fusion, channel attention, residual link | SSIM improves from 0.556 to 0.589 on OCTA-3M Proj$16$8 |
| Wound tissue segmentation (Kabir et al., 19 May 2026) | Spatially-preserving coarse-to-fine aggregation head with convolutional refinement | 81.9% Dice, up to 4.3 Dice points over baselines |
| Multi-frame pose estimation (Pace et al., 14 Jan 2025) | Pyramid pooling on layers 9, 21, and final output, multi-head self-attention across layers | 88.3 mAP on PoseTrack21 and 87.8 on PoseTrack18 |
| Semantic segmentation (Meng et al., 2022) | Extended feature space to $16$9, BiFPN, weighted sum over upsampled levels | 76.0% mIoU at 189 FPS for ESeg-Lite-S |
| UAV object detection (Wang et al., 29 Jan 2025) | Fusion Down Sample, Fusion Up Sample, Fusion Multi-Head Self-Attention | 2% improvement in average precision (AP) with the same number of parameters |
Outside these examples, the same principle appears in online knowledge distillation, where MFEF combines divide-and-concatenate channel grouping, dual attention, and fusion across students (Zou et al., 2022); in multi-frame depth estimation, where fused multi-scale features are inserted into feature matching and cost-volume construction (Zhong et al., 2023); in neural video representation, where MSNeRV introduces multi-resolution supervision, hybrid upsampling, multi-kernel fusion, and high-frequency boosting (Zhu et al., 18 Jun 2025); and in object-centric learning, where inter-scale and intra-scale quantized fusion improve slot guidance for objects of varying sizes (Zhao et al., 2024). This breadth indicates that “scale” may refer to spatial resolution, receptive field, semantic depth, temporal window, or even the granularity of latent object codes.
5. Empirical regularities and ablation evidence
Ablation studies repeatedly show that naive inclusion of additional scales is not sufficient. In person re-identification, directly fusing $32$0–$32$1 reduces performance relative to the ResNet50 baseline, while fusing $32$2–$32$3 after lateral and top-down propagation improves rank-1 from 90.4 to 91.5, and adding MSFF yields the best result, 92.3 rank-1 and 81.9 mAP (Wang, 2020). This suggests that shallow features must be contextually aligned and further abstracted before fusion.
The XOCT ablation makes a similar point in a different modality. Adding MSFF improves 3D Reconstruction SSIM from 0.885 to 0.893, improves perceptual loss from 0.198 to 0.184, and reduces model size from 64.6M to 52.7M parameters (Khosravi et al., 9 Sep 2025). The paper further states that, in isolation, MSFF improves 3D metrics more than projection scores, which is consistent with the role of anisotropic and large-kernel branches in volumetric continuity.
In Poseidon, the ablation on PoseTrack21 shows that adding Cross-Attention and AFW raises mean mAP from 87.1 to 87.5, adding Cross-Attention and MSFF raises it to 88.0, and the full model reaches 88.3 (Pace et al., 14 Jan 2025). The paper explicitly identifies MSFF as the largest single boost among the tested additions.
In semantic segmentation, ESeg shows that extending the feature space from $32$4–$32$5 to $32$6–$32$7 raises mIoU from 78.3 to 80.1 with only a small increase from 6.4M to 6.9M parameters and from 34.3B to 34.5B FLOPs; replacing FPN with BiFPN further improves the richer pyramid setting from 79.1 to 80.1 mIoU (Meng et al., 2022). The data support the paper’s claim that a stronger fusion network can be more important than high internal resolution.
Detection-oriented results follow the same pattern. In seismic noise localization, MobileNet + FPN improves AP@[0.5:0.05:0.95] from 32.80% to 41.12%, and the full MobileNet+FPN+Focal configuration reaches 45.62% on the validation set (Busson et al., 2020). In UAV detection, the proposed framework improves AP by 2% while maintaining the same number of parameters (Wang et al., 29 Jan 2025). Across tasks, the recurring empirical pattern is that scale diversity helps only when accompanied by explicit alignment, attention, or structured routing.
6. Misconceptions, trade-offs, and current directions
Several papers explicitly challenge common assumptions. “Revisiting Multi-Scale Feature Fusion for Semantic Segmentation” states that it is “commonly believed that high internal resolution combined with expensive operations (e.g. atrous convolutions) are necessary,” then argues that “neither high internal resolution nor atrous convolutions are necessary” if the multi-scale feature space is richer and the fusion network stronger (Meng et al., 2022). WoundFormer similarly argues that SegFormer’s token flattening “disrupts spatial/topological relationships,” and its replacement decoder is justified on precisely spatial grounds (Kabir et al., 19 May 2026).
Another recurring correction concerns overly simple fusion operators. The saliency-detection MSFA framework argues that one-way fusion can cause “blurring or inaccurate localization of saliency maps” (Song, 2022). FuseUNet argues that standard U-Net skip connections “lack effective interaction between features at different scales” and are constrained by “simple concatenation or addition operations” (He et al., 6 Jun 2025). Person re-identification ablations show that direct fusion of shallow features can even harm performance (Wang, 2020). These works collectively reject the assumption that more fused scales automatically imply better representations.
The trade-offs are mostly computational and structural rather than conceptual. XOCT reports that $32$8 kernels offered only “minor accuracy improvements at a much higher computational cost,” so $32$9 was selected (Khosravi et al., 9 Sep 2025). WoundFormer characterizes its added fusion operations as “lightweight” and with “comparable computational cost” (Kabir et al., 19 May 2026). FuseUNet reports large parameter and GFLOPS reductions but also notes that memory rises due to multi-step history (He et al., 6 Jun 2025). This suggests that MSFF design is increasingly about routing efficiency and feature utility, not only about adding branches or pyramid levels.
A plausible implication is that future MSFF research will continue to merge three strands that are already visible in the surveyed work: richer scale spaces, more selective fusion operators, and supervision that explicitly constrains how information should move across scales. The direction is already apparent in MSNeRV’s scale-adaptive loss, in object-centric inter-/intra-scale code fusion, and in multi-frequency channel attention for remote sensing (Zhu et al., 18 Jun 2025, Zhao et al., 2024, Cao et al., 2024). In that sense, MSFF has evolved from a decoder pattern into a broader framework for representation design, alignment, and optimization across heterogeneous scales.