Multi-Scale Decoder Fusion (MSDF)

• Multi-Scale Decoder Fusion (MSDF) is a strategy that integrates features from multiple resolutions, combining fine spatial detail with coarse semantic context for improved dense prediction.
• MSDF employs diverse architectural patterns such as top-down pyramidal fusion, coarse-to-fine concatenation, and attention-gated mechanisms to enhance feature integration.
• Recent works demonstrate that MSDF leads to significant performance gains in segmentation, registration, restoration, and other dense prediction tasks compared to traditional skip connections.

Multi-Scale Decoder Fusion (MSDF) denotes a family of decoder-centric or decoder-adjacent fusion strategies that integrate features from multiple resolutions so that fine spatial detail and coarse semantic context are jointly available to dense prediction or reconstruction modules. Across recent arXiv literature, the term is used both explicitly and as a precise explanatory lens for architectures in segmentation, localization, registration, restoration, video representation, and change detection. Its implementations range from top-down addition-and-concatenation schemes, to attention-gated all-scale fusion, to topology-preserving convolutional aggregation, to bottleneck deformable fusion, and even to predictor–corrector updates derived from linear multistep methods (Zhu et al., 2022, Kabir et al., 19 May 2026, Xu et al., 2022, He et al., 6 Jun 2025, Segu et al., 16 Oct 2025, Srivastava et al., 2021).

1. Definition and conceptual scope

MSDF is not a single operator. In the cited literature it may refer to fusing hierarchical encoder features inside a conventional decoder, as in the ConvNeXt-based tampering localization framework where hierarchical features are fused in the decoder through a top-down FPN path and a final all-scale aggregation (Zhu et al., 2022). It may also refer to a decoder head that replaces an existing lightweight head, as in WoundFormer, where a spatially-preserving multi-scale aggregation head replaces SegFormer’s All-MLP decoder and keeps feature maps in $2$D throughout fusion (Kabir et al., 19 May 2026). In transformer decoders, it can be instantiated as repeated attention-based fusion between decoder features and same-scale encoder features, as in MUSTER’s Multi-Head Skip Attention (MSKA) units and FuseUpsample operation (Xu et al., 2022).

The scope is broader still. In GMSRF-Net, the primary innovation is not a heavy decoder, but dense global multi-scale residual fusion before a vanilla decoder; the decoder then performs classic upsample-and-concatenate skip fusion over already enriched representations (Srivastava et al., 2021). In FuseUNet, MSDF is a mathematically grounded reformulation of skip fusion itself, where the decoder is treated as solving an initial value problem and the fusion across scales is performed by Adams–Bashforth and Adams–Moulton updates rather than by concatenation or addition (He et al., 6 Jun 2025). In MOBIUS, multi-scale and multi-modal information is fused efficiently into a single bottleneck pixel decoder, after which the transformer decoder becomes single-scale, showing that MSDF need not imply a multi-scale transformer decoder at inference time (Segu et al., 16 Oct 2025).

A recurring theme across these works is that ordinary one-to-one skip connections are often considered insufficient. Several papers explicitly identify semantic gaps, weak cross-scale interaction, or delayed fusion as limitations of standard encoder–decoder designs, and position MSDF as a remedy that increases the frequency, breadth, or adaptivity of scale interaction (Srivastava et al., 2021, He et al., 6 Jun 2025, Zhang et al., 2023).

2. Architectural patterns of fusion

The design space of MSDF can be organized by where fusion occurs, what is fused, and which operator performs the fusion.

Representative system	Fusion locus	Characteristic operator
GMSRF-Net (Srivastava et al., 2021)	Pre-decoder plus vanilla decoder	Dense all-scale residual fusion with CMSA and MSFS
ConvNeXt tampering localization (Zhu et al., 2022)	Decoder	FPN top-down addition plus final all-scale concatenation
WoundFormer (Kabir et al., 19 May 2026)	Decoder head	Coarse-to-fine concatenation with $1\times1$ conv and terminal $3\times3$ refinement
MUSTER (Xu et al., 2022)	Transformer decoder	MSKA and FuseUpsample with PixelShuffle
FuseUNet (He et al., 6 Jun 2025)	Decoder dynamics	Adams–Bashforth/Moulton multistep fusion over all skips
MOBIUS (Segu et al., 16 Oct 2025)	Bottleneck pixel decoder	Bidirectional cross-attention, DeformSA, and MSDeformCA

One major pattern is top-down pyramidal fusion. The tampering localization scheme based on ConvNeXt uses a UPerNet-inspired decoder with a Pyramid Pooling Module on the coarsest feature, lateral $1\times1$ projections, element-wise addition during top-down propagation, and a final bilinear alignment plus concatenation of all decoded scales before classification (Zhu et al., 2022). CEDNet generalizes the idea by repeating encoder–decoder stages in cascade; each stage fuses multi-scale features in its decoder, and the fused high-resolution output becomes the input to the next stage’s encoder, which lets high-level features from early stages guide low-level feature learning in subsequent stages (Zhang et al., 2023).

A second pattern is progressive coarse-to-fine concatenation with spatial preservation. WoundFormer projects MiT-B5 features to a unified channel dimension $C=128$, upsamples the coarsest feature by bilinear interpolation, concatenates it with the next finer feature, applies $1\times1$ convolution plus BatchNorm and ReLU, and repeats this process until the finest stage, followed by a $3\times3$ refinement convolution (Kabir et al., 19 May 2026). The explicit claim is that preserving feature topology during cross-scale integration improves boundary localization and discrimination between visually similar tissue categories.

A third pattern is attention-gated decoder fusion. GMSRF-Net uses Cross Multi-Scale Attention (CMSA) to derive spatial attention for each scale from the other scales and then applies Multi-Scale Feature Selection (MSFS), described as a squeeze-and-excitation channel-wise attention, after CMSA-gated fusion (Srivastava et al., 2021). MUSTER fuses the current decoder features with encoder features at the same scale through windowed and shifted-window skip attention, then upsamples using encoder-guided FuseUpsample (Xu et al., 2022). In CHMFFN, decoder-side fusion occurs through cross-attention from decoder queries to encoder outputs, and later cross-level aggregation is reinforced by STCFL and AFAF (Sheng et al., 21 Sep 2025).

A fourth pattern is bottleneck fusion before simplified decoding. MOBIUS selects a single bottleneck scale, fuses $\{S_3,S_4,S_5,S_6\}$ and language tokens $E_{\text{text}}$ into that bottleneck through bidirectional cross-attention, DeformSA, and MSDeformCA, and then runs a single-scale transformer decoder on the fused bottleneck alone (Segu et al., 16 Oct 2025). This architecture makes a strong point: decoder fusion can become more powerful while the downstream decoder becomes simpler.

A fifth pattern is fusion as numerical integration. FuseUNet replaces skip concatenation or addition with an adaptive ODE method based on linear multistep integration, treating skip connections as discrete nodes and propagating a memory flow $Y_i$ across scales (He et al., 6 Jun 2025). This is the most explicit mathematical reinterpretation of decoder fusion among the cited works.

3. Mathematical formulations and operators

Although the architectural manifestations differ, the core mathematics of MSDF usually reduces to alignment, merge, refinement, and supervision.

In GMSRF-Net, decoder fusion is deliberately simple after heavy pre-decoder enrichment. The decoder recurrence is

$1\times1$0

while the pre-decoder global multi-scale residual fusion at layer $1\times1$1 and scale $1\times1$2 is

$1\times1$3

followed by attention modulation

$1\times1$4

This formulation makes repeated all-scale interaction the central mechanism rather than the decoder itself (Srivastava et al., 2021).

In the ConvNeXt tampering localization model, MSDF is explicitly formulated as top-down FPN merging followed by final all-scale decoder fusion: $1\times1$5

$1\times1$6

$1\times1$7

A notable detail is that fusion uses element-wise addition in the top-down path and concatenation only in the final aggregation; the paper states that no attention or learned scalar weights are used (Zhu et al., 2022).

WoundFormer’s MSDF head adopts a coarse-to-fine recursive fusion: $1\times1$8

$1\times1$9

$3\times3$0

The paper explicitly contrasts this with token-flattening All-MLP decoding and attributes gains to topology-preserving fusion plus spatial convolutional refinement (Kabir et al., 19 May 2026).

FuseUNet recasts decoder fusion as state evolution. Its nodewise derivative is

$3\times3$1

and one of its corrector updates is

$3\times3$2

with the final fusion given by an explicit AB4 step: $3\times3$3 This formulation makes multi-scale fusion a weighted accumulation over multiple past scales rather than a local merge of only the current skip and the previous decoder state (He et al., 6 Jun 2025).

In MOBIUS, the central decoder-side fusion block is the bottleneck update

$3\times3$4

preceded by bidirectional cross-attention with text and followed by an FFN. The resulting fused bottleneck $3\times3$5 is upsampled and merged with $3\times3$6 to form the mask embedding map

$3\times3$7

The paper then computes language-guided confidence by scaled cosine similarity for top-$3\times3$8 query selection and uses a language-guided uncertainty calibration loss for adaptive decoder pruning (Segu et al., 16 Oct 2025).

The loss functions attached to MSDF are similarly task-dependent. They include weighted IoU plus BCE with deep supervision in GMSRF-Net (Srivastava et al., 2021), Focal plus Lovasz in tampering localization (Zhu et al., 2022), cross-entropy in WoundFormer where Focal+Dice underperformed (Kabir et al., 19 May 2026), scale-adaptive multi-resolution supervision with high-frequency boosting in MSNeRV (Zhu et al., 18 Jun 2025), BCE plus Dice plus TV in PINN-EMFNet (Ding et al., 2024), and calibration loss aligned with IoU in MOBIUS (Segu et al., 16 Oct 2025).

4. Application domains and reported performance

Medical segmentation supplies several strong illustrations of MSDF. GMSRF-Net was proposed to address cross-dataset generalizability in polyp segmentation and reported gains on unseen datasets: when trained on Kvasir-SEG and tested on unseen CVC-ClinicDB, Dice rose from $3\times3$9 to $1\times1$0, a reported $1\times1$1; when trained on CVC-ClinicDB and tested on unseen Kvasir-SEG, Dice rose from $1\times1$2 to $1\times1$3, a reported $1\times1$4 (Srivastava et al., 2021). WoundFormer reported an overall Dice score of $1\times1$5 on WoundTissueSeg and $1\times1$6 with augmentation, with a $1\times1$7 overall Dice improvement over SegFormer-B5 on WoundTissueSeg (Kabir et al., 19 May 2026). PINN-EMFNet reported mean $1\times1$8 std results of DSC $1\times1$9 on BUSIS and $C=128$0 on BUSI, and attributed improved robustness under low contrast and speckle noise to the Multi-Scale Feature Refinement Decoder, PCAM, deep supervision, and TV-based PINN regularization (Ding et al., 2024).

For semantic segmentation and dense prediction, MUSTER and CEDNet exemplify two different decoder-fusion philosophies. MUSTER emphasizes transformer-based decoder design with MSKA and FuseUpsample, achieving single-scale and multi-scale mIoU of $C=128$1 and $C=128$2 on ADE20K in its best model, while reporting a FLOPs reduction of $C=128$3 (Xu et al., 2022). CEDNet instead pushes fusion earlier and repeatedly inside stagewise decoders; with RetinaNet on COCO val2017, CEDNet-NeXt-T and CEDNet-NeXt-S achieved $C=128$4 and $C=128$5 AP$C=128$6 versus ConvNeXt-T and ConvNeXt-S at $C=128$7 and $C=128$8, and on ADE20K CEDNet-NeXt-T/S/B reported $C=128$9, $1\times1$0, and $1\times1$1 mIoU(ms) (Zhang et al., 2023).

MSDF has also been effective outside segmentation. In tampering localization, the ConvNeXt-based method reported average performance across test sets of F1 $1\times1$2, IoU $1\times1$3, and AUC $1\times1$4, outperforming OSN and MVSS-Net on average F1 and IoU; its ablation showed that using all four scales $1\times1$5 performed best (Zhu et al., 2022). In deformable image registration, the shared-encoder plus auxiliary-decoder design with an MSFB reported Dice $1\times1$6, TRE $1\times1$7, HdDist95 $1\times1$8, and NDV $1\times1$9, outperforming VoxelMorph and surpassing TransMorph on Dice and HdDist95 while remaining smoother in terms of NDV (Zhou et al., 2024). In hyperspectral change detection, CHMFFN reported improvements over MSDFFN across four public datasets, including on River where OA increased from $3\times3$0 to $3\times3$1, KC from $3\times3$2 to $3\times3$3, and F1 from $3\times3$4 to $3\times3$5 (Sheng et al., 21 Sep 2025).

Restoration and representation tasks further broaden the meaning of MSDF. The Multi-Scale Boosted Dehazing Network with Dense Feature Fusion combined an S-O-S boosted decoder with progressive back-projection and reported $3\times3$6 dB PSNR and $3\times3$7 SSIM on SOTS, $3\times3$8 CIEDE2000 and $3\times3$9 SSIM on HazeRD, and the best reported performance in its comparisons on I-HAZE and O-HAZE (Dong et al., 2020). The self-supervised monocular depth method using deep feature fusion reported on KITTI 2015 Stereo 200 training data Abs Rel $\{S_3,S_4,S_5,S_6\}$0, Sq Rel $\{S_3,S_4,S_5,S_6\}$1, RMSE $\{S_3,S_4,S_5,S_6\}$2, RMSE log $\{S_3,S_4,S_5,S_6\}$3, and D1-all $\{S_3,S_4,S_5,S_6\}$4, while ablations showed consistent degradation for “no fusion” and “no CoordConv” variants (Kaushik et al., 2020). MSNeRV introduced a multi-scale spatial decoder with scale-adaptive loss and reported average PSNR $\{S_3,S_4,S_5,S_6\}$5 dB in small configurations and $\{S_3,S_4,S_5,S_6\}$6 dB in larger configurations across HEVC ClassB and UVG, together with superior representation capability among INR-based approaches and better compression efficiency than VTM-23.7 (Random Access) in dynamic scenarios (Zhu et al., 18 Jun 2025).

MOBIUS shows that MSDF can also be a scaling strategy for deployment. It reported reductions in pixel and transformer decoder FLOPs by up to $\{S_3,S_4,S_5,S_6\}$7 and $\{S_3,S_4,S_5,S_6\}$8, respectively, while maintaining state-of-the-art performance in just a third of the training iterations, and provided mobile latencies including $\{S_3,S_4,S_5,S_6\}$9 ms on Samsung S24 for MOBIUS d (Segu et al., 16 Oct 2025).

5. Relation to neighboring architectures

MSDF overlaps with, but is not identical to, U-Net skip connections, FPN/BiFPN necks, HRNet-style exchanges, and transformer pixel decoders.

Relative to classic U-Net, many MSDF papers explicitly argue that same-scale skip concatenation or addition is too local. FuseUNet makes this critique formal by describing concatenation or addition as analogous to an explicit Euler step that is first-order accurate and strictly local (He et al., 6 Jun 2025). WoundFormer similarly differentiates itself from SegFormer’s All-MLP decoder by retaining $E_{\text{text}}$0D topology and explicitly modeling local neighborhoods via convolution after cross-scale concatenation (Kabir et al., 19 May 2026).

Relative to FPN, some MSDF variants extend it and others depart from it. The ConvNeXt tampering localization decoder is FPN-like but adds a PPM global prior on $E_{\text{text}}$1 and a final all-scale concatenation stage not present in vanilla FPN (Zhu et al., 2022). CEDNet criticizes lightweight necks such as FPN and BiFPN for delayed fusion and instead distributes fusion inside every stage decoder of the backbone (Zhang et al., 2023). MOBIUS takes the opposite route: it compresses all multi-scale information into a single expressive bottleneck so that the transformer decoder can remain single-scale (Segu et al., 16 Oct 2025).

Relative to HRNet, several papers adopt the principle of repeated exchange or high-resolution preservation without reproducing HRNet exactly. GMSRF-Net is explicitly described as maintaining high-resolution representations while performing multi-scale fusion operations for all resolution scales, and the paper situates itself near HRNet and MSRF-Net while intensifying fusion frequency via dense global fusion and attention (Srivastava et al., 2021). CEDNet instead uses repeated encoder–decoder stages rather than persistent parallel streams (Zhang et al., 2023).

Relative to transformer decoders, MSDF may enter through skip attention, deformable cross-attention, or bottleneck fusion. MUSTER’s MSKA units are same-scale skip attention modules inside Swin-style windows (Xu et al., 2022). CHMFFN injects encoder information into decoder tokens through standard cross-attention (Sheng et al., 21 Sep 2025). MOBIUS fuses language and multi-scale vision features into a bottleneck and then simplifies the transformer decoder, demonstrating that fusion capacity and decoder scale count are separable design choices (Segu et al., 16 Oct 2025).

A common misconception is that MSDF must involve attention. The literature does not support that view. The ConvNeXt tampering localization model explicitly states that no attention or learned scalar weights are used in its fusion (Zhu et al., 2022), whereas GMSRF-Net, MUSTER, CHMFFN, MOBIUS, and PINN-EMFNet all use attention or gating of some form (Srivastava et al., 2021, Xu et al., 2022, Sheng et al., 21 Sep 2025, Segu et al., 16 Oct 2025, Ding et al., 2024). Another misconception is that MSDF must occur only inside the decoder; GMSRF-Net and MOBIUS both show that heavy pre-decoder or bottleneck fusion can be the dominant mechanism, with a comparatively light downstream decoder (Srivastava et al., 2021, Segu et al., 16 Oct 2025).

6. Limitations, misconceptions, and future directions

The literature also makes clear that MSDF is not a universally solved design problem. Several papers omit detailed efficiency statistics: GMSRF-Net does not provide parameter counts, FLOPs, or inference time (Srivastava et al., 2021); the deformable registration paper does not report parameter count, runtime, or memory footprint (Zhou et al., 2024); the tampering localization letter does not report model complexity, FLOPs, speed, or memory (Zhu et al., 2022). This makes cross-paper efficiency comparisons approximate even when accuracy gains are clear.

A second limitation is incomplete module isolation. GMSRF-Net does not report ablations isolating CMSA and MSFS (Srivastava et al., 2021). The deformable registration work attributes gains to the multi-scale fusion and auxiliary decoder but does not provide separate ablation numbers (Zhou et al., 2024). In such cases, the role of specific submodules is inferential even when the full system improves performance.

A third issue is that some gains are task- and distribution-dependent. The ConvNeXt tampering localization paper explicitly notes that DSO remains challenging due to distribution shift from close-up human faces and body splices (Zhu et al., 2022). WoundFormer improves minority tissue classes on WoundTissueSeg but does not surpass DFUTissueSegNet on DFUTissue without or with augmentation, reporting $E_{\text{text}}$2 versus $E_{\text{text}}$3 without augmentation and $E_{\text{text}}$4 versus $E_{\text{text}}$5 with augmentation (Kabir et al., 19 May 2026). Such results suggest that the benefits of MSDF are strong but not uniform across benchmarks.

The cited papers also point toward several open directions. Repeatedly proposed extensions include transformer-style cross-scale attention for GMSRF-Net, boundary-aware losses or edge decoders for the ConvNeXt tampering localization model, learnable high-pass filters and adaptive scale-wise loss schedules for MSNeRV, dynamic scale selection for FuseUNet, boundary-aware losses or a final upsampling stage for WoundFormer, and more advanced entropy coding or quantization for MOBIUS (Srivastava et al., 2021, Zhu et al., 2022, Zhu et al., 18 Jun 2025, He et al., 6 Jun 2025, Kabir et al., 19 May 2026, Segu et al., 16 Oct 2025). A plausible implication is that future MSDF work will continue to separate three concerns that are often entangled in current architectures: where fusion happens, how scales are weighted, and how much of the final decoder actually needs to remain multi-scale at inference time.

Taken together, the arXiv literature presents MSDF not as a narrow module class but as a general decoder-design principle: repeated, aligned, and task-aware interaction among heterogeneous scales. Its concrete realizations range from simple top-down addition and all-scale concatenation to deformable multimodal bottlenecks and ODE-derived predictor–corrector updates, but the central objective remains consistent—improving the transfer of information between high-resolution detail and low-resolution semantics in ways that ordinary skip connections or delayed neck fusion do not fully capture (Zhu et al., 2022, Zhang et al., 2023, He et al., 6 Jun 2025, Segu et al., 16 Oct 2025).