Multi-Scale Fusion U-Shaped Mamba (MSF-UM)

• The paper introduces a U-shaped encoder–decoder that integrates Mamba-based state-space modeling with multi-scale fusion for guided depth map super-resolution.
• RGB-guided branches and RDCB blocks are used to recover high-frequency depth details while balancing local feature extraction with global context modeling.
• Empirical evaluations on multiple datasets show that MSF-UM achieves competitive RMSE scores with significantly fewer parameters than transformer-based alternatives.

Searching arXiv for the exact topic and closely related papers to ground the article. Multi-Scale Fusion U-Shaped Mamba (MSF-UM) denotes a class of U-shaped encoder–decoder architectures that couple multi-scale feature fusion with Mamba-based state-space modeling, and it is also the explicit name of a guided depth map super-resolution framework introduced in "Guided Depth Map Super-Resolution via Multi-Scale Fusion U-shaped Mamba Network" (Guo et al., 1 Aug 2025). In that formulation, the model integrates Mamba's efficient state-space modeling capabilities into a multi-scale U-shaped fusion structure guided by a color image, combining residual dense channel attention blocks with Mamba modules to restore high-frequency depth detail while modeling long-range dependencies (Guo et al., 1 Aug 2025). A plausible broader interpretation, supported by adjacent literature, is that MSF-UM has become a reusable architectural pattern for tasks in segmentation, reconstruction, and salient object detection whenever local detail preservation, cross-scale communication, and linear-complexity global modeling must be balanced (Xu et al., 14 Jun 2025).

1. Definition and conceptual scope

In its strictest usage, MSF-UM is the model proposed for guided depth map super-resolution, where a low-resolution depth map $D_{ir} \in \mathbb{R}^{h \times w \times 1}$ and a registered high-resolution color image $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$ are used to predict a high-resolution depth map $D_{sr}$ close to $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$ (Guo et al., 1 Aug 2025). The motivation is that RGB edges and textures correlate with depth discontinuities, so the guidance image can help restore high-frequency detail that is difficult to reconstruct from the depth signal alone (Guo et al., 1 Aug 2025).

The phrase also functions as a descriptive shorthand in neighboring work. In "MS-UMamba: An Improved Vision Mamba Unet for Fetal Abdominal Medical Image Segmentation," MSF-UM is not explicitly used as the paper’s formal model name, but it is described as a reasonable shorthand because “Multi-Scale Fusion” corresponds to the proposed ADFF module and “U-Shaped Mamba” corresponds to the U-Net-style encoder–decoder built from visual state-space blocks (Xu et al., 14 Jun 2025). This suggests that MSF-UM can be understood both as a specific depth super-resolution model and as a broader design family centered on multi-scale skips, hierarchical fusion, and Mamba-based long-range modeling.

2. Architectural organization

The depth super-resolution MSF-UM is a U-shaped encoder–decoder guided by a color image and equipped with RDCB-Mamba blocks (Guo et al., 1 Aug 2025). The encoder downsamples depth features with stride-2 convolutions, increasing channels while decreasing spatial size, and the decoder upsamples with PixelShuffle while fusing three inputs at each scale: skip connections from the encoder, upsampled decoder features, and features from an RGB guidance branch (Guo et al., 1 Aug 2025). The paper adopts a UNet-style multi-scale hierarchy with symmetric downsampling and upsampling paths, but it does not specify the exact number of scales or channels per level (Guo et al., 1 Aug 2025).

The color guidance branch extracts multi-scale features from the high-resolution color image using RDCB-Mamba before the downsampling step so that high-frequency textures are well captured, and these RGB features are fused with depth features at multiple decoder scales (Guo et al., 1 Aug 2025). The central computational unit is the RDCB-Mamba block, which stacks two RDCBs and two Mamba modules: local features are first extracted and enhanced via two RDCBs with channel attention, then two Mamba modules perform global context modeling (Guo et al., 1 Aug 2025). Reconstruction is carried out progressively in the decoder, with a final convolution producing the high-resolution depth map (Guo et al., 1 Aug 2025).

A recurring architectural principle in related systems is the replacement of naive skip concatenation with more selective fusion. In CVMH-UNet for remote sensing segmentation, MFMSBlock replaces standard skip fusion by combining multi-frequency global descriptors from 2D DCT and local multi-scale descriptors to compute fusion weights (Cao et al., 2024). In U$^2$Mamba, MMUBs introduce an inner U-shaped fusion inside each stage while outer encoder–decoder skips perform top-level multi-scale integration (Li et al., 18 Jun 2026). These examples indicate that the “multi-scale fusion” component of MSF-UM is not incidental but structurally central.

3. State-space modeling and local–global coupling

MSF-UM uses Mamba as its long-range dependency mechanism. The continuous-time state-space model is given as

$$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t),$$

and the discrete-time form is written as

$$x_{k+1} = \bar{A} x_k + \bar{B} u_k, \quad y_k = C x_k + D u_k,$$

with $\bar{A} = \exp(A\Delta)$ and $\bar{B} = \int_0^\Delta \exp(A\tau)\, d\tau\, B$ (Guo et al., 1 Aug 2025). The 2D feature map $F \in \mathbb{R}^{H \times W \times C}$ is flattened into a sequence $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$0 with $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$1, then processed by a Mamba module after LayerNorm (Guo et al., 1 Aug 2025).

The reported Mamba block uses two branches. One branch applies linear expansion to $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$2, 1D convolution, SiLU, and an SSM layer; the other applies linear expansion to $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$3 and SiLU; the two branch outputs are multiplied, projected back to length $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$4, and reshaped to $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$5 (Guo et al., 1 Aug 2025). This yields linear time and memory in $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$6, in contrast to the quadratic cost of Transformer self-attention, which is why the model is positioned as suitable for high-resolution depth maps (Guo et al., 1 Aug 2025).

Local representation is provided by RDCB. As described in the paper, an RDCB contains six densely connected convolution layers, followed by a feature fusion convolution, channel attention, and a residual connection (Guo et al., 1 Aug 2025). The RDCB-Mamba block therefore couples dense local aggregation and channel reweighting with linear-complexity global context modeling. In related architectures, the same local–global split is realized with different block names but similar logic: SS-MCAT-SSM in MS-UMamba splits features into a CNN branch and a selective SSM branch before channel concatenation and shuffle (Xu et al., 14 Jun 2025), while M3SR’s MPF block combines spatial SS2D, frequency-domain VSS, and spectral Mamba/S6 within a U-shaped reconstruction pipeline (Zhang et al., 13 Jan 2026).

4. Multi-scale and cross-modal fusion

The “multi-scale fusion” part of MSF-UM is realized by fusing encoder features from depth, upsampled decoder features, and RGB guidance features at multiple decoder scales using RDCB-Mamba (Guo et al., 1 Aug 2025). The paper emphasizes concatenation-based multi-level fusion and channel attention inside RDCB-Mamba, but it does not define a separate cross-attention operator (Guo et al., 1 Aug 2025). Functionally, the RGB branch supplies high-resolution texture and edge cues, while the depth branch supplies geometry and low-frequency structure; the U-shaped decoder then integrates them progressively.

Upsampling is explicitly performed with PixelShuffle (Guo et al., 1 Aug 2025). For scale $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$7, given a feature tensor $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$8, PixelShuffle rearranges it into $I_{rgb} \in \mathbb{R}^{sh \times sw \times 3}$9 according to

$D_{sr}$0

This is followed by RDCB-Mamba refinement at each decoder level (Guo et al., 1 Aug 2025).

Related papers instantiate the same fusion idea with different operators. ADFF in MS-UMamba computes spatial attention

$D_{sr}$1

channel attention over concatenated encoder–decoder features, and a fused output $D_{sr}$2 (Xu et al., 14 Jun 2025). In CVMH-UNet, MFMSBlock computes

$D_{sr}$3

where $D_{sr}$4 is derived from multi-frequency 2D DCT descriptors and local point-wise convolutions (Cao et al., 2024). These formulations show that MSF-UM is less a single fusion formula than a consistent strategy: fuse same-scale encoder and decoder information through learned, content-adaptive weighting rather than direct addition or raw concatenation.

5. Empirical profile in guided depth map super-resolution

MSF-UM is evaluated on NYUv2, Middlebury, Lu, and RGBDD, with scaling factors $D_{sr}$5, $D_{sr}$6, and $D_{sr}$7 (Guo et al., 1 Aug 2025). On NYUv2, the reported RMSE values are 1.15 at $D_{sr}$8, 2.48 at $D_{sr}$9, and 4.67 at $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$0; on Middlebury, 1.19, 1.82, and 3.14; on Lu, 1.05, 1.82, and 3.82; and on RGBDD, 1.17, 1.70, and 2.68 (Guo et al., 1 Aug 2025). The paper states that MSF-UM is especially strong at large-scale depth map super-resolution, often achieving the best RMSE at $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$1 across datasets (Guo et al., 1 Aug 2025).

The parameter count is notably small relative to Transformer-based alternatives. The paper reports 1.27M parameters for the $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$2 model, 1.74M for $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$3, and 2.12M for $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$4, whereas SUFT is reported with 22.01M, 39.95M, and 97.36M parameters for the same scales (Guo et al., 1 Aug 2025). This anchors the claim that the architecture reduces parameter count while remaining competitive or superior in reconstruction accuracy (Guo et al., 1 Aug 2025).

Ablation results isolate the contributions of the three main components. On NYUv2 at $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$5, the baseline RMSE is 7.29; adding the color guidance branch reduces it to 5.85; adding RDCB yields 5.27; adding Mamba yields 5.15; and the full model reaches 4.67 (Guo et al., 1 Aug 2025). On RGBDD at $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$6, the corresponding progression is 3.57, 3.04, 2.91, 2.82, and 2.68 (Guo et al., 1 Aug 2025). These numbers indicate that color guidance, residual dense channel attention, and Mamba-based long-range modeling each contribute independently, with the best performance obtained only when all three are combined.

6. Broader significance, related variants, and limitations

The broader literature suggests that MSF-UM is not confined to depth map super-resolution. In fetal abdominal ultrasound segmentation, MS-UMamba reports mIoU 67.62%, mDice 79.82%, mSen 84.78%, mSpe 99.31%, and mPre 76.11%, using a U-shaped encoder–decoder with SS-MCAT-SSM blocks and ADFF-based multi-scale fusion (Xu et al., 14 Jun 2025). In echocardiography segmentation, MSV-Mamba combines a residual encoder, large-window multiscale Mamba decoder modules, and hierarchical dual-attention fusion, reaching 95.01 and 93.36 for $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$7 and 87.35 and 87.80 for $D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$8 on CAMUS (Yang et al., 13 Jan 2025). In spectral reconstruction, M3SR uses a U-shaped encoder–decoder with multi-perceptual fusion blocks and reports 31.3995 dB PSNR on NTIRE2022 with 2.166M parameters and 100.924G FLOPs (Zhang et al., 13 Jan 2026). In salient object detection, U$D_{hr} \in \mathbb{R}^{sh \times sw \times 1}$9Mamba combines nested U-structures and Mamba bottlenecks, reaching 0.904 maxF$^2$0 and 0.024 MAE on DUTS-TE (Li et al., 18 Jun 2026). This suggests that MSF-UM is best understood as a transferable architectural pattern rather than a domain-locked design.

The explicit depth super-resolution paper also exposes several limitations. It notes potential sensitivity to RGB–depth misalignment, the possibility that noisy or low-light RGB may misguide detail transfer, and the risk of texture-copy artifacts when RGB texture is strong but depth contrast is weak (Guo et al., 1 Aug 2025). It also states that the exact number of U-Net levels, channels per stage, and the precise training loss are not specified, even though the implementation uses PyTorch, an NVIDIA A40 GPU, Adam with $^2$1, $^2$2, initial learning rate $^2$3, decay by 0.1 every 150 epochs, and batch size 2 (Guo et al., 1 Aug 2025). The stated future direction is to optimize Vision Mamba for multimodal super-resolution tasks and improve its integration within UNet-style fusion frameworks (Guo et al., 1 Aug 2025).

Taken together, the available evidence supports a technically specific definition. MSF-UM is a U-shaped, multi-scale fusion architecture in which Mamba provides linear-complexity long-range dependency modeling, residual dense or attention-weighted blocks preserve local detail, and skip-based multi-scale fusion mediates the interaction between coarse semantic structure and fine spatial detail. The named depth super-resolution model provides the clearest canonical instance of the term, while related works show that the same architectural logic extends naturally to segmentation, reconstruction, and dense prediction across disparate imaging modalities (Guo et al., 1 Aug 2025).