---
title: 'MSSFNet: Multi-Scale Fusion Neural Architecture'
url: https://www.emergentmind.com/topics/mssfnet
type: topic
---

# MSSFNet: Multi-Scale Fusion Neural Architecture

MSSFNet refers to a family of neural architectures emphasizing multi-scale and multi-stream/multi-branch fusion, predominantly for edge detection and stereo image super-resolution. The term “MSSFNet” is most notably defined as “Msmsfnet: a multi-stream and multi-scale fusion net for edge detection” [2404.04856], but also appears (sometimes as “MSSF-Net”) in stereoSR [2406.16001] and building interpretation [2504.00759] literature. This article reviews the core formulation, mathematical structure, training regime, and empirical impact of MSSFNet for edge detection, with brief comparative notes on related usages.

## 1. Network Architecture Overview

MSSFNet, in the edge detection setting [2404.04856], is a fully convolutional encoder of depth 74, constructed by recursive stacking of a “msmsfblock” that embodies parallel convolutional streams with diverse receptive fields. Five max-pooling layers (3×3, stride 2) are interleaved to yield multi-resolution feature hierarchies.

The main block, msmsfblock, takes as input $X\in\mathbb R^{C\times H\times W}$ and processes it along four parallel convolutional streams:
- **Stream 1** (receptive field 1×1): $T_1 = W_1^{1\times1} * X$
- **Stream 2** (3×3): $T_2 = W_2^{3\times1} * (W_2^{1\times3} * X)$
- **Stream 3** (5×5): $T_3 = W_3^{5\times1} * (W_3^{1\times5} * X)$
- **Stream 4** (7×7): $T_4 = W_4^{7\times1} * (W_4^{1\times7} * X)$

Following the multi-stream convolution, the features are fused in a two-stage system:
- **First fusion:** $S_\ell = T_1 + T_2$, $S_h = T_3 + T_4$
- **Final fusion:** $Y = \mathrm{ReLU}(W_f^{3\times1} * (S_\ell + S_h))$

No batch normalization is used; only the final convolution within each block is followed by a ReLU. Deep supervision is applied via four side outputs after select blocks, mimicking the HED paradigm.

## 2. Multi-Scale Fusion Principle

Central to MSSFNet is the msmsfblock, which simultaneously encodes fine and coarse spatial features across scales $\{1,3,5,7\}$. All larger convolutions (3×3, 5×5, 7×7) are expanded into factorized $1\times n$ and $n\times 1$ layers to control parameter growth. Table 1 below summarizes stream structure:

| Stream | Receptive Field | Parameterization   |
|--------|-----------------|-------------------|
| 1      | 1×1             | $1\times1$ conv   |
| 2      | 3×3             | $1\times3$, $3\times1$ |
| 3      | 5×5             | $1\times5$, $5\times1$ |
| 4      | 7×7             | $1\times7$, $7\times1$ |

This design ensures that both local boundaries and broader context are encoded, addressing the trade-off between localization and context aggregation in edge detection.

## 3. Objective Functions and Optimization

Let $u$ denote an input image, with pixelwise binary ground-truth edges $\mathbb G_j$. Defining $\lambda = |\mathbb G_-|/|\mathbb G|$ as class balance, the side-output losses at pixel $j$ and block $m\in\{1,2,3\}$ are

$$
\ell_{\mathrm{side}}^{(m)} = -\,\lambda\,\sum_{j\in\mathbb G_+} \log \sigma(a_j^{(m)}) - 1.1(1-\lambda) \sum_{j\in\mathbb G_-} \log(1-\sigma(a_j^{(m)}))
$$

where $a_j^{(m)}$ are the pre-sigmoid activations, and the negative class is upweighted by 1.1 (as in RCF). The total side loss, plus a fused output loss after a final 3×3 conv, is minimized over all weights.

No batch normalization or pre-trained initialization is used: all models are optimized from scratch with Adam (weight decay $1\times10^{-12}$), mini-batch size 6, and learning rate $1\times10^{-4}$, decayed after a set number of epochs depending on the dataset.

## 4. Empirical Performance and Generalization

When trained from scratch—circumventing all reliance on ImageNet pretraining—MSSFNet surpasses canonical edge detectors in standard metrics on BIPEDv2, BSDS500, and NYUDv2. Results for the main metrics (ODS, OIS, AP) are as follows:

| Dataset     | Model      | ODS    | OIS    | AP    |
|-------------|------------|--------|--------|-------|
| BIPEDv2     | MSSFNet    | 0.897  | 0.901  | 0.936 |
| BSDS500     | MSSFNet    | 0.816  | 0.835  | 0.859 |
| NYUDv2 RGB  | MSSFNet    | 0.732  | 0.747  | 0.744 |

On BSDS500, MSSFNet's ODS ($0.816$) exceeds the reported human benchmark ($\approx0.803$) in this regime. The network demonstrates comparable inference speed and complexity to RCF or BDCN, due to heavy reuse of asymmetric convolutions and minimal channel expansion.

Generalization to different modalities is an explicit design goal: MSSFNet abandons pretraining to facilitate application to non-optical data such as SAR imagery, though quantitative SAR benchmarks remain for future work.

## 5. Related Multi-Scale/Multi-Stream Fusion Networks in Vision

The MSSFNet moniker has also been adopted for architectures targeting:
- **Stereo Image Super-Resolution:** “Mixed-Scale Selective Fusion Network (MSSFNet)” incorporates mixed-scale blocks, selective fusion attention (SFAM), and fast Fourier convolution (FFCB) [2406.16001]. It achieves state-of-the-art PSNR/SSIM on standard datasets (e.g., MSSFNet-S: 35.77 dB/0.9555 on Middlebury for 2× upscaling), with ablations confirming the necessity of each component.
- **Remote Sensing Building Interpretation:** “MSSFC-Net” (sometimes “MSSFNet”) fuses a dual-branch multi-scale contextual encoder with spatial-spectral attention and a differential temporal fusion module [2504.00759], delivering $\Delta\text{IoU}$ gains ($+1.07\%$ on WHU, $+0.50\%$ on LEVIR-CD) over previous methods.

While architectural details diverge across domains, the unifying theme remains explicit multi-scale convolutional encoding, channel/stream fusion, and intentional avoidance of pre-trained weights to unlock broad generalizability.

## 6. Practical Recommendations and Prospects

In edge detection, MSSFNet demonstrates that systematic multi-scale fusion via parallel asymmetric convolutions can achieve, and even surpass, the performance of pre-trained deep backbones—when paired with carefully balanced loss terms and well-tuned optimization from random initialization.

A plausible implication is strong suitability for imaging domains lacking large annotation corpora (e.g., SAR or medical), where pretraining is unavailable. Future research directions include empirical validation of MSSFNet on such non-optical datasets and further analysis of multi-stream versus transformer-based fusion paradigms in low-data or cross-modal settings.

## 7. Summary Table: Distinct MSSFNet Variants

| Domain & Paper                                                | Purpose                        | Key Modules/Fusions                                    |
|--------------------------------------------------------------|-------------------------------|--------------------------------------------------------|
| Edge detection [2404.04856]                                  | Edge detection from scratch   | msmsfblock: 4-stream parallel asymmetric conv fusion   |
| StereoSR [2406.16001]                                        | Stereo image SR               | Mixed-scale block, SFAM, FFCB                          |
| Building extraction/change detection [2504.00759]            | Remote sensing dual tasks     | DMFE (dual branch), SSFC, MDFM                         |

Each MSSFNet instantiation is defined by the use of explicit multi-scale fusion, either within blocks or across parallel streams, often combined with specialized attention or inter-feature weighting. Empirical results support the conclusion that this architectural philosophy affords strong performance, namely precision, recall, and intersection-over-union, in highly challenging pixel-wise vision tasks.

Source: https://www.emergentmind.com/topics/mssfnet