SMMM: Structural-aware Multi-scale Masking Module

• SMMM is a module that refines skip connections by fusing multi-scale features with structural saliency for improved medical image segmentation.
• It employs parallel depthwise-separable convolutions and a spatial saliency gating system to capture both fine and coarse spatial contexts.
• The approach enhances boundary delineation, reducing noise in segmentation outputs and achieving sharper localization of lesions and organs.

The Structural-aware Multi-scale Masking Module (SMMM) constitutes a refinement mechanism for skip connections in deep encoder–decoder architectures, particularly in the context of medical image segmentation. Instead of employing naïve addition or concatenation of encoder/decoder features, SMMM introduces a multi-scale, boundary-sensitive, and structurally selective fusion. It achieves this via (i) parallel depthwise-separable convolutional streams that extract both fine and coarse spatial context, (ii) a spatial saliency gating system that highlights structural boundaries, and (iii) reweighting and fusing the masked features prior to integration into the decoder. SMMM works synergistically within structured-aware decoders—alongside attention mechanisms like ACFA and frequency–spatial fusion modules like TFFA—by filtering out redundant and noisy activations, accentuating salient edges, and adapting its receptive field dynamically to image detail scale (Zhang et al., 5 Dec 2025).

1. Motivations and Principle of Operation

In conventional U-Net and similar architectures, skip connections transfer spatial features directly from encoder to decoder, preserving localization but indiscriminately propagating both semantic and noisy information. This often results in blurry object boundaries or misidentification of lesion regions. SMMM was developed to address this by implementing: (a) multi-scale context extraction using depthwise-separable 3×3 and 5×5 convolutions; (b) structural saliency filtering via learned gating responses; (c) spatially-aware fusion of masked features. Its goal is to maximize the semantic interaction quality at each skip connection, especially for tasks demanding precise geometric fidelity (e.g., organ and tumor boundary delineation).

2. Architectural Composition and Mathematical Formulation

SMMM operates on paired encoder and decoder feature maps of identical shape, $X, Y \in \mathbb{R}^{B \times C \times H \times W}$, producing a refined output $Z$.

Multi-Scale Feature Streams

Each input undergoes $1 \times 1$ convolution for channel alignment: $$\widehat{X}_0 = f^{Conv}_{1\times1}(X), \quad \widehat{Y}_0 = f^{Conv}_{1\times1}(Y)$$ Two-stage, dual-kernel depthwise-separable subnetworks are applied: $$\begin{align*} \widehat{F}_1 &= \text{ReLU}(f^{DWConv}_{3\times3}(F_0)) \ \widehat{F}_2 &= \text{ReLU}(f^{DWConv}_{5\times5}(F_0)) \ S_1 &= \text{ReLU}(f^{DWConv}_{3\times3}(\mathrm{Cat}(\widehat{F}_1, \widehat{F}_2))) \ S_2 &= \text{ReLU}(f^{DWConv}_{5\times5}(\mathrm{Cat}(\widehat{F}_1, \widehat{F}_2))) \ \widehat{F} &= f^{Conv}_{1\times1}(\mathrm{Cat}(S_1, S_2)) \end{align*}$$ where Cat denotes channel concatenation.

Structural Saliency Filtering

From $\widehat{X}$ and $\widehat{Y}$, three channel-gating responses $$G_i \in \mathbb{R}^{B \times 1 \times H \times W}, i\in\{1,2,3\}$$ are computed. Per spatial position $(x, y)$: $s_i(x, y) = [G_i]_{:,1,x,y}$ Normalized softmax gating produces pixel-wise masks $\alpha_i(x, y)$: $$\alpha_i(x, y) = \frac{\exp(s_i(x, y))}{\sum_{j=1}^{3} \exp(s_j(x, y))}$$

Masked Feature Fusion

Masked features for each stream: $M_i = \widehat{F} \odot \alpha_i$ Summation yields the fused map: $M = \sum_{i=1}^{3} M_i$ Context aggregation is performed by dilated convolution (kernel $3 \times 3$, dilation $2$), followed by batch normalization and final $1 \times 1$ convolution: $F_{dil} = f^{DilConv}_{3\times3, d=2}(M)$

$Z = f^{Conv}_{1\times1}(\mathrm{BN}(F_{dil}))$

3. Implementation Workflow

The following pseudocode encapsulates the SMMM operations for each skip-connection:

def SMMM(X, Y): # X, Y: B×C×H×W X0 = Conv1x1(X) Y0 = Conv1x1(Y) def multi_scale(F0): F1 = ReLU(DWConv3x3(F0)) F2 = ReLU(DWConv5x5(F0)) S1 = ReLU(DWConv3x3(cat(F1, F2))) S2 = ReLU(DWConv5x5(cat(F1, F2))) return Conv1x1(cat(S1, S2)) X_hat = multi_scale(X0) Y_hat = multi_scale(Y0) G1 = Gate1(X_hat + Y_hat) G2 = Gate2(X_hat + Y_hat) G3 = Gate3(X_hat + Y_hat) S = Softmax(concat(G1, G2, G3), dim='channel') M = S[1]*(X_hat+Y_hat) + S[2]*(X_hat+Y_hat) + S[3]*(X_hat+Y_hat) Fd = DilatedConv3x3_d2(M) Z = Conv1x1(BatchNorm(Fd)) return Z

4. Training Protocols and Hyperparameterization

SMMM does not require specialized loss functions; it is trained end-to-end via the standard Dice and cross-entropy losses used for overall segmentation. The gating softmax ensures spatial normalization, and the dilated convolution promotes structural continuity. Key hyperparameters are:

Component	Value/Type	Comment
Scales (kernels)	3×3, 5×5	Parallel, two stages per input
Number of gates	3	Independent gating filters
Dilation (final conv)	2	For context aggregation
Activation	ReLU	Standard rectification
Channel count	$C$	Preserved throughout
Param overhead	+10.51M	32.01M ⇒ 42.52M total
Compute increase	+4.54 GMac	13.75 ⇒ 18.29 GMac
Initialization	Xavier or PyTorch	Gating inherits encoder init (PVTv2-b2)

5. Empirical Performance and Observed Effects

Ablation studies evidence the contribution of SMMM to segmentation accuracy and geometric localization on benchmark datasets.

• Synapse (Multi-Organ CT):
  • With only ACFA+TFFA, DSC = 83.23%, HD95 = 16.72 mm.
  • Adding SMMM: DSC = 83.92% (Δ+0.69%), HD95 = 18.91 mm.
  • This suggests SMMM enables sharper boundary localization even when organ interfaces are complex (Zhang et al., 5 Dec 2025).
• ISIC 2017 (Dermoscopic Lesions):
  • Without SMMM: DSC = 89.15%, SE = 89.83%, ACC = 96.85%.
  • With SMMM: DSC = 91.40% (Δ+2.25%), SE = 92.75%, ACC = 97.26%.
  • The saliency mask mechanism demonstrably increases both dice and sensitivity, improving edge-lock onto lesions.

Qualitative visualization (see “Synapse.pdf,” “ISIC2017.pdf”) reveals that SMMM produces skip-connection activations tightly contouring organ and lesion boundaries, in contrast to the diffuse, error-prone activations of naïve fusion.

6. Contextual Role within Structured-aware Decoders

Within the structured-aware decoder studied in (Zhang et al., 5 Dec 2025), SMMM is designed to operate in concert with ACFA (directional attention) and TFFA (frequency–spatial fusion). ACFA enhances directional edge response, TFFA consolidates frequency-spatial representations, and SMMM selectively amplifies structural saliency while suppressing background noise during skip connection fusion. A plausible implication is that this modular arrangement permits explicit disentangling of edge, texture, and context sensitivity, thereby improving both segmentation accuracy and generalization in medical imaging tasks.

7. Computational Impact and Practical Considerations

SMMM introduces moderate overhead (+10.51M parameters; +4.54 GMac compute), but its lightweight design (depthwise-separable convolutions, gated masking) supports efficient deployment. No bottleneck splits are introduced, and the module can inherit initialization from the encoder. There is no requirement for auxiliary training objectives, nor evidence of adverse effects on overall optimization. The main practical benefit, substantiated both quantitatively and visually, is consistently sharper boundary prediction and semantic fusion at each decoder stage, especially in images with complex or blurred anatomical interfaces.