MSA2-Net: Adaptive Multi-Scale Segmentation

• MSA2-Net is a hybrid encoder-decoder framework that adaptively fuses local and global features through multi-scale skip connections for precise medical image segmentation.
• It integrates a pretrained MaxViT encoder, LKA-based shallow decoder, and DAE-Former deep decoder to balance high-resolution detail with long-range semantic context.
• The innovative MASAG module dynamically performs multi-scale fusion, spatial selection, and recalibration to enhance boundary delineation and suppress background noise.

Searching arXiv for the specified paper and closely related work to ground the article. MSA$^2$Net, short for Multi-scale Adaptive Attention-guided Network, is a hybrid encoder–decoder framework for medical image segmentation that targets variation in the size, shape, and density of anatomical structures and lesions. It was introduced as a method for integrating local features such as edges, texture, fine boundaries, and small structures with global features such as organ-level shape, long-range relationships, and semantic context. The defining architectural claim is that many earlier systems concentrated on improving the encoder or decoder while leaving skip connections comparatively simple; MSA$^2$Net instead redesigns the skip pathway through a Multi-Scale Adaptive Spatial Attention Gate (MASAG) that adaptively fuses encoder and decoder features across scales (Kolahi et al., 2024).

1. Problem setting and design rationale

Medical image segmentation assigns each pixel or voxel to an anatomical structure or lesion. The target setting includes organs in CT and skin lesions in dermoscopy, where segmentation is complicated by large variation in size, shape, appearance/density, texture and contrast, and background clutter / low contrast boundaries (Kolahi et al., 2024). The paper frames these difficulties as a multi-scale representation problem: fine boundaries and small structures require strong local sensitivity, whereas organ-level structure and disambiguation from clutter require broader semantic context.

The method is positioned between two established design tendencies. CNN-only models, including U-Net-style systems, are described as effective at local pattern extraction but limited by static / fixed receptive fields, reduced capacity for long-range dependencies, and difficulty integrating context across distant regions. Transformer-only models provide global context through self-attention, but in dense medical segmentation they are described as having a weaker local inductive bias, higher computational cost, especially due to quadratic self-attention, and lower efficiency in preserving fine-grained spatial detail (Kolahi et al., 2024).

MSA$^2$Net is therefore organized around a specific synthesis: combine CNN-like local processing and transformer-like global modeling, and redesign the skip connections so that coarse semantic information and fine spatial information are fused adaptively rather than by fixed concatenation or addition. This suggests that the architecture treats the skip pathway not as a passive transport channel but as an active site of feature selection and recalibration.

2. Architectural composition

The network is a hybrid segmentation architecture with three named elements: a hierarchical transformer encoder, a mixed decoder, and the MASAG skip module (Kolahi et al., 2024).

Component	Instantiation	Stated role
Encoder	pretrained MaxViT	hierarchical multi-scale representations
Shallow decoder	LKA modules	efficient high-resolution feature processing
Deep decoder	DAE-Former blocks	preserve long-range dependencies at lower resolution
Skip pathway	MASAG	adaptive, multi-scale, spatially selective fusion

The encoder uses a pretrained MaxViT backbone. In the description provided, MaxViT is hierarchical and captures multi-scale representations through a combination of convolutional blocks and attention. This places the encoder in the class of transformer backbones that still preserve some convolutional structure, aligning with the paper’s objective of mixing local and global cues.

The decoder is heterogeneous. Shallow decoder layers use LKA (Large Kernel Attention) modules for efficient high-resolution processing, while deeper decoder layers use DAE-Former blocks to preserve long-range dependencies at lower spatial resolution while combining spatial and channel attention (Kolahi et al., 2024). This split implies an explicit division of labor across the decoder hierarchy: high-resolution stages focus on detail-sensitive processing, and low-resolution stages maintain broader semantic relationships.

The skip connections distinguish MSA$^2$Net from standard encoder–decoder designs. Encoder feature maps are not simply copied and concatenated with decoder activations. Instead, they are passed through MASAG, which fuses encoder feature maps $\mathbf{X}$ with decoder feature maps $\mathbf{G}$ so that the decoder context can guide the relevance of encoder detail, and encoder detail can refine decoder semantics. The stated goal is to facilitate feature fusion by dynamically weighting and combining information across scales (Kolahi et al., 2024).

3. MASAG: Multi-Scale Adaptive Spatial Attention Gate

MASAG is the primary novelty of MSA$^2$Net. It is defined as a spatial attention-based gating mechanism that dynamically adjusts receptive fields and selectively fuses features across scales. The module has four stages: multi-scale feature fusion, spatial selection, spatial interaction and cross-modulation, and recalibration (Kolahi et al., 2024).

Multi-scale feature fusion

Let $\mathbf{X}$ denote the encoder feature map and $\mathbf{G}$ the decoder feature map. MASAG first combines local context from the encoder with global context from the decoder:

$$\mathbf{U} = \text{Conv}_{1 \times 1}\big(\text{DW-D}(\text{DW}(\mathbf{X}))\big) + \text{Conv}_{1 \times 1}\left(\left[\text{P}_{Avg}(\mathbf{G}); \text{P}_{Max}(\mathbf{G})\right]\right).$$

Here, $^2$0 is depthwise convolution, $^2$1 is dilated depthwise convolution, and $^2$2 and $^2$3 are average pooling and max pooling, respectively (Kolahi et al., 2024). In the paper’s interpretation, this produces a feature representation containing both local fine detail from the encoder and global semantic context from the decoder.

Spatial selection

The fused map $^2$4 is then projected to two channels and normalized using a channel-wise softmax:

$^2$5

where $^2$6 is the softmax across the two channels. These spatial weights refine the two streams as

$^2$7

The operator $^2$8 denotes element-wise multiplication (Kolahi et al., 2024). The residual additions are explicitly described as preserving gradient flow and original information. Functionally, the gate learns, for each spatial position, whether encoder detail or decoder semantics should dominate.

Spatial interaction and cross-modulation

MASAG then performs bidirectional modulation:

$^2$9

followed by

$^2$0

with $^2$1 denoting the sigmoid (Kolahi et al., 2024). This stage is described as a bidirectional attention exchange in which encoder features are refined by decoder-derived global context and decoder features are refined by encoder-derived local detail.

Recalibration

Finally, the interaction output is converted into an attention map and used to recalibrate the encoder feature map:

$^2$2

The paper’s interpretation is that this reinforces the most relevant spatial regions while suppressing background regions (Kolahi et al., 2024). That target behavior is especially pertinent in settings where lesions or organs are small, ambiguous, or surrounded by clutter.

The paper further states that MASAG addresses three recurrent segmentation problems: fixed receptive fields, weak fusion in skip connections, and background noise / false positives. A supplementary frequency analysis is said to suggest that MASAG helps preserve high-frequency detail that can otherwise be lost in deep transformer layers (Kolahi et al., 2024). This suggests that the skip-gating mechanism has implications not only for semantic fusion but also for structural fidelity at object boundaries.

4. Experimental protocol

The reported evaluation uses two benchmark datasets selected to represent distinct segmentation regimes: Synapse multi-organ CT and ISIC 2018 dermoscopy (Kolahi et al., 2024).

Dataset	Composition	Reported metrics
Synapse	30 cases; 3779 axial abdominal CT images; spleen, right kidney, left kidney, gallbladder, liver, stomach, aorta, pancreas	DSC, HD95
ISIC 2018	2594 skin lesion images with ground truth annotations	DSC, SE, SP, ACC

For Synapse, the metrics are Dice Similarity Coefficient (DSC) and 95th percentile Hausdorff Distance (HD95), reported per organ and averaged. For ISIC 2018, the reported metrics are DSC, sensitivity (SE), specificity (SP), and accuracy (ACC) (Kolahi et al., 2024).

The implementation details are explicit. The framework is PyTorch, the hardware is NVIDIA RTX 3090, the input resolution is 224 × 224, and the encoder is a pretrained MaxViT (Kolahi et al., 2024).

For Synapse, training uses:

• batch size 20
• optimizer SGD
• learning rate 0.05
• momentum 0.9
• weight decay 0.0001
• 700 epochs

For ISIC 2018, training uses:

• batch size 16
• optimizer Adam
• learning rate 0.0001
• 50 epochs

The loss is Boundary Difference over Union (BDoU) loss, which is introduced to improve boundary awareness (Kolahi et al., 2024). Given the centrality of fine delineation in both abdominal CT and dermoscopy, the use of a boundary-aware loss is consistent with the architectural emphasis on preserving high-frequency and contour information.

5. Quantitative results and ablation evidence

On Synapse, MSA$^2$3Net achieves an Average DSC of 84.75 and an Average HD95 of 13.29. These are reported as the best HD95 in the table and the best average DSC among the listed methods (Kolahi et al., 2024). The comparison set includes TransUNet, Swin-UNet, MISSFormer, ScaleFormer, HiFormer-B, DAEFormer, PVT-CASCADE, and 2D D-LKA Net.

The reported comparison highlights are specific:

• it beats 2D D-LKA Net by 0.48 DSC and improves HD95 substantially,
• it beats DAEFormer by 2.21 DSC,
• it beats HiFormer-B by 4.36 DSC and improves HD95 by 1.41 (Kolahi et al., 2024).

The paper also reports strong organ-level gains, especially for the pancreas, aorta, liver, spleen, and kidneys. It emphasizes that the gains are meaningful for both small organs and larger organs, suggesting that the adaptive receptive-field mechanism helps across scales (Kolahi et al., 2024).

On ISIC 2018, MSA$^2$4Net achieves:

• DSC: 0.9129
• SE: 0.8840
• SP: 0.9557
• ACC: 0.9640

These values yield the best DSC and best ACC in the reported table (Kolahi et al., 2024). The comparison set includes U-Net, AttU-Net, TransUNet, FAT-Net, Swin-UNet, UCTransNet, and DermoSegDiff. The paper attributes the dermoscopy performance to improved handling of lesion boundaries and background suppression, which is central in images with irregular lesion boundaries and confusing skin textures.

The ablation study on Synapse is used to isolate the contribution of the major components. The stated observations are that MASAG improves both Dice and HD95, and that combining MASAG + LKA + DAE-Former gives the best result, namely Dice: 84.75 and HD95: 13.29 (Kolahi et al., 2024). The component roles are summarized as complementary:

• MASAG improves adaptive feature fusion in skip connections,
• LKA helps local-global handling in high-resolution decoder stages,
• DAE-Former preserves long-range dependencies in deeper decoder stages.

A supplementary parameter analysis reports that all three MASAG modules together have 9.44M parameters, approximately 8% of the full model (Kolahi et al., 2024). This supports the interpretation that the skip-path redesign is a targeted intervention rather than the dominant source of model size.

6. Position within the literature and nomenclature

MSA$^2$5Net belongs to a strand of medical segmentation research that seeks to combine local detail sensitivity and global context modeling without committing exclusively to convolutional or transformer mechanisms. Its specific contribution is not merely hybridization of encoder and decoder blocks, but the argument that skip connections themselves should be made adaptive, multi-scale, and spatially selective (Kolahi et al., 2024). A plausible implication is that the paper re-centers the skip pathway as a learnable fusion operator rather than a static architectural convenience.

The name can create bibliographic ambiguity. A later paper titled "MSA2-Net: Utilizing Self-Adaptive Convolution Module to Extract Multi-Scale Information in Medical Image Segmentation" introduces a different architecture based on a Self-Adaptive Convolution Module, CSWin Transformer, MSConvBridge, and MSADecoder, and reports results on Synapse, ACDC, Kvasir-SEG, and ISIC2017 (Deng et al., 1 Sep 2025). That model is distinct from the 2024 MSA$^2$6Net defined by MaxViT, LKA, DAE-Former, and MASAG.

A separate naming overlap occurs with MSANet, the Multi-Scale Adaptive Network for Single Image Denoising, which addresses image restoration rather than medical segmentation and is built around AFeB, AMB, and AFuB (Gou et al., 2022). Consequently, references to “MSA2-Net” in secondary sources may denote materially different systems unless the corresponding arXiv identifier is checked.

Within its own formulation, MSA$^2$7Net is best understood as a medical segmentation architecture in which adaptive skip fusion is the central organizing principle. Its reported performance on Synapse and ISIC 2018, together with the ablation evidence for MASAG, places it among methods that treat multi-scale fusion and boundary-aware selection as first-class design problems rather than byproducts of a conventional encoder–decoder stack (Kolahi et al., 2024).