Multi-scale Complementary Feature Extraction Module

• MCFEM is a neural module design that extracts features at multiple scales using complementary CNN and transformer mechanisms for robust local and global pattern fusion.
• It addresses the limitation of single receptive fields by combining fine-grained edge detection with broad semantic context to boost feature hierarchies.
• Its implementations—such as pyramidal CNN–transformer fusion, cascaded window-based attention, and convolutional multi-branch extraction—drive advances in classification, pose estimation, and segmentation.

A Multi-scale Complementary Feature Extraction Module (MCFEM) denotes a class of neural components that jointly learn and fuse features that capture local spatial patterns at multiple granularities, such as edges, textures, and shapes, with global, long-range dependencies, such as context and semantics, across different resolutions or strides. In this usage, “complementary” refers to the fact that convolutional neural networks and transformers encode different inductive biases: CNNs provide locality, translation equivariance, and hierarchical aggregation, while transformers provide dynamic, content-dependent receptive fields via attention that capture cross-region correlations. Recent arXiv work instantiates this design space in several distinct ways: CTA-Net frames MCFEM around a CNN pyramid, Light Weight Multi-Scale Feature Fusion Multi-Head Self-Attention, and Reverse Reconstruction CNN-Variants (Meng et al., 2024); CMSA realizes a resolution-preserving, grouped, window-based, cascaded attention mechanism for low-resolution inputs (Lu et al., 2024); and a dermoscopic segmentation network realizes the same principle through a Residual Multi-Scale Module with dual attention refinement (Chowdary et al., 2021).

1. Definition and problem setting

MCFEM addresses the general problem that a single receptive field is seldom sufficient. Small kernels are sensitive to local edge and texture cues, but miss broader context; larger kernels gather coarse semantic information, but can smooth away fine boundary detail. In hybrid CNN–ViT settings, the same asymmetry appears at the architectural level: on small datasets, pure transformers often struggle to generalize because limited data are available to learn positional and locality priors, whereas CNNs can underperform on tasks requiring long-range reasoning (Meng et al., 2024).

This leads to a unified definition of MCFEM across the cited work. In CTA-Net, multi-scale complementary feature extraction is the process of jointly learning and fusing local spatial patterns at multiple granularities with global, long-range dependencies across different resolutions or strides. In CMSA, the same objective is expressed as extracting multi-scale features complementarily for each spatial location, integrating and interacting across scales, and preserving resolution inside the module so that low-resolution inputs are not further degraded (Lu et al., 2024). In the dermoscopic segmentation model, the same principle appears as a multi-scale module whose branches provide distinct receptive fields and whose dual attention modules selectively weight non-redundant responses to improve robustness to poor contrast, irregular shapes, fuzzy boundaries, and artifacts (Chowdary et al., 2021).

A common misconception is that multi-scale extraction necessarily requires explicit spatial downsampling inside the feature extraction block. CMSA directly contradicts that assumption: it changes effective receptive field sizes via windowing, including one global group, rather than spatial reduction within the block (Lu et al., 2024). Another misconception is that “complementary” implies an explicit diversity regularizer. The dermoscopic segmentation work states that no explicit diversity regularizer is used, and that complementarity is promoted implicitly by design through different kernels, residual concatenation, and attention weights (Chowdary et al., 2021). This suggests that MCFEM is better understood as a design principle than as a single canonical operator.

2. Canonical architectural patterns

Across the cited work, MCFEM appears in three recurring patterns: pyramidal CNN–Transformer fusion, fixed-resolution cascaded attention, and convolutional multi-branch extraction with residual concatenation.

Instantiation	Core operators	Integration pattern
CTA-Net	CNN pyramid, LMF-MHSA, RRCV	CNN backbone produces $\{S_1,S_2,S_3,S_4\}$, tokens are fused, locality is re-injected, then classified
CMSA	Grouped MHSA, window-based local attention, CF, SF	Stage-internal attention block preserves spatial size, with stage-wise patch embedding between stages
RMSM-based segmentation network	$1\times1$, $3\times3$, $5\times5$ branches, DF-RAM, EF-RAM	U-Net-like encoder–bridge–decoder with refined skips and post-upsampling refinement

In CTA-Net, the high-level pipeline is explicitly specified: an input image resized to $224\times224$ passes through a CNN backbone that builds a feature pyramid $\{S_1,S_2,S_3,S_4\}$ with strides $\{4,8,16,32\}$ and channels $\{C_1,C_2,C_3,C_4\}$; each scale is projected to token embeddings with positional encodings; the Light-weight Multi-scale Multi-Head Self-Attention module performs multi-scale fusion; the Reverse Reconstruction CNN-Variants module maps tokens back to spatial maps, applies residual CNN variants, and re-embeds them to tokens; and a classifier head uses global pooling or a CLS token with an MLP classifier (Meng et al., 2024).

The interface is equally explicit. CNN stages produce $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$, $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$, $1\times1$0, and $1\times1$1. Each $1\times1$2 is aligned to a target embedding dimension $1\times1$3 via a $1\times1$4 convolution, $1\times1$5, then flattened to tokens $1\times1$6 with $1\times1$7, after which positional encodings are added: $1\times1$8 (Meng et al., 2024).

CMSA places MCFEM inside a hybrid block between a conditional positional encoding and a standard feed-forward network with residual connections. The overall network remains hierarchical and pyramidal, but within each stage CMSA preserves the current spatial resolution and performs multi-scale extraction through different window sizes only (Lu et al., 2024). By contrast, the dermoscopic segmentation network embeds its multi-scale module in a 4-level U-Net-like encoder–bridge–decoder: three RMSM modules in the encoder, one RMSM in the bridge, and RMSM plus attention refinement in each decoder stage (Chowdary et al., 2021).

3. Attention-based multi-scale fusion

The attention-centric version of MCFEM is most explicit in CTA-Net and CMSA. CTA-Net starts from standard multi-head self-attention. For tokens $1\times1$9, the projections are $3\times3$0, $3\times3$1, $3\times3$2, where $3\times3$3, and one head computes

$3\times3$4

LMF-MHSA extends this to multiple scales by constructing multi-scale streams from the CNN pyramid or from multi-resolution tokens derived within a single scale. A practical instantiation splits $3\times3$5 attention heads across $3\times3$6 scales, computes $3\times3$7, $3\times3$8, $3\times3$9, and forms

$5\times5$0

Fusion is then performed either by concatenation and projection,

$5\times5$1

or by a gated weighted sum,

$5\times5$2

with optional adjacent-scale cross-attention as a cost-aware refinement (Meng et al., 2024).

CMSA also partitions attention into groups, but its scale index is realized by window size rather than by an explicit CNN feature pyramid. Let $5\times5$3 be the input. Point-wise convolutions produce $5\times5$4, $5\times5$5, and $5\times5$6, channels are split into $5\times5$7 groups, and each group attends over non-overlapping windows of different sizes $5\times5$8. The largest group uses a global window, typically $5\times5$9, while subsequent groups use smaller windows such as $224\times224$0 and $224\times224$1 (Lu et al., 2024).

The defining feature of CMSA is its cascaded fusion mechanism. For group $224\times224$2, the previous group output $224\times224$3 is fused into the current key/value stream by first concatenating

$224\times224$4

then applying Channel Fusion and Spatial Fusion:

$224\times224$5

For $224\times224$6,

$224\times224$7

Windowed attention is then computed inside each group, and final aggregation uses

$224\times224$8

This mechanism enforces complementary multi-scale interaction because successive groups use both their own queries and fused keys/values enriched by the previous scale (Lu et al., 2024).

The two formulations differ in implementation detail but converge on the same principle: separate scale-specific processing, lightweight parameterization, and a learned fusion operator that preserves or restores cross-scale complementarity. This suggests that an MCFEM can be indexed either by pyramid level or by window scale, provided that interaction between scales is not reduced to naive averaging.

4. Convolutional refinement, residual paths, and locality priors

A second major theme is that multi-scale attention alone is not treated as sufficient. CTA-Net introduces Reverse Reconstruction CNN-Variants specifically to reintroduce CNN inductive biases—local connectivity and spatial hierarchies—into the transformer stream. Given tokens $224\times224$9 and a scale shape $\{S_1,S_2,S_3,S_4\}$0, the module reshapes them to a map $\{S_1,S_2,S_3,S_4\}$1, applies a lightweight residual CNN block

$\{S_1,S_2,S_3,S_4\}$2

then flattens and stabilizes the result through

$\{S_1,S_2,S_3,S_4\}$3

An optional residual gating with the attention output is

$\{S_1,S_2,S_3,S_4\}$4

where $\{S_1,S_2,S_3,S_4\}$5 is a learned scalar gate initialized small, for example $\{S_1,S_2,S_3,S_4\}$6 (Meng et al., 2024).

The same source states that ablations show ResNet consistently outperforms standard CNNs and depthwise-only variants across APTOS2019, RFMiD2020, CIFAR-10, and CIFAR-100, with gains including $\{S_1,S_2,S_3,S_4\}$7 over standard CNNs and $\{S_1,S_2,S_3,S_4\}$8 over DW-Conv on CIFAR-10, and $\{S_1,S_2,S_3,S_4\}$9 and $\{4,8,16,32\}$0 on CIFAR-100 (Meng et al., 2024). Within that framework, ResNet variants are therefore the preferred local refiner in RRCV.

The convolutional formulation of MCFEM is explicit in the Residual Multi-Scale Module used for skin lesion segmentation. For an input feature map $\{4,8,16,32\}$1, the module contains three parallel branches:

$\{4,8,16,32\}$2

Their outputs are concatenated, $\{4,8,16,32\}$3, compressed with a bottleneck convolution, $\{4,8,16,32\}$4, and combined with the identity path by residual concatenation:

$\{4,8,16,32\}$5

In encoder stages used for downsampling, the fusion $\{4,8,16,32\}$6 uses stride $\{4,8,16,32\}$7 (Chowdary et al., 2021).

That segmentation network couples RMSM with two attention mechanisms. DF-RAM refines encoder skip features using decoder semantics before concatenation, and EF-RAM refines post-upsampling decoder features to counter interpolation-induced noise (Chowdary et al., 2021). The paper states that these modules have complementary roles rather than redundant ones: DF-RAM aligns skip features with decoder semantics, whereas EF-RAM refines noisy upsampled features.

5. Efficiency, complexity, and empirical performance

Efficiency is a first-order concern in the MCFEM literature, and the cited work gives explicit mechanisms for reducing cost. In CTA-Net, parameter reduction in LMF-MHSA comes from depthwise separable token convolutions for local enhancement of $\{4,8,16,32\}$8 and $\{4,8,16,32\}$9, grouped or shared projections, low-rank factorization of projection matrices, and head-wise window attention for local tokens (Meng et al., 2024). The parameter comparison is concrete: depthwise-separable parameters scale as $\{C_1,C_2,C_3,C_4\}$0 versus standard $\{C_1,C_2,C_3,C_4\}$1; for $\{C_1,C_2,C_3,C_4\}$2 and $\{C_1,C_2,C_3,C_4\}$3, this is approximately $\{C_1,C_2,C_3,C_4\}$4 versus approximately $\{C_1,C_2,C_3,C_4\}$5 (Meng et al., 2024).

CTA-Net reports approximately $\{C_1,C_2,C_3,C_4\}$6 parameters and approximately $\{C_1,C_2,C_3,C_4\}$7 FLOPs at $\{C_1,C_2,C_3,C_4\}$8 with Top-1 approximately $\{C_1,C_2,C_3,C_4\}$9 on CIFAR-10 for the best ResNet-based RRCV variant with batch $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$0, depth $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$1, and heads $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$2 (Meng et al., 2024). The same source reports that multi-scale convolutional streams consistently outperform single-scale kernels across parameter configurations PC1–PC4 on APTOS2019 and RFMiD2020, and that these findings imply that LMF-MHSA plus RRCV is preferable to standard MHSA or CNN-only or Transformer-only designs for small-scale data (Meng et al., 2024).

CMSA derives its efficiency from window-based local attention. For group $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$3, with window size $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$4 and number of windows $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$5, the per-head complexity is $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$6, and the total complexity is roughly

$X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$7

rather than global $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$8 (Lu et al., 2024). Parameter counts are given for several variants: CMSA-S at approximately $X_1 \in \mathbb{R}^{H/4\times W/4\times C_1}$9–$X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$0, CMSA-B at approximately $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$1–$X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$2, and CMSA-L at approximately $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$3–$X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$4, depending on task variant (Lu et al., 2024).

The empirical results for CMSA are extensive. On low-resolution human pose estimation on COCO 2017 val, bottom-up, CMSA-L reaches AP $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$5 with $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$6 parameters at $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$7, AP $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$8 with $X_2 \in \mathbb{R}^{H/8\times W/8\times C_2}$9 at $1\times1$00, and AP $1\times1$01 with $1\times1$02 at $1\times1$03; corresponding larger baselines such as HRNet-W48, HRNet-W32, and ViTPose-B use $1\times1$04, $1\times1$05, and $1\times1$06 parameters, respectively (Lu et al., 2024). On head pose estimation, CMSA-L reaches MAE $1\times1$07 on AFLW2000 at $1\times1$08 and MAE $1\times1$09 on BIWI at $1\times1$10 (Lu et al., 2024). On CIFAR-100, CMSA-L reaches $1\times1$11 with $1\times1$12 parameters (Lu et al., 2024). The ablation at COCO $1\times1$13 is especially diagnostic: standard attention gives AP $1\times1$14, grouped attention only AP $1\times1$15, grouped plus cascade AP $1\times1$16, plus CF AP $1\times1$17, and plus CF plus SF, that is, full CMSA, AP $1\times1$18 (Lu et al., 2024).

The dermoscopic segmentation network reports state-of-the-art quantitative results on both evaluation sets considered. On ISBI2017 test, it achieves ACC $1\times1$19, SPE $1\times1$20, REC $1\times1$21, DC $1\times1$22, and JSI $1\times1$23; on ISIC2018 test, ACC $1\times1$24, SPE $1\times1$25, REC $1\times1$26, DC $1\times1$27, and JSI $1\times1$28 (Chowdary et al., 2021). Its ablation on ISBI2017 validation isolates the contribution of the multi-scale and attention components: U-Net gives DC $1\times1$29, JSI $1\times1$30, REC $1\times1$31; adding RMSM yields DC $1\times1$32, JSI $1\times1$33, REC $1\times1$34; adding DF-RAM or EF-RAM separately yields further gains; and the full combination reaches DC $1\times1$35, JSI $1\times1$36, REC $1\times1$37 (Chowdary et al., 2021). These results support the interpretation that multi-scale extraction and attention refinement contribute distinct improvements rather than duplicating one another.

6. Applications, design guidance, and limitations

The application range of MCFEM is broad, but the cited work clusters around three regimes. In small-scale image classification, CTA-Net positions MCFEM as a mid-level fusion block before the classifier, using either a CLS token or global average over tokens to produce logits (Meng et al., 2024). In low-resolution recognition, CMSA is designed for tasks such as human pose estimation and head pose estimation, where preserving information density within the block is essential (Lu et al., 2024). In dense prediction, RMSM-based MCFEM is embedded in a U-Net-like encoder–decoder to support lesion segmentation under low contrast, artifacts, and fuzzy boundaries (Chowdary et al., 2021).

Practical guidance also converges across the papers. CTA-Net recommends a lightweight ResNet-18/34 variant or MobileNetV2-style inverted bottlenecks for the CNN pyramid, $1\times1$38–$1\times1$39 scales with strides $1\times1$40, and unified embedding dimensions such as $1\times1$41 or $1\times1$42; its training settings for small datasets include AdamW, learning rate $1\times1$43 with cosine decay, warmup for $1\times1$44 epochs, weight decay $1\times1$45, DropPath or Stochastic Depth $1\times1$46–$1\times1$47, label smoothing $1\times1$48, Mixup $1\times1$49, CutMix $1\times1$50 with probability $1\times1$51, RandAugment or AutoAugment, gradient clipping $1\times1$52, and optional EMA of weights (Meng et al., 2024). CMSA recommends always including one global group, choosing $1\times1$53–$1\times1$54 groups per stage, keeping heads per group small, using CF and SF, normalizing with BN before attention and LN before FFN, and using GeLU in SF and FFN (Lu et al., 2024). The dermoscopic segmentation model uses Adam with initial learning rate $1\times1$55, reduction by a factor of $1\times1$56 every $1\times1$57 epochs, $1\times1$58 total epochs, batch size $1\times1$59, and a Dice-plus-Focal objective with $1\times1$60 (Chowdary et al., 2021).

The limitations are equally consistent. CTA-Net notes alignment overhead from folding and unfolding tokens and from scale alignment, as well as overfitting risk on very small datasets; it recommends relying on RRCV, augmentations, and regularization (Meng et al., 2024). CMSA notes that windows that are too small may over-localize and miss cross-region cues, and that excessive groups or heads can increase overhead without gains (Lu et al., 2024). The dermoscopic segmentation study reports that extremely indistinct boundaries at the lesion periphery can still cause minor under-segmentation or over-segmentation (Chowdary et al., 2021).

A final point concerns terminology. One cited paper explicitly states that it does not use the term “MCFEM,” but that CMSA is effectively an MCFEM: a drop-in module that extracts multi-scale features at fixed resolution and fuses them complementarily via cascaded attention (Lu et al., 2024). The same characterization also fits CTA-Net’s LMF-MHSA plus RRCV and the RMSM plus dual-attention formulation. In that sense, MCFEM is best viewed as a reusable architectural motif whose concrete realization may be attention-dominant, convolution-dominant, or hybrid, provided that it preserves the core objective: diverse yet synergistic multi-scale representations.