Attention-Based Multi-Scale Preprocessing

• Attention-based multi-scale preprocessing is a method that transforms inputs into multiple scale-specific views and fuses them with learnable, spatially adaptive attention rather than fixed pooling techniques.
• It improves performance in semantic segmentation and related tasks by dynamically weighting features, leading to measurable mIoU gains on benchmarks such as PASCAL VOC and ADE20K.
• The approach is versatile across domains—extending to graph embedding, dehazing, and hybrid CNN-Transformer models—while optimizing computational efficiency through hierarchical and structured attention mechanisms.

Attention-based multi-scale preprocessing denotes a class of representation-learning procedures in which an input signal, or an intermediate feature tensor, is converted into multiple scale-specific views and then fused by learned attention, gating, or structured message passing rather than by fixed averaging or max selection. In computer vision, an early and influential formulation used multiple resized copies of an image, a shared fully convolutional network, and a pixel-wise softmax over scales to build scale-aware semantic segmentation logits (Chen et al., 2015). Subsequent work extended the same design principle to hypercolumn fusion, hierarchical scale chaining, local-global attention, grouped windowed attention, scale-space convolutions, graph embedding, text encoding, and hybrid CNN-Transformer-Mamba architectures (Yang et al., 2018).

1. Canonical formulation

The canonical image-domain formulation begins with a finite set of scales $s \in \{1,\dots,S\}$. For each scale, the original image $I$ is resized to $I_s = \mathrm{Resize}(I,s)$ and passed through a shared backbone, producing per-scale score maps or feature maps. In the semantic-segmentation formulation of "Attention to Scale" (Chen et al., 2015), the score map at spatial location $x$ and class $c$ is

$$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$

and all scale-specific outputs are bilinearly upsampled to the finest spatial resolution before fusion.

Attention is then defined across scales at each spatial location. The attention model produces logits $e_s(x)$, normalized by a per-location softmax,

$$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$

and the fused score is a pixel-wise weighted sum,

$g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$

Final class probabilities follow from a class-wise softmax,

$p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$

This formulation establishes two important equivalences. If $I$0 for all $I$1, the fusion reduces to average pooling over scales. If the weighted sum is replaced by a max and the winning scale receives weight $I$2, the procedure reduces to max-pooling over scales. The distinctive contribution of attention is therefore not merely multi-scale inference, but the replacement of fixed scale aggregation by spatially adaptive weighting (Chen et al., 2015).

A recurrent companion technique is deep supervision. In the same segmentation framework, a cross-entropy loss is applied not only to the fused output but also to each per-scale score map: $I$3

$I$4

$I$5

In that study, the extra supervision was reported as essential whenever $I$6 (Chen et al., 2015).

2. Semantic segmentation as the formative use case

Semantic segmentation provided the earliest systematic demonstrations that attention-based multi-scale preprocessing is more than a cosmetic replacement for heuristic scale fusion. In "Attention to Scale" (Chen et al., 2015), the shared backbone is DeepLab-LargeFOV, and the auxiliary attention FCN is attached on top of the convolutionalized $I$7 features. That attention FCN uses two convolutional layers: $I$8 kernels of size $I$9, followed by $I_s = \mathrm{Resize}(I,s)$0 kernels of size $I_s = \mathrm{Resize}(I,s)$1, yielding one attention logit per scale and spatial location. On PASCAL-Person-Part validation with scales $I_s = \mathrm{Resize}(I,s)$2 and no extra supervision, max-pooling achieved $I_s = \mathrm{Resize}(I,s)$3 mIoU, average-pooling $I_s = \mathrm{Resize}(I,s)$4, and attention $I_s = \mathrm{Resize}(I,s)$5. With extra supervision, the attention model reached $I_s = \mathrm{Resize}(I,s)$6, compared with $I_s = \mathrm{Resize}(I,s)$7 for max-pooling and $I_s = \mathrm{Resize}(I,s)$8 for average-pooling. Similar gains of about $I_s = \mathrm{Resize}(I,s)$9 absolute were reported on PASCAL-VOC2012 and a COCO subset, both with and without COCO pre-training.

A later refinement, "Attention to Refine through Multi-Scales for Semantic Segmentation" (Yang et al., 2018), replaced direct score fusion by a richer two-branch design. Each resized input is processed by a shared backbone in Siamese fashion, and hypercolumn-style features from multiple stages are bilinearly upsampled to a common resolution and concatenated: $x$0 Scale-specific dilated convolutions are then applied, with $x$1 and $x$2, after which all scale outputs are concatenated into a tensor $x$3. A location-attention branch predicts per-pixel scale weights

$x$4

while a parallel recalibrating branch predicts per-pixel, per-class sigmoid gates $x$5. Each scale-specific score map $x$6 is first recalibrated,

$x$7

then weighted by location attention,

$x$8

and finally summed: $x$9 On PASCAL VOC 2012 validation, a VGG16-LFOV baseline at $c$0 mIoU rose to $c$1 with average-pool fusion, $c$2 with Attention-to-Scale, and $c$3 with location attention plus recalibration. On ADE20K validation, a ResNet34-dil8 baseline at $c$4 mIoU and $c$5 pixel accuracy rose to $c$6 mIoU and $c$7 with the new method, or $c$8 mIoU and $c$9 when combined with PSP.

"Hierarchical Multi-Scale Attention for Semantic Segmentation" introduced a different answer to the cost of explicit multi-scale attention (Tao et al., 2020). Instead of training a flat attention head over all inference scales, it learns a relative attention mask between adjacent scales and composes these masks hierarchically. For a coarse scale $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$0 and finer scale $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$1, the fused logits are

$$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$2

With scales $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$3, an explicit flat approach has relative cost $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$4, whereas hierarchical training on $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$5 has cost $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$6, giving roughly a $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$7 improvement in efficiency. The method achieved $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$8 IOU on Mapillary with scales $$f^{(s)}_{x,c} = \mathrm{FCN}(\mathrm{Resize}(I,s))_{x,c},$$9, and $e_s(x)$0 IOU on the Cityscapes test set (Tao et al., 2020).

In weakly supervised semantic segmentation, the same principle appears in the form of multi-scale class-aware attention fusion rather than supervised logit fusion. "A Self-Training Framework Based on Multi-Scale Attention Fusion for Weakly Supervised Semantic Segmentation" resizes each image to $e_s(x)$1, extracts teacher attention maps $e_s(x)$2, bilinearly resizes them to a common grid, sums them channel-wise, and normalizes each channel: $e_s(x)$3 Absent classes are zeroed, and a reactivation step renormalizes each pixel across classes,

$e_s(x)$4

The student is trained with binary classification loss and a multi-scale attention consistency term

$e_s(x)$5

The reported segmentation performance on PASCAL VOC 2012 was $e_s(x)$6 mIoU on both validation and test sets (Yang et al., 2023).

3. Sources of scale

Although early work treated scale as a set of resized input images, later systems show that attention-based multi-scale preprocessing is not tied to image pyramids. The notion of scale can arise from sampling resolution, feature depth, window geometry, transition operators, or even learned continuous scale-space parameters.

The image-pyramid formulation remains the most direct. It appears in semantic segmentation systems that use scales such as $e_s(x)$7, $e_s(x)$8, or $e_s(x)$9, with a shared network applied to each resized image and later alignment by bilinear interpolation (Chen et al., 2015). This same principle underlies hierarchical inference schemes that train on $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$0 but test on longer scale chains (Tao et al., 2020).

A different construction appears in Atlas, where scales are created internally by strided summarization rather than by resizing the raw image (Agrawal et al., 16 Mar 2025). Starting from a patchified feature map $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$1, the model repeatedly downsamples with

$$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$2

where $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$3 is implemented as strided max-pooling with stride $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$4, $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$5, and the process stops when the coarsest map has at most $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$6 tokens. This yields $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$7 scales. The scale hierarchy is therefore generated by token summarization rather than by explicit multi-resolution copies of the image.

SDA-$$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$8Net defines scale through receptive-field depth inside a stage (Guo et al., 2022). In a ResNet-style stage, blocks share the same spatial resolution but have progressively larger receptive fields. If the stage outputs are $$\alpha_s(x)=\frac{\exp(e_s(x))}{\sum_{t=1}^{S}\exp(e_t(x))},$$9, the attention branch forms a summary

$g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$0

passes $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$1 through an SE-style bottleneck, reshapes the result into $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$2 channel vectors, and applies softmax over the depth index: $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$3 The stage output is

$g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$4

Here scale is indexed by block depth, not by image resolution.

CMSA, designed for low-resolution CNN-ViT hybrids, defines scale through grouped attention windows (Lu et al., 2024). After computing $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$5, the channels are split into $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$6 groups, one per scale, and each group attends within windows of size $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$7. Group $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$8 uses a global window with $g_{x,c}=\sum_{s=1}^{S}\alpha_s(x)\,f^{(s)}_{x,c}.$9, $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$0, while later groups use increasingly local windows. The model therefore simulates multi-scale receptive fields entirely in the attention domain, with no explicit downsampling of $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$1.

In natural language processing, "Multi-Scale Self-Attention for Text Classification" defines scale as the half-width $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$2 of a restricted attention window (Guo et al., 2019). For each head, only tokens in a neighborhood of size $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$3 are visible: $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$4 Different heads within the same layer can be assigned different $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$5, yielding multi-scale multi-head self-attention. This suggests that multi-scale preprocessing is not inherently spatial; it can also be phrased as a prior over the support of self-attention.

SCAN-CNN goes further by learning scales continuously from data through scale-space convolution (Shi et al., 2022). For filter $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$6, the response is

$p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$7

where $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$8 is a learnable scale parameter and $p_{x,c}=\mathrm{SoftMax}_c(g_{x,\cdot}).$9 is a Gaussian kernel. Each output channel then corresponds to one learned scale, and after training the scale-aware convolution can be fused into a standard static kernel with zero additional inference cost.

4. Mechanisms for cross-scale interaction

Attention-based multi-scale preprocessing differs most sharply across works in how information is allowed to flow between scales. Some methods use simple convex fusion, some use directional cross-attention, and others embed gating into structured graphical or cascaded interactions.

Atlas is the most explicit bidirectional design (Agrawal et al., 16 Mar 2025). In each MSA block, fine-to-coarse summarization is followed by coarse-to-fine top-down cross-attention,

$I$00

where each window at scale $I$01 attends jointly to itself and all coarser windows, and then by fine-to-coarse bottom-up cross-attention,

$I$02

where each coarse window attends to its direct fine-scale parent. The reported per-block complexity is $I$03, which becomes effectively $I$04 when $I$05 and $I$06 are constant, and $I$07 hops suffice to mix information between any two tokens.

CMSA uses a cascaded rather than symmetric interaction pattern (Lu et al., 2024). After grouped projections, the output of group $I$08 is concatenated with the current group’s $I$09 and $I$10, then transformed by channel fusion and spatial fusion: $I$11 Each scale therefore inherits context from the preceding, coarser group before computing its own local attention. The authors emphasize that this preserves all spatial details of low-resolution inputs because no explicit downsampling of $I$12 is introduced inside CMSA.

"Learning Deep Structured Multi-Scale Features using Attention-Gated CRFs for Contour Prediction" formulates cross-scale interaction as latent gated message passing (Xu et al., 2018). Each emitter scale $I$13 sends messages to a receiver scale $I$14, but the transmission is modulated by binary gate variables $I$15. Under mean-field inference, the gate expectation becomes an attention weight

$I$16

and the hidden feature update is

$I$17

The two-level hierarchy first fuses three intra-layer scales and then fuses the refined layer outputs across network depth.

A sequential factorization of interaction appears in multi-task scene understanding (Kim et al., 2022). Because direct cross-attention over every task-scale pair has complexity $I$18, the architecture first applies cross-task attention at fixed scale and then cross-scale attention within each task. For a task $I$19, coarse-scale features are aggregated by

$I$20

after which cross-scale attention computes

$I$21

The paper reports that CTAM alone gave a modest gain on NYUD-v2, whereas adding CSAM increased mIoU from $I$22 to $I$23, supporting the sequential ordering (Kim et al., 2022).

In object detection, Local-Global Attention fuses scale-specific convolutional features and then blends local and global attention branches by learned scalars $I$24 and $I$25 (Shao, 2024). Small object detection under complex backgrounds adopts a sparse aligned interaction pattern: P4 queries attend to sampled positions from P3, P4, and P5 using learned offsets $I$26, with weights normalized over level and sample index (Tao et al., 4 Mar 2026). These designs indicate that cross-scale attention is increasingly implemented as structured, computationally constrained communication rather than as dense all-to-all fusion.

5. Domain-specific realizations

The abstraction is portable across domains because the core ingredients—multi-scale views, attention-based weighting, and supervised or self-supervised fusion—do not depend on any single prediction task.

Domain	Representative realization	Reported outcome
Graph embedding	Attention over $I$27-th order proximity vectors with adversarial autoencoding (Sang et al., 2018)	2–5 percentage-point gains over baselines on Cora, Citeseer, Wiki
Text classification	Per-head restricted attention windows with layer-wise scale control (Guo et al., 2019)	SNLI $I$28; MTL-16 $I$29
Face recognition	Learned per-filter scale-space convolution with fused one-shot inference (Shi et al., 2022)	BLUFR @ FAR $I$30: $I$31
Dehazing	Trainable pre-processing plus grid attention-based multi-scale backbone (Liu et al., 2019)	Replacing learned pre-processing with hand-crafted inputs drops PSNR by about $I$32 dB on SOTS
Lightweight detection	Multi-scale residual convolutions, positional encoding, local/global attention fusion (Shao, 2024)	TinyPerson: $I$33, FLOPs $I$34

In graph representation learning, AAANE computes $I$35 structural views $I$36 from powers of the row-normalized adjacency matrix $I$37, forms a global context $I$38, scores each scale by

$I$39

and aggregates them into

$I$40

An adversarially regularized autoencoder then produces the final embedding, and the method was reported to outperform DeepWalk, LINE, GraRep, node2vec, and AIDW by $I$41 to $I$42 percentage points across datasets and labeling ratios (Sang et al., 2018).

In dehazing, GridDehazeNet treats preprocessing itself as learnable multi-scale input formation (Liu et al., 2019). A $I$43 convolution and residual dense block produce $I$44, after which a grid backbone with $I$45 rows and $I$46 columns performs channel-wise attention fusion at each junction. For channel $I$47, pooled row and column descriptors $I$48 and $I$49 are mapped to a two-way softmax

$I$50

and the fused channel is

$I$51

The paper argues that this learned pre-processor produces more diverse and pertinent features than hand-selected image transforms.

Medical segmentation offers a hybrid branch-level version of the same concept (Bui et al., 4 Oct 2025). MambaCAFU uses a three-branch encoder consisting of ResNet-18, PVTv2, and a Mamba-based Attention Fusion branch. At scale $I$52, the next fused feature is

$I$53

where spatial attention, attention gates, channel attention, and a 2D state-space module cooperate to combine local, global, and long-range cues. The model variants reported $I$54 M and $I$55 M parameters, with the MAF branch adding about $I$56 M parameters (Bui et al., 4 Oct 2025).

Industrial detection systems also adopt the pattern. SMR-Net inserts CBAM after each residual block of ResNet-34, extracts three feature maps, processes them with standard and dilated convolutions, and predicts adaptive fusion weights by an MLP followed by softmax: $I$57 On Type A and Type B snap datasets, the paper reports IoU gains of $I$58 and $I$59, and mAP gains of $I$60 and $I$61, relative to Faster R-CNN (Hou, 1 Mar 2026).

6. Methodological issues, misconceptions, and open directions

A recurring misconception is that multi-scale preprocessing is equivalent to merely enlarging receptive fields. Several papers explicitly reject that simplification. SDA-$I$62Net begins from the claim that existing multi-scale solutions risk increasing receptive field sizes while neglecting small receptive fields (Guo et al., 2022). MATCNN makes the parallel claim that attention-based global-feature methods can compromise local feature capture when diverse receptive-field extraction is absent (Liu et al., 4 Feb 2025). These arguments place attention-based multi-scale preprocessing not at one extreme of the local-global spectrum, but at the interface between them.

A second misconception is that any scale fusion suffices once multiple scales are present. The segmentation evidence contradicts this directly. Average pooling enforces equal contribution from all scales, even when only one scale is appropriate at a pixel; max-pooling discards complementary cues from non-maximal scales. The soft attention formulation in (Chen et al., 2015) was proposed precisely to avoid those two failure modes, and the later recalibration branch in (Yang et al., 2018) further shows that scale weighting alone may be insufficient without class-wise contextual gating.

Computational tractability is another central theme. Naïve cross-attention over all task-scale pairs has $I$63 complexity in multi-task learning (Kim et al., 2022); flat explicit multi-scale attention over $I$64 requires $I$65 the cost of single-scale training, versus $I$66 for hierarchical training on $I$67 (Tao et al., 2020); Atlas targets $I$68 rather than $I$69 long-context image modeling (Agrawal et al., 16 Mar 2025); SCAN-CNN collapses its learned scale-space kernels into ordinary convolutions at inference time, with no runtime increase over the base CNN (Shi et al., 2022). Lightweight modules such as LGA report only $I$70–$I$71 M extra parameters and about a $I$72 FLOP increase on one detection backbone (Shao, 2024), whereas low-resolution CMSA notes that its additional CF/SF modules introduce modest compute overhead and extra hyperparameters (Lu et al., 2024).

The training signal required for stable scale learning is also nontrivial. Extra per-scale supervision was reported as essential whenever $I$73 in early segmentation work (Chen et al., 2015). Weakly supervised segmentation compensates for incomplete labels by denoising absent classes and reactivating under-emphasized channels across scales (Yang et al., 2023). SCAN-CNN introduces a response-maximization objective

$I$74

to force each learnable scale parameter toward a response peak (Shi et al., 2022). This suggests that scale attention is often not self-stabilizing; it frequently requires architectural or loss-level scaffolding.

A plausible implication is that the field is moving away from rigid, hand-designed pyramids toward learned scale generation and structured cross-scale communication. The evidence in the cited work points in that direction: logarithmic internal scales in Atlas, grouped window scales in CMSA, depth-indexed scales in SDA-$I$75Net, continuous scale parameters in SCAN-CNN, and sparse aligned cross-scale attention in small-object detection all replace or augment the classic resized-image pyramid (Agrawal et al., 16 Mar 2025). Even so, the image-pyramid formulation remains a strong reference point because it makes the semantics of scale explicit and keeps the fusion objective directly interpretable.