ScaleFormer: Multi-Scale Transformer Designs

• ScaleFormer is a design paradigm that explicitly models scale as a structural variable across various applications.
• It implements iterative multi-scale refinements using shared modules, cross-scale normalization, and dual-axis attention.
• Practical variants enhance performance in time series forecasting, medical segmentation, and long-context summarization through specialized fusion strategies.

Searching arXiv for papers on “ScaleFormer” and closely related usages to ground the article in published work. ScaleFormer is an overloaded name in the arXiv literature rather than a single canonical architecture. It has been used for an iterative multi-scale refinement framework for time series forecasting (Shabani et al., 2022), a scale-wise Transformer backbone for medical image segmentation (Huang et al., 2022), a plug-and-play long-context wrapper for encoder-decoder LLMs based on span representation cumulation (Du et al., 13 Nov 2025), a hierarchical multi-scale backbone inside the chaotic-systems foundation model ChaosNexus (Liu et al., 26 Sep 2025), and a cross-scale pansharpening architecture that reframes resolution generalization as sequence-length generalization (Cao et al., 28 Feb 2026). Taken together, these works suggest that “ScaleFormer” denotes a recurring design stance—explicit modeling of scale as a structural variable—rather than a single research lineage.

1. Nomenclature and scope

The term has been attached to multiple unrelated methods, each defined by its own task, data regime, and notion of scale.

Paper	Domain	Core mechanism
"Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting" (Shabani et al., 2022)	Long-term time series forecasting	Shared-weight coarse-to-fine refinement across temporal scales
"ScaleFormer: Revisiting the Transformer-based Backbones from a Scale-wise Perspective for Medical Image Segmentation" (Huang et al., 2022)	Medical image segmentation	Intra-scale and inter-scale Transformers over CNN features
"ScaleFormer: Span Representation Cumulation for Long-Context Transformer" (Du et al., 13 Nov 2025)	Long-document summarization	Chunk boundary cumulation with parameter-free directional fusion
"ChaosNexus: A Foundation Model for Universal Chaotic System Forecasting with Multi-scale Representations" (Liu et al., 26 Sep 2025)	Chaotic-system forecasting	Hierarchical Transformer with dual axial attention and MoE
"Cross-Scale Pansharpening via ScaleFormer and the PanScale Benchmark" (Cao et al., 28 Feb 2026)	Remote-sensing pansharpening	Scale-aware patch tokenization and decoupled spatial/sequence Transformers

This multiplicity is important bibliographically. Several scale-related Transformer papers are close in spirit but are not named ScaleFormer: TorchScale is an open-source toolkit for scaling Transformers (Ma et al., 2022), Scalable Transformers for NMT introduces a crop-able super-network (Gao et al., 2021), FactFormer is a factorized Transformer for PDE surrogate modeling (Li et al., 2023), and PRO-SCALE is a progressive token-length strategy for Mask2Former-style encoders (Aich et al., 2024). In consequence, the name cannot be interpreted without task context.

2. Recurring architectural motif

Despite the absence of a single canonical implementation, the named variants share a recognizable pattern. Each treats scale as an explicit computational axis rather than a latent side effect of depth alone. In time-series forecasting, scale is a hierarchy of temporal resolutions traversed from coarse to fine (Shabani et al., 2022). In medical segmentation, scale appears as both intra-scale local/global coupling and inter-scale interaction among spatially corresponding regions (Huang et al., 2022). In long-context summarization, document structure is compressed through chunk boundaries enriched by cumulative signals from preceding and succeeding chunks (Du et al., 13 Nov 2025). In chaotic forecasting, hierarchical temporal scales are encoded and decoded in a U-Net-like Transformer with patch merging and expansion (Liu et al., 26 Sep 2025). In pansharpening, image resolution is recast as variable token-sequence length under fixed patch resolution (Cao et al., 28 Feb 2026).

A second shared trait is that cross-scale interaction is almost always constrained rather than fully dense. The time-series variant reuses one forecasting backbone across scales and normalizes each step to control drift (Shabani et al., 2022). The segmentation variant replaces full spatial attention with Dual-Axis MSA and restricts inter-scale attention to matched patches (Huang et al., 2022). The long-context variant compresses each chunk to boundaries and optional sampled middle tokens instead of passing all token-level states (Du et al., 13 Nov 2025). The chaotic-system variant factorizes attention into variable and temporal axes and adds sparse expert routing (Liu et al., 26 Sep 2025). The pansharpening variant separates intra-patch spatial modeling from inter-patch sequential modeling and uses RoPE for extrapolation to unseen lengths (Cao et al., 28 Feb 2026). This suggests that “ScaleFormer” is best understood as a family resemblance term for scale-aware structured computation.

3. Time-series forecasting variant

The 2022 time-series model, styled as “Scaleformer,” is a model-agnostic multi-scale refinement framework for long-term forecasting (Shabani et al., 2022). It does not redesign attention itself; instead, it changes the staging of prediction. Given a look-back window $X^{(L)}$ of length $L_L$ and a horizon window $X^{(H)}$ of length $L_H$, it uses a scale set $S=\{s^m,\dots,s^2,s^1,1\}$, typically with $s=2$ and $m=[\log_s L_L]-1$. At each scale, the encoder input is obtained by average pooling the original look-back window, while the decoder input is initialized with zeros at the first step and then replaced by a linearly upsampled version of the previous forecast. The same forecasting module is reused at every scale, so multi-scale behavior is realized by iterative application with shared parameters rather than by separate branches.

A central technical contribution is cross-scale normalization. At scale $i$, the mean $\mu_i$ is computed over the temporal dimension of the concatenated encoder and decoder inputs, and both inputs are shifted by this mean. The paper motivates this by two distribution shifts: look-back versus forecast covariate shift, and scale-to-scale drift induced by reusing upsampled predictions. The embedding is also made scale-aware: value embeddings append indicator values $0$, $L_L$0, and $L_L$1 for look-back observations, zero initialization, and previous-step predictions, while temporal embeddings append the current-scale scalar $L_L$2. The method further replaces MSE with the adaptive loss of Barron (2019), with learned $L_L$3 and $L_L$4, to improve robustness to outliers (Shabani et al., 2022).

The framework is attached to Informer, Autoformer, FEDformer, Reformer, and Performer, and evaluated on Electricity, Traffic, Weather, Exchange-Rate, and ILI. Main experiments use look-back window $L_L$5, horizon lengths $L_L$6 corresponding to the standard long-horizon setup $L_L$7, hidden dimension $L_L$8, batch size $L_L$9, Adam, learning rate $X^{(H)}$0 for the forecasting model and $X^{(H)}$1 for the adaptive loss, 2 encoder layers, 1 decoder layer, and $X^{(H)}$2, with 5 repeated runs. The reported average MSE reductions are about $X^{(H)}$3 over FEDformer, about $X^{(H)}$4 over Autoformer, and about $X^{(H)}$5 over Informer; the abstract summarizes gains ranging from $X^{(H)}$6 to $X^{(H)}$7 across datasets and backbones. Ablations show that multi-scale refinement without normalization often degrades performance, that $X^{(H)}$8 is the best tested scale factor, and that adaptive loss and the multi-scale framework are complementary (Shabani et al., 2022).

The same paper also extends the idea to probabilistic forecasting and to non-Transformer backbones such as NHits and FiLM. A plausible implication is that, in this usage, Scaleformer is less a Transformer architecture than a coarse-to-fine forecasting prior that can be wrapped around heterogeneous sequence models.

4. Medical image segmentation variant

The medical segmentation ScaleFormer is a scale-wise hybrid backbone built on a ResUNet-style CNN encoder-decoder (Huang et al., 2022). Its starting point is a two-part critique of existing Transformer backbones. The paper identifies an intra-scale problem—failure to couple CNN local features with Transformer global cues within each scale—and an inter-scale problem—failure to model distinctive cross-scale information among objects whose size, shape, and location vary widely. To address these, it introduces two modules: a scale-wise intra-scale Transformer and a spatial-aware inter-scale Transformer.

The encoder comprises five CNN stages built from ResNet-34 blocks. At selected stages, CNN features are passed through an intra-scale Transformer. The first block uses same-stage CNN features; later blocks receive the concatenation of same-stage CNN features and downsampled previous Transformer outputs. The attention mechanism is Dual-Axis MSA, which decomposes full spatial attention on $X^{(H)}$9 from $L_H$0 to $L_H$1 by modeling row-wise and column-wise dependencies separately. This preserves horizontal and vertical structure while remaining tractable on high-resolution medical feature maps. After attention, an enhanced MLP with recursive skip connections is applied (Huang et al., 2022).

The inter-scale module addresses scale correspondence explicitly. Let $L_H$2 denote the $L_H$3-th patch at scale $L_H$4. Through patch matching, the model identifies aligned patches at neighboring or coarser scales, concatenates the matched tokens, applies a standard Transformer block to the fused sequence, and then splits the result back into scale-specific outputs. The paper describes this as interaction among consensual regions rather than all-to-all multi-scale fusion. Final training uses cross-entropy plus Dice loss, with SGD on Synapse and ACDC, Adam on MoNuSeg, momentum $L_H$5, weight decay $L_H$6, random rotation and flipping, and no pretraining on large external datasets (Huang et al., 2022).

Evaluation covers Synapse, MoNuSeg, and ACDC. On Synapse, the reported results are DSC $L_H$7 and HD $L_H$8 mm for the full model, versus DSC $L_H$9 and HD $S=\{s^m,\dots,s^2,s^1,1\}$0 mm without the inter-scale module; the inter-scale Transformer therefore adds approximately $S=\{s^m,\dots,s^2,s^1,1\}$1 DSC and $S=\{s^m,\dots,s^2,s^1,1\}$2 mm HD. On ACDC, the reported average DSC rises from $S=\{s^m,\dots,s^2,s^1,1\}$3 without inter-scale interaction to $S=\{s^m,\dots,s^2,s^1,1\}$4 with it. Ablations show the best efficiency-performance tradeoff when intra-scale Transformers are inserted at Stage 3–Stage 6, while adding Stage 2 is more expensive and improves little. They also show that combining same-stage CNN features with previous Transformer outputs is better than either input alone, and that Dual-Axis MSA compares favorably with CMT, PVT, and Axial Attention in both cost and accuracy (Huang et al., 2022).

In this formulation, ScaleFormer names a scale-wise backbone in the literal sense: it reorganizes segmentation around within-scale local/global fusion and across-scale spatial correspondence.

5. Long-context encoder-decoder variant

The 2025 ScaleFormer is a plug-and-play framework for adapting off-the-shelf pre-trained encoder-decoder PLMs such as BART or T5 to long-context summarization without modifying the model internals (Du et al., 13 Nov 2025). The method segments a long input sequence $S=\{s^m,\dots,s^2,s^1,1\}$5 into $S=\{s^m,\dots,s^2,s^1,1\}$6 overlapping chunks $S=\{s^m,\dots,s^2,s^1,1\}$7, with fixed chunk length $S=\{s^m,\dots,s^2,s^1,1\}$8 and overlap $S=\{s^m,\dots,s^2,s^1,1\}$9. In the reported experiments, $s=2$0 and $s=2$1. Each chunk is encoded independently, and from its top-layer hidden states $s=2$2 the method extracts only the left and right boundary tokens, denoted $s=2$3 and $s=2$4. Experiments use $s=2$5, so each chunk contributes one left and one right boundary vector.

The distinctive step is span representation cumulation. For each chunk, a backward context summarizes the current left boundary together with all boundary vectors from preceding chunks, and a forward context summarizes the current right boundary together with all boundary vectors from succeeding chunks. These cumulative vectors are fused with the local boundaries through a parameter-free convex combination governed by $s=2$6, producing $s=2$7 and $s=2$8. When $s=2$9, the representation is purely local; when $m=[\log_s L_L]-1$0, local boundaries are fully replaced by cumulative context. The paper’s ablation reports that performance is poor at the extremes and peaks around $m=[\log_s L_L]-1$1, indicating that chunk-local semantics and directional document context are complementary (Du et al., 13 Nov 2025).

To avoid excessive compression, the model may also sample $m=[\log_s L_L]-1$2 middle tokens from each chunk interior, $m=[\log_s L_L]-1$3. In the main full model, $m=[\log_s L_L]-1$4 and sampling is random. The final decoder input is the concatenation of fused boundaries and sampled interiors, with total length $m=[\log_s L_L]-1$5, substantially shorter than passing all chunk encodings. Because $m=[\log_s L_L]-1$6, total encoding cost is approximately $m=[\log_s L_L]-1$7, which is effectively linear in $m=[\log_s L_L]-1$8 when $m=[\log_s L_L]-1$9 is fixed, and the cumulation step is only a pass over chunk boundaries (Du et al., 13 Nov 2025).

The method is evaluated on SummScreen, GovReport, and BookSum with BART-base and T5-base backbones. For BART-base, the full ScaleFormer + Middle variant reports SummScreen test ROUGE $i$0, GovReport test $i$1, and BookSum test $i$2. Without middle tokens it already surpasses SLED in several cases, including GovReport test ROUGE-1 $i$3 versus $i$4 and SummScreen test ROUGE-1 $i$5 versus $i$6. For T5-base, ScaleFormer + Middle reports SummScreen test $i$7 and GovReport test $i$8. Baselines include standard truncation, SLED, Memorizing Transformers, and Unlimiformer, and the paper emphasizes that the method requires neither architectural modification nor external retrieval (Du et al., 13 Nov 2025).

This version uses “ScaleFormer” in a different sense from the 2022 models. Here the central scale variable is document length and chunk position, and the main contribution is a compressed decoder-facing representation that carries explicit narrative directionality.

6. Later multi-scale variants in dynamical systems and remote sensing

In ChaosNexus, ScaleFormer is the core backbone for universal chaotic-system forecasting (Liu et al., 26 Sep 2025). The model takes a multivariate time series $i$9, segments it into non-overlapping temporal patches, enriches each patch with random polynomial features and random Fourier features, and projects the result to an embedding $\mu_i$0. The ScaleFormer block then applies variable attention, temporal attention, and an MoE layer under RMS normalization and residual connections. The paper states that flattening time and variables together would incur complexity $\mu_i$1, whereas dual axial attention reduces this to $\mu_i$2. The encoder-decoder is hierarchical and U-Net-like: patch merging halves temporal resolution and doubles channel dimension, patch expansion reverses this, and skip connections are implemented with a 1D residual convolutional module. The MoE uses $\mu_i$3 specialist experts, top-$\mu_i$4 routing with $\mu_i$5, one shared expert, and an auxiliary load-balancing loss. ScaleFormer outputs are combined with a wavelet-scattering frequency fingerprint, and training uses MSE, the MoE balancing term, and an MMD regularizer weighted by $\mu_i$6 and $\mu_i$7. On a synthetic test set of 9.3K unseen chaotic systems, ChaosNexus reports $\mu_i$8 better correlation dimension error and $\mu_i$9 better KL divergence between attractors than the best baseline, and on WEATHER-5K it achieves zero-shot MAE below $0$0C for 5-day temperature forecasting (Liu et al., 26 Sep 2025).

The 2026 pansharpening ScaleFormer addresses cross-scale generalization in remote sensing (Cao et al., 28 Feb 2026). Its premise is that models trained on small fixed-resolution crops often fail on much larger images because of both computational scaling and scale-induced distribution shift. The model therefore treats resolution generalization as sequence-length generalization. Given high-resolution PAN $0$1 and low-resolution multispectral input $0$2, Scale-Aware Patchify produces patch sequences $0$3, where scale changes mainly affect token count $0$4, not patch statistics. The architecture decouples intra-patch spatial learning from inter-patch sequential dependency modeling through separate Spatial Transformer and Sequence Transformer modules, and it introduces corresponding spatial and sequence cross-attention modules between PAN and MS streams. RoPE is used in the sequence attention modules to improve extrapolation to unseen lengths. Training uses a simple $0$5 loss, PyTorch, Adam, cosine learning-rate decay from $0$6 to $0$7, 500 epochs, gradient clipping 4, and feature channels 32. The accompanying PanScale dataset is built from Jilin-1, Landsat-9, and Skysat using Google Earth Engine, and PanScale-Bench evaluates both reduced-resolution and full-resolution settings with PSNR, SSIM, SAM, ERGAS, SCC, Q, $0$8, $0$9, and QNR. The reported model size is $L_L$00M parameters and $L_L$01 GFLOPs at $L_L$02, with averaged reduced-resolution results including Jilin-Ave PSNR $L_L$03, Landsat-Ave PSNR $L_L$04, and Skysat-Ave PSNR $L_L$05 (Cao et al., 28 Feb 2026).

These two later uses broaden the name considerably. One centers hierarchical dynamical representation learning with MoE specialization and attractor-aware regularization; the other centers cross-scale image fusion via scale-aware tokenization and resolution-length extrapolation.

7. Related terms, misconceptions, and disambiguation

A persistent source of confusion is that “ScaleFormer” is often conflated with any Transformer method that scales well. That interpretation is too broad. TorchScale, for example, is a toolkit for scaling Transformers from the modeling perspective through a Magneto backbone, DeepNorm or Sub-LN stabilization, X-MoE sparse scaling, and SparseClip gradient clipping; it is explicitly not a method named ScaleFormer (Ma et al., 2022). Scalable Transformers for NMT likewise introduces a shared-parameter super-network that can be cropped into many sub-Transformers of different widths and trained with a three-stage self-distillation curriculum, but its official name is Scalable Transformers or ST, not ScaleFormer (Gao et al., 2021). FactFormer is a factorized Transformer for PDE surrogate modeling based on a learnable projection operator and axial factorized kernel integrals, and PRO-SCALE is an efficiency strategy for progressively increasing token length with encoder depth in Mask2Former-style segmentation; both are strongly scale-aware yet distinct from the ScaleFormer papers (Li et al., 2023, Aich et al., 2024).

The same caution applies outside deep learning. “Projective Decomposition and Matrix Equivalence up to Scale” introduces a matrix normalization framework for ratio-scale data, not a Transformer architecture (Robinson, 2019). “Shape complexity in cluster analysis” proposes optimization-based feature rescaling for clustering via shape complexity, again unrelated to Transformer backbones (Aguilar et al., 2022). A practical consequence is that bibliographic searches for “ScaleFormer” can return official model names, informal analogies, or mistaken shorthand.

Taken as a whole, the literature supports a narrow and a broad reading. In the narrow reading, ScaleFormer denotes a specific named model within a particular application paper. In the broader reading, it names a class of scale-explicit architectures that replace flat processing with structured multi-scale computation. The former is precise; the latter is comparative and should be stated as interpretation rather than official nomenclature.