Conv-like ScaleFusion Transformer

• Conv-like ScaleFusion Transformer is a multi-scale framework that couples convolution-like local processing with explicit cross-scale interactions to enhance feature representation across modalities.
• It employs mechanisms like re-patching, pyramidal downsampling, and cross-scale attention to reduce computational complexity while maintaining robust performance in time series, text, vision, and graph tasks.
• Empirical results report significant efficiency gains, including up to 99.5% reduction in attention computations and competitive accuracy in forecasting, summarization, and image classification.

to describe architectures that combine convolution-like locality and positional bias with explicit scale-wise interaction across hierarchical representations"—that is supported across details. Might mark as editor's synthesis/inference.

Need include mathematical notation when appears. Could compare formulations: ScaleFormer long text cumulation equations, time series CTL equations, CoaT conv attention, medical ScaleFormer dual-axis attention, CLEAR mask. Need keep article comprehensive and organized. Need avoid too much formula because user wants article comprehensive. Should have 6 sections maybe: definition and scope; core design principles; architectural realizations across modalities; fusion mechanisms; complexity/training; limitations/prospects. But headings numbered.

Need maybe include table listing representative instances and modalities. Since detailed explanations should be outside table. Fine.

Need cite papers at most once per section. Could cite multiple IDs in a section. We can have introduction cite all primary. Then sections cite relevant ones. Need not overcite.

Need ensure no claim from arXiv search results beyond data. But maybe cite only provided IDs; search can just verify titles. Let's use search tool minimal. Searching arXiv for the cited papers to verify identifiers and titles. The Conv-like ScaleFusion Transformer denotes a class of Transformer architectures that couple convolution-like locality with explicit multi-scale feature interaction. In a strict sense, the phrase names a Multi-Scale Representation Learning Framework based on a Conv-like ScaleFusion Transformer for variable-length long time series (Zhang et al., 22 Sep 2025). More broadly, closely related work applies the same design logic across long-context text processing, image classification, medical image segmentation, spatio-temporal graph learning, and diffusion transformers: progressive or hierarchical token compression, local inductive bias inside attention or parallel convolution branches, and explicit fusion across scales or representation levels (Du et al., 13 Nov 2025, Xu et al., 2021, Huang et al., 2022, Su et al., 2022, EL-Assiouti et al., 2024, Liu et al., 2024, Mourchid et al., 2023).

1. Definition and scope

A Conv-like ScaleFusion Transformer combines two architectural commitments. The first is conv-like processing: local receptive fields, pyramidal downsampling, depthwise convolutional positional encoding, local-window attention, or explicit convolution branches. The second is ScaleFusion: interaction among features at different scales, resolutions, chunk levels, or abstraction depths, typically through cross-scale attention, interpolation, matched-patch fusion, or adaptive weighting (Zhang et al., 22 Sep 2025, Xu et al., 2021, Huang et al., 2022).

A concise synthesis of representative realizations is given below.

Paper	Domain	Defining mechanism
"ScaleFormer: Span Representation Cumulation for Long-Context Transformer" (Du et al., 13 Nov 2025)	Long-context text	Overlapping chunks with parameter-free cumulative boundary fusion
"Conv-like Scale-Fusion Time Series Transformer: A Multi-Scale Representation for Variable-Length Long Time Series" (Zhang et al., 22 Sep 2025)	Time series	Conv-like Transformer Layers, cross-scale attention, log-space normalization
"Co-Scale Conv-Attentional Image Transformers" (Xu et al., 2021)	Vision	Conv-attention plus co-scale parallel fusion
"ScaleFormer: Revisiting the Transformer-based Backbones from a Scale-wise Perspective for Medical Image Segmentation" (Huang et al., 2022)	Medical segmentation	Intra-scale local-global coupling and spatial-aware inter-scale transformer
"Adaptive Split-Fusion Transformer" (Su et al., 2022) and "CTRL-F" (EL-Assiouti et al., 2024)	Hybrid visual recognition	Parallel convolution and transformer branches with learned fusion
"CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up" (Liu et al., 2024)	Diffusion transformers	Local softmax attention with convolution-like receptive fields

Within this family, “scale” is not restricted to image resolution. In long-context NLP it may denote chunk order and cumulative document context; in time series it may denote progressively re-patched temporal resolutions; in graph models it may denote multi-hop neighborhoods and multiple temporal kernel sizes. This suggests that the unifying property is not a single canonical block, but an architectural pattern in which locality and hierarchy are made explicit and then fused rather than left implicit in a flat token sequence (Du et al., 13 Nov 2025, Zhang et al., 22 Sep 2025, Mourchid et al., 2023).

2. Core architectural principles

The clearest formalization appears in the time-series framework of (Zhang et al., 22 Sep 2025). A raw series $\mathbf{X} = [x_1, x_2, \dots, x_T]$ is first patched and embedded, then processed by a stack of Conv-like Transformer Layers. Each layer performs re-patching along the patch dimension, reducing the number of tokens while expanding channel capacity. If the previous level has $P^{l-1}$ patch tokens, the next level has

$$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$

while the feature dimension expands as

$d^l = (l_{\text{rp}})^l \times d^0.$

The architecture is explicitly pyramidal: temporal resolution decreases, channels increase, and the feature volume $P^l \times d^l$ is kept approximately constant (Zhang et al., 22 Sep 2025).

A different but related principle appears in long-context ScaleFormer (Du et al., 13 Nov 2025). There, a long sequence is partitioned into overlapping segments $S_1,\dots,S_C$, each processed independently by a pre-trained encoder. Rather than altering the backbone attention, the model compresses each chunk into boundary and optional middle representations, then fuses boundaries with cumulative directional context:

$$\text{ctx}^{\text{back}}_i = \frac{1}{2i - 1}\left( L_i + \sum_{j=1}^{i-1} (L_j + R_j) \right),$$

$$\text{ctx}^{\text{fwd}}_i = \frac{1}{2(C - i) + 1}\left( R_i + \sum_{j=i+1}^{C} (L_j + R_j) \right).$$

The fused boundaries are

$$L'_i = \alpha \cdot L_i + (1 - \alpha)\cdot \text{ctx}^{\text{back}}_i,$$

$$R'_i = \alpha \cdot R_i + (1 - \alpha)\cdot \text{ctx}^{\text{fwd}}_i.$$

This is parameter-free and structurally analogous to directional convolution over chunk-level features, even though the aggregation is prefix/suffix cumulative rather than a learned finite kernel (Du et al., 13 Nov 2025).

In vision, CoaT builds the conv-like aspect into attention itself. Its conv-attentional mechanism combines factorized attention with convolutional relative position encoding:

$P^{l-1}$0

where $P^{l-1}$1 is implemented with depthwise convolution, giving the attention block a local, convolution-like positional branch alongside global content mixing (Xu et al., 2021). Medical ScaleFormer uses a different efficiency strategy: Dual-Axis MSA, which decomposes full 2D attention into row-wise and column-wise attention,

$P^{l-1}$2

reducing complexity from $P^{l-1}$3 to $P^{l-1}$4 while preserving structured long-range dependency at each scale (Huang et al., 2022).

Across these instantiations, the recurrent design pattern is a replacement of undifferentiated global token interaction with structured locality plus explicit hierarchical communication.

3. ScaleFusion mechanisms across domains

The principal distinction among Conv-like ScaleFusion Transformer variants lies in how they define and fuse scales. In the time-series framework, each coarse representation $P^{l-1}$5 queries the previous finer scale $P^{l-1}$6 through multi-head cross-attention:

$P^{l-1}$7

and the fused output is

$P^{l-1}$8

This replaces simple residual transmission with content-adaptive cross-scale interaction, and the paper attributes to it improved feature independence and reduced redundancy (Zhang et al., 22 Sep 2025).

CoaT realizes ScaleFusion through co-scale parallel blocks. Multiple scales are maintained simultaneously, and each scale receives interpolated features from the others before a shared feed-forward update. The result is a symmetric feature-level fusion pathway across resolutions rather than a purely fine-to-coarse hierarchy (Xu et al., 2021). Medical ScaleFormer is more spatially constrained: it matches corresponding patches across scales, concatenates them,

$P^{l-1}$9

applies inter-scale self-attention and MLP, then splits and reshapes the outputs back into scale-specific feature maps (Huang et al., 2022). This makes the fusion explicitly spatial-aware, limiting cross-scale interaction to consensual regions rather than all possible token pairs.

Hybrid visual classifiers adopt branch-level rather than scale-level fusion. ASF-former splits channels equally into a convolutional path and an attention path, then fuses them with adaptive weights,

$$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$0

$$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$1

$$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$2

The fusion weights are content-aware rather than fixed, and the design is explicitly motivated by different local/global requirements across depth and category (Su et al., 2022). CTRL-F uses a related but not identical strategy: multi-level convolutional features from two stages are tokenized into small-patch and large-patch branches, and the branches exchange information via cross-attention in the Multi-Level Feature Cross-Attention module, after which convolutional and transformer predictions are merged by Adaptive Knowledge Fusion or Collaborative Knowledge Fusion (EL-Assiouti et al., 2024).

A common misconception is that ScaleFusion is equivalent to ordinary skip connections. Several of these architectures reject that equivalence explicitly: (Zhang et al., 22 Sep 2025) replaces residual scale transfer with cross-scale attention, (Huang et al., 2022) introduces a dedicated inter-scale transformer because skip connections alone do not model cross-scale dependency, and (Su et al., 2022) shows that context-agnostic fusion underperforms content-aware adaptive fusion.

4. Conv-like locality inside Transformer processing

The “conv-like” qualifier refers to multiple, technically distinct mechanisms. In the long-context setting, it refers to sliding-window segmentation with overlap and cumulative aggregation over chunk boundaries, which behaves like a document-level convolution over chunk representations while remaining parameter-free (Du et al., 13 Nov 2025). In the time-series setting, it refers to re-patching-based temporal downsampling followed by global self-attention over compacted tokens, producing local aggregation through re-patching and global aggregation through attention at each pyramid level (Zhang et al., 22 Sep 2025).

In CoaT, conv-like behavior is encoded directly in attention. Convolutional Position Encoding and Convolutional Relative Position Encoding use depthwise convolutions, with kernel sizes $$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$3, $$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$4, and $$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$5 across head groups, so that locality bias is injected without abandoning Transformer-native content-adaptive mixing (Xu et al., 2021). Medical ScaleFormer similarly retains CNN blocks as the primary local feature extractors and augments them with scale-wise transformer blocks; its Enhanced MLP also includes depth-wise convolution before channel mixing, reinforcing the local-global hybrid character (Huang et al., 2022).

CLEAR gives the most explicit attention-level formulation of “conv-like” locality for pre-trained diffusion transformers. The attention function remains standard softmax scaled dot-product attention, but the visibility mask is local for image queries:

$$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$6

Thus each image token attends to all text tokens and only a circular local neighborhood of image tokens, while text queries remain global. The paper argues that successful linearization of pre-trained diffusion transformers depends on four properties—locality, formulation consistency, high-rank attention maps, and feature integrity—and presents CLEAR as satisfying all four (Liu et al., 2024). This is important because it counters another common misconception: conv-like attention need not imply abandoning softmax attention in favor of kernelized linear attention. In CLEAR, the formulation is preserved; only the attention pattern is localized.

Graph-based spatio-temporal models extend the same idea beyond Euclidean grids. D-STGCNT uses multi-hop adjacency matrices

$$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$7

graph convolution

$$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$8

multi-scale temporal convolutions with kernels $$P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,$$9, $d^l = (l_{\text{rp}})^l \times d^0.$0, and $d^l = (l_{\text{rp}})^l \times d^0.$1, and a transformer encoder on top of dense STGC-GRU blocks (Mourchid et al., 2023). Here the conv-like component is graph convolution plus ConvGRU, and the fusion occurs across hops, temporal scales, and network depth.

5. Complexity, objectives, and reported empirical behavior

A central motivation of this family is improved scaling. Long-context ScaleFormer keeps the native encoder length fixed at chunk length $d^l = (l_{\text{rp}})^l \times d^0.$2 and processes $d^l = (l_{\text{rp}})^l \times d^0.$3 chunks, yielding total encoder cost $d^l = (l_{\text{rp}})^l \times d^0.$4 and therefore effective linear scaling in the original sequence length $d^l = (l_{\text{rp}})^l \times d^0.$5 when $d^l = (l_{\text{rp}})^l \times d^0.$6 is fixed (Du et al., 13 Nov 2025). The time-series Conv-like ScaleFusion Transformer reduces token length exponentially with depth, activates a number of Conv-like Transformer Layers proportional to $d^l = (l_{\text{rp}})^l \times d^0.$7, and therefore makes depth itself length-aware (Zhang et al., 22 Sep 2025). CoaT reduces the cost of self-attention from quadratic token-token interaction to factorized attention with complexity $d^l = (l_{\text{rp}})^l \times d^0.$8 for fixed channel dimension, while also adding efficient convolution-like positional branches (Xu et al., 2021). CLEAR makes the most explicit complexity claim: it limits each image query to a fixed local support and reports 99.5% reduction in attention computations together with 6.3× acceleration for generating 8K-resolution images (Liu et al., 2024).

The optimization objectives also vary by domain. The time-series framework introduces a self-supervised representation objective composed of reconstruction and independence regularization,

$d^l = (l_{\text{rp}})^l \times d^0.$9

with downstream forecasting heads trained by L1 loss and classification heads by cross-entropy (Zhang et al., 22 Sep 2025). CLEAR distills a pre-trained quadratic teacher into a local-attention student using flow matching, prediction consistency, and attention-output consistency:

$P^l \times d^l$0

Only the attention-layer parameters are trainable, and the reported setup uses 10K self-generated samples for 10K iterations (Liu et al., 2024).

The reported empirical behavior is broadly consistent with the architectural claims. On variable-length forecasting, the time-series framework reports on ETTm1 that Ours: MSE-uf = 0.390, MAE-uf = 0.394, compared with PatchTST (second-best): MSE-uf = 0.398, MAE-uf = 0.403, and also reports markedly lower redundancy metrics such as Pearson-abs: Ours: 0.31 and Mutual Information: Ours: 2.45 (Zhang et al., 22 Sep 2025). In long-document summarization, ScaleFormer + Middle reports GovReport test ROUGE-1: 57.0, ROUGE-L: 27.7, and BERT-Score: 68.4, while on BookSum test it reports ROUGE-1: 39.2 vs Unlimiformer 37.3 (Du et al., 13 Nov 2025). CoaT-Lite-Medium reaches 84.5% on ImageNet at 384² with 45M parameters, while ASF-former-B reports 83.9% on ImageNet-1K under 12.9G MACs/56.7M Params (Xu et al., 2021, Su et al., 2022). These results do not establish a single universal best design, but they do indicate that locality plus scale-aware fusion is a recurring route to stronger efficiency-accuracy trade-offs.

6. Limitations, misconceptions, and future directions

Several limitations recur across the literature. Long-context ScaleFormer encodes chunks independently, so token-level self-attention never crosses chunk boundaries; cross-chunk structure is reconstructed only at the representation level, and the uniform cumulative kernel may over-smooth salient segments (Du et al., 13 Nov 2025). The time-series framework depends on hyperparameters such as $P^l \times d^l$1, $P^l \times d^l$2, $P^l \times d^l$3, and $P^l \times d^l$4, and the paper notes that irregularly sampled or event-based sequences may require adaptive patching (Zhang et al., 22 Sep 2025). CoaT reports that parallel co-scale groups increase latency and memory, especially at high resolutions (Xu et al., 2021). ASF-former retains a fixed 50/50 channel split and a global fusion gate, which limits the granularity of local/global allocation (Su et al., 2022). CLEAR preserves quality under local masking, but its sparse implementation can be less favorable at modest resolutions, where dense hardware-aware kernels such as FlashAttention already perform well (Liu et al., 2024).

The literature also clarifies several misunderstandings. One is that replacing quadratic attention necessarily requires low-rank kernelization; CLEAR argues instead for preserving the original softmax formulation and altering only the attention mask (Liu et al., 2024). Another is that multi-scale modeling is adequately handled by hierarchical downsampling alone; the medical and time-series ScaleFormer variants both argue that explicit inter-scale interaction is necessary because skip connections or coarse pyramids do not by themselves model cross-scale dependency (Huang et al., 2022, Zhang et al., 22 Sep 2025). A third is that convolution and attention can simply be juxtaposed; ASF-former and CTRL-F both emphasize that the fusion rule itself is decisive, with adaptive or collaborative fusion outperforming naive combination (Su et al., 2022, EL-Assiouti et al., 2024).

Future directions described in these works are largely convergent. Long-context ScaleFormer explicitly motivates replacing parameter-free global averages with local, learnable fusion kernels, multi-scale pyramids, hybrid conv + sparse attention, or directional dilated convolutions (Du et al., 13 Nov 2025). The time-series framework points toward richer self-supervision and broader multivariate modeling (Zhang et al., 22 Sep 2025). CoaT suggests further optimization of kernel sizes, number of scales, and parallelization (Xu et al., 2021). CLEAR implies that a multi-scale variant could combine CLEAR-style local neighborhoods at fine scales with coarser global or larger-radius interactions at coarse scales (Liu et al., 2024). Taken together, these proposals suggest that the next stage of Conv-like ScaleFusion Transformer research is likely to focus on more expressive yet still structured cross-scale operators, preserving locality and feature integrity while making the fusion mechanism itself more adaptive.