---
title: Conv-like ScaleFusion Transformer
url: https://www.emergentmind.com/topics/conv-like-scalefusion-transformer
type: topic
---

# Conv-like ScaleFusion Transformer

to describe architectures that combine convolution-like locality and positional bias with explicit scale-wise interaction across hierarchical representations"—that is supported across details. Might mark as editor's synthesis/inference.

Need include mathematical notation when appears. Could compare formulations: ScaleFormer long text cumulation equations, time series CTL equations, CoaT conv attention, medical ScaleFormer dual-axis attention, CLEAR mask. Need keep article comprehensive and organized. Need avoid too much formula because user wants article comprehensive. Should have 6 sections maybe: definition and scope; core design principles; architectural realizations across modalities; fusion mechanisms; complexity/training; limitations/prospects. But headings numbered.

Need maybe include table listing representative instances and modalities. Since detailed explanations should be outside table. Fine.

Need cite papers at most once per section. Could cite multiple IDs in a section. We can have introduction cite all primary. Then sections cite relevant ones. Need not overcite.

Need ensure no claim from arXiv search results beyond data. But maybe cite only provided IDs; search can just verify titles. Let's use search tool minimal.
Searching arXiv for the cited papers to verify identifiers and titles.
The **Conv-like ScaleFusion Transformer** denotes a class of Transformer architectures that couple convolution-like locality with explicit multi-scale feature interaction. In a strict sense, the phrase names a **Multi-Scale Representation Learning Framework based on a Conv-like ScaleFusion Transformer** for **variable-length long time series** [2509.17845]. More broadly, closely related work applies the same design logic across long-context text processing, image classification, medical image segmentation, spatio-temporal graph learning, and diffusion transformers: progressive or hierarchical token compression, local inductive bias inside attention or parallel convolution branches, and explicit fusion across scales or representation levels [2511.10029; 2104.06399; 2207.14552; 2204.12196; 2407.06673; 2412.16112; 2401.06150].

## 1. Definition and scope

A Conv-like ScaleFusion Transformer combines two architectural commitments. The first is **conv-like processing**: local receptive fields, pyramidal downsampling, depthwise convolutional positional encoding, local-window attention, or explicit convolution branches. The second is **ScaleFusion**: interaction among features at different scales, resolutions, chunk levels, or abstraction depths, typically through cross-scale attention, interpolation, matched-patch fusion, or adaptive weighting [2509.17845; 2104.06399; 2207.14552].

A concise synthesis of representative realizations is given below.

| Paper | Domain | Defining mechanism |
|---|---|---|
| "ScaleFormer: Span Representation Cumulation for Long-Context Transformer" [2511.10029] | Long-context text | Overlapping chunks with parameter-free cumulative boundary fusion |
| "Conv-like Scale-Fusion Time Series Transformer: A Multi-Scale Representation for Variable-Length Long Time Series" [2509.17845] | Time series | Conv-like Transformer Layers, cross-scale attention, log-space normalization |
| "Co-Scale Conv-Attentional Image Transformers" [2104.06399] | Vision | Conv-attention plus co-scale parallel fusion |
| "ScaleFormer: Revisiting the Transformer-based Backbones from a Scale-wise Perspective for Medical Image Segmentation" [2207.14552] | Medical segmentation | Intra-scale local-global coupling and spatial-aware inter-scale transformer |
| "Adaptive Split-Fusion Transformer" [2204.12196] and "CTRL-F" [2407.06673] | Hybrid visual recognition | Parallel convolution and transformer branches with learned fusion |
| "CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up" [2412.16112] | Diffusion transformers | Local softmax attention with convolution-like receptive fields |

Within this family, “scale” is not restricted to image resolution. In long-context NLP it may denote chunk order and cumulative document context; in time series it may denote progressively re-patched temporal resolutions; in graph models it may denote multi-hop neighborhoods and multiple temporal kernel sizes. This suggests that the unifying property is not a single canonical block, but an architectural pattern in which locality and hierarchy are made explicit and then fused rather than left implicit in a flat token sequence [2511.10029; 2509.17845; 2401.06150].

## 2. Core architectural principles

The clearest formalization appears in the time-series framework of [2509.17845]. A raw series $\mathbf{X} = [x_1, x_2, \dots, x_T]$ is first patched and embedded, then processed by a stack of **Conv-like Transformer Layers**. Each layer performs **re-patching** along the patch dimension, reducing the number of tokens while expanding channel capacity. If the previous level has $P^{l-1}$ patch tokens, the next level has
$$
P^l = \left\lceil \frac{P^{l-1}}{l_{\text{rp}}} \right\rceil,
$$
while the feature dimension expands as
$$
d^l = (l_{\text{rp}})^l \times d^0.
$$
The architecture is explicitly pyramidal: temporal resolution decreases, channels increase, and the feature volume $P^l \times d^l$ is kept approximately constant [2509.17845].

A different but related principle appears in long-context ScaleFormer [2511.10029]. There, a long sequence is partitioned into overlapping segments $S_1,\dots,S_C$, each processed independently by a pre-trained encoder. Rather than altering the backbone attention, the model compresses each chunk into boundary and optional middle representations, then fuses boundaries with cumulative directional context:
$$
\text{ctx}^{\text{back}}_i = \frac{1}{2i - 1}\left( L_i + \sum_{j=1}^{i-1} (L_j + R_j) \right),
$$
$$
\text{ctx}^{\text{fwd}}_i = \frac{1}{2(C - i) + 1}\left( R_i + \sum_{j=i+1}^{C} (L_j + R_j) \right).
$$
The fused boundaries are
$$
L'_i = \alpha \cdot L_i + (1 - \alpha)\cdot \text{ctx}^{\text{back}}_i,
$$
$$
R'_i = \alpha \cdot R_i + (1 - \alpha)\cdot \text{ctx}^{\text{fwd}}_i.
$$
This is parameter-free and structurally analogous to directional convolution over chunk-level features, even though the aggregation is prefix/suffix cumulative rather than a learned finite kernel [2511.10029].

In vision, CoaT builds the conv-like aspect into attention itself. Its conv-attentional mechanism combines factorized attention with convolutional relative position encoding:
$$
\text{ConvAtt}(X) = \frac{Q}{\sqrt{C}}\Big(\mathrm{softmax}(K)^\top V\Big) + \hat{EV},
$$
where $\hat{EV}$ is implemented with depthwise convolution, giving the attention block a local, convolution-like positional branch alongside global content mixing [2104.06399]. Medical ScaleFormer uses a different efficiency strategy: **Dual-Axis MSA**, which decomposes full 2D attention into row-wise and column-wise attention,
$$
A(Q, K, V) = \text{softmax}\left( \frac{Q_{rp} K_{rp}^\top}{\sqrt{d_k}} \right)V + \text{softmax}\left( \frac{Q_{cp} K_{cp}^\top}{\sqrt{d_k}} \right)V,
$$
reducing complexity from $\mathcal{O}(H^2W^2C)$ to $\mathcal{O}(H^2WC + HW^2C)$ while preserving structured long-range dependency at each scale [2207.14552].

Across these instantiations, the recurrent design pattern is a replacement of undifferentiated global token interaction with structured locality plus explicit hierarchical communication.

## 3. ScaleFusion mechanisms across domains

The principal distinction among Conv-like ScaleFusion Transformer variants lies in how they define and fuse scales. In the time-series framework, each coarse representation $\mathbf{H}_l^{*}$ queries the previous finer scale $\mathbf{H}_{l-1}$ through multi-head cross-attention:
$$
\mathbf{Q}_{l}^{h} = \mathbf{W}_{l}^{Q,h} \mathbf{H}_{l}^{*}, \quad
\mathbf{K}_{l}^{h} = \mathbf{W}_{l}^{K,h} \mathbf{H}_{l-1}, \quad
\mathbf{V}_{l}^{h} = \mathbf{W}_{l}^{V,h} \mathbf{H}_{l-1},
$$
and the fused output is
$$
\mathbf{H}_{l} = \text{LayerNorm}(\mathbf{H}_{l}^{*} + \mathbf{O}_{l}).
$$
This replaces simple residual transmission with content-adaptive cross-scale interaction, and the paper attributes to it improved feature independence and reduced redundancy [2509.17845].

CoaT realizes ScaleFusion through **co-scale parallel blocks**. Multiple scales are maintained simultaneously, and each scale receives interpolated features from the others before a shared feed-forward update. The result is a symmetric feature-level fusion pathway across resolutions rather than a purely fine-to-coarse hierarchy [2104.06399]. Medical ScaleFormer is more spatially constrained: it matches corresponding patches across scales, concatenates them,
$$
[\text{flatten}(p_j^{(i)}),\, \text{flatten}(p_j^{(i+1)}),\, \text{flatten}(p_j^{(i+2)})] \rightarrow p_j^{\text{cat}},
$$
applies inter-scale self-attention and MLP, then splits and reshapes the outputs back into scale-specific feature maps [2207.14552]. This makes the fusion explicitly **spatial-aware**, limiting cross-scale interaction to consensual regions rather than all possible token pairs.

Hybrid visual classifiers adopt branch-level rather than scale-level fusion. ASF-former splits channels equally into a convolutional path and an attention path, then fuses them with adaptive weights,
$$
\boldsymbol{S} = \hat{\boldsymbol{T}}^{(a)} + \hat{\boldsymbol{T}}^{(b)},
$$
$$
\alpha = \sigma(f_w(\boldsymbol{S})), \quad \beta = 1 - \alpha,
$$
$$
\hat{\boldsymbol{T}} = \alpha \cdot \hat{\boldsymbol{T}}^{(a)} + \beta \cdot \hat{\boldsymbol{T}}^{(b)} + \boldsymbol{S}.
$$
The fusion weights are content-aware rather than fixed, and the design is explicitly motivated by different local/global requirements across depth and category [2204.12196]. CTRL-F uses a related but not identical strategy: multi-level convolutional features from two stages are tokenized into small-patch and large-patch branches, and the branches exchange information via cross-attention in the **Multi-Level Feature Cross-Attention** module, after which convolutional and transformer predictions are merged by **Adaptive Knowledge Fusion** or **Collaborative Knowledge Fusion** [2407.06673].

A common misconception is that ScaleFusion is equivalent to ordinary skip connections. Several of these architectures reject that equivalence explicitly: [2509.17845] replaces residual scale transfer with cross-scale attention, [2207.14552] introduces a dedicated inter-scale transformer because skip connections alone do not model cross-scale dependency, and [2204.12196] shows that context-agnostic fusion underperforms content-aware adaptive fusion.

## 4. Conv-like locality inside Transformer processing

The “conv-like” qualifier refers to multiple, technically distinct mechanisms. In the long-context setting, it refers to **sliding-window segmentation** with overlap and cumulative aggregation over chunk boundaries, which behaves like a document-level convolution over chunk representations while remaining parameter-free [2511.10029]. In the time-series setting, it refers to re-patching-based temporal downsampling followed by global self-attention over compacted tokens, producing local aggregation through re-patching and global aggregation through attention at each pyramid level [2509.17845].

In CoaT, conv-like behavior is encoded directly in attention. Convolutional Position Encoding and Convolutional Relative Position Encoding use depthwise convolutions, with kernel sizes $3$, $5$, and $7$ across head groups, so that locality bias is injected without abandoning Transformer-native content-adaptive mixing [2104.06399]. Medical ScaleFormer similarly retains CNN blocks as the primary local feature extractors and augments them with scale-wise transformer blocks; its Enhanced MLP also includes depth-wise convolution before channel mixing, reinforcing the local-global hybrid character [2207.14552].

CLEAR gives the most explicit attention-level formulation of “conv-like” locality for pre-trained diffusion transformers. The attention function remains standard softmax scaled dot-product attention, but the visibility mask is local for image queries:
$$
M_{ij} =
\begin{cases}
1, & \text{if } i \le n_{\text{text}} \text{ or } j \le n_{\text{text}} \text{ or } d_{ij}^{(x)2} + d_{ij}^{(y)2} < r^2, \\
0, & \text{otherwise}.
\end{cases}
$$
Thus each image token attends to all text tokens and only a circular local neighborhood of image tokens, while text queries remain global. The paper argues that successful linearization of pre-trained diffusion transformers depends on four properties—**locality, formulation consistency, high-rank attention maps, and feature integrity**—and presents CLEAR as satisfying all four [2412.16112]. This is important because it counters another common misconception: conv-like attention need not imply abandoning softmax attention in favor of kernelized linear attention. In CLEAR, the formulation is preserved; only the attention pattern is localized.

Graph-based spatio-temporal models extend the same idea beyond Euclidean grids. D-STGCNT uses multi-hop adjacency matrices
$$
A_k = D_k^{-1/2} (\tilde{A}_k + I) D_k^{-1/2},
$$
graph convolution
$$
G(P) = \sum_{k}^{\Gamma_a} (P A_k) W_k,
$$
multi-scale temporal convolutions with kernels $(9,1)$, $(15,1)$, and $(20,1)$, and a transformer encoder on top of dense STGC-GRU blocks [2401.06150]. Here the conv-like component is graph convolution plus ConvGRU, and the fusion occurs across hops, temporal scales, and network depth.

## 5. Complexity, objectives, and reported empirical behavior

A central motivation of this family is improved scaling. Long-context ScaleFormer keeps the native encoder length fixed at chunk length $L$ and processes $C \approx \frac{N}{L-O}$ chunks, yielding total encoder cost $O(N \cdot L)$ and therefore effective linear scaling in the original sequence length $N$ when $L$ is fixed [2511.10029]. The time-series Conv-like ScaleFusion Transformer reduces token length exponentially with depth, activates a number of Conv-like Transformer Layers proportional to $\log_{l_{\text{rp}}} T$, and therefore makes depth itself length-aware [2509.17845]. CoaT reduces the cost of self-attention from quadratic token-token interaction to factorized attention with complexity $O(NC^2)$ for fixed channel dimension, while also adding efficient convolution-like positional branches [2104.06399]. CLEAR makes the most explicit complexity claim: it limits each image query to a fixed local support and reports **99.5% reduction in attention computations** together with **6.3× acceleration** for generating **8K-resolution images** [2412.16112].

The optimization objectives also vary by domain. The time-series framework introduces a self-supervised representation objective composed of reconstruction and independence regularization,
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{recon}} + \alpha \, \mathcal{L}_{\text{indep}},
$$
with downstream forecasting heads trained by L1 loss and classification heads by cross-entropy [2509.17845]. CLEAR distills a pre-trained quadratic teacher into a local-attention student using flow matching, prediction consistency, and attention-output consistency:
$$
\min_{\theta} \mathbb{E}\left[ \mathcal{L}_{fm} + \alpha \mathcal{L}_{pred} + \beta \mathcal{L}_{attn} \right].
$$
Only the attention-layer parameters are trainable, and the reported setup uses **10K self-generated samples for 10K iterations** [2412.16112].

The reported empirical behavior is broadly consistent with the architectural claims. On variable-length forecasting, the time-series framework reports on ETTm1 that **Ours: MSE-uf = 0.390, MAE-uf = 0.394**, compared with **PatchTST (second-best): MSE-uf = 0.398, MAE-uf = 0.403**, and also reports markedly lower redundancy metrics such as **Pearson-abs: Ours: 0.31** and **Mutual Information: Ours: 2.45** [2509.17845]. In long-document summarization, ScaleFormer + Middle reports **GovReport test ROUGE-1: 57.0**, **ROUGE-L: 27.7**, and **BERT-Score: 68.4**, while on BookSum test it reports **ROUGE-1: 39.2 vs Unlimiformer 37.3** [2511.10029]. CoaT-Lite-Medium reaches **84.5%** on ImageNet at **384²** with **45M** parameters, while ASF-former-B reports **83.9% on ImageNet-1K** under **12.9G MACs/56.7M Params** [2104.06399; 2204.12196]. These results do not establish a single universal best design, but they do indicate that locality plus scale-aware fusion is a recurring route to stronger efficiency-accuracy trade-offs.

## 6. Limitations, misconceptions, and future directions

Several limitations recur across the literature. Long-context ScaleFormer encodes chunks independently, so token-level self-attention never crosses chunk boundaries; cross-chunk structure is reconstructed only at the representation level, and the uniform cumulative kernel may over-smooth salient segments [2511.10029]. The time-series framework depends on hyperparameters such as $l_p$, $s_p$, $l_{\text{rp}}$, and $L_{\text{max}}$, and the paper notes that irregularly sampled or event-based sequences may require adaptive patching [2509.17845]. CoaT reports that parallel co-scale groups increase latency and memory, especially at high resolutions [2104.06399]. ASF-former retains a fixed 50/50 channel split and a global fusion gate, which limits the granularity of local/global allocation [2204.12196]. CLEAR preserves quality under local masking, but its sparse implementation can be less favorable at modest resolutions, where dense hardware-aware kernels such as FlashAttention already perform well [2412.16112].

The literature also clarifies several misunderstandings. One is that replacing quadratic attention necessarily requires low-rank kernelization; CLEAR argues instead for preserving the original softmax formulation and altering only the attention mask [2412.16112]. Another is that multi-scale modeling is adequately handled by hierarchical downsampling alone; the medical and time-series ScaleFormer variants both argue that explicit inter-scale interaction is necessary because skip connections or coarse pyramids do not by themselves model cross-scale dependency [2207.14552; 2509.17845]. A third is that convolution and attention can simply be juxtaposed; ASF-former and CTRL-F both emphasize that the fusion rule itself is decisive, with adaptive or collaborative fusion outperforming naive combination [2204.12196; 2407.06673].

Future directions described in these works are largely convergent. Long-context ScaleFormer explicitly motivates replacing parameter-free global averages with **local, learnable fusion kernels**, **multi-scale pyramids**, **hybrid conv + sparse attention**, or **directional dilated convolutions** [2511.10029]. The time-series framework points toward richer self-supervision and broader multivariate modeling [2509.17845]. CoaT suggests further optimization of kernel sizes, number of scales, and parallelization [2104.06399]. CLEAR implies that a multi-scale variant could combine CLEAR-style local neighborhoods at fine scales with coarser global or larger-radius interactions at coarse scales [2412.16112]. Taken together, these proposals suggest that the next stage of Conv-like ScaleFusion Transformer research is likely to focus on more expressive yet still structured cross-scale operators, preserving locality and feature integrity while making the fusion mechanism itself more adaptive.

Source: https://www.emergentmind.com/topics/conv-like-scalefusion-transformer