Multi-Scale Fusion Transformer

Updated 8 July 2026

Multi-scale fusion transformers are architectural principles that integrate multi-resolution representations with transformer-based attention to capture both local details and global context.
They combine CNN or windowed operations for local feature extraction with self- or cross-attention mechanisms for long-range dependency modeling, enhancing performance in diverse tasks.
Architectural variants such as dual-branch, hierarchical encoder-decoder, and parallel processing paths enable adaptive feature selection, benefiting applications like image fusion, speech separation, and semantic segmentation.

Multi-scale fusion transformer denotes a family of architectures in which representations are extracted at multiple resolutions, temporal granularities, or semantic levels and then fused with transformer-style attention mechanisms. Across the literature, the pattern appears in image fusion, semantic and medical segmentation, visual recognition, speech separation, time-series modeling, trajectory prediction, multimodal sentiment analysis, continuous sign language recognition, and LiDAR place recognition. A common objective is to combine local detail, typically emphasized by convolutional or windowed operators, with long-range dependencies or global context, typically emphasized by self-attention or cross-attention (VS et al., 2021, Wang et al., 2022, Xu et al., 2022).

1. Conceptual scope and evolution

The term emerged in several adjacent but not identical lines of work. In image fusion, the "Image Fusion Transformer" introduced a transformer-based multi-scale fusion strategy in a two-stage framework, with multi-scale encoder features fused by a Spatio-Transformer block that combines a CNN branch and a transformer branch (VS et al., 2021). In fine-grained visual recognition, AFTrans used a global-and-local multi-scale pipeline supervised by a weight-sharing encoder, with attention-derived region selection rather than explicit box annotations (Zhang et al., 2021). In general visual recognition, MAFormer formalized multi-scale attention fusion as a dual-stream transformer block with local window attention, global learning with down-sampling, and attention-based fusion (Wang et al., 2022).

By 2022 and after, the idea had broadened beyond images. MSFFT-Net used multiple parallel processing paths at different temporal scales for speech separation, rather than a single dual-path sequence model (Xu et al., 2022). FeSeFormer introduced scale-level feature selection and full-scale feature fusion for semantic segmentation, explicitly selecting informative subsets from the entire multi-scale feature set for each query feature (Lin et al., 2022). Later work extended the same design logic to incomplete vehicle trajectories, skin-lesion segmentation, 3D spine segmentation, spiking vision transformers, variable-length long time series, and cross-view LiDAR descriptors (Liu et al., 2024, Qamar et al., 5 Mar 2025, Xiang et al., 17 Mar 2025, Hua et al., 19 May 2025, Zhang et al., 22 Sep 2025, Li et al., 6 Apr 2026).

This distribution suggests that "multi-scale fusion transformer" is better understood as an architectural principle than as a single standardized network. The principle is the explicit interaction of representations computed at multiple scales, with attention used not merely for encoding but also for scale coupling, adaptive weighting, or query-conditioned selection.

2. Architectural archetypes

Several recurrent architectural forms appear across the literature.

Archetype	Representative papers	Distinguishing mechanism
Dual-branch local/global fusion	IFT, MATCNN, MAFormer	CNN or local-attention branch plus transformer or global-attention branch
Hierarchical encoder-decoder fusion	ScaleFusionNet, WoundFormer, RefineFormer3D	Multi-scale skip integration with cross-attention or spatial aggregation
Parallel multi-resolution paths	MSFFT-Net, sign-language MSFT	Simultaneous fine and coarse temporal paths with explicit exchange or concatenation
Pyramid or scale-selective fusion	FeSeFormer, PST, MPTF-Net	Query-conditioned selection, sparse tokens, or cross-view pyramid attention

In dual-branch designs, one branch is responsible for local structure and the other for long-range context. IFT makes this division explicit: the spatial branch consists of convolutional layers plus