SFormer: Multi-Domain Transformer Variants

• SFormer is an umbrella term for distinct Transformer-derived architectures tailored to specific domains such as language modeling, spiking neural conversion, weather forecasting, and underwater image enhancement.
• In language contexts, Semformer mitigates teacher forcing issues by using planning tokens to generate latent semantic representations, achieving near-perfect graph path-finding and improved benchmark results.
• Other SFormer variants employ domain-specific techniques—adaptive scaling in spiking models, lead-time conditioning in weather forecasting, and frequency-guided modules in image enhancement—to boost efficiency, accuracy, and energy reduction.

SFormer is not a single canonical architecture in contemporary arXiv usage, but an overloaded designation applied to several distinct Transformer-derived models in different subfields. The label appears most prominently in at least four technically unrelated contexts: semantic-planning language modeling, single-timestep spiking Transformers for ANN-to-SNN conversion, a lead-time-conditioned weather forecasting backbone in AI-based data assimilation, and an SNR-guided frequency-domain model for underwater image enhancement. Around these exact or near-exact uses sits a larger family of similarly named architectures—such as SGFormer, SSformer, SASFormer, DenSformer, DSFormer, and G-SFormer—whose shared suffix indicates Transformer lineage rather than common method design (Yin et al., 2024, Chen et al., 27 Oct 2025, Wang et al., 2024, Tian et al., 26 Aug 2025).

1. Disambiguation and naming

The first technical fact about SFormer is terminological: the name does not identify a single model family. In practice, correct interpretation depends on domain, task, and exact capitalization. Some papers use SFormer exactly; others use Sformer or near-matches such as Semformer and DSFormer.

Label in paper	Domain	Core idea
Semformer	Autoregressive language modeling	Planning tokens predict latent semantic representations of the future response
SFormer	ANN-to-SNN conversion for Transformers	Scale-and-Fire Neurons enable single-timestep spiking inference
Sformer	Weather forecasting in DABench	Lead-time-conditioned Swin Transformer used as benchmark forecasting backbone
SFormer	Underwater image enhancement	Dual-branch U-shaped network with SNR-guided frequency-domain modules

This multiplicity is not superficial nomenclature. Each use attaches the name to a different modeling problem, loss construction, and inductive bias. A common misconception is therefore to treat SFormer as a coherent architecture lineage. In the broader literature represented here, the short form also overlaps with unrelated near-name models: SASFormer for sparsely annotated semantic segmentation (Su et al., 2022), SGFormer for large-graph learning (Wu et al., 2023), SSformer for semantic segmentation (Shi et al., 2022), FG-STFormer with FG-SFormer for skeleton-based action recognition (Gao et al., 2022), DenSformer for image denoising (Yao et al., 2022), DSFormer/SFormer for hyperspectral image classification (Xu et al., 2024), SGFormer for 3D scene graph generation (Lv et al., 2023), and G-SFormer for 3D human pose estimation (Cui et al., 2024).

2. Semformer in language modeling

In NLP contexts, the short query often points to Semformer, a decoder-only Transformer training framework that augments next-token prediction with an explicit semantic-planning objective. Its starting point is a critique of teacher forcing: because future target tokens become part of the revealed prefix during training, a model can exploit shortcuts instead of learning genuinely future-aware internal computation. Semformer addresses this by splitting a sequence into prefix and response, inserting planning tokens $d=\{d_1,\dots,d_k\}$ after the prefix, and training the hidden states at those positions to predict a latent semantic representation $Z=\{z_1,\dots,z_k\}$ of the future response induced by an autoencoder. The full objective is

$$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$

The method is designed so that planning tokens are not ordinary lexical outputs. Rather, they act as designated internal computation slots. During training, the autoencoder encodes the response suffix into compact latent vectors through a cross-attention bottleneck, and a linear predictor regresses planning-token hidden states onto these vectors. During inference, the model simply appends planning tokens to the prefix and performs standard autoregressive decoding; there is no separate planning module at test time.

Empirically, the strongest evidence comes from graph path-finding, where standard teacher-forced GPT2-Large training achieves only 49.2 on $G(2,20)$, 20.1 on $G(5,20)$, and 10.1 on $G(10,20)$, whereas Semformer reports 99.9, 99.9, and 99.6 on the same settings. In 125M-parameter pretraining on OpenWebText, it improves Wikitext perplexity from 37.5 to 35.6 and LAMBADA from 42.5 / 32.1 accuracy to 38.8 / 33.5, while also improving in-context learning on SST-2 and MRPC and ROUGE-L in summarization on XSum, SAMSum, and DialogSum. The paper also reports limitations: OOD transfer from $G(20,10)$ to $G(25,10)$ drops to 2.8, and performance on the longer-path graph $G(10,40)$ is 10.0, indicating that shortcut mitigation does not imply broad extrapolative generalization (Yin et al., 2024).

3. SFormer as a spiking Transformer

A different exact use of the name appears in ANN-to-SNN conversion. Here SFormer is the Transformer-specific realization of the Scale-and-Fire Neuron (SFN) framework for single-timestep spiking inference. The motivating problem is that conventional ANN-to-SNN conversion preserves accuracy through multi-timestep rate coding, but pays a large latency and compute penalty because expensive non-neuronal operations—especially matrix multiplications in Transformer blocks—must be executed repeatedly across timesteps.

The theoretical basis is the paper’s Temporal-to-Spatial Equivalence Theory, which shows that under suitable conditions a multi-timestep integrate-and-fire neuron can be replaced by a single-timestep multi-threshold neuron. SFN operationalizes this with adaptive scaling and a piecewise-constant fire function,

$o(t) = \lambda\theta \, G_{\lambda\theta}(h(t)),$

where $Z=\{z_1,\dots,z_k\}$0 is tuned by Bayesian optimization. SFormer then specializes this design to Transformer activations, whose cross-layer magnitude variation and post-SoftMax skew make naive thresholding ineffective. Its firing levels mix linear and exponential growth, it uses separate positive and negative thresholds, and in the reported setting $Z=\{z_1,\dots,z_k\}$1, so each SFN has $Z=\{z_1,\dots,z_k\}$2 thresholds total. A further Transformer-specific adaptation sets dedicated post-SoftMax thresholds so that decisive attention peaks are preserved rather than clipped into crude binary patterns.

The resulting converted spiking Transformers report 70.7% top-1 on ImageNet-1K for ViT-Base/16 at $Z=\{z_1,\dots,z_k\}$3, 82.4% for ViT-Large/16, and 88.8% for EVA, with the EVA result only 0.8% below the 89.6% ANN baseline. On COCO-2017, the same framework reports 60.3 [email protected]:0.95 for object detection and 50.7 [email protected]:0.95 for instance segmentation at $Z=\{z_1,\dots,z_k\}$4. The energy ratios are 0.17 for ViT-Base/16, 0.15 for ViT-Large/16, and 0.19 for EVA, which for EVA corresponds to about an 81% energy reduction relative to the ANN baseline ratio 1.00. The ablation study is especially sharp: without scaling, all tested fire functions collapse to very low accuracy, whereas scaling plus the full Transformer-tailored SFN fire function yields the 88.8% headline result (Chen et al., 27 Oct 2025).

4. Sformer in AI-based weather forecasting and data assimilation

In DABench, Sformer is neither a LLM nor a vision backbone in the usual sense. It is a pre-trained Transformer-based global weather forecasting model supplied as the benchmark’s standardized forecast engine. Its role is to generate background fields for data assimilation and to measure how strongly a DA system’s analysis quality affects downstream forecasts. In the benchmark formulation, the background evolves by

$Z=\{z_1,\dots,z_k\}$5

with $Z=\{z_1,\dots,z_k\}$6 instantiated by Sformer.

Architecturally, Sformer is a Swin Transformer-based global forecaster operating on upper-air variables $Z=\{z_1,\dots,z_k\}$7 across nine pressure levels and surface variables T2M, U10, V10, and MSLP on a $Z=\{z_1,\dots,z_k\}$8 global grid. It uses separate patch embedding layers for different modalities, concatenates them, processes them with four SwinTransformer blocks, and conditions forecasting on lead time through adaptive LayerNorm. This allows a single model to support multiple lead times and hierarchical temporal aggregation, borrowing the efficiency idea of Pangu-Weather while avoiding a bank of separately trained lead-time models. Training uses a two-phase latitude-weighted $Z=\{z_1,\dots,z_k\}$9 objective, with $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$0 in phase 1 and $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$1 in phase 2.

Functionally, Sformer is central to the benchmark because it converts DA quality into forecast quality. Initialized directly from ERA5, it achieves a skillful forecast lead time of 9 days. In the end-to-end system that couples 4DVarFormerV2 + Sformer, the benchmark reports stable one-year DA cycling and up to 7 days of skillful forecast lead time using real observations. The reference ERA5-initialized Sformer forecasts also report 3-day / 5-day Z500 RMSE of $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$2, T850 RMSE of $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$3, and T2M RMSE of $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$4, together with 3-day / 5-day Z500 ACC of $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$5 (Wang et al., 2024).

5. SFormer for underwater image enhancement

In low-level vision, SFormer denotes the underwater image enhancement architecture introduced by Xin Tian, Yingtie Lei, Xiujun Zhang, Zimeng Li, Chi-Man Pun, and Xuhang Chen. This model targets the limitations of spatial-domain SNR priors in underwater restoration, specifically their inability to separate cross-channel interference effectively and their limited power to amplify informative structure while suppressing noise. Its remedy is to move SNR guidance into the frequency domain.

The network is a dual-branch U-shaped architecture with a conventional RGB stream and an SNR-guided stream. Its two defining modules are FAST—the Fourier Attention SNR-prior Transformer—at skip connections, and FAT—the Frequency Adaptive Transformer—at the bottleneck. In FAST, RGB and SNR features are projected to $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$6, $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$7, and $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$8, transformed by FFT, decomposed into amplitude and phase spectra, and recombined through inverse FFT after elementwise amplitude- and phase-wise interactions. The resulting spectral attention modulates $$L = L_{\mathrm{LM}} + L_{\mathrm{AE}} + \alpha L_{\mathrm{RP}}.$$9 and is followed by an MLP refinement. In FAT, bottleneck RGB tokens cross-attend to SNR tokens, then pass through Transformer layers whose adaptive feed-forward network separates low-frequency and high-frequency components, refines them with channel and spatial attention respectively, and fuses them through gated depth-wise convolution. The total loss combines spatial, frequency, perceptual, LAB, and LCH terms: $G(2,20)$0 with $G(2,20)$1.

The model is trained on 4,800 paired images from UIEB, EUVP, and LSUI. Reported results are strongest on all three benchmarks: on EUVP it reaches PSNR 27.451, SSIM 0.936, LPIPS 0.163, UCIQE 0.863, and $G(2,20)$2; on LSUI, PSNR 26.946 and SSIM 0.956; on UIEB, PSNR 23.680 and SSIM 0.964. The ablations also show clear complementarity between FAST and FAT: on EUVP, a baseline U-Net yields PSNR 21.145, adding only FAT gives 24.543, adding only FAST gives 24.852, and the full model reaches 27.451 (Tian et al., 26 Aug 2025).

6. Related and similarly named architectures

The instability of the term is reinforced by a broader ecosystem of near-name models whose technical content is unrelated. In graph learning, SGFormer means a simplified graph transformer for large-scale node representation learning, built around a single global linear-attention layer plus a GCN branch rather than any of the exact SFormer variants (Wu et al., 2023). In point cloud scene understanding, SGFormer instead expands to Semantic Graph Transformer and combines a Graph Embedding Layer with a Semantic Injection Layer using ChatGPT-generated class descriptions encoded by CLIP for 3D scene graph generation (Lv et al., 2023). In semantic segmentation, SSformer is a lightweight Swin-based encoder-decoder with an all-MLP decoder (Shi et al., 2022), while SASFormer targets sparsely annotated semantic segmentation and uses hierarchical patch attention maps plus an affinity loss on top of SegFormer (Su et al., 2022).

A second cluster appears in structured vision and signal processing. FG-STFormer contains FG-SFormer, a focal-joints and global-parts coupling spatial transformer for skeleton-based action recognition (Gao et al., 2022). DenSformer is a dense residual Transformer for image denoising whose internal repeated unit is itself called an Sformer block or group (Yao et al., 2022). In hyperspectral imaging, the model queried as DSFormer is described in the paper as SFormer, built from a Kernel Selective Transformer Block and a Token Selective Transformer Block for adaptive receptive-field fusion and top-$G(2,20)$3 token selection (Xu et al., 2024). In 3D human pose estimation, G-SFormer combines a part-based spatial graph with a Skipped Transformer for efficient temporal modeling (Cui et al., 2024). A plausible implication is that “SFormer” functions less as a stable architectural family name than as a recurrent suffix attached to highly task-specific Transformer variants across arXiv domains.