Transformer-SL: A Multifaceted Research Landscape

• Transformer-SL is a polysemous term applied across fields such as synthetic lethality prediction, split learning, sign language processing, and more, each with distinct domain-specific implementations.
• It typically employs hybrid architectures that combine transformer modules with GNNs, ResNets, or convolutional components to mitigate issues like over-smoothing and bottlenecking.
• Key applications range from gene-edge classification and privacy-preserving split learning to video-to-text translation in sign language, reflecting its broad impact and design variability.

“Transformer-SL” is not a single standardized model name in the arXiv literature. The designation is used across multiple, technically unrelated research programs, including synthetic lethality prediction, split learning with Vision Transformers, sign language translation and continuous sign language recognition, self-stabilizing distributed graph algorithms for locally checkable labeling problems, super-resolution calibration in magnetic particle imaging, self-supervised accelerated MR reconstruction, and lightweight speech-to-intent modeling. In the synthetic lethality literature, the MSGT-SL paper explicitly notes that it does not cite any prior SL model named “Transformer-SL,” and presents its own method as distinct rather than equivalent to an earlier canonical transformer baseline (Zhao et al., 2023).

1. Terminological scope

The expression “Transformer-SL” is polysemous. In some papers, “SL” denotes synthetic lethality; in others it denotes split learning, sign language, self-stabilizing local problems, super-resolution learning, self-supervised learning, or speech language understanding. The term therefore cannot be understood independently of domain context. This suggests that “Transformer-SL” functions less as the name of a single architecture than as a domain-dependent shorthand for “a transformer method applied to an SL-labeled task” (Zhao et al., 2023, Almalik et al., 2023, Ruiz et al., 4 Feb 2025, Rastgoo et al., 2024, Bitton et al., 2021, Güngör et al., 2021, S. et al., 2023, Wang et al., 2020, Yin et al., 2020, Baek et al., 2022).

Domain	“SL” meaning	Representative paper
Computational biology	Synthetic lethality	MSGT-SL (Zhao et al., 2023)
Distributed learning	Split learning	FeSViBS (Almalik et al., 2023), CutMixSL (Baek et al., 2022)
Sign language	Sign language / sign language translation	Spatio-temporal Transformer (Ruiz et al., 4 Feb 2025), STMC-Transformer (Yin et al., 2020), stochastic SLT transformer (Voskou et al., 2021), boundary detection transformer (Rastgoo et al., 2024)
Distributed algorithms	Self-stabilizing local problems	Fully adaptive self-stabilizing transformer (Bitton et al., 2021)
Imaging inverse problems	Super-resolution learning / self-supervised learning	TranSMS (Güngör et al., 2021), SDLFormer (S. et al., 2023)
Spoken language systems	Speech language understanding	Light transformer for speech-to-intent (Wang et al., 2020)

The ambiguity is not merely lexical. Each usage carries a different mathematical object, supervisory signal, and systems assumption. In synthetic lethality, the primitive object is an edge on a gene graph; in split learning it is the smashed representation at a model cut; in sign language translation it is a video-to-text or gloss-to-text sequence mapping; in self-stabilization it is a compiler from fault-free LCL algorithms to self-stabilizing ones; in MRI and MPI it is a reconstruction or calibration operator constrained by a forward model.

2. Transformer-SL as synthetic lethality prediction

In the synthetic lethality setting, MSGT-SL formulates SL prediction as binary edge classification over a cell-line-specific core gene graph $G_1=(V_1,E_1)$, augmented by four omics graphs for physical interactions, genetic interactions, expression co-correlation, and co-essentiality. The full multi-view graph is $\mathcal{G}=\{G_1,\dots,G_M\}$ with $M=5$. The two SL datasets are K562, with 448 core genes and 1,523 SL edges, and Jurkat, with 387 core genes and 373 SL edges; positive SL pairs are defined by interaction scores below $-3$. The SL view is split into train/validation/test with ratios $0.7/0.1/0.2$, and evaluation uses both leave-gene-pair-out and leave-gene-out protocols (Zhao et al., 2023).

The architecture is hybrid rather than purely transformer-based. A shallow multi-view GNN performs $L=2$ GCN layers with hidden dimensions $128 \rightarrow 64$ on each view,

$F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$

producing view-specific node embeddings. Parallel random walks of length 10 are then launched from batches of $B=100$ core genes across the omics graphs, and the resulting sampled set is capped at $N'=500$ nodes per batch. Multi-view embeddings for sampled genes are concatenated,

$\mathcal{G}=\{G_1,\dots,G_M\}$0

and passed through a single-layer multi-head self-attention block with $\mathcal{G}=\{G_1,\dots,G_M\}$1 and $\mathcal{G}=\{G_1,\dots,G_M\}$2. The paper adds no explicit positional encoding, instead treating multi-view concatenation as an implicit structural encoding. Edge scoring is performed by concatenating the transformer outputs of two genes and applying a linear softmax classifier with cross-entropy loss (Zhao et al., 2023).

The method is motivated by specific GNN pathologies. Over-smoothing is defined as the loss of class separability under repeated neighborhood averaging, and over-squashing as compression of long-range information through narrow graph bottlenecks. MSGT-SL mitigates these by keeping the MVGNN shallow and using global self-attention over sampled subgraphs to create direct long-range communication paths. Empirically, the method outperforms GCN, GAT, DDGCN, GCATSL, KG4SL, PiLSL, and MVGCN-iSL. Under leave-gene-pair-out, MSGT-SL achieves $\mathcal{G}=\{G_1,\dots,G_M\}$3 Acc, $\mathcal{G}=\{G_1,\dots,G_M\}$4 F1, and $\mathcal{G}=\{G_1,\dots,G_M\}$5 ROC-AUC on K562, and $\mathcal{G}=\{G_1,\dots,G_M\}$6, $\mathcal{G}=\{G_1,\dots,G_M\}$7, and $\mathcal{G}=\{G_1,\dots,G_M\}$8 on Jurkat. Under leave-gene-out, the corresponding results are $\mathcal{G}=\{G_1,\dots,G_M\}$9, $M=5$0, $M=5$1 for K562 and $M=5$2, $M=5$3, $M=5$4 for Jurkat. Ablations identify the removal of MVGNN as the worst variant under leave-gene-pair-out, and show that removing the graph transformer particularly hurts leave-gene-out generalization (Zhao et al., 2023).

A recurring misconception is that MSGT-SL is merely a rebranding of an earlier “Transformer-SL.” The paper explicitly rejects that equivalence: it states that no prior SL model named “Transformer-SL” is cited in its related work and characterizes its own contribution as the first attempt to capture long-range gene dependencies using the GT framework in SL prediction. Its distinctive elements are the shallow MVGNN pre-encoder, cross-omics structure-aware random-walk sampling, concatenation across views as implicit structural encoding, and standard self-attention over sampled subgraphs (Zhao et al., 2023).

3. Transformer-SL as split learning with Vision Transformers

In distributed learning, “Transformer-SL” refers to split learning formulations in which a transformer is partitioned across clients and a server. FeSViBS is a U-shaped split-learning framework for medical image classification with $M=5$5 clients, a server-hosted ViT-B/16 body with $M=5$6 blocks, embedding dimension $M=5$7, $M=5$8 self-attention heads, and a client-side ResNet-50 head plus MLP tail. Its defining mechanism is server-side block sampling: for each client-round, the server samples a block index $M=5$9, runs the ViT body to that depth, and compresses the $-3$0 patch tokens into a single pseudo class token $-3$1 using a projection network with two convolution layers and a skip connection. Training proceeds for 200 rounds with Adam, learning rate $-3$2, batch size 32, and FedAvg of client heads and tails every 2 rounds (Almalik et al., 2023).

FeSViBS operationalizes split learning by transmitting smashed representations $-3$3 from client to server and returning $-3$4 to the client. The task loss is cross-entropy at the client tail, and no auxiliary consistency loss is imposed between the pseudo token and the original ViT class token. The method is evaluated on HAM10000, BloodMNIST, and Fed-ISIC2019. Balanced accuracy reaches $-3$5 on HAM10000, $-3$6 on BloodMNIST, and $-3$7 on Fed-ISIC2019, outperforming FeSTA, SLViT, SViBS, FedAvg, FedProx, MOON, and SCAFFOLD. The paper also reports that FeSViBS maintains performance under differential privacy noise at the client head output and does not increase client-server message sizes relative to SLViT or FeSTA because the pseudo token has the same size as the final class token (Almalik et al., 2023).

A second split-learning interpretation appears in CutMixSL for ViTs. There the client computes patch-token smashed data $-3$8, then transmits only a subset of tokens after random punching and packing into “CutSmashed data.” At the mixer level, non-overlapping token masks are assigned across grouped clients, producing feature-level CutMix on transformer tokens before server-side processing. On CIFAR-10 with ViT-Tiny, Patch CutMix achieves 75.55% top-1 accuracy versus 57.05% for parallel SL, while upload payload is reduced by 20–50% relative to parallel SL depending on the grouping and mask ratio. Privacy is evaluated by reconstruction attacks: smashed data yield MSE values of approximately 0.0091 and 0.0056, whereas CutSmashed data increase those values to approximately 0.0920 and 0.0829, and Shuffled CutMix to approximately 0.1233 and 0.1250 (Baek et al., 2022).

These two split-learning strands share neither architecture nor objective. FeSViBS emphasizes block-sampled pseudo class tokens and federated split optimization for medical imaging. CutMixSL emphasizes communication compression, privacy leakage reduction, and augmentation at the patch-token level for ViTs. Their commonality is therefore infrastructural: both use transformers under the split-learning communication pattern rather than defining a unique “Transformer-SL” model family (Almalik et al., 2023, Baek et al., 2022).

4. Transformer-SL in sign language research

Sign language is the domain in which the label most often appears as a task name rather than an optimization regime. One line of work uses transformer architectures directly for sign language translation. A spatio-temporal transformer for automatic sign language translation encodes per-frame $-3$9 ResNet-18 feature maps with 2D sinusoidal positional encoding, SAGAN-style pixel-wise 2D self-attention, and a 2D FFN built from two $0.7/0.1/0.2$0 convolutions with batch normalization. Optical flow is computed from RGB frames, the encoder is a single spatially enhanced layer, and the decoder is a standard Transformer decoder with four attention heads. The model is trained with cross-entropy for translation plus an auxiliary CTC loss on glosses, using batch size 1, 10 epochs, Adam, and an NVIDIA Titan V. It reports BLEU-4 $0.7/0.1/0.2$1 on CoL-SLTD and BLEU-4 $0.7/0.1/0.2$2 on PHOENIX14T; ablations identify 2D positional encoding as the single largest contributor, with full-model BLEU-4 $0.7/0.1/0.2$3 on CoL-SLTD and clear degradations after removing 2D positional encodings, 2D attention, or 2D FFN (Ruiz et al., 4 Feb 2025).

A second line treats continuous sign language recognition and boundary detection. The boundary-detection transformer uses OpenPose-derived 3D hand keypoints as the only input representation, with 20 normalized keypoints per hand and 120 features per frame. It trains an encoder-only Transformer on isolated sign videos equalized to 50 frames, then applies the trained model to continuous sign streams using a sliding window of 50 frames, stride 1, a threshold of 0.51 on the maximum softmax probability, and first-occurrence suppression of repeated labels. The best configuration uses 12 layers and 8 heads, achieving 99.80% isolated-sign accuracy on RKS-PERSIANSIGN and 95.80% on ASLLVD. On synthetic continuous videos, the paper reports average softmax outputs of 0.9930 on RKS-PERSIANSIGN and 0.6820 on ASLLVD with post-processing, together with 9 and 8 false recognitions respectively (Rastgoo et al., 2024).

A third line addresses translation with or without gloss supervision. “Better Sign Language Translation with STMC-Transformer” couples an STMC sign recognition front-end with a 2-layer transformer encoder-decoder using $0.7/0.1/0.2$4, $0.7/0.1/0.2$5, $0.7/0.1/0.2$6, sinusoidal positional encodings, dropout $0.7/0.1/0.2$7, and label smoothing $0.7/0.1/0.2$8. On PHOENIX G2T it reports BLEU-4 $0.7/0.1/0.2$9 for the ensemble model; on ASLG-PC12 it reports BLEU-4 $L=2$0. For video-to-text translation on PHOENIX, the STMC-Transformer ensemble reaches BLEU-4 $L=2$1, exceeding both the contemporaneous joint-transformer baseline and the paper’s own GT gloss translation result. The paper explicitly argues that this contradicts the claim that GT gloss translation acts as an upper bound for SLT performance (Yin et al., 2020).

A fourth line removes gloss supervision entirely. The stochastic transformer with LWTA layers replaces standard ReLU FFN sublayers by stochastic local winner-takes-all blocks, learns Gaussian posteriors over weights by variational inference, and performs posterior-driven compression at inference time. With embedding size 512, 8 heads, and depths $L=2$2, $L=2$3, and $L=2$4, the best single model on PHOENIX 2014T achieves test BLEU-4 $L=2$5 at depth $L=2$6, while an ensemble of 8 models reaches BLEU-4 $L=2$7. Compression reduces memory footprint by more than 70%, with average bits per weight of 8.8–9.4 and only small BLEU degradation for the $L=2$8 model (Voskou et al., 2021).

Taken together, these papers show that sign-language “Transformer-SL” refers not to a fixed architecture but to a family of video-to-text, gloss-to-text, or boundary-detection systems using transformers under very different representational assumptions: RGB plus optical flow, 3D hand keypoints, gloss sequences, or stochastic token-level encoders (Ruiz et al., 4 Feb 2025, Rastgoo et al., 2024, Yin et al., 2020, Voskou et al., 2021).

5. Other domain-specific meanings

Outside biology, split learning, and sign language, the term appears in several technically specialized settings. In distributed graph algorithms, the “Fully Adaptive Self-Stabilizing Transformer for LCL Problems” is a compiler that converts an eligible fault-free synchronous phase-based algorithm for a locally checkable labeling problem into an anonymous, size-uniform, randomized self-stabilizing algorithm. Its main guarantees are message size $L=2$9 and expected fully adaptive stabilization time

$128 \rightarrow 64$0

for edge-LCLs, with an analogous bound involving $128 \rightarrow 64$1 for node-LCLs. For classic problems such as MIS, maximal matching, and several coloring variants, the resulting algorithms achieve $128 \rightarrow 64$2 expected stabilization time under the stated eligibility conditions (Bitton et al., 2021).

In magnetic particle imaging, TranSMS realizes transformer-based super-resolution learning for system-matrix calibration. The forward model is $128 \rightarrow 64$3, low-resolution calibration corresponds to box-car downsampling $128 \rightarrow 64$4, and the learned mapping is $128 \rightarrow 64$5. The model combines a vision transformer module with convolutional token embeddings, a residual-dense CNN path, and a data-consistency projection that enforces $128 \rightarrow 64$6. It is demonstrated for $128 \rightarrow 64$7, $128 \rightarrow 64$8, and $128 \rightarrow 64$9 super-resolution, enabling up to 64-fold acceleration in two-dimensional calibration. On simulated data, TranSMS achieves nRMSE values of 0.41%, 0.58%, and 3.71% for $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$0, $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$1, and $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$2 SR, outperforming SRCNN, VDSR, 2d-SMRnet, CS, and bicubic interpolation (Güngör et al., 2021).

In accelerated MRI reconstruction, SDLFormer uses self-supervised learning by splitting acquired multi-coil $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$3-space into disjoint subsets $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$4 and $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$5, enforcing data consistency on one split and an $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$6 SSDU loss on the withheld measurements. The reconstruction network combines a 5-layer $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$7-space CNN with sparse dilated neighborhood attention, dense neighborhood attention, depth-wise convolutional locality modules, and locally enhanced feed-forward layers. Across coronal PD, coronal PDFS, and axial T2 at $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$8 and $F_i^l=\sigma(A_iF_i^{l-1}W_i^l),$9 undersampling, the paper reports average gains of around 1.40 dB PSNR and around 0.028 SSIM over other architectures, and around 1.44 dB PSNR and around 0.029 SSIM over the parallel-domain self-supervised learning baseline (S. et al., 2023).

In spoken language systems, “Transformer-SL” is a light transformer speech encoder for user-taught speech language understanding. The encoder has 4 layers, 8 heads, head dimension 64, two input $B=100$0 convolution layers for $B=100$1 temporal downsampling, local self-attention window size 5, shared parameters across layers, and a low-dimensional 6-D concatenative relative position encoding. It replaces a bidirectional GRU encoder in a multitask capsule-based SLU system and reduces parameter count from 2.36M to 1.31M. On FluentSpeech Commands it reaches 98.8% intent accuracy with the full training set, while the paper reports roughly half the original model size and training time relative to the prior recurrent system (Wang et al., 2020).

6. Recurring design patterns, misconceptions, and future directions

Across these heterogeneous meanings, several technical patterns recur. First, most methods are hybrid rather than transformer-pure. MSGT-SL prepends a shallow MVGNN before self-attention (Zhao et al., 2023); FeSViBS sandwiches a server-side ViT body between a client ResNet-50 head and MLP tail (Almalik et al., 2023); the spatio-temporal sign LLM retains a ResNet-18 spatial backbone and convolutional FFN (Ruiz et al., 4 Feb 2025); the boundary-detection system uses hand-keypoint extraction and post-processing rather than an end-to-end video transformer (Rastgoo et al., 2024); TranSMS fuses a transformer branch, residual-dense convolutions, and an explicit data-consistency projection (Güngör et al., 2021); SDLFormer inserts depth-wise convolutions into attention and FFN blocks (S. et al., 2023); and the speech-to-intent system combines a light transformer with a capsule decoder (Wang et al., 2020). A plausible implication is that, in every domain represented here, transformer components are being stabilized by domain-specific inductive biases rather than replacing them wholesale.

Second, the meaning of “SL” is a persistent source of confusion. In the biology paper it means synthetic lethality; in FeSViBS and CutMixSL it means split learning; in several vision-language papers it means sign language; in the LCL transformer it concerns self-stabilization of local algorithms; in TranSMS it denotes super-resolution learning; in SDLFormer it denotes self-supervised learning; and in the speech paper it refers to speech language understanding (Zhao et al., 2023, Almalik et al., 2023, Ruiz et al., 4 Feb 2025, Rastgoo et al., 2024, Bitton et al., 2021, Güngör et al., 2021, S. et al., 2023, Wang et al., 2020). The common misconception that “Transformer-SL” names a single recognizable architecture is therefore not supported by the literature assembled here.

Third, several papers use the term to challenge prior assumptions within their own subfields. MSGT-SL emphasizes that no prior SL model named “Transformer-SL” is present in its own related work, underscoring terminological instability in synthetic lethality prediction (Zhao et al., 2023). STMC-Transformer argues that GT gloss translation is not an upper bound for sign language translation, because predicted glosses from a strong STMC front-end can yield higher BLEU than translation from GT glosses (Yin et al., 2020). FeSViBS positions intermediate-block token distillation as an alternative to relying on the final ViT class token in split learning (Almalik et al., 2023). TranSMS and SDLFormer both argue that transformers alone are insufficient for inverse problems unless paired with measurement-aware constraints (Güngör et al., 2021, S. et al., 2023).

The future directions are equally domain-specific. MSGT-SL proposes wet-lab validation and improved handling of unseen genes (Zhao et al., 2023). FeSViBS suggests formal privacy analysis, adaptive block-sampling schedules, and efficient communication schemes (Almalik et al., 2023). The spatio-temporal sign-language transformer points to temporal self-attention, multimodal cues such as pose and facial landmarks, and regularization strategies (Ruiz et al., 4 Feb 2025). The boundary-detection transformer identifies the need for real continuous datasets and stronger CSLR metrics such as boundary precision, recall, and F1 (Rastgoo et al., 2024). The self-stabilizing transformer raises open questions about reducing the $B=100$2 factor and extending PPS to asynchronous models (Bitton et al., 2021). TranSMS proposes 3D extension and adaptation to other inverse problems (Güngör et al., 2021). SDLFormer leaves open broader validation on non-Cartesian and 3D MRI (S. et al., 2023).

In that sense, “Transformer-SL” is best understood as a historically contingent label attached to multiple transformer-centered research programs rather than as a canonical model class. Its encyclopedia value lies precisely in that heterogeneity: the same surface term indexes graph transformers for gene-pair prediction, split-learning systems for ViTs, sign-language translation and recognition architectures, self-stabilizing graph-algorithm transformers, and physics-informed reconstruction models, each with its own objective, formalism, and empirical regime.