Transformer-Based Models (TBMs)

• Transformer-Based Models (TBMs) are neural architectures that use self-attention and feed-forward networks to aggregate token information for diverse tasks.
• They leverage large-scale pretraining with objectives like masked language modeling and autoregressive prediction, followed by task-specific fine-tuning for improved performance.
• TBMs incorporate advanced positional encoding, memory mechanisms, and system-level optimizations to overcome quadratic attention costs and enhance training efficiency.

Searching arXiv for recent transformer-based model references to ground the article. Transformer-Based Models (TBMs) are neural architectures built on the Transformer paradigm, in which self-attention aggregates information across tokens and position-wise feed-forward networks refine per-token features; in practice, this design has been instantiated as encoder-only, decoder-only, and encoder–decoder systems for language, vision, spatio-temporal modeling, biomedical NLP, reinforcement learning, and other domains (Turner, 2023, Torre, 2023). Across the literature, TBMs are characterized by content-dependent token mixing, large-scale pretraining followed by task-specific fine-tuning, and a flexible tokenization interface that can operate on words, subwords, image patches, spectrogram segments, structured clinical codes, or symbolic expressions (Kalyan et al., 2021, Melis et al., 15 Jan 2026).

1. Core architecture and mathematical structure

A standard Transformer stack repeatedly applies two operations: token mixing through self-attention and feature mixing through a position-wise multilayer perceptron. In conventional notation, with input sequence $X \in \mathbb{R}^{n \times d_{\text{model}}}$, one attention head forms

$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$

then computes

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$

and multi-head attention concatenates per-head outputs before a learned output projection (Turner, 2023). The same scaled dot-product form appears across application papers, including headline generation, clinical relation extraction, cognitive radio modulation synthesis, GPU training systems, and long-context representation analysis (Bukhtiyarov et al., 2020, Yang et al., 2021, Melis et al., 15 Jan 2026, Wang et al., 2021, Hoang et al., 8 Oct 2025).

Within a block, residual connections and layer normalization stabilize optimization. A common pre-norm formulation is

$$Y = X + \mathrm{MHA}(\mathrm{LayerNorm}(X)), \qquad Z = Y + \mathrm{FFN}(\mathrm{LayerNorm}(Y)),$$

where $\mathrm{FFN}(x)=W_2 \sigma(W_1 x+b_1)+b_2$ (Turner, 2023). The survey in Spanish likewise presents encoder and decoder stacks as compositions of multi-head self-attention, residual pathways, normalization, and position-wise MLPs, and emphasizes that these modules replace recurrence with direct interactions between distant sequence elements (Torre, 2023).

Two structural consequences recur across the literature. First, self-attention provides a full receptive field with content-dependent weights, which differentiates TBMs from CNNs and RNNs (Turner, 2023). Second, standard full attention has quadratic sequence-length scaling in time and memory, $O(n^2 d)$ and $O(n^2)$ for the attention map, which is a persistent systems and modeling bottleneck for long contexts (Hoang et al., 8 Oct 2025, Fang et al., 2020).

2. Positional information, memory, and representation flow

Because self-attention is permutation-equivariant without positional information, TBMs require explicit mechanisms to encode order or temporal layout (Turner, 2023). Absolute sinusoidal encodings remain canonical:

$$PE(pos,2i)=\sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE(pos,2i+1)=\cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right),$$

while learned absolute embeddings, relative biases, rotary schemes, ALiBi-style linear biases, and hybrid position-content formulations are all used in practice (Turner, 2023, Irani et al., 17 Feb 2025). For time series, positional encoding is not merely an ordering tag: the survey on time-series TBMs frames it as a carrier of periodicity, trend, seasonality, multiscale dependencies, and irregular sampling structure, and reports that advanced methods such as SPE and TUPE outperform fixed sinusoidal baselines on long and high-dimensional classification tasks (Irani et al., 17 Feb 2025).

Several papers extend positional structure into explicit memory mechanisms. “Memory Transformer” prepends trainable memory tokens to the sequence, studies a dedicated memory controller, and introduces a memory bottleneck in which token-to-token global communication is routed through memory, reducing encoder complexity from $O(N^2)$ to $O(NM)$ for fixed memory size $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$0 (Burtsev et al., 2020). The reported attention maps suggest interpretable write, process, copy, and read operations over the memory bank, rather than leaving all local and global information in the same token states (Burtsev et al., 2020).

Recent work on long-context representation dynamics adds a more critical perspective. A unified comparison of TBMs and state-space models reports that GPT-Neo-2.7B and Pythia-2.8B rapidly homogenize token representations early in depth, with high inter-token cosine similarity across most layers and late-layer re-diversification near the output (Hoang et al., 8 Oct 2025). The paper attributes oversmoothing in TBMs primarily to architectural mixing through attention and MLP blocks rather than to training alone, and shows that the phenomenon persists even at random initialization (Hoang et al., 8 Oct 2025). This suggests that positional encoding and memory design are not only sequence-order devices but also regulators of how distinct token identities survive through depth.

3. Pretraining and task objectives

A defining property of modern TBMs is the separation between large-scale self-supervised pretraining and downstream supervised fine-tuning (Yang et al., 2021, Kalyan et al., 2021). In encoder-only biomedical and clinical models, masked language modeling is the dominant pretraining objective:

$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$1

with variants such as whole-word masking, whole-entity masking, and replaced token detection in ELECTRA-style models (Kalyan et al., 2021). In XLNet, permutation language modeling replaces masking, while BERT originally combines MLM with next-sentence prediction (Yang et al., 2021).

Decoder-only TBMs use autoregressive next-token prediction,

$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$2

which is also the objective used when GPT-2 is fine-tuned on symbolic modulation formulas in cognitive radio (Melis et al., 15 Jan 2026). Encoder–decoder TBMs for generation use teacher forcing with sequence cross-entropy,

$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$3

as in Russian news headline generation with mBART and BertSumAbs (Bukhtiyarov et al., 2020). mBART adds denoising pretraining with sentence order shuffling, document rotation, and text infilling, while BertSumAbs combines a pretrained RuBERT encoder with a randomly initialized Transformer decoder and therefore requires separate optimizers to mitigate encoder–decoder mismatch during fine-tuning (Bukhtiyarov et al., 2020).

Domain adaptation strategies vary systematically. The biomedical PLM survey distinguishes mixed-domain continual pretraining, simultaneous pretraining, domain-specific pretraining from scratch, and task-adaptive pretraining (Kalyan et al., 2021). Clinical relation extraction provides a concrete downstream case: BERT, RoBERTa, and XLNet variants are fine-tuned with explicit entity markers, binary versus multi-class classification heads, and candidate-relation pruning by semantic type and cross-sentence distance; the study reports that binary classification consistently outperforms multi-class classification on MADE1.0 and n2c2 (Yang et al., 2021).

Not all TBM training objectives are token reconstruction objectives. In model-based reinforcement learning, TWISTER argues that next-state prediction alone underuses masked self-attention, because adjacent Atari frames are too similar in latent space; it adds action-conditioned Contrastive Predictive Coding over longer horizons and achieves a human-normalized mean score of $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$4 on Atari 100k among methods without look-ahead search (Burchi et al., 6 Mar 2025). This broadens the role of TBMs from sequence likelihood estimators to latent temporal predictors optimized for downstream control.

4. Systems, compression, and training efficiency

The systems literature treats TBMs as a distinctive workload: large GEMMs are interleaved with many small reduction-heavy kernels, variable sequence lengths induce padding and memory fragmentation, and training stores substantial activations for backward passes (Wang et al., 2021, Fang et al., 2020, Andoorveedu et al., 2022). LightSeq2 addresses this by fusing non-GEMM kernels, rewriting LayerNorm and softmax reductions for GPU efficiency, batching optimizer updates through symbolic tensor linking, and reusing temporary buffers with a dangling-tensor-aware memory manager (Wang et al., 2021). Across Transformer, BERT, GPT-2, and ViT workloads, it reports end-to-end speedups of $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$5–$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$6, with “308% training speedup” on WMT14 English–German interpreted by the paper as about $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$7 faster throughput (Wang et al., 2021).

Tempo focuses on activation memory during training. It introduces in-place GELU, in-place LayerNorm, and attention-layer changes that store only the softmax output and dropout mask, thereby reducing memory enough to enable up to $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$8 higher batch sizes and speedups of $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,$9 on BERT Large pre-training, $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$0 on GPT-2, and $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$1 on RoBERTa relative to the baseline settings studied (Andoorveedu et al., 2022). The method is notable because it is presented as a drop-in replacement for standard Transformer layers rather than an architectural redesign (Andoorveedu et al., 2022).

For inference serving, TurboTransformers couples optimized GPU reduction kernels with a sequence-length-aware memory allocator and a dynamic-programming batch scheduler for variable-length requests (Fang et al., 2020). On BERT with variable-length workloads, it reports an intermediate-tensor peak footprint of $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$2 MB and end-to-end peak GPU memory of $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$3 MB, compared with $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$4 MB for PyTorch and $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$5 MB for ONNX Runtime; throughput reaches $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$6 responses/sec on short variable-length workloads with DP batching on RTX 2060 (Fang et al., 2020).

Model compression is another systems axis. Tensor Train Matrix parameterization replaces dense GPT-2 fully connected layers with tensorized cores, reducing parameters by up to about $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$7 while keeping perplexity near the dense baseline at moderate ranks; in GPT-2 medium, the TTM-72 model uses $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$8 of baseline parameters and substantially outperforms similarly sized truncated-SVD and distilled baselines on out-of-domain language modeling (Chekalina et al., 2023). A separate post-training scheme, BBCT, is described as block-wise bit-compression of embeddings, matrix multiplication, GELU, softmax, layer normalization, and intermediate results, with the arXiv entry noting “less than 1% accuracy drop in most tasks” on GLUE for an efficient BERT case (Dong et al., 2023).

5. Domain-specific instantiations

TBMs now appear across a heterogeneous set of scientific and engineering problems. The common architectural substrate is stable, but tokenization, conditioning, and evaluation metrics vary sharply by domain.

Domain	TBM formulation	Representative result
News summarization	mBART, BertSumAbs encoder–decoder generation	BertSumAbs reaches R-mean 39.0 on RIA (Bukhtiyarov et al., 2020)
Clinical NLP	BERT, RoBERTa, XLNet relation extraction	XLNet-clinical reaches F1 0.9610 on n2c2 (Yang et al., 2021)
Biomedical NLP	Domain-pretrained encoder-only PLMs	BioELECTRA reports BLURB 82.60 (Kalyan et al., 2021)
Medical imaging	UNETR, SwinUNETR, UNETR++	UNETR++ is strongest TBM on RATIC, average DSC 0.934 (Bayer et al., 19 Mar 2026)
Cognitive radio	GPT-2 symbolic modulation synthesis	3 of 20 generated formulas are syntactically valid (Melis et al., 15 Jan 2026)
Topology optimization	ViT with class-token conditioning	ViT-Small reaches 1.88% static compliance error (Lutheran et al., 6 Sep 2025)
Model-based RL	Transformer world model with AC-CPC	TWISTER reaches 162% Atari 100k mean score (Burchi et al., 6 Mar 2025)

In NLP, pretrained TBMs advance extreme abstractive summarization: on Russian news, BertSumAbs improves R-mean by $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$9 over PBATrans on RIA and by $$Y = X + \mathrm{MHA}(\mathrm{LayerNorm}(X)), \qquad Z = Y + \mathrm{FFN}(\mathrm{LayerNorm}(Y)),$$0 over CopyNet on Lenta, while human evaluation finds BertSumAbs headlines better than human-written ones in $$Y = X + \mathrm{MHA}(\mathrm{LayerNorm}(X)), \qquad Z = Y + \mathrm{FFN}(\mathrm{LayerNorm}(Y)),$$1 of paired cases (Bukhtiyarov et al., 2020). In biomedical language processing, domain-specific corpora and vocabularies matter: the survey contrasts BioBERT, PubMedBERT, BlueBERT, ClinicalBERT, BioALBERT, BioELECTRA, SciBERT, SciFive, and code-centric EHR models such as BEHRT and MedBERT, and links improvements to domain-specific pretraining and knowledge integration from resources such as UMLS (Kalyan et al., 2021).

Outside text, hybrid TBMs operate on image patches or volumetric blocks. On abdominal multi-organ segmentation over the heterogeneous RATIC dataset, UNETR, SwinUNETR, and UNETR++ are benchmarked against SegResNet under identical training conditions; UNETR++ is the most competitive TBM with average DSC 0.934, but the CNN baseline achieves the highest average DSC at 0.945 and is best for each organ (Bayer et al., 19 Mar 2026). In topology optimization, a ViT tokenizes a $$Y = X + \mathrm{MHA}(\mathrm{LayerNorm}(X)), \qquad Z = Y + \mathrm{FFN}(\mathrm{LayerNorm}(Y)),$$2 field of strain energy density and von Mises stress into 64 tokens, conditions on supports, loads, and target volume fraction through a class token, and produces iteration-free designs with large speedups relative to SIMP-based optimization (Lutheran et al., 6 Sep 2025).

In symbolic engineering tasks, TBMs are used as structured sequence generators rather than LLMs in the narrow sense. GPT-2 fine-tuned on mathematical modulation expressions produces candidate waveforms that are syntax-checked and simulated under AWGN and fading, although the low validity rate and physically questionable outputs such as singularities in one candidate highlight the gap between syntactic plausibility and deployable designs (Melis et al., 15 Jan 2026). In reinforcement learning, the world-model formulation makes TBMs latent dynamics estimators operating over state–action tokens rather than text tokens (Burchi et al., 6 Mar 2025).

6. Limitations, controversies, and research directions

Several limitations recur across the literature. The most fundamental is the quadratic cost of full self-attention, which stresses both training and serving systems and motivates fused kernels, memory managers, activation rematerialization, efficient positional schemes, and alternative sequence models (Wang et al., 2021, Fang et al., 2020, Hoang et al., 8 Oct 2025). A second limitation is that TBMs do not dominate every data regime: on RATIC, a well-configured CNN baseline outperforms all three hybrid transformer-based segmentation models, suggesting that strong locality priors and data efficiency can still be decisive on small- to medium-sized heterogeneous datasets (Bayer et al., 19 Mar 2026).

Generalization under domain or temporal shift is another unresolved issue. In headline generation, RIA lacks strictly time-based splits, and both mBART and BertSumAbs degrade on 2020 articles because of unseen entities; language-specific pretraining with RuBERT is advantageous for Russian, but multilingual pretraining may offer slightly better robustness to entity drift (Bukhtiyarov et al., 2020). Clinical relation extraction shows a different manifestation: expanding cross-sentence distance sharply increases negative candidate pairs, destabilizing training and harming precision unless schema-aware pruning or distance-specific models are used (Yang et al., 2021).

A more conceptual controversy concerns what TBMs learn internally. The long-context analysis of TBMs versus state-space models argues that oversmoothing in TBMs is architectural, not merely optimization-induced, and recommends intermediate-layer readouts, diversity-preserving regularizers, and hybrid TBM–SSM stacks that preserve token uniqueness early and apply global restructuring later (Hoang et al., 8 Oct 2025). This suggests that improvements may require not only larger models or more data, but changes to how information is mixed across depth.

Application-specific papers expose additional constraints. Transformer-generated modulation formulas have low validity rates and incomplete physical constraints during decoding (Melis et al., 15 Jan 2026). Dynamic topology optimization still shows high floating-material rates despite auxiliary losses (Lutheran et al., 6 Sep 2025). Medical and biomedical TBMs face privacy, fairness, and domain adaptation challenges, including vocabulary fragmentation, limited labeled data, and the need for larger clinical pretraining corpora or continual pretraining on target institutions (Kalyan et al., 2021).

The emerging direction across these studies is not a single universal remedy but a layered program: stronger domain-specific pretraining where corpora permit it, tighter inductive bias through positional or memory design, systems work that reduces activation and serving overhead, and objective functions that better align TBM capacity with the structure of the target problem (Bukhtiyarov et al., 2020, Andoorveedu et al., 2022, Burchi et al., 6 Mar 2025). A plausible implication is that the future of TBMs lies less in treating the Transformer as a fixed blueprint than in treating self-attention, positional structure, memory, and optimization as composable design axes whose best configuration is domain-dependent.