Transformers (Neural Network)

• Transformers are neural network architectures designed to process sequences, sets, spatial structures, or heterogeneous observations through attention mechanisms. They excel in applications like natural language processing, computer vision, audio analysis, robotics, and more.
• Transformers enable long-range interactions between inputs, parallelizable training, and multifaceted token representations.
• Positively impacting many areas but grappling with challenges like quadratic attention cost, high data and computational demands, reliance on specific encodings, and interpretability issues.
• Transformers were initially developed to handle limitations in recurrent sequence models by replacing sequential processing with a mechanism based on self and cross-attention.
• The efficiency of a Transformer model depends on its architecture, especially its ability to compute token interactions and process feature transformations efficiently.
• Transformers adapt to different data types through tokenization, enabling versatile applications.
• Potential future improvements in Transformer models could focus on enhancing computational efficiency, reducing data requirements, refining positional representations, integrating better multiplication, and ensuring fairness, robustness, integrity, and interpretability.

Transformers are neural-network architectures that represent sequences, sets, spatial structures, or heterogeneous observations as collections of vector-valued tokens and repeatedly apply attention-based token interaction, feature-wise nonlinear transformation, residual connections, and normalization. Their defining operation, scaled dot-product attention, computes context-dependent weighted combinations of values according to query–key compatibility. Originally introduced for sequence-to-sequence natural-language processing, Transformers are now used in language, computer vision, audio, video, scientific modeling, healthcare, robotics, optimization, graph learning, and compiler analysis. The architecture’s principal advantages are direct long-range interactions, parallelizable training, flexible tokenization, multimodal integration, and scalability; its principal limitations include quadratic attention cost, substantial data and compute requirements, dependence on positional and domain-specific representations, and unresolved issues of interpretability, robustness, fairness, and reliability.

1. Architectural foundations and historical development

Transformers were introduced to address limitations of recurrent sequence models. RNNs, including LSTMs and GRUs, process tokens sequentially, which restricts training parallelism and makes long-range dependencies pass through many recurrent transitions. This can cause optimization difficulties associated with vanishing or exploding gradients and creates a hidden-state bottleneck for long sequences. Earlier attention mechanisms allowed a decoder to consult relevant encoder states rather than relying only on a single final encoder vector. The Transformer removed recurrence and convolution from the central sequence-processing mechanism, replacing them with self-attention, cross-attention, residual connections, normalization, and position-wise feed-forward networks (Torre, 2023).

A Transformer receives token vectors $\mathbf{x}_1,\ldots,\mathbf{x}_N$, usually collected in a matrix $X$. Tokens may be words or subwords, image patches, audio segments, time-series observations, graph nodes, object instances, physical states, molecular substructures, or robot observations. A Transformer block alternates two operations:

1. Token interaction: self-attention mixes information across token positions.
2. Feature transformation: a position-wise multilayer perceptron transforms each token independently.

This decomposition gives Transformers a relationship-learning mechanism that is distinct from the local, fixed connectivity of convolution and the sequential state propagation of recurrent networks. A Transformer can also be interpreted as a fully connected, data-dependent graph neural network: tokens are nodes, attention coefficients are message weights, and the feed-forward network performs node-wise feature updating (Turner, 2023).

The original architecture is encoder–decoder. The encoder applies bidirectional self-attention to the source sequence. The decoder applies masked self-attention to previously generated target tokens, cross-attention to encoder representations, and a feed-forward network. Subsequent systems use three broad configurations:

• Encoder-only: bidirectional contextual representation, used for classification, tagging, retrieval, extraction, and representation learning.
• Decoder-only: causal autoregressive generation, used by GPT-family models and other generative systems.
• Encoder–decoder: conditional sequence-to-sequence prediction, used for translation, captioning, report generation, and related tasks.

Transformers are architectures rather than synonymous with LLMs or generative AI. An encoder-only Transformer can perform discriminative scientific classification, while a decoder-only Transformer can generate text, images, actions, or continuous-valued sequences. Generative models may also be based on CNNs, GANs, diffusion models, or other architectures (Tanoglidis et al., 2023).

2. Scaled dot-product attention

Attention transforms queries, keys, and values through learned linear projections:

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$

A query represents the information sought by a token, a key represents the properties offered by a candidate source token, and a value contains the information that will be aggregated. The standard attention operation is

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$

where $d_k$ is the key dimension. The matrix $QK^\top$ contains pairwise query–key similarities. The softmax is applied row-wise, so the attention weights associated with each query are nonnegative and sum to one. For query position $i$,

$$\mathbf{z}_i=\sum_{j=1}^{N}\alpha_{ij}\mathbf{v}_j.$$

The output is therefore a similarity-weighted aggregation of value vectors. The scale factor $1/\sqrt{d_k}$ prevents dot products from becoming excessively large as the key dimension increases. Without scaling, the softmax can become sharply concentrated, producing small gradients and unstable optimization (Turner, 2023).

Self-attention and cross-attention

In self-attention, queries, keys, and values are derived from the same sequence:

$$\operatorname{SA}(X) = \operatorname{Attention}(XW_Q,XW_K,XW_V).$$

Every token can therefore incorporate information from every other token in the same input. In text, this enables long-distance syntactic and semantic interactions. In images, one patch can directly interact with a distant patch. In time series, a measurement can use information from observations separated by long intervals.

Cross-attention uses different sources for queries and for keys and values:

$X$0

It is used when one representation queries another, such as a decoder attending to an encoded source sentence, a text representation attending to image regions, or a robot situation decoder attending to a demonstration-derived preference representation.

Multi-head attention

Multi-head attention computes several attention operations in parallel:

$X$1

The head outputs are concatenated and projected:

$X$2

Separate heads can learn different relational subspaces, including local texture, long-range correspondence, syntactic dependency, temporal association, anatomical correspondence, or cross-modal alignment. Attention heads should not automatically be interpreted as semantically disentangled components; their functional specialization is task- and model-dependent.

Residual connections, normalization, and feed-forward networks

A typical pre-normalized block can be written schematically as

$X$3

$X$4

The feed-forward network is applied independently at each token:

$X$5

where $X$6 may be ReLU or GELU. Attention mixes tokens; the feed-forward network mixes features within each token. Residual connections facilitate information and gradient flow, while layer normalization normalizes the feature dimensions of each token rather than statistics across a minibatch (Turner, 2023).

3. Position, masking, and sequence computation

Self-attention without positional information is permutation-equivariant. If the input tokens are permuted by a permutation matrix $X$7, the output is permuted correspondingly:

$X$8

This property is useful when the input is genuinely an unordered set, but it is inappropriate when order, spatial arrangement, temporal position, or graph direction carries meaning. Position information is therefore added to token embeddings or incorporated into attention scores.

The original Transformer used sinusoidal positional encodings:

$X$9

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$0

Other systems use learned absolute embeddings, relative position biases, rotary or coordinate-based encodings, two-dimensional spatial encodings, temporal embeddings, or domain-specific representations. For irregular scientific time series, the actual observation time may be more informative than an integer token index. For directed graphs, direction-aware positional representations can preserve orientation that ordinary Laplacian encodings lose (Geisler et al., 2023).

Causal masking

Autoregressive prediction requires a token to depend only on earlier tokens. A causal mask sets forbidden logits to $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$1 before softmax:

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$2

During training, the entire target sequence can still be processed in parallel because the mask prevents future information leakage. This is commonly combined with teacher forcing: the model receives the correct preceding tokens and predicts all next-token distributions simultaneously.

During autoregressive inference, the dependency remains sequential. A token is generated, appended to the context, and used to predict the next token. Key–value caching avoids recomputing all previous projections, but standard causal attention still stores a sequence-length-dependent state and compares each new query with a growing set of keys.

Encoder–decoder computation

For a source sequence $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$3 and a shifted target sequence $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$4, an encoder–decoder Transformer performs:

1. tokenization and embedding;
2. addition of positional information;
3. encoder self-attention;
4. masked decoder self-attention;
5. decoder cross-attention to encoder representations;
6. position-wise feed-forward transformation;
7. output projection and normalization.

For discrete targets, autoregressive likelihood is commonly factorized as

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$5

Encoder-only models instead use bidirectional self-attention, frequently with masked-language-modeling or related self-supervised objectives. Decoder-only models use causal self-attention and next-token prediction.

4. Computational complexity and efficient variants

For $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$6 tokens and model dimension $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$7, dense self-attention forms an $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$8 interaction matrix. Its dominant time complexity is approximately

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$9

and its attention-matrix memory complexity is

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$0

The quadratic dependence affects long documents, high-resolution images, volumetric medical data, long genomic sequences, videos, dense scientific fields, and long EHR histories. The position-wise feed-forward network has approximate cost $$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$1 and can dominate when the model dimension is large relative to sequence length (Nerella et al., 2023).

Several strategies modify this cost:

• Local or windowed attention restricts interactions to neighborhoods.
• Shifted windows allow communication across local partitions over successive layers.
• Sparse or deformable attention selects a limited set of positions.
• Axial attention factorizes spatial interaction along separate axes.
• Hierarchical token merging progressively reduces token count.
• Latent bottlenecks compress large inputs into a smaller latent sequence.
• Low-rank and kernelized attention reorder computations to avoid explicit $$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$2 matrices.
• Token pruning and pooling reduce the sequence dynamically.
• Hybrid CNN–Transformer architectures use convolution for local structure and attention for global interaction.

The linear-attention formulation in “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention” replaces the softmax kernel with a feature-map factorization. If

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$3

then the attention computation can be reordered from

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$4

to

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$5

The sequence-dependent intermediate is $$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$6, whose dimensions do not depend on $$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$7. The resulting computation has linear sequence-length cost under fixed feature dimensions and admits a recurrent state update for causal inference (Katharopoulos et al., 2020). Its limitation is that finite-dimensional feature maps generally do not reproduce the original softmax kernel exactly. The model therefore changes the attention function and may lose information that explicit token-by-token softmax attention would preserve.

The distinction between training and inference is central. Linear attention can be parallelized during training using cumulative sums or scan operations, while autoregressive inference remains sequential at the outer generation loop. Its advantage is a fixed-size state and constant computation per generated token for the attention component, not elimination of autoregressive dependence.

5. Representations across modalities and domains

The generality of Transformers follows from tokenization: a domain-specific input is converted into a sequence or set of vectors, after which a common attention mechanism can be applied.

Language

LLMs tokenize words, subwords, or characters. BPE, Unigram, and WordPiece are subword tokenization methods. Encoder-only systems such as BERT produce bidirectional contextual representations. Decoder-only systems such as GPT models predict tokens autoregressively. Encoder–decoder systems perform conditional sequence transformation, including translation, summarization, and question answering.

Large-scale pretraining commonly uses masked language modeling, next-token prediction, denoising, or related self-supervised objectives. Domain-specific biomedical models, including BioBERT, ClinicalBERT, PubMedBERT, and GatorTron, adapt pretraining corpora to biomedical literature, clinical notes, and EHR data (Nerella et al., 2023).

Vision

Vision Transformers represent images as patch tokens. An image of height $$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$8 and width $$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$9, divided into $d_k$0 patches, produces

$d_k$1

tokens. Each flattened patch is projected into the model dimension, positional information is added, and an encoder processes the sequence. A class token or pooled representation can be used for classification.

Pure ViTs provide global interactions and scale effectively with model size and data, but have weaker visual inductive biases than CNNs. CNNs encode locality, translation equivariance, weight sharing, and multiscale receptive fields. Consequently, ViTs often require large pretraining datasets, while CNNs can be more data-efficient on smaller datasets. DeiT uses distillation from a CNN teacher; ConViT introduces a soft convolutional bias; CCT uses convolutional patch extraction; and Swin Transformer uses shifted local windows (Khan et al., 2021).

Transformers are used for image classification, object detection, semantic and instance segmentation, image generation, super-resolution, denoising, colorization, and visual grounding. DETR formulates detection as set prediction using object queries and bipartite matching, eliminating region proposals and nonmaximum suppression. Deformable DETR attends to sparse multiscale feature locations to improve convergence and small-object behavior.

Video, audio, and multimodal data

Video tokens may represent spatial patches, temporal segments, or spatiotemporal volumes. Full space–time attention is expensive because token count grows with both spatial and temporal resolution. TimeSformer factorizes spatial and temporal attention; ViViT uses factorized spatial-temporal encoders; and Perceiver-style models use latent bottlenecks.

Audio systems tokenize windows, spectrogram regions, or learned acoustic features. Whisper uses an encoder–decoder architecture for speech transcription. Wav2Vec 2.0 combines convolutional feature extraction, Transformer temporal modeling, and masking. Transformers have also been used for speech recognition, source separation, physiological signals, EEG, ECG, and activity recognition.

Multimodal systems may concatenate tokens from multiple modalities, process each modality separately before fusion, or use cross-attention. Vision-LLMs such as ViLBERT use separate visual and linguistic streams with co-attention. CLIP jointly trains image and text encoders using contrastive learning. Flamingo uses a vision encoder, a Perceiver resampler, and a LLM conditioned through cross-attention.

Scientific and physical systems

Physical trajectories can be represented as sequences of embedded states. “Transformers for Modeling Physical Systems” uses a learned Koopman-based embedding to map physical states into a lower-dimensional vector space, freezes the embedding network, and trains an autoregressive Transformer over the resulting sequence. The approach was evaluated on the Lorenz system, two-dimensional cylinder flow, and the three-dimensional Gray–Scott reaction-diffusion system. Its results suggest that self-attention can model long-range and multiscale temporal structure, but the approach remains dependent on embedding quality, training data, finite context, and autoregressive error (Geneva et al., 2020).

Scientific Transformers have also been applied to astronomical light curves, images, spectra, and multimodal observations. Astronomical time series require explicit handling of irregular sampling, measurement uncertainty, wavelength or band identity, and missing data. A generic integer position encoding is not sufficient when observation times are irregular (Tanoglidis et al., 2023).

Graphs and directed structures

A standard Transformer applied to graph nodes requires structural information because attention alone does not know adjacency, reachability, graph distance, or direction. “Transformers Meet Directed Graphs” introduces Magnetic-Laplacian eigenvectors and directional random-walk encodings. Magnetic-Laplacian positional encodings preserve direction through complex phases while retaining a Hermitian operator with real eigenvalues and orthogonal complex eigenvectors. Directional random walks encode forward and reverse landing probabilities, reachability, and longer-range connectivity (Geisler et al., 2023).

These encodings are useful for sorting-network correctness and source-code understanding. A data-flow-centric graph construction represents dependencies, control flow, calls, AST relations, and operand order rather than relying solely on textual statement order. The resulting model improved Code2 function-name prediction relative to prior systems in the reported experiments. Limitations include eigendecomposition cost, eigenvector ambiguities, graph-construction errors, and the continued quadratic cost of full attention.

Healthcare

Healthcare Transformers process medical images, clinical text, structured EHRs, social-media data, physiological signals, genomic and protein sequences, and molecular representations. Applications include diagnosis, prognosis, clinical information extraction, report generation, image reconstruction, risk prediction, drug–drug interaction modeling, retrosynthesis, protein modeling, and multimodal patient representation (Nerella et al., 2023).

Their benefits include long-range dependency modeling, self-supervised pretraining on unlabeled medical data, multimodal fusion, and flexible adaptation to variable-length sequences. Their limitations are especially consequential in healthcare: domain shift across institutions, limited labeled datasets, privacy constraints, hallucinated reports, calibration problems, demographic bias, interpretability limitations, and the absence of prospective evidence for autonomous clinical use.

Robotics, optimization, and planning

Transformers can represent object-centric states, actions, demonstrations, and preferences. The Transformer Task Planner encodes a human demonstration of dishwasher-loading behavior as a preference prompt. A situation decoder cross-attends to this fixed-length prompt representation and selects object and placement instances through dot-product matching. The system generalized to unseen preferences from a single demonstration in simulation and was demonstrated with a Franka Panda arm, while depending on perception, predefined placement candidates, and separate low-level controllers (Jain et al., 2022).

Transformers have also been used as learned decision layers for mixed-integer programs. In the CLSP setting, a sequence-to-sequence Transformer predicts binary setup variables; these variables are fixed, and the remaining continuous production and inventory problem is solved as a linear program. The reported benchmark achieved zero observed infeasibility and zero observed optimality gap after post-processing, but these are empirical in-distribution results rather than worst-case approximation guarantees (Cooper et al., 2024).

6. Probabilistic, theoretical, and alternative formulations

Probabilistic interpretation of attention

“Probabilistic Transformers” interprets the core attention operation as inference in a Gaussian mixture model. Under the model, keys are Gaussian means in query space, values are Gaussian means in value space, attention weights are posterior responsibilities, and the weighted value sum is a maximum-posterior or EM-style estimate. Under tied precisions, special mixture priors, and the limit $d_k$2, the posterior weights reduce to

$d_k$3

which has the form of softmax dot-product attention (Movellan et al., 2020).

The result applies directly to the query–key–value attention operation. It does not establish that a complete modern Transformer—including residual connections, layer normalization, positional embeddings, multiple heads, feed-forward networks, and masking—is exactly a Gaussian-mixture inference system. The probabilistic model suggests extensions involving non-isotropic covariance, heavy-tailed distributions, sparse distributions, multilayer mixtures, and inference-time adaptation, but these are proposed generalizations rather than properties of ordinary attention.

Formal-language and circuit complexity

The computational power of Transformers depends strongly on attention type, numerical representation, positional encoding, and depth. “Saturated Transformers are Constant-Depth Threshold Circuits” distinguishes:

• Hard attention, which selects one maximizing position.
• Saturated attention, which averages all positions tied for the maximum.
• Soft attention, which assigns continuously varying weights through softmax.

Finite-precision saturated Transformers can recognize majority and therefore exceed the $d_k$4 upper bound associated with hard attention. The paper establishes

$d_k$5

where $d_k$6 denotes the paper’s finite-precision float model. Under arbitrary exact rational representations and highly permissive embeddings and activation functions, saturated Transformers can recognize every formal language, illustrating the importance of finite precision and representation-size assumptions (Merrill et al., 2021).

“Transformers as Transducers” studies sequence-to-sequence functions rather than language recognition. It introduces B-RASP, B-RASP[pos], and S-RASP, progressively adding Boolean selection, positional arithmetic, and prefix sums. B-RASP corresponds to first-order rational transductions for the functions considered; B-RASP[pos] contains first-order regular functions and can perform positional operations such as copying the first half of a string; S-RASP contains first-order polyregular functions and can perform prefix-sum-based computations. Masked average-hard-attention Transformers can simulate S-RASP under structured positional encodings (Strobl et al., 2024).

These results are formal expressivity theorems. They rely on exact hard or average-hard attention, carefully designed positional encodings, finite symbolic representations, and constructive weights. They do not establish that ordinary soft-attention Transformers will learn the corresponding algorithms or generalize them reliably.

Transformers as compiler systems

“Transformers are Efficient Compilers, Provably” constructs encoder-only Transformers for AST construction, symbol resolution, and simplified type analysis in Mini-Husky, a C-like language. Under bounded AST depth $d_k$7 and bounded type-inference depth $d_k$8, the constructions use architectural resources scaling as

$d_k$9

for AST construction and symbol resolution, and

$QK^\top$0

for type analysis, where $QK^\top$1 is the input sequence length (Zhai et al., 2024).

The Cybertron domain-specific language expresses compiler algorithms as typed compositions of local feed-forward operations and global attention primitives such as nearest-left retrieval, indexed lookup, scope propagation, and filtered attention. The paper contrasts these constructions with an RNN lower bound of $QK^\top$2 state-space size for the considered type-analysis task under suitable discretization assumptions.

The results depend on bounded nesting, bounded inference depth, idealized hard attention, simplified positional representations, exact discrete encodings, and a restricted language. They establish existence of Transformer parameters, not successful optimization or replacement of industrial compilers.

Quantum Transformers

Quantum Vision Transformers replace selected classical operations with Hamming-weight-preserving quantum circuits and quantum orthogonal layers. The proposed models use unary vector encodings, matrix loading, RBS gates, orthogonal transformations, quantum inner-product estimation, and compound-matrix operations. The Compound Transformer acts on a Hamming-weight-two subspace and provides a theoretical asymptotic advantage for a particular attention operation under the paper’s model (Cherrat et al., 2022).

Experiments on small MedMNIST datasets produced competitive results relative to classical ViT baselines and used fewer attention-layer parameters. Small experiments were also run on IBM superconducting quantum processors, with meaningful results reported for four- to six-qubit configurations. The work does not demonstrate an end-to-end wall-clock quantum speedup. Unary encoding, matrix loading, tomography, sampling, post-selection, circuit noise, and limited qubit counts remain substantial constraints.

7. Applications, limitations, and research directions

Transformers have become a general-purpose architecture for tasks in which context-dependent relationships among elements are important. Their applications include language understanding and generation, image classification and detection, video recognition, multimodal alignment, scientific surrogate modeling, healthcare prediction, robotic planning, graph reasoning, compiler analysis, combinatorial optimization, and quantum machine learning.

The main advantages are:

• Direct long-range interaction: distant tokens can communicate in one attention layer.
• Parallel training: non-autoregressive token computation is possible during training.
• Flexible tokenization: words, patches, events, graph nodes, physical states, and actions can use a common architecture.
• Multimodal fusion: cross-attention and shared token spaces connect heterogeneous data.
• Scalability: model capacity and pretraining data can be increased systematically.
• Representation transfer: self-supervised pretraining can produce reusable contextual features.
• Adaptability: masks, positional representations, tokenization, and attention sparsity can encode domain structure.

The principal limitations are:

• Quadratic dense attention: time and memory grow with the square of token count.
• Data hunger: weak built-in inductive biases can require large datasets.
• Autoregressive latency: causal generation remains sequential even when attention is efficient.
• Representation dependence: tokenization, positional information, and embeddings strongly affect performance.
• Long-horizon error: autoregressive physical and robotic models accumulate prediction errors.
• Interpretability uncertainty: attention maps are not automatically faithful explanations.
• Distribution shift: performance can degrade across institutions, sensors, users, object counts, graph constructions, or physical parameters.
• Privacy, fairness, and safety risks: particularly acute in healthcare and other high-stakes settings.
• Environmental cost: large-scale pretraining and inference require substantial compute and energy.
• Theoretical–empirical gap: formal expressivity results often use idealized attention, exact arithmetic, special positional encodings, or existential constructions that do not describe standard trained models.

Current research directions include efficient and sparse attention, linear and kernelized alternatives, hierarchical and latent-bottleneck architectures, token pruning and merging, better positional representations, hybrid CNN–Transformer models, structured graph and physical encodings, self-supervised learning with limited labels, parameter-efficient adaptation, uncertainty estimation, mechanistic interpretability, privacy-preserving training, fairness auditing, and hardware-aware deployment.

The central design problem is not simply increasing Transformer size. It is balancing global, content-dependent interaction against computational cost, inductive bias, data availability, representation fidelity, reliability, and domain-specific constraints. Transformers provide a common computational abstraction for many modalities, but their effectiveness depends on whether the token representation and attention structure preserve the information that the task requires.