Transformer Architecture Search

• Transformer Architecture Search is the automated discovery of optimal Transformer models using neural architecture search methods.
• It employs differentiable, reinforcement learning, evolutionary, and training-free approaches to explore vast design spaces.
• The methodology has been applied in vision, NLP, multi-task, and hardware-constrained settings to achieve state-of-the-art performance.

Transformer Architecture Search (TAS) refers to the automated discovery of optimal or efficient Transformer architectures under task-specific constraints, using neural architecture search (NAS) methodologies. TAS frameworks span differentiable, evolutionary, reinforcement learning, and training-free paradigms, and have been deployed in diverse domains including vision (classification, detection, segmentation), NLP (sequence modeling, translation), multi-task learning, remote sensing, and hardware-constrained settings. This article systematically reviews the key search spaces, search algorithms, efficient proxy indicators, and TAS systems, synthesizing the state of the art with reference to the underlying technical literature.

1. Search Space Design: Dimensions, Operators, and Parametrization

Defining a rich yet tractable search space (𝒜) is fundamental to TAS. The canonical Transformer block consists of multi-head self-attention (MSA) and MLP sublayers, but modern TAS spaces admit substantial heterogeneity:

Core parametrization:

• Depth (number of blocks/layers): Each stage or block can vary in depth (e.g., $L\in\{4,6,12,16\}$) (Chen et al., 2021, Lee et al., 13 Dec 2025).
• Embedding dimension per stage ($d$): Block-wise or stage-wise embedding/channel sizes (Chen et al., 2021, Lee et al., 13 Dec 2025, Zhang et al., 2022, Liu et al., 2021).
• MLP expansion ratio ($r$): Ratio of hidden-to-input size in feed-forward (Chen et al., 2021, Lee et al., 13 Dec 2025).
• Attention head count ($h$): Block-level or depth-wise attention head numbers (Chen et al., 2021, Lee et al., 13 Dec 2025).
• Operator choices and hybrids: Inclusion of convolutions, local or global self-attention, windowed attention, and MLP-mixers (Liu et al., 2021, Liu et al., 2022, Zhang et al., 2022).

Advanced axes:

• Attention mechanism type: Per-layer selection between softmax-attention, kernel-linear (efficient) attention, dynamic convolution, or other variants (Liu et al., 2022, Serianni et al., 2023).
• Downsampling modules: Parameterizable DSMs (local convolutional, global self-attention, local-global) to bridge transitions between blocks of different operator types (Liu et al., 2021, Liu et al., 2022).
• Multi-resolution/multi-branching: Cells and fusing nodes governing flow between multiple resolutions or branches (as in multi-scale segmentation networks) (Yu et al., 15 Mar 2024).
• Cross-task skeletons and shared/unshared routing: Multi-task settings allow the skeleton (branch-out locations for decoders) to be searched (Liu et al., 2023).

Formulation: The typical architecture is represented as a tuple of all these choices. For example, in UniNet:

$$(o_0,e_0,s_0,r_0,\ldots,o_4,e_4,s_4,r_4,DSM_0,\ldots,DSM_4)$$

with $o_i$ ∈ {DWConv, Conv3×3, SA, LSA, MLP}; expansion, scaling, and DSM selected at each of $K=5$ stages (Liu et al., 2021).

Search spaces can be enormous (e.g., |𝒜| ≈ 10¹⁶ for standard ViT-like spaces (Lee et al., 13 Dec 2025)), necessitating both algorithmic and statistical parsimony in search.

2. Search Algorithms: Differentiable, Evolutionary, RL, and Training-Free Approaches

TAS leverages multiple classes of search algorithms according to task, resource, and supervision regimes.

Differentiable NAS

• DARTS-style relaxation: Architecture weights (α) assigned to candidate operations are optimized jointly by gradient descent, followed by discretization. Memory bottlenecks for transformers (large $d$, large $|\mathscr{O}|$) have been mitigated by introducing multi-split reversible layers and reconstruction-based backpropagation, halving memory cost without sacrificing search space expressivity (Zhao et al., 2021).
• Supernet/sandwich strategies: Weights are shared across sub-architectures, with sandwich rules (always sampling smallest, largest, and random mids) stabilizing optimization (Chen et al., 2021, Lee et al., 13 Dec 2025).

Reinforcement Learning (RL)

• Controllers (often LSTMs) sample architectures; rewards incorporate hold-out accuracy and computational cost (e.g., $R = \text{Acc} - \alpha\, [\log F - \log t]^+$) (Liu et al., 2021, Liu et al., 2022).
• Policy gradient or PPO is used to update the controller; reward functions target FLOPs, latency, or hardware-aware proxies.

Evolutionary and Genetic Algorithms

• Multi-objective genetic algorithms (MOEA/D, NSGA-II): Variables encode block types, head counts, FFN widths, decoder-encoder connections. Objectives include BLEU (or mIoU) and perplexity (or latency), and evolutionary operators implement variable-length, block-aware crossover and mutation (Wang et al., 2 May 2025, Yu et al., 15 Mar 2024).
• Pareto optimization: Many frameworks explicitly search for the Pareto frontier of accuracy vs. cost (FLOPs, latency, memory) (Javaheripi et al., 2022, Yu et al., 15 Mar 2024).
• Zero-shot evolutionary search: Fitness is based on maximum-entropy or entropy-like signal (e.g., differential entropy for backbone blocks); no training is performed during search (Gu et al., 10 May 2025).

Training-Free and Proxy-Guided Search

• Zero-cost proxies: Metrics correlated with final accuracy are computed from randomly-initialized architectures. Popular proxies include synaptic diversity and saliency (DSS-indicator) for ViTs (Zhou et al., 2022), hidden covariance (Serianni et al., 2023), Jacobian covariance, GradNorm, SNIP, NASWOT, ZiCo, and their fusions (Zhou et al., 23 Jul 2024).
• Parameter-count proxy: For GPT-style LMs, decoder parameter count nearly perfectly (Spearman ρ≈0.97–0.99) predicts perplexity, enabling search to be run on target hardware without training (Javaheripi et al., 2022).
• Hybrid predictors: Some works advocate two-stage zero-cost pipelines combining architectural heuristics (e.g., parameter budget) and structural proxies, optionally ensemble-metamodels (e.g., random forests over proxy + architecture features) (Zhou et al., 23 Jul 2024).

3. Unified Hybrid Search Spaces and Modular Operators

Hybrid TAS spaces allow dynamic composition of convolutional, self-attention, and MLP modules under unified templates:

• Operator unification: All blocks instantiated as inverted bottlenecks with expansion, residual addition, and a core operator (e.g., Conv, SA, MLP) (Liu et al., 2021, Liu et al., 2022).
• Context-aware downsampling: The DSM search allows the controller to bridge between local (conv) and global (SA/MLP) operators by picking among local-DSM, global-DSM, or mixed DSM adapted to following operator context (Liu et al., 2021, Liu et al., 2022).
• Multi-resolution fusion: In segmentation and panoptic applications, the search space includes selection of branch count, memory-efficient self-attention vs. lightweight convolution blocks, and learned node-level fusion (Yu et al., 15 Mar 2024).
• Graph-structured search: Architectural DAGs (as in VTCAS or cell-based NLP spaces) support rewiring of attention and FFN placements at sub-block scale, enhancing representational diversity (Zhang et al., 2022, Serianni et al., 2023).
• Task-specific macro structures: For multi-task networks, the skeleton (branching and head-insertion points) is a discrete macro-variable, subject to Gumbel-Softmax relaxation and subsequent hard selection (Liu et al., 2023).

4. Efficient Evaluation Proxies and Training-Free TAS

Training-free TAS achieves several orders-of-magnitude reduction in search cost by substituting full or partial training with scalar indicators:

• DSS-indicator: $$S_{\text{DSS}}(\mathcal{A}) = \sum_l D^{l}_{\text{MSA}} + \sum_k S^{k}_{\text{MLP}}$$, summing nuclear-norm-based diversity of untrained MSA blocks and saliency of MLP weights, offers strong correlation with test accuracy in ViTs (Kendall's τ ≈ 0.65–0.70) and dramatically reduces total GPU-days (∼0.5 vs. 24+) (Zhou et al., 2022).
• ZiCo and ZiCo++: Gradient-based proxies combining Fisher-activation statistics and layer-decay weighting (ZiCo++) outperform single-score approaches in ranking HSI-transformers, with Spearman ρ≃0.73–0.75 (Zhou et al., 23 Jul 2024).
• Decoder-parameter proxy: Pure model parameter count suffices to select high-quality LMs under hardware constraints, exploiting monotonicity between count and PPL (Javaheripi et al., 2022).
• Statistical fusion and meta-models: Combining multiple zero-cost proxies and simple architecture features via a small metamodel (e.g., Random Forest) can further improve ranking accuracy, with ρ ≈ 0.80 using only 50 samples (Zhou et al., 23 Jul 2024).

A plausible implication is that further advances will require proxies sensitive not only to parameter scale and diversity but also to task-specific feature representations and data variance, especially in challenging modalities such as hyperspectral imaging and multi-modal learning.

5. Empirical Results and Discovered Architectural Patterns

Results from major TAS frameworks demonstrate state-of-the-art performance, with hybrid configurations consistently outperforming pure ConvNets or ViTs under equivalent resource constraints:

Model/Method	Domain	Params/FLOPs	Accuracy/Top-1	Key Discovery	Ref.
UniNet-B0 to B6	ImageNet/Det	0.56G–51G	79.1–87.4%	Early DWConv, late global SA, dynamic DSM	(Liu et al., 2021, Liu et al., 2022)
S3 (S3-T/S/B)	Vision	28M/50M/71M	82–84%	Search-space evolution, block allocation, increasing window size	(Chen et al., 2021)
VTCAS	Classification	31M/5.2G	82.0%	Conv + shifted-MSA, DAG block search	(Zhang et al., 2022)
DARTSformer	Seq2Seq	65M (base)	28.4 BLEU	Multi-split reversible memory, operation mixing (conv, SA, FFN)	(Zhao et al., 2021)
LTS	LM/Hardware	10⁵⁴ archs	ρ ≈ 0.97/0.99	Decoder param proxy, on-device search, Pareto frontier extraction	(Javaheripi et al., 2022)
GrowTAS	ViT, Transfer	5.9–54M	75.2–82.7%	Progressive small-to-large curriculum, fine-tuning of large-only params	(Lee et al., 13 Dec 2025)
AutoTaskFormer	Multi-task	22–109M	Δ_T ≈ 7–9%	Gumbel-Softmax skeleton search, task-adaptive branching	(Liu et al., 2023)
NAS-DETR (MaxEntropy)	Detection	GFLOPs ≈145	mmAP .538	Maximum entropy zero-shot backbone, entropy-based mutations	(Gu et al., 10 May 2025)

Experimental ablation confirms:

• Hybrid pipelines enable the NAS controller to assign local operations to early layers, global (SA/MLP) to deep layers, with DSM assignment closely tracking operator transitions (Liu et al., 2021, Liu et al., 2022).
• Mixed attention (Softmax/Linear) layers outperform vanilla variants for compute-constrained language and vision tasks (Liu et al., 2022).
• Progressive subspace expansion (GrowTAS) mitigates training interference between small and large subnets, with feature-space stability analysis confirming knowledge is better preserved when starting with small subnets (Lee et al., 13 Dec 2025).
• Pareto-efficient architectures—balancing accuracy, latency, and memory—are efficiently extractable by combining zero-cost proxies and hardware profiling (Javaheripi et al., 2022, Yu et al., 15 Mar 2024).

6. Applications Across Modalities and Transfer Settings

TAS has been instantiated in a variety of specialized contexts:

• Vision (ImageNet, COCO, ADE20K, ExDark, HSI): Hybrid Conv-Transformer-MLP backbones, context-aware DSMs, and Pareto search yield SOTA classification, detection, segmentation, and low-light robustness (Liu et al., 2021, Liu et al., 2022, Yu et al., 15 Mar 2024, Zhou et al., 23 Jul 2024).
• Language modeling (WikiText-103, LM1B, GLUE): Hardware-constrained search, decoder-param proxies, and highly-heterogeneous search spaces (Javaheripi et al., 2022, Serianni et al., 2023).
• Multi-task learning (Taskonomy, Cityscapes, NYUv2): Automatic discovery of per-task head placement and cross-task sharing (Liu et al., 2023).
• Spiking Vision Transformers: Architectures optimized by energy-accuracy balanced indicators for ultra-low-power platforms (Che et al., 2023).
• Remote sensing: Zero-proxy guided search tailored for hyperspectral transformer classification (Zhou et al., 23 Jul 2024).
• Autonomous and embedded systems: Search explicitly targets latency and memory using on-device profiling (Javaheripi et al., 2022, Yu et al., 15 Mar 2024).

7. Limitations, Design Guidelines, and Future Directions

Critical lessons and open challenges from TAS research include:

• Search space expresses performance trade-offs: Excessive space "homogeneity" (e.g., stacking identical attention + FFN) erodes the utility of most zero-cost proxies, with parameter count dominating (Serianni et al., 2023, Zhou et al., 23 Jul 2024). Heterogeneous, cell- or block-based search spaces with operator, depth/width, and connection diversity are essential.
• Proxies must normalize for size: Without normalization, proxies regress redundantly to parameter count; modular, seed-invariant, and task-adaptive proxies are recommended (Zhou et al., 2022, Serianni et al., 2023, Zhou et al., 23 Jul 2024).
• Curriculum and progressive expansion: Progressive training from small to large subnets reduces destructive interference, particularly when using shared weights (Lee et al., 13 Dec 2025).
• Integration of data-driven and structural indicators: Current zero-proxy methods are largely data-agnostic and may show bias toward deeper/wider models; future methods should incorporate data signals and avoid overfitting to component type (Zhou et al., 23 Jul 2024).

Architects are advised to jointly design search spaces and proxies, embed target hardware signals and task requirements, and use multi-objective or meta-learned fitness functions when possible. State-of-the-art frameworks have set benchmarks for efficiency (sub-hour or sub-GPU-day search), flexibility (hybrid operator spaces), and transferability (strong downstream/generalization performance), yet continued progress in proxy design, adaptive search, and domain-specific constraints is a frontier of ongoing research.

---

Principal references: (Liu et al., 2021, Liu et al., 2022, Lee et al., 13 Dec 2025, Chen et al., 2021, Zhou et al., 2022, Zhang et al., 2022, Zhou et al., 23 Jul 2024, Che et al., 2023, Liu et al., 2023, Liu et al., 2022, Serianni et al., 2023, Gu et al., 10 May 2025, Javaheripi et al., 2022, Zhao et al., 2021, Wang et al., 2 May 2025, Yu et al., 15 Mar 2024).