--- title: Transformer Architecture Search url: https://www.emergentmind.com/topics/transformer-architecture-search-tas type: topic --- # Transformer Architecture Search Transformer Architecture Search (TAS) refers to the automated discovery of optimal or efficient Transformer architectures under task-specific constraints, using neural architecture search (NAS) methodologies. TAS frameworks span differentiable, evolutionary, reinforcement learning, and training-free paradigms, and have been deployed in diverse domains including vision (classification, detection, segmentation), NLP (sequence modeling, translation), multi-task learning, remote sensing, and hardware-constrained settings. This article systematically reviews the key search spaces, search algorithms, efficient proxy indicators, and TAS systems, synthesizing the state of the art with reference to the underlying technical literature. ## 1. Search Space Design: Dimensions, Operators, and Parametrization Defining a rich yet tractable search space (𝒜) is fundamental to TAS. The canonical Transformer block consists of multi-head self-attention (MSA) and MLP sublayers, but modern TAS spaces admit substantial heterogeneity: **Core parametrization**: - **Depth (number of blocks/layers):** Each stage or block can vary in depth (e.g., $L\in\{4,6,12,16\}$) [2111.14725], [2512.12296]. - **Embedding dimension per stage ($d$):** Block-wise or stage-wise embedding/channel sizes [2111.14725], [2512.12296], [2203.10435], [2110.04035]. - **MLP expansion ratio ($r$):** Ratio of hidden-to-input size in feed-forward [2111.14725], [2512.12296]. - **Attention head count ($h$):** Block-level or depth-wise attention head numbers [2111.14725], [2512.12296]. - **Operator choices and hybrids:** Inclusion of convolutions, local or global self-attention, windowed attention, and MLP-mixers [2110.04035], [2207.05420], [2203.10435]. **Advanced axes**: - **Attention mechanism type:** Per-layer selection between softmax-attention, kernel-linear (efficient) attention, dynamic convolution, or other variants [2207.13955], [2306.00288]. - **Downsampling modules:** Parameterizable DSMs (local convolutional, global self-attention, local-global) to bridge transitions between blocks of different operator types [2110.04035], [2207.05420]. - **Multi-resolution/multi-branching:** Cells and fusing nodes governing flow between multiple resolutions or branches (as in multi-scale segmentation networks) [2403.10413]. - **Cross-task skeletons and shared/unshared routing:** Multi-task settings allow the skeleton (branch-out locations for decoders) to be searched [2304.08756]. **Formulation**: The typical architecture is represented as a tuple of all these choices. For example, in UniNet: $$(o_0,e_0,s_0,r_0,\ldots,o_4,e_4,s_4,r_4,DSM_0,\ldots,DSM_4)$$ with $o_i$ ∈ {DWConv, Conv3×3, SA, LSA, MLP}; expansion, scaling, and DSM selected at each of $K=5$ stages [2110.04035]. Search spaces can be enormous (e.g., |𝒜| ≈ 10¹⁶ for standard ViT-like spaces [2512.12296]), necessitating both algorithmic and statistical parsimony in search. ## 2. Search Algorithms: Differentiable, Evolutionary, RL, and Training-Free Approaches TAS leverages multiple classes of search algorithms according to task, resource, and supervision regimes. ### Differentiable NAS - **DARTS-style relaxation:** Architecture weights (α) assigned to candidate operations are optimized jointly by gradient descent, followed by discretization. Memory bottlenecks for transformers (large $d$, large $|\mathscr{O}|$) have been mitigated by introducing multi-split reversible layers and reconstruction-based backpropagation, halving memory cost without sacrificing search space expressivity [2105.14669]. - **Supernet/sandwich strategies:** Weights are shared across sub-architectures, with sandwich rules (always sampling smallest, largest, and random mids) stabilizing optimization [2111.14725], [2512.12296]. ### Reinforcement Learning (RL) - Controllers (often LSTMs) sample architectures; rewards incorporate hold-out accuracy and computational cost (e.g., $R = \text{Acc} - \alpha\, [\log F - \log t]^+$) [2110.04035], [2207.05420]. - Policy gradient or PPO is used to update the controller; reward functions target FLOPs, latency, or hardware-aware proxies. ### Evolutionary and Genetic Algorithms - **Multi-objective genetic algorithms (MOEA/D, NSGA-II):** Variables encode block types, head counts, FFN widths, decoder-encoder connections. Objectives include BLEU (or mIoU) and perplexity (or latency), and evolutionary operators implement variable-length, block-aware crossover and mutation [2505.01314], [2403.10413]. - **Pareto optimization:** Many frameworks explicitly search for the Pareto frontier of accuracy vs. cost (FLOPs, latency, memory) [2203.02094], [2403.10413]. - **Zero-shot evolutionary search:** Fitness is based on maximum-entropy or entropy-like signal (e.g., differential entropy for backbone blocks); no training is performed during search [2505.06694]. ### Training-Free and Proxy-Guided Search - **Zero-cost proxies:** Metrics correlated with final accuracy are computed from randomly-initialized architectures. Popular proxies include synaptic diversity and saliency (DSS-indicator) for ViTs [2203.12217], hidden covariance [2306.00288], Jacobian covariance, GradNorm, SNIP, NASWOT, ZiCo, and their fusions [2407.16269]. - **Parameter-count proxy:** For GPT-style LMs, decoder parameter count nearly perfectly (Spearman ρ≈0.97–0.99) predicts perplexity, enabling search to be run on target hardware without training [2203.02094]. - **Hybrid predictors:** Some works advocate two-stage zero-cost pipelines combining architectural heuristics (e.g., parameter budget) and structural proxies, optionally ensemble-metamodels (e.g., random forests over proxy + architecture features) [2407.16269]. ## 3. Unified Hybrid Search Spaces and Modular Operators Hybrid TAS spaces allow dynamic composition of convolutional, self-attention, and MLP modules under unified templates: - **Operator unification:** All blocks instantiated as inverted bottlenecks with expansion, residual addition, and a core operator (e.g., Conv, SA, MLP) [2110.04035], [2207.05420]. - **Context-aware downsampling:** The DSM search allows the controller to bridge between local (conv) and global (SA/MLP) operators by picking among local-DSM, global-DSM, or mixed DSM adapted to following operator context [2110.04035], [2207.05420]. - **Multi-resolution fusion:** In segmentation and panoptic applications, the search space includes selection of branch count, memory-efficient self-attention vs. lightweight convolution blocks, and learned node-level fusion [2403.10413]. - **Graph-structured search:** Architectural DAGs (as in VTCAS or cell-based NLP spaces) support rewiring of attention and FFN placements at sub-block scale, enhancing representational diversity [2203.10435], [2306.00288]. - **Task-specific macro structures:** For multi-task networks, the skeleton (branching and head-insertion points) is a discrete macro-variable, subject to Gumbel-Softmax relaxation and subsequent hard selection [2304.08756]. ## 4. Efficient Evaluation Proxies and Training-Free TAS Training-free TAS achieves several orders-of-magnitude reduction in search cost by substituting full or partial training with scalar indicators: - **DSS-indicator:** $S_{\text{DSS}}(\mathcal{A}) = \sum_l D^{l}_{\text{MSA}} + \sum_k S^{k}_{\text{MLP}}$, summing nuclear-norm-based diversity of untrained MSA blocks and saliency of MLP weights, offers strong correlation with test accuracy in ViTs (Kendall's τ ≈ 0.65–0.70) and dramatically reduces total GPU-days (∼0.5 vs. 24+) [2203.12217]. - **ZiCo and ZiCo++:** Gradient-based proxies combining Fisher-activation statistics and layer-decay weighting (ZiCo++) outperform single-score approaches in ranking HSI-transformers, with Spearman ρ≃0.73–0.75 [2407.16269]. - **Decoder-parameter proxy:** Pure model parameter count suffices to select high-quality LMs under hardware constraints, exploiting monotonicity between count and PPL [2203.02094]. - **Statistical fusion and meta-models:** Combining multiple zero-cost proxies and simple architecture features via a small metamodel (e.g., Random Forest) can further improve ranking accuracy, with ρ ≈ 0.80 using only 50 samples [2407.16269]. A plausible implication is that further advances will require proxies sensitive not only to parameter scale and diversity but also to task-specific feature representations and data variance, especially in challenging modalities such as hyperspectral imaging and multi-modal learning. ## 5. Empirical Results and Discovered Architectural Patterns Results from major TAS frameworks demonstrate state-of-the-art performance, with hybrid configurations consistently outperforming pure ConvNets or ViTs under equivalent resource constraints: | Model/Method | Domain | Params/FLOPs | Accuracy/Top-1 | Key Discovery | Ref. | |------------------------------|---------------|----------------|----------------|------------------------------------------------------------------------|-------------------| | UniNet-B0 to B6 | ImageNet/Det | 0.56G–51G | 79.1–87.4% | Early DWConv, late global SA, dynamic DSM | [2110.04035], [2207.05420] | | S3 (S3-T/S/B) | Vision | 28M/50M/71M | 82–84% | Search-space evolution, block allocation, increasing window size | [2111.14725] | | VTCAS | Classification| 31M/5.2G | 82.0% | Conv + shifted-MSA, DAG block search | [2203.10435] | | DARTSformer | Seq2Seq | 65M (base) | 28.4 BLEU | Multi-split reversible memory, operation mixing (conv, SA, FFN) | [2105.14669] | | LTS | LM/Hardware | 10⁵⁴ archs | ρ ≈ 0.97/0.99 | Decoder param proxy, on-device search, Pareto frontier extraction | [2203.02094] | | GrowTAS | ViT, Transfer | 5.9–54M | 75.2–82.7% | Progressive small-to-large curriculum, fine-tuning of large-only params | [2512.12296] | | AutoTaskFormer | Multi-task | 22–109M | Δ_T ≈ 7–9% | Gumbel-Softmax skeleton search, task-adaptive branching | [2304.08756] | | NAS-DETR (MaxEntropy) | Detection | GFLOPs ≈145 | mmAP .538 | Maximum entropy zero-shot backbone, entropy-based mutations | [2505.06694] | Experimental ablation confirms: - Hybrid pipelines enable the NAS controller to assign local operations to early layers, global (SA/MLP) to deep layers, with DSM assignment closely tracking operator transitions [2110.04035], [2207.05420]. - Mixed attention (Softmax/Linear) layers outperform vanilla variants for compute-constrained language and vision tasks [2207.13955]. - Progressive subspace expansion (GrowTAS) mitigates training interference between small and large subnets, with feature-space stability analysis confirming knowledge is better preserved when starting with small subnets [2512.12296]. - Pareto-efficient architectures—balancing accuracy, latency, and memory—are efficiently extractable by combining zero-cost proxies and hardware profiling [2203.02094], [2403.10413]. ## 6. Applications Across Modalities and Transfer Settings TAS has been instantiated in a variety of specialized contexts: - **Vision (ImageNet, COCO, ADE20K, ExDark, HSI):** Hybrid Conv-Transformer-MLP backbones, context-aware DSMs, and Pareto search yield SOTA classification, detection, segmentation, and low-light robustness [2110.04035], [2207.05420], [2403.10413], [2407.16269]. - **Language modeling (WikiText-103, LM1B, GLUE):** Hardware-constrained search, decoder-param proxies, and highly-heterogeneous search spaces [2203.02094], [2306.00288]. - **Multi-task learning (Taskonomy, Cityscapes, NYUv2):** Automatic discovery of per-task head placement and cross-task sharing [2304.08756]. - **Spiking Vision Transformers:** Architectures optimized by energy-accuracy balanced indicators for ultra-low-power platforms [2306.00807]. - **Remote sensing:** Zero-proxy guided search tailored for hyperspectral transformer classification [2407.16269]. - **Autonomous and embedded systems:** Search explicitly targets latency and memory using on-device profiling [2203.02094], [2403.10413]. ## 7. Limitations, Design Guidelines, and Future Directions Critical lessons and open challenges from TAS research include: - **Search space expresses performance trade-offs:** Excessive space "homogeneity" (e.g., stacking identical attention + FFN) erodes the utility of most zero-cost proxies, with parameter count dominating [2306.00288], [2407.16269]. Heterogeneous, cell- or block-based search spaces with operator, depth/width, and connection diversity are essential. - **Proxies must normalize for size:** Without normalization, proxies regress redundantly to parameter count; modular, seed-invariant, and task-adaptive proxies are recommended [2203.12217], [2306.00288], [2407.16269]. - **Curriculum and progressive expansion:** Progressive training from small to large subnets reduces destructive interference, particularly when using shared weights [2512.12296]. - **Integration of data-driven and structural indicators:** Current zero-proxy methods are largely data-agnostic and may show bias toward deeper/wider models; future methods should incorporate data signals and avoid overfitting to component type [2407.16269]. Architects are advised to jointly design search spaces and proxies, embed target hardware signals and task requirements, and use multi-objective or meta-learned fitness functions when possible. State-of-the-art frameworks have set benchmarks for efficiency (sub-hour or sub-GPU-day search), flexibility (hybrid operator spaces), and transferability (strong downstream/generalization performance), yet continued progress in proxy design, adaptive search, and domain-specific constraints is a frontier of ongoing research. --- **Principal references**: [2110.04035], [2207.05420], [2512.12296], [2111.14725], [2203.12217], [2203.10435], [2407.16269], [2306.00807], [2304.08756], [2207.13955], [2306.00288], [2505.06694], [2203.02094], [2105.14669], [2505.01314], [2403.10413]. Source: https://www.emergentmind.com/topics/transformer-architecture-search-tas