DASViT: Differentiable NAS for Vision Transformers

• DASViT is a framework that combines differentiable neural architecture search with hardware-aware dynamic memory mapping to optimize Vision Transformer structures.
• It employs a continuous relaxation approach by representing the ViT topology as a directed acyclic graph with candidate operations and fairness regularization to prevent trivial solutions.
• The framework achieves significant efficiency gains by progressively increasing depth, pruning low-weight operations, and leveraging dynamic allocation on multi-core RISC-V clusters.

DASViT (Differentiable Architecture Search for Vision Transformer) refers collectively to recent methods extending differentiable neural architecture search (NAS) to Vision Transformer (ViT) models, as well as to specialized hardware runtime allocation schemes for ViT deployment at kilo-core scales. The term encompasses both a novel algorithmic framework for efficient architecture discovery in pure self-attention transformers (Wu et al., 17 Jul 2025) and a hardware-oriented dynamic memory mapping approach for high-throughput ViT inference (Wang et al., 2 Aug 2025). This entry provides an integrated view of DASViT, focusing on core principles, mathematical formulations, experimental findings, and system-level implications. Unless otherwise noted, DASViT identifies the differentiable ViT NAS methodology.

1. Motivation and Problem Scope

The search for efficient, high-performing neural network designs has driven broad adoption of NAS. Most NAS work targets convolutional neural networks (CNNs) with reinforcement learning, evolutionary algorithms, or differentiable relaxation. With the widespread adoption of ViT architectures—in which model capacity is dominated by multi-head self-attention (MSA) and interleaved MLP blocks—directly searching the combinatorially larger, less regular ViT design space becomes attractive. Standard NAS methods, however, typically rely on macro-level, hand-picked ViT layer types and discrete evolutionary search, which often underexplore novel connection or operation patterns due to resource constraints.

Simultaneously, hardware platforms for parallel ViT inference, such as large RISC-V clusters with shared L1 memories, struggle to efficiently map attention-based kernels due to memory bank contention and suboptimal locality when scaled to thousands of processing elements (PEs).

DASViT addresses algorithmic architecture search for Vision Transformers and hardware dynamic allocation for scalable deployment, aiming for model innovations and system throughput improvements, respectively (Wu et al., 17 Jul 2025, Wang et al., 2 Aug 2025).

2. Differentiable Architecture Search for ViTs

At the heart of algorithmic DASViT is a continuous relaxation of the ViT encoder-layer topology as a directed acyclic graph of nodes, where edges represent candidate operations drawn from a defined set: $$\mathcal{O} = \{\text{Zero},\,\text{Identity},\,\mathrm{MSA}_h\;(h\in\{8,12,16\}),\,\mathrm{MLP}_r\;(r\in\{0.5,3,4\})\}$$ Each node $x_j$ is computed from its predecessors by summing the outputs of mixed operations parameterized by softmax-normalized real-valued architecture variables $\alpha_{i,j}^k$. Layer computation proceeds as

$$x_j = \sum_{i < j} \overline{o}_{i,j}(x_i), \quad \overline{o}_{i,j}(x_i) = \sum_{o_k \in \mathcal{O}}\frac{\exp(\alpha_{i,j}^k)}{\sum_{o_{k'}\in \mathcal{O}}\exp(\alpha_{i,j}^{k'})} o_k(x_i)$$

This continuous “supernet” is trained jointly over network weights $\mathbf{w}$ and architecture parameters $\boldsymbol{\alpha}$ in a bilevel optimization framework: $$\min_{\boldsymbol{\alpha}}\, [\mathcal{L}_{val}(\mathbf{w}^*(\boldsymbol{\alpha}), \boldsymbol{\alpha}) + \mathcal{L}_{fair}(\boldsymbol{\alpha})],\quad \mathbf{w}^*(\boldsymbol{\alpha}) = \arg\min_{\mathbf{w}}\,\mathcal{L}_{train}(\mathbf{w},\boldsymbol{\alpha})$$ Updates alternate between gradient descent on $\mathbf{w}$ (training data) and $\boldsymbol{\alpha}$ (validation plus "fairness" regularization), using one-step unrolled weight updates for architecture gradients.

3. Search Space, Algorithm Design, and Fairness Regularization

The DASViT methodology introduces several architectural and algorithmic improvements over DARTS-type NAS for ViTs:

• Progressive Depth and Pruning: Rather than optimize the entire depth at once, DASViT grows layer depth in stages (e.g., 2→4→6), inheriting weights and pruning candidate operations with lowest $\alpha$ weights each stage.
• Operation Fairness Regularization:
  • $x_j$0
  • $x_j$1 penalizes the mean skip (identity) operation weight to prevent dominance by trivial connections.
  • $x_j$2 constrains total weights for each operation type (MSA, MLP, skip, zero) within preset bounds, encouraging diverse operation usage.
• Attention-based Partial Token Selection: For memory efficiency, only a top-$x_j$3 subset (by QK score) of tokens participates in each MSA computation during search, dramatically reducing memory footprint.
• Discretization: After training, the architecture is derived by selecting the maximum-weighted operation on each edge, yielding the final discrete ViT topology.

DASViT thus enables efficient, resource-conscious differentiable search for self-attention model structure, overcoming pathologies of skip-connection collapse and unbalanced operation selection (Wu et al., 17 Jul 2025).

4. Discovered Transformer Architectures and Empirical Findings

DASViT-optimized architectures systematically deviate from the standard “MSA→add→MLP→add” encoder macro found in manual ViT designs. A typical discovered block, especially on CIFAR-10, follows: $x_j$4 This cross-layer “add-and-reuse” structure leverages both shallow and deep features, enhancing expressiveness without significant parameter overhead.

Comparative empirical results (training from scratch, no pre-training) highlight the gains:

Model	CIFAR-10 Top-1	#Params	FLOPs	ImageNet-100 Top-1
ViT-B/16	78.8%	85.8M	12.0G	39.7%
DASViT	80.1%	50.4M	9.9G	46.8%

On CIFAR-100, DASViT achieves 54.4% versus 45.7% for ViT-B/16 (Top-1). The search results therefore demonstrate both accuracy improvements and efficiency gains (−41% parameters, −17.5% FLOPs vs ViT-B/16), with the discovered architectures narrowing the performance gap to strong convolutional baselines (Wu et al., 17 Jul 2025).

5. System Architecture: Dynamic Allocation Scheme for ViTs

DASViT also refers to a hardware-level Dynamic Allocation Scheme (DAS) for large RISC-V clusters deploying attention mechanisms (Wang et al., 2 Aug 2025). This hardware augmentation comprises:

• Programmable Address Mapper: A combinatorial mapping unit at each PE’s load-store interface, configured via per-region CSRs ($x_j$5: partition granularity, $x_j$6: region size exponent, Base: region offset).
• Unified Dynamic Allocator: Runtime-managed heap partitioning and allocator (linked list free-list), reserving “DAS-regions” in L1; regions map PE-contiguous accesses to physically localized bank subsets.
• DMA Mapper Parity: A matching address-mapper in the DMA data splitter for consistent block copy semantics.

This design exploits the arithmetic intensity and memory access patterns of GEMV, GEMM, and self-attention kernels, dynamically carving L1 banks to maximize bank locality for distributed outputs (e.g., ViT queries, outputs) and interleaving shared vectors/matrices to balance global demand. For example, ViT-L/16 encoder layers run in 5.67 ms (176 images/s) on a 1024-PE setup, with 0.81 PE utilization, nearly doubling the baseline throughput and halving LSU stalls, all at <0.1% chip area overhead.

6. Quantitative Impact, Trade-offs, and Limitations

Algorithmic DASViT consistently outperforms both hand-tuned and NAS-discovered ViTs (ViT-B/16, AutoFormer-B) in accuracy and efficiency on CIFAR-10, CIFAR-100, and ImageNet-100 using fewer parameters and FLOPs, without large-scale pre-training (Wu et al., 17 Jul 2025). Efficiency is multiplicatively increased by pruning non-essential operations, tailoring MLP ratios, and leveraging top-$x_j$7 token selection. Architectures discovered exhibit improved reuse of features across layers, conferring greater representational capacity per parameter.

The hardware DAS achieves a 1.94× speedup on ViT-L/16 workloads relative to word-level interleaved baselines, nearly doubles PE utilization (0.40 to 0.80), and requires only a 0.06 mm² logic addition (<0.1% area) on a 68.9 mm² 1024-PE RISC-V cluster (Wang et al., 2 Aug 2025). This is accomplished without changing PE ISA or memory porting, thus easing system integration.

Current limitations include the potential for heap fragmentation when many small DAS-regions are used, the absence of multi-dimensional bank-mapping logic for non-square/causal mask attention, and the challenge of extending beyond single-cluster shared-L1 to multi-tile mesh topologies. The general mapping logic and allocator extend to LLMs—including GPT or BERT—without modification.

7. Broader Context and Extensions

DASViT’s differentiable NAS and hardware techniques are tied to the broader NAS and system design literature but distinguish themselves by, respectively: (a) bringing continuous-relaxation search to pure self-attention transformer topologies using operation fairness and memory balancing; and (b) enabling runtime-tunable, fine-grained memory partitioning for kilo-core ViT inference at minimal area overhead.

Both architectures have implications for efficient, hardware-scalable deployment of advanced transformer models in vision and language tasks, with plausible applicability to sparse attention, rotary embeddings, and mixture-of-experts (MoE) layers. Extensions to more dynamic partitioning, advanced bin-packing, region coalescing, and integration across mesh backbones represent natural directions for subsequent research and system design innovation (Wu et al., 17 Jul 2025, Wang et al., 2 Aug 2025).