NAS-LoRA: Efficient PEFT for Vision Models

• NAS-LoRA is a parameter-efficient fine-tuning method that combines neural architecture search with LoRA to dynamically optimize adapter structures in large pre-trained models.
• It introduces lightweight, differentiable NAS cells that fuse candidate operations to inject spatial inductive biases and adapt model capacity for various tasks.
• The method employs a two-stage training process alternating weight and architecture updates to achieve significant training cost reductions while maintaining zero inference overhead.

Neural Architecture Search for Low-Rank Adaptation (NAS-LoRA) refers to a family of methodologies that leverage neural architecture search (NAS) to dynamically optimize or structure parameter-efficient fine-tuning (PEFT) adapters in large pre-trained models, notably Vision Transformers (ViTs) and Vision-LLMs (VLMs). By integrating lightweight NAS modules with the Low-Rank Adaptation (LoRA) paradigm, NAS-LoRA aims to (1) introduce inductive biases, (2) adapt model capacity with respect to downstream task requirements, and (3) enhance sample efficiency and resource savings—all while retaining the zero inference cost characterizing LoRA-based fine-tuning. The term encompasses methods such as the NAS-LoRA framework for visual foundation models and its variable-rank adaptation counterpart, LangVision-LoRA-NAS for VLMs (Chen et al., 3 Dec 2025, Chitty-Venkata et al., 17 Aug 2025).

1. LoRA Fundamentals and Motivating Limitations

Low-Rank Adaptation (LoRA) is a PEFT method where, for a frozen pre-trained weight matrix $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$, adaptation is realized by injecting two trainable matrices $W_e \in \mathbb{R}^{r \times d_{\text{in}}}$, $W_d \in \mathbb{R}^{d_{\text{out}} \times r}$ of low rank $r \ll \min(d_{\text{in}}, d_{\text{out}})$. During fine-tuning, the forward computation is: $h = W_0 x + W_d W_e x$ and at inference, the low-rank update $\Delta W$ can be merged into $W_0$, yielding no inference overhead. However, plain LoRA lacks explicit inductive bias and typically uses fixed-rank adapters per layer or operator. This uniformity may hinder domain adaptation or task specialization, particularly when domain-specific priors are absent or when computational resource allocation across layers is suboptimal (Chen et al., 3 Dec 2025, Chitty-Venkata et al., 17 Aug 2025).

2. NAS-LoRA for Vision Transformers: Methodology

NAS-LoRA augments standard LoRA in ViT-based segmentation models by introducing a differentiable NAS cell between each LoRA encoder and decoder. Specifically, after encoding with $W_e$, the resulting activation traverses a mixture of candidate operations $\mathcal{O} = \{O_1, ..., O_K\}$ such as depthwise separable convolutions (3×3, 5×5), dilated convolutions, average/max pooling, skip connections, and no-op. The architectural choice is controlled via softmax-normalized parameters $\alpha \in \mathbb{R}^K$:

$$h = W_0 x + W_d \left( \sum_{i=1}^K \frac{\exp(\alpha_i)}{\sum_{j=1}^K \exp(\alpha_j)} O_i (W_e x) \right)$$

This soft routing allows concurrent exploration and injection of local spatial inductive biases, addressing the lack of spatial priors in ViTs. The NAS block is always lightweight (≤0.63% extra trainable parameters) and task-adaptive. At inference, all operations—parameterized by learned $\alpha$—are fused into a single dense matrix, and the computational cost collapses to that of the original ViT (Chen et al., 3 Dec 2025).

3. Optimization Strategy and Training Dynamics

The NAS-LoRA training regime alternates between two stages to stabilize joint optimization of millions of pretrained weights and a handful of architectural ($\alpha$) parameters:

• Stage 1 (Weight Update): Fix $\alpha$, update low-rank weights $w = \{W_e, W_d\}$ using segmentation loss $\mathcal{L}(w, \alpha)$.
• Stage 2 (Architecture Update): After an initial "warmup" period (empirically, 10 out of 40 epochs), fix $w$, then update $\alpha$ by differentiating through the aggregated loss.

This stage-wise schedule prevents premature architectural specialization and oscillatory dynamics, empirically boosting stability and final accuracy. When using partial-channel sampling (NAS-PC-LoRA), the regularization effect further enhances robustness to noisy data (Chen et al., 3 Dec 2025).

4. Variable-Rank NAS-LoRA for Vision–LLMs

LangVision-LoRA-NAS generalizes NAS-LoRA to VLMs by searching for optimal LoRA rank configurations per layer and module. The framework constructs a supernetwork with rank-maximal LoRA adapters (e.g., $r=64$) per module—across Q, K, V, MLP up/down, etc.—and learns softmax-parameterized architectural weights $\alpha_{i,j}$ for a set of allowed ranks $R = \{4,8,16,32,64\}$:

$$W^*_{A,i} = \sum_{j=1}^{|R|} \alpha_{i,j} \, \text{slice}(A_i, r=R_j), \quad W^*_{B,i} = \sum_{j=1}^{|R|} \alpha_{i,j} \, \text{slice}(B_i, r=R_j)$$

Bilevel training alternates LoRA weight optimization (fixed $\alpha$), and architecture parameter updates (fixed LoRA weights), using validation perplexity or a composite resource-aware loss as the objective. The final architecture deterministically selects per-module rank maximizing $\alpha$, yielding a compressed, mixed-rank LoRA model (Chitty-Venkata et al., 17 Aug 2025).

5. Computational Efficiency and Inference Overhead

NAS-LoRA maintains parameter and compute efficiency throughout. Ablative analysis reveals that, compared to Conv-LoRA and standard LoRA, NAS-LoRA achieves approximately 24.1% reduction in training cost (e.g., 21.1 min/epoch for weight step, 25.2 min/epoch for combined weight+$\alpha$ on ISIC 2017) at no increase in inference time or FLOPs. At deployment, all NAS operations are fully merged; the operational graph and runtime match that of standard LoRA/ViT (Chen et al., 3 Dec 2025). In LangVision-LoRA-NAS, adapter count reductions by 2–5× are routinely achieved with negligible impact on validation perplexity, decreasing both parameter count and per-epoch fine-tuning time (e.g., 268.7M to 103.3M parameters; 1,815.3s to 1,786.2s per epoch) (Chitty-Venkata et al., 17 Aug 2025).

6. Empirical Performance Across Visual Tasks

NAS-LoRA has been empirically validated across diverse segmentation benchmarks, consistently outperforming LoRA, Conv-LoRA, and other PEFT baselines. Key gains include:

• Binary medical segmentation: +1.2–1.8 Dice/Jaccard points over LoRA (e.g., 78.5 Dice vs. 76.6% on ISIC 2017).
• Camouflaged object datasets: surpassing $F_\beta^w$ 84.2 vs. best Conv-LoRA 84.0.
• Multi-class segmentation: 96.56 mIoU on Trans10K v1 "easy" and 95.28% accuracy/67.86 mIoU on Trans10K v2. Ablations confirm the benefits of (1) mixture search space (local convolutions and poolings), (2) partial-channel sampling, and (3) delayed (non-joint) architecture updates (Chen et al., 3 Dec 2025). In LangVision-LoRA-NAS, layerwise rank selection patterns reveal that attention Q/K projections in early layers utilize higher ranks, while later or MLP layers favor compression (Chitty-Venkata et al., 17 Aug 2025).

7. Broader Implications and Extensions

The NAS-LoRA philosophy extends beyond vision-only models to other modalities: VLMs, LLMs, encoder–decoder models, and even other adapter schemes (e.g., DoRA, oLoRA, Pissa). The generality of the supernetwork + NAS approach provides a systematic recipe for balancing task-specific inductive bias, resource budgets, and performance, tunable through differentiable search and composite loss objectives. A plausible implication is that further integration of NAS with adapter-based PEFT will continue to enhance the adaptability and efficiency of large foundation models across modalities (Chen et al., 3 Dec 2025, Chitty-Venkata et al., 17 Aug 2025).