VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Experts

Abstract: Vision-language-action (VLA) models inherit rich visual-semantic priors from pre-trained vision-language backbones, but adapting them to robotic control remains challenging. Full fine-tuning (FFT) is prone to overfitting on downstream robotic data and catastrophic forgetting of pretrained vision-language capabilities. Parameter-efficient fine-tuning (PEFT) better preserves pre-trained knowledge, yet existing PEFT methods still struggle to adapt effectively to robot control tasks. To address this gap, we propose VLA-GSE, a parameter-efficient VLA fine-tuning framework that improves control adaptation while retaining PEFT's knowledge preservation advantage. Specifically, VLA-GSE (Generalized and Specialized Experts) is initialized by spectrally decomposing the frozen backbone, assigning leading singular components to generalized experts (shared experts) and disjoint residual components to specialized experts (routed experts). This decomposition improves adaptation capacity under a fixed trainable-parameter budget. Under a comparable parameter budget, VLA-GSE updates only 2.51% of the full model parameters and consistently outperforms strong FFT and PEFT baselines. It achieves 81.2% average zero-shot success on LIBERO-Plus, preserves pre-trained VLM capability comparably to LoRA on multimodal understanding benchmarks, and improves real-world manipulation success under multiple distribution shifts. Code is available at: https://github.com/YuhuaJiang2002/VLA-GSE

• The paper introduces an SVD-based framework that decomposes VLA model adaptation into a generalized expert for preserving pretrained knowledge and specialized experts for task-specific adaptation.
• Experiments reveal that tuning just 2.51% of parameters, VLA-GSE achieves up to 81.2% zero-shot success, outperforming FFT and LoRA methods by 4.4–11.9 points.
• The approach mitigates catastrophic forgetting and balances gradient dynamics through load-balancing losses and trace-inverse scaling, ensuring robust performance in real-world scenarios.

VLA-GSE: Parameter-Efficient Fine-Tuning in Vision-Language-Action Models via Generalized and Specialized Experts

Introduction and Motivation

Parameter-efficient adaptation of large-scale vision-LLMs (VLMs) to the embodied control regime is an open problem in robotics and multimodal learning. Full-parameter fine-tuning (FFT) of vision-language-action (VLA) policies leads to overfitting and catastrophic forgetting of pretrained representations, while traditional parameter-efficient fine-tuning (PEFT) schemes such as LoRA are insufficiently expressive for high-fidelity control. The work "VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Experts" (2605.06175) introduces a structured adaptation framework for VLA models that leverages SVD-based spectral decomposition to jointly learn generalized and specialized experts, maximizing adaptation headroom while strongly preserving pretrained capacity. The approach is directly motivated by empirical and theoretical limitations of both full and LoRA-based VLM adaptation in robot learning.

Methodology: Generalized-Specialized Expert Framework

VLA-GSE decomposes VLM adaptation into two explicit parameter-efficient paths: a generalized expert, initialized from the leading singular vectors of pretrained layers, and multiple specialized experts, each responsible for a disjoint lower-energy spectral region of the backbone. The generalized expert is always active, supporting robust knowledge preservation and transfer. Specialized experts are dynamically routed per input using a learned gating mechanism, allowing for context-sensitive adaptation required by robotic policy learning.

Figure 1: Overview of the VLA-GSE framework, showing SVD-based decomposition into generalized and multiple specialized experts, each covering different spectral regions of the frozen backbone weight matrix.

At initialization, the SVD yields a block assignment of spectral components: the top-$r_g$ components parameterize the generalized expert ($B_g, A_g$), while the succeeding blocks of size $d$ parameterize each specialized expert ($B_s^i, A_s^i$ for expert $i$). Specialized experts are selected via a trainable router $W_z$, employing sparse top-$k$ routing for computational and adaptation efficiency.

Gradient imbalance arising from spectral scale heterogeneity across expert blocks is rigorously addressed by a trace-inverse scaling rule and expert-wise gradient normalization (see Theorem 1 in the work), ensuring equitable optimization dynamics even in highly skewed singular value spectra.

Specialized expert routing collapse is mitigated via a per-block load-balancing auxiliary loss, which encourages soft and hard expert assignment frequencies to match, thereby preventing expert under-utilization and maintaining effective modular specialization.

To preserve the retained knowledge of the backbone at initialization, the frozen weight tensor is adjusted to exactly compensate for the mean perturbation introduced by the experts (in expectation over router stochasticity).

Experimental Evaluation

Simulation: LIBERO-Plus and LIBERO

VLA-GSE was benchmarked on the LIBERO-Plus simulation suite, a systematic generalization testbed spanning compositional OODs across camera, robot, language, illumination, background, sensor noise, and spatial layout axes.

On LIBERO-Plus, VLA-GSE, tuning only 2.51% of the backbone and action head parameters, achieves 81.2% average zero-shot success, outperforming all prior SOTA PEFT (GOAT: 76.8%) and even full fine-tuning (FFT: 74.9%) (see main text Table 2). Notably, VLA-GSE exhibits across-the-board robustness, excelling especially in light and background OODs, and demonstrates the lowest catastrophic forgetting on auxiliary multimodal benchmarks.

Compared to LoRA, rsLoRA, DoRA, PiSSA, MoLoRA, AdaMoLE, HydraLoRA, and MiLoRA variants—all matched in adaptation budget—VLA-GSE's structured expert composition yields a consistent 4.4–11.9 point margin in generalization success.

Attention Dynamics

Qualitative inspection of attention maps under severe visual shift (e.g., dark lighting) reveals that VLA-GSE sharply localizes to manipulation-relevant object edges, while FFT and LoRA baselines fail to maintain focused grounding and attention collapses (Figure 2).

Figure 2: Model attention under challenging visual shift; VLA-GSE maintains sharp, affordance-aligned attention, while baselines are distracted or mislocalize.

This effect is robust across all generalization axes (camera, object layout, lighting, language, sensor noise, and backgrounds), as confirmed by extended attention visualizations (Figure 3).

Real-World Generalization and Latency

VLA-GSE delivers high real-world generalization on four manipulation tasks under four OOD axes (lighting, language, scene layout, background), achieving 82.5% success, outperforming all prior OpenVLA-OFT, $\pi_0$, and FFT/LoRA-based VLA deployments by 7–40 points (Figure 4; see text Table 3). Importantly, VLA-GSE introduces minimal inference latency overhead (~2nd fastest among all methods) on industrial hardware, supporting practical deployment demands (Figure 5).

Figure 4: Real-world task configurations and OOD axes for systematic evaluation.

Figure 5: VLA-GSE achieves near-minimal inference latency, closely matching standard FFT deployments.

Ablations and Theoretical Analysis

Ablation studies highlight the necessity of each core design. Removing SVD-based initialization, either expert type, load-balance loss, or expert-wise scaling significantly degrades robustness and downstream performance (drops up to 13 points). Theoretical analysis justifies the trace-inverse scaling for expert gradients, tying expected learning rates to the singular value spectrum and guaranteeing balanced expert optimization.

Knowledge Retention and Catastrophic Forgetting

On general multimodal benchmarks detached from robot data, FFT and co-trained VLA policies exhibit sharp catastrophic forgetting, with up to 30–50 point drops versus the base VLM. By contrast, both VLA-GSE and LoRA can match the original VLM's capacity, confirming that adaptation expressivity is not achieved by sacrificing pretrained generalization.

Practical and Theoretical Implications

VLA-GSE advances parameter-efficient backbone adaptation, strongly positioning the SVD-based expert decomposition as the new baseline for embodied VLM-to-VLA transfer. Architecturally, VLA-GSE decouples the preservation of multimodal priors from local task specialization, thereby circumventing the overspecialization or under-adaptation failure modes of prior FFT and LoRA-style methods.

The generalized-specialized mechanism is broadly applicable to other domains requiring simultaneous retention of transferable knowledge and high adaptation expressivity; its extension to domains such as medical reasoning, financial data, or other high-precision multimodal environments is a promising future direction.

On the systems front, VLA-GSE's low adaptation budget allows for scalable distributed training and fast iteration, as well as safety- and deployment-critical real-world bridging where catastrophic forgetting is unacceptable.

Conclusion

VLA-GSE provides an effective, theoretically justified, and empirically validated framework for parameter-efficient adaptation of VLMs to high-fidelity embodied action. Through SVD-initialized generalized and routed specialized experts, load-balance regularization, and expert-wise scaling, it achieves higher robustness, superior generalization, and minimal inference penalty compared to all prior FFT and PEFT schemes. This architecture marks a substantial step toward practical and reliable transfer of foundation models into embodied domains, supporting both industrial deployment and ongoing theoretical investigation.