MergeVLA: Cross-Skill Model Merging

• The paper introduces MergeVLA, a vision-language-action framework that fuses independently finetuned robotic experts into a generalist policy without requiring joint retraining.
• It employs task-conditioned sparse activation through LoRA masks and a cross-attention-only design in the action expert to suppress conflicting task-specific updates.
• Empirical evaluations on simulators and real-world robotic tasks demonstrate improved multi-skill performance and effective mitigation of parameter conflicts.

Searching arXiv for MergeVLA and closely related VLA token-merging work to ground the article in current papers. MergeVLA is a vision-language-action (VLA) architecture and model-merging framework designed for combining independently finetuned robotic experts into a single generalist policy without joint retraining or access to the original data. In the formulation introduced in "MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent" (Fu et al., 24 Nov 2025), the central problem is that standard post-hoc merging methods that are often effective in NLP and vision fail almost completely on VLAs: directly merging VLA experts trained on different tasks results in near-zero success rates. MergeVLA therefore treats mergeability as an architectural property to be preserved during finetuning, rather than as a purely downstream parameter-averaging problem.

1. Problem formulation and failure of naïve VLA merging

MergeVLA is posed in a setting with a collection of single-skill imitation datasets,

$$\mathfrak{D}=\{\mathcal{D}_m\}_{m=1}^M, \quad \mathcal{D}_m = \{ (\mathbf{I}^v_t,\mathbf{I}^w_t,L)_t \},$$

where each $\mathcal{D}_m$ corresponds to one manipulation task. Finetuning a pretrained VLA on each $\mathcal{D}_m$ yields task-specific experts with weights $\Theta_1,\ldots,\Theta_M$. The objective is to construct a single generalist $\Theta_{\mathrm{merge}}$ that performs all tasks without joint retraining or access to the original data (Fu et al., 24 Nov 2025).

The motivating empirical result is negative. On LIBERO, merging four separately finetuned VLA experts with Task Arithmetic or related methods often yields $0\%$ success across all tasks. For OpenVLA, merging all components with Task Arithmetic gives near-zero performance, and for VLA-Adapter, merging all components also results in $0\%$ success. This establishes the paper’s core claim that multi-skill VLA composition is structurally harder than model merging in domains with discrete outputs (Fu et al., 24 Nov 2025).

A common misconception is that VLA merging should follow directly from the success of weight averaging, task arithmetic, or TIES-style methods in LLMs. The MergeVLA analysis argues otherwise. In continuous control, small discrepancies in internal representations can accumulate into catastrophic action errors, so the constraints on mergeability are tighter than in text generation. This suggests that the merge failure is not merely an artifact of a poor merging rule, but of the way existing VLAs localize or diffuse task-specific information during finetuning (Fu et al., 24 Nov 2025).

2. Empirical diagnosis of non-mergeability

The MergeVLA paper identifies two principal sources of non-mergeability: divergent LoRA directions in the VLM backbone and inter-block dependency in action experts (Fu et al., 24 Nov 2025).

For the VLM backbone, let $\Theta_0$ denote the pretrained VLM and $\Theta_m$ the LoRA-finetuned weights for task $m$. The task vector is

$\mathcal{D}_m$0

Standard merging methods construct

$\mathcal{D}_m$1

where $\mathcal{D}_m$2 is an aggregation rule such as average, TIES, TSV, or related operators. MergeVLA reports that task-specific LoRA updates activate largely disjoint subsets of channels, and that when merging only four tasks, more than $\mathcal{D}_m$3 of LoRA parameters are “selfish,” meaning that they are used by exactly one task. The paper defines the selfish ratio as

$\mathcal{D}_m$4

with $\mathcal{D}_m$5 the number of parameters and $\mathcal{D}_m$6 a binary mask for task $\mathcal{D}_m$7. Empirically, $\mathcal{D}_m$8 for LIBERO with $\mathcal{D}_m$9 to $\mathcal{D}_m$0 merged tasks. The interpretation offered in the paper is destructive LoRA parameter interference: naïve merging re-activates many irrelevant or conflicting updates, corrupting the pretrained VLM manifold (Fu et al., 24 Nov 2025).

The second source of non-mergeability lies in the action expert. In architectures such as VLA-Adapter, the action expert is a transformer trained from scratch and contains self-attention, cross-attention on a task stream $\mathcal{D}_m$1, cross-attention on an action stream $\mathcal{D}_m$2, feed-forward blocks, and a gating function. MergeVLA argues that self-attention creates inter-block feedback: task-specific signal produced in shallow blocks feeds into deeper blocks, becomes amplified, and spreads across layers. The paper reports that average relative $\mathcal{D}_m$3 distance between action-expert blocks across tasks increases sharply in deeper blocks, which become highly task-specialized and therefore resistant to modular recombination (Fu et al., 24 Nov 2025).

The diagnostic decomposition on OpenVLA makes this especially explicit. The model is partitioned into A: vision backbone, B: vision-to-language projector, C: LLM body, and D: LLM head. Using Iso-CTS to merge four experts on each component separately, the paper reports $\mathcal{D}_m$4 average success for the finetuned baseline, $\mathcal{D}_m$5 when merging only D, and $\mathcal{D}_m$6 on all suites when merging C. Progressive merging of the first $\mathcal{D}_m$7 language-model blocks shows performance staying around $\mathcal{D}_m$8 up to approximately $\mathcal{D}_m$9, then collapsing to approximately $\Theta_1,\ldots,\Theta_M$0 beyond $\Theta_1,\ldots,\Theta_M$1 blocks. The resulting conclusion is that deep action-decoding blocks are strongly task-specific and incompatible under direct merging (Fu et al., 24 Nov 2025).

3. Merge-oriented architecture

MergeVLA addresses the two failure modes with a mergeability-by-design architecture. In the VLM backbone it introduces sparsely activated LoRA adapters via task masks; in the action expert it removes self-attention and retains only cross-attention blocks, reserving the final block or blocks as task-specific expert heads (Fu et al., 24 Nov 2025).

VLM masking with task-conditioned sparse activation

Instead of using a single global merged task vector for all tasks, MergeVLA defines task-specific binary masks $\Theta_1,\ldots,\Theta_M$2. For task $\Theta_1,\ldots,\Theta_M$3, the effective merged VLM is

$\Theta_1,\ldots,\Theta_M$4

where $\Theta_1,\ldots,\Theta_M$5 denotes element-wise multiplication. The mask is constructed by

$\Theta_1,\ldots,\Theta_M$6

with mask ratio hyperparameter $\Theta_1,\ldots,\Theta_M$7 in the reported default setting. The intended effect is to keep only merged parameters that remain consistent for task $\Theta_1,\ldots,\Theta_M$8, while reverting inconsistent positions to the pretrained weights $\Theta_1,\ldots,\Theta_M$9 (Fu et al., 24 Nov 2025).

This converts a single merged update into a family of task-conditioned masked variants. The paper’s interpretation is that these sparsely activated LoRA adapters isolate task-consistent subspaces, suppress cross-task conflicts, and preserve pretrained visual-language knowledge more effectively than a one-size-fits-all merged adapter. The ablation summarized in Figure 1 reports that when $\Theta_{\mathrm{merge}}$0, success on LIBERO-Long is approximately $\Theta_{\mathrm{merge}}$1, whereas for $\Theta_{\mathrm{merge}}$2, success on Long exceeds $\Theta_{\mathrm{merge}}$3, indicating that moderate sparsity is essential (Fu et al., 24 Nov 2025).

Cross-attention-only action expert

To prevent inter-block task-information diffusion, MergeVLA redesigns the action expert so that each block removes self-attention entirely and uses only cross-attention from internal action tokens to the task hidden states $\Theta_{\mathrm{merge}}$4 and action hidden states $\Theta_{\mathrm{merge}}$5. It also replaces the $\Theta_{\mathrm{merge}}$6 gate on the task stream with a sigmoid gate (Fu et al., 24 Nov 2025).

The action expert therefore becomes anchored to shared VLM features rather than recursively amplifying its own task-specific outputs. The paper states that, after removing self-attention and changing the gate, shallow action-expert blocks across tasks become close enough to be averaged directly, whereas deeper blocks remain specialized. The deeper specialized tail is formalized as the expert head $\Theta_{\mathrm{merge}}$7, with most experiments setting $\Theta_{\mathrm{merge}}$8, so that only the final block remains task-specific and unmerged. In more difficult cross-embodiment settings, the retained head may be $\Theta_{\mathrm{merge}}$9 or $0\%$0 (Fu et al., 24 Nov 2025).

The resulting specialization hierarchy is concise:

Component	Merge strategy	Role
VLM LoRA	Merge then mask	Task-consistent backbone adaptation
Action blocks $0\%$1	Weight averaging	Generic cross-task control layers
Action head $0\%$2	Unmerged per task	Task-specific control specialization

This architecture should be distinguished from approaches that attempt to merge arbitrary pretrained VLA experts after the fact. In MergeVLA, the architectural decomposition is itself the mechanism that makes later merging feasible (Fu et al., 24 Nov 2025).

4. Merging pipeline and test-time routing

The MergeVLA procedure consists of independent per-task finetuning, LoRA task-vector merging, mask computation, shallow action-block averaging, and test-time routing (Fu et al., 24 Nov 2025).

For each task $0\%$3, the method finetunes LoRA parameters in the VLM backbone and the entire cross-attention-only action expert from scratch. The reported hyperparameters are learning rate $0\%$4, LoRA rank $0\%$5, batch size $0\%$6, and $0\%$7k gradient steps, increased to $0\%$8k for LIBERO-Long. After finetuning, task vectors $0\%$9 are formed only over LoRA parameters. Any existing merging operator $0\%$0 designed for low-rank adapters may then be applied: $0\%$1 with default $0\%$2. The paper evaluates Task Arithmetic, TIES, TSV, KnOTS, WUDI, EMR, and related rules within the MergeVLA architecture (Fu et al., 24 Nov 2025).

The early action-expert blocks $0\%$3 are merged by parameter-wise averaging across tasks, while the tail blocks $0\%$4 remain as task-specific expert heads. This selective merging is central: it avoids forcing the most control-specialized layers into a single shared representation (Fu et al., 24 Nov 2025).

When task identity is unknown at inference time, MergeVLA uses a training-free router. The router exploits the observation that fine-tuned experts occupy distinct subspaces in the value projections of the action expert. At block $0\%$5, value matrices for task and action cross-attention are decomposed by SVD: $0\%$6 The top $0\%$7 right singular vectors define dominant value subspaces $0\%$8 and $0\%$9. For each candidate task $\Theta_0$0, the initial observation is passed through the masked VLM $\Theta_0$1, producing hidden states $\Theta_0$2 and $\Theta_0$3. The router computes

$\Theta_0$4

combines them as $\Theta_0$5, and selects

$\Theta_0$6

The selected mask $\Theta_0$7 and expert head $\Theta_0$8 are then fixed for the full episode (Fu et al., 24 Nov 2025).

A frequent misunderstanding is that MergeVLA requires explicit task IDs at deployment. In the mixed-task regime described in the paper, it does not: the router performs unsupervised task inference from the initial observation. Table 5 reports that routing with only the value subspace $\Theta_0$9 achieves $\Theta_m$0 average success, whereas using only key subspaces gives $\Theta_m$1, making value-based routing the default design (Fu et al., 24 Nov 2025).

5. Empirical performance across tasks, embodiments, and environments

MergeVLA is evaluated on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world experiments on the SO-101 robotic arm (Fu et al., 24 Nov 2025).

On LIBERO multi-task merging, the paper reports the following average success rates. Single-task finetuned OpenVLA reaches $\Theta_m$2, VLA-Adapter reaches $\Theta_m$3, and single-task finetuned MergeVLA reaches $\Theta_m$4. Direct merging remains ineffective: OpenVLA + Task Arithmetic over all parts yields $\Theta_m$5, and VLA-Adapter + Task Arithmetic also yields $\Theta_m$6. Within the MergeVLA architecture, however, different merging rules become viable. MergeVLA$\Theta_m$7 reaches $\Theta_m$8, MergeVLA$\Theta_m$9 $m$0, MergeVLA$m$1 $m$2, MergeVLA$m$3 $m$4, MergeVLA$m$5 $m$6, and MergeVLA$m$7 $m$8, with per-suite scores $m$9. This places the best merged model $\mathcal{D}_m$00 points below the single-task finetuned MergeVLA upper bound while dramatically outperforming direct merges on previous architectures (Fu et al., 24 Nov 2025).

On LIBERO-Plus, which introduces seven perturbation types including background textures, camera viewpoints, language instructions, lighting conditions, object layout, robot states, and sensor noise, the paper reports $\mathcal{D}_m$01 average success for single-task finetuned MergeVLA. Merged variants remain competitive: MergeVLA$\mathcal{D}_m$02 reaches $\mathcal{D}_m$03, MergeVLA$\mathcal{D}_m$04 $\mathcal{D}_m$05, and MergeVLA$\mathcal{D}_m$06 $\mathcal{D}_m$07. The paper also states that removing self-attention and replacing $\mathcal{D}_m$08 with sigmoid improves LIBERO-Plus robustness by $\mathcal{D}_m$09 versus VLA-Adapter, from $\mathcal{D}_m$10 to $\mathcal{D}_m$11 average success (Fu et al., 24 Nov 2025).

On RoboTwin 2.0, MergeVLA is evaluated in two regimes. In Setting A, the same task $\mathcal{D}_m$12 is executed on three embodiments; single-task finetuning yields $\mathcal{D}_m$13 average success, while MergeVLA$\mathcal{D}_m$14 reaches $\mathcal{D}_m$15, effectively matching that baseline. In Setting B, which combines cross-embodiment and cross-task variation, single-task finetuning yields $\mathcal{D}_m$16; using a deeper retained expert head $\mathcal{D}_m$17 with TIES reaches $\mathcal{D}_m$18, indicating that harder composition problems require more unmerged task-specific depth (Fu et al., 24 Nov 2025).

On the real SO-101 arm, the paper evaluates three cube-based tasks with $\mathcal{D}_m$19 rollouts per task. Single-task finetuned MergeVLA reaches $\mathcal{D}_m$20 average success, MergeVLA$\mathcal{D}_m$21 reaches $\mathcal{D}_m$22, and MergeVLA$\mathcal{D}_m$23 reaches $\mathcal{D}_m$24, matching the single-task performance. This is presented as evidence that the framework is not limited to simulation (Fu et al., 24 Nov 2025).

6. Relation to adjacent merge-based VLA research and limitations

MergeVLA occupies one branch of a broader merging literature in embodied multimodal systems, but it should not be conflated with token-compression methods or continual-learning merging schemes.

A first neighboring line of work merges visual tokens rather than model parameters. ST-Merge, introduced in "Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs" (Chen et al., 28 Jun 2026), is a plug-and-play, training-free framework for fusing redundant visual tokens during visual encoding. It constructs 3D spatiotemporal coordinates $\mathcal{D}_m$25, uses Gaussian neighborhood weights

$\mathcal{D}_m$26

applies multi-queue parallel matching, and performs RoPE-aware positional correction. In Qwen2.5-VL video QA it achieves a $\mathcal{D}_m$27 inference speedup with only a tiny $\mathcal{D}_m$28 loss in precision, and on the $\mathcal{D}_m$29 VLA policy it achieves an $\mathcal{D}_m$30 speedup at $\mathcal{D}_m$31 resolution while matching the baseline success rate. This is a different use of “merge” from MergeVLA proper: ST-Merge compresses observations to reduce latency rather than merging cross-skill policies (Chen et al., 28 Jun 2026).

A second neighboring line is TEAM-VLA, "Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Models" (Ye et al., 10 Dec 2025). TEAM-VLA combines early token expansion around language-relevant anchors with action-guided soft bipartite token merging at an intermediate layer. On LIBERO it reduces CUDA latency from $\mathcal{D}_m$32 ms to $\mathcal{D}_m$33 ms, uses about $\mathcal{D}_m$34 of the baseline FLOPs, and maintains the same $\mathcal{D}_m$35 average success rate as OpenVLA-OFT. Again, the operative object of merging is the token sequence rather than the parameterized policy family (Ye et al., 10 Dec 2025).

A third related but conceptually distinct line is merge-then-realign continual learning. "Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs" (Zhang et al., 8 Mar 2025) proposes MERA, which merges modality-agnostic backbone weights by cumulative moving average and then realigns modality-specific connectors using a replay set. MERA is not a VLA method, but its decomposition into shared versus modality-specific components offers a general conceptual template for later embodied systems. This suggests a broader pattern across multimodal learning: mergeability often depends less on raw averaging than on how shared cores and task-specific interfaces are structurally separated and subsequently realigned (Zhang et al., 8 Mar 2025).

Within this landscape, MergeVLA’s specific contribution is architectural co-design for cross-skill and cross-embodiment parameter merging. The paper’s limitations follow directly from that design. Storage grows linearly with the number of tasks because each task requires its own mask $\mathcal{D}_m$36 and expert head $\mathcal{D}_m$37. The choice of split depth $\mathcal{D}_m$38 is workload-dependent; more heterogeneous task or embodiment combinations require larger unmerged heads, reducing the proportion of shared parameters. The router requires running the VLM $\mathcal{D}_m$39 times on the initial observation, which may become costly for large $\mathcal{D}_m$40. Experiments use Qwen2.5-0.5B as the backbone, so scaling behavior for substantially larger VLMs remains open. The paper explicitly suggests larger backbones, more diverse robot datasets, and broader domains such as humanoids and mobile manipulators as future directions (Fu et al., 24 Nov 2025).