Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent
Yuxia Fu, Zhizhen Zhang, Yuqi Zhang, Zijian Wang, Zi Huang, Yadan Luo · UQMM Lab, The University of Queensland
将多个技能专用的 VLA 专家合并为单一通才模型长期以来面临近零成功率的困境。
MergeVLA 通过稀疏任务掩码激活 LoRA 参数，并将 action expert 重新设计为纯 cross-attention 结构，
首次实现了跨技能 VLA 专家的高质量合并——合并后模型在 LIBERO 基准上达到 90.2% 平均成功率，
与单独微调的专家（96.7%）仅差 6.5%，且在真实机器人上同样达到 90% 成功率。Merging multiple skill-specific VLA experts into a single generalist model has long been plagued by near-zero success rates. MergeVLA activates LoRA parameters through sparse task masks and redesigns the action expert as a pure cross-attention structure, achieving for the first time high-quality merging of cross-skill VLA experts: the merged model reaches 90.2% average success rate on the LIBERO benchmark, only 6.5% below the individually fine-tuned experts (96.7%), and likewise attains 90% success rate on a real robot.
arXiv 2511.18810LIBERO · RoboTwin · SO-101University of Queensland📄 arXiv:2511.18810PDF
当前 VLA（Vision-Language-Action）模型通常针对单一任务或单一机器人形态进行微调，难以扩展至多技能通才场景。
模型合并（model merging）是一种无需重新训练即可整合多个专家知识的技术，
但直接将多个 VLA 专家合并会导致几乎为零的成功率，根本原因在于两类不可合并性。Current VLA (Vision-Language-Action) models are usually fine-tuned for a single task or a single robot embodiment, and are hard to scale to multi-skill generalist scenarios. Model merging is a technique that integrates the knowledge of multiple experts without retraining, but directly merging multiple VLA experts yields an almost zero success rate, which is rooted in two kinds of un-mergeability.
"directly merging VLA experts trained on different tasks results in near-zero success rates."
MergeVLA 框架总览：通过任务掩码、重设计的 action expert 以及测试时任务路由，将多个 VLA 专家无损合并为单一通才模型。Overview of the MergeVLA framework: through task masking, a redesigned action expert and test-time task routing, multiple VLA experts are losslessly merged into a single generalist model.
不同任务的 LoRA 微调更新激活了几乎互不相交的参数子集。
分析显示，超过 75% 的参数是"自私的"（selfish）——仅与一个任务相关，
直接合并时这些参数会互相干扰，导致性能崩溃。LoRA fine-tuning updates of different tasks activate almost disjoint parameter subsets. The analysis shows that more than 75% of the parameters are "selfish" — relevant to only one task — so that direct merging makes them interfere with each other and collapses performance.
传统 VLA-Adapter 中 action expert 含有 self-attention 层，
训练过程中不同任务的 self-attention 会在各层之间积累任务特定的依赖关系（inter-block dependencies），
导致合并后信息混乱、难以区分任务。In the conventional VLA-Adapter the action expert contains self-attention layers; during training, the self-attention of different tasks accumulates task-specific inter-block dependencies across layers, so that after merging the information becomes entangled and tasks are hard to tell apart.
~0%直接合并 VLA-Adapter 的成功率Success rate of directly merging VLA-Adapter
75%+参数为"selfish"（仅属于单一任务）of the parameters are "selfish" (belonging to a single task only)
90.2%MergeVLA 合并后 LIBERO 平均成功率Average LIBERO success rate of merged MergeVLA
90%真实 SO-101 机械臂平均成功率Average success rate on the real SO-101 arm
02方法Method
MergeVLA 由三个核心组件构成：（1）稀疏任务掩码用于 VLM 部分的无干扰合并，
（2）重新设计的 action expert消除跨块依赖，
（3）测试时任务路由器在无监督条件下实时识别当前任务并激活对应掩码。MergeVLA consists of three core components: (1) sparse task masks for interference-free merging of the VLM part, (2) a redesigned action expert that removes inter-block dependencies, and (3) a test-time task router that identifies the current task online without supervision and activates the corresponding mask.
不同 VLA 架构对比：OpenVLA 使用标准 VLM 生成 token 化动作；VLA-Adapter 增加了含 cross-attention 和 self-attention 的 action expert；MergeVLA 移除了不可合并的 self-attention 层，仅保留 cross-attention。Comparison of VLA architectures: OpenVLA uses a standard VLM to produce tokenized actions; VLA-Adapter adds an action expert containing cross-attention and self-attention; MergeVLA removes the un-mergeable self-attention layers and keeps cross-attention only.MergeVLA 完整架构：(1) 任务掩码选择性激活合并后的 LoRA 参数；(2) 重设计的 action expert 仅含 cross-attention 与 sigmoid 门控；(3) 训练无关的测试时任务路由器动态选择任务组件。The full MergeVLA architecture: (1) task masks selectively activate the merged LoRA parameters; (2) the redesigned action expert contains only cross-attention and sigmoid gating; (3) a training-free test-time task router dynamically selects the task components.
任务掩码用于 VLM 合并（Task Masking for VLM Merging）Task Masking for VLM Merging
对于 VLM 部分，MergeVLA 在 TIES 或 WUDI 等基础合并算法之上引入二值任务掩码（binary task mask）。
对每个任务 m，掩码构造方式为：For the VLM part, MergeVLA introduces a binary task mask on top of base merging algorithms such as TIES or WUDI. For each task m, the mask is constructed as:
Sm = I[|τm| > λ|τmerge − τm|]
即仅保留任务特定更新显著且与整体合并向量对齐的参数（由超参数 λ 控制掩码比率）。
推理时，对应任务的掩码被激活，从而屏蔽其他任务的参数干扰。That is, only those parameters whose task-specific update is significant and aligned with the overall merged vector are kept (the mask ratio is controlled by the hyperparameter λ). At inference time the mask of the corresponding task is activated, thereby shielding the model from parameter interference of the other tasks.
针对 action expert 的不可合并性，MergeVLA 做出两处关键架构修改：To address the un-mergeability of the action expert, MergeVLA makes two key architectural modifications:
移除 self-attention 层：消除跨块的任务特定依赖，action expert 完全依赖 VLM 的 cross-attention 特征，具有更好的泛化性。Removing the self-attention layers: this eliminates task-specific inter-block dependencies; the action expert relies entirely on the cross-attention features of the VLM and generalizes better.
tanh → sigmoid 门控：确保 VLM 信息不会被压制（tanh 会将值域限制在 [−1,1]，而 sigmoid 保证信号始终正向流通），维持均衡的特征保留。tanh → sigmoid gating: this ensures that VLM information is not suppressed (tanh bounds the range to [−1,1], whereas sigmoid keeps the signal always flowing positively), maintaining balanced feature retention.
对于跨任务差异较大的场景（如跨形态），最后 1–2 个 action expert 块（记为 H(L-1→L) 或 H(L-2→L)）保持未合并，以保留任务特异性。For scenarios with large cross-task differences (such as cross-embodiment), the last 1–2 action expert blocks (denoted H(L-1→L) or H(L-2→L)) are kept unmerged in order to preserve task specificity.
该路由器无需训练，仅利用初始观测帧进行任务识别：The router requires no training and performs task identification using only the initial observation frame:
对每个候选任务掩码，运行 VLM 的 cross-attention 块得到隐状态；for each candidate task mask, run the cross-attention blocks of the VLM to obtain hidden states;
对 value projection 矩阵做 SVD，将隐状态投影到主成分子空间；apply SVD to the value projection matrix and project the hidden states onto the principal subspace;
通过 softmax 计算各任务的激活分数，选择得分最高的任务并激活其掩码。compute the activation score of each task via softmax, then pick the highest-scoring task and activate its mask.
消融实验表明，使用 value（V）投影子空间的路由效果（LIBERO 平均 89.7%）显著优于 key（K）投影（53.6%），选择 V 是最终设计。Ablations show that routing in the value (V) projection subspace (89.7% LIBERO average) clearly outperforms the key (K) projection (53.6%), so V is adopted in the final design.
左：随任务数增加，TA 和 TIES 合并方法中"selfish"参数（仅与单一任务相关）的比例均超过 75%，证明直接合并的失败根源。右：不同 action expert 块之间的相对 L2 距离分析，揭示 self-attention 引入的跨块依赖。Left: as the number of tasks grows, the proportion of "selfish" parameters (relevant to a single task only) exceeds 75% for both the TA and TIES merging methods, revealing the root cause of the failure of direct merging. Right: relative L2 distance analysis between action expert blocks, exposing the inter-block dependencies introduced by self-attention.
03实验Experiments
实验覆盖三个基准：LIBERO（四套仿真任务组）、LIBERO-Plus（七类分布外扰动测试）、
RoboTwin（跨形态跨任务）及真实 SO-101 机械臂。
基线包括 OpenVLA、π0、VLA-Adapter 等。The experiments span three benchmarks: LIBERO (four simulated task suites), LIBERO-Plus (seven kinds of out-of-distribution perturbation tests), RoboTwin (cross-embodiment and cross-task), plus the real SO-101 arm. Baselines include OpenVLA, π0, VLA-Adapter and others.
LIBERO 基准（单任务微调 vs 合并模型）LIBERO Benchmark (Single-Task Fine-Tuning vs. Merged Model)
LIBERO-Plus 鲁棒性测试（七类扰动）LIBERO-Plus Robustness Test (Seven Kinds of Perturbation)
方法Method
LIBERO-Plus 平均成功率LIBERO-Plus average success rate
OpenVLA
16.3%
π0
56.3%
VLA-Adapter
59.0%
MergeVLA（合并后）MergeVLA (merged)
62.5%
在颜色变化、光照变化、视角偏移、指令改写等七类分布外扰动下，MergeVLA 全面超越所有基线，展示出优异的泛化能力。Under seven kinds of out-of-distribution perturbation, including color change, lighting change, viewpoint shift and instruction rewriting, MergeVLA surpasses all baselines across the board, demonstrating excellent generalization.
LIBERO-Plus 的七类视觉与语言扰动示例，用于评估模型在分布外场景下的鲁棒性。Examples of the seven kinds of visual and language perturbations in LIBERO-Plus, used to evaluate model robustness in out-of-distribution scenarios.
Setting A（相同任务，不同形态）Setting A (same task, different embodiments)
88.0%
88.7%（HL-1→L 未合并）88.7% (HL-1→L unmerged)
Setting B（不同任务，不同形态）Setting B (different tasks, different embodiments)
76.0%
70.7%（HL-2→L 未合并）70.7% (HL-2→L unmerged)
Setting A 中合并模型（88.7%）已超越单任务基线（88.0%）；Setting B 中跨任务跨形态的组合挑战导致性能有所下降，但仍验证了跨形态泛化能力。In Setting A the merged model (88.7%) already surpasses the single-task baseline (88.0%); in Setting B the combined challenge of cross-task and cross-embodiment causes some degradation, yet cross-embodiment generalization is still verified.
RoboTwin 跨形态实验：三种不同双臂机器人形态上测试 MergeVLA 的多任务合并效果。RoboTwin cross-embodiment experiments: the multi-task merging of MergeVLA is tested on three different dual-arm robot embodiments.
真实机器人 SO-101 验证Real-Robot Validation on SO-101
任务Task
MergeVLA TIES 成功率MergeVLA TIES success rate
Pick & Place（含颜色分布外测试）Pick & Place (incl. out-of-distribution color test)
90.0%
Push Cube
90.0%
Stack Cube
90.0%
平均Average
90.0%
消融实验Ablation Studies
掩码比率 λ 的影响：λ 过小（如 0.2）导致几乎 0% 成功率，λ 在 [0.6, 0.9] 区间时性能稳定在 70% 以上，表明方法对该超参数有一定鲁棒性。Effect of the mask ratio λ: too small a λ (e.g. 0.2) leads to a nearly 0% success rate, while λ in the interval [0.6, 0.9] keeps performance stably above 70%, indicating that the method is fairly robust to this hyperparameter.
路由子空间选择：仅用 K 投影子空间路由时 LIBERO 平均 53.6%，K+V 组合为 65.1%，仅用 V 达到最优 89.7%。Choice of routing subspace: routing with the K projection subspace alone gives a LIBERO average of 53.6%, the K+V combination gives 65.1%, and V alone reaches the best 89.7%.
Expert head 深度：跨任务合并时保留最后 1 个块（HL-1→L）足以应对同形态场景，跨形态时则需保留最后 2 个块（HL-2→L）。Expert head depth: for cross-task merging, keeping the last 1 block (HL-1→L) is enough in the same-embodiment scenario, whereas cross-embodiment requires keeping the last 2 blocks (HL-2→L).
04局限性Limitations
注：论文未设置显式的 Limitations 章节。以下各点部分源自论文对实验结果的讨论（stated），部分为从设计中推断（inferred）。Note: The paper has no explicit Limitations section. Some of the points below come from the paper's discussion of its experimental results (stated), others are inferred from its design (inferred).
内存开销随任务数线性增长（inferred）Memory overhead grows linearly with the number of tasks (inferred)
MergeVLA 需要为每个任务维护一套独立的二值任务掩码和未合并的 expert head 块。
随任务数 M 增加，存储的掩码和 head 数量线性增长，对大规模多任务场景的可扩展性构成限制。MergeVLA has to maintain a separate binary task mask and unmerged expert head blocks for every task. As the number of tasks M grows, the number of stored masks and heads grows linearly, which limits scalability to large-scale multi-task scenarios.
跨任务跨形态合并性能下降（stated）Merging performance degrades across tasks and embodiments (stated)
在 RoboTwin Setting B（不同任务 + 不同机器人形态）中，MergeVLA 合并后的平均成功率（70.7%）低于单任务基线（76.0%）。
论文指出此设置需保留更多层（HL-2→L）未合并，降低了模型压缩收益。
对于需要双臂协调的 handover 类任务，跨任务跨形态的挑战尤为突出。In RoboTwin Setting B (different tasks + different robot embodiments), the average success rate of merged MergeVLA (70.7%) is below the single-task baseline (76.0%). The paper points out that this setting requires keeping more layers (HL-2→L) unmerged, which reduces the model-compression benefit. For handover-style tasks that need dual-arm coordination, the cross-task cross-embodiment challenge is particularly pronounced.
主干模型规模受限（inferred）The backbone model scale is limited (inferred)
主要实验基于 Qwen2.5-0.5B 的小型 VLM 骨干网络，对更大规模 VLM 骨干（如 7B+）的扩展性尚未验证。
更大规模模型的 selfish 参数分布和 action expert 合并特性可能有所不同。The main experiments are based on a small VLM backbone from Qwen2.5-0.5B; scalability to larger VLM backbones (such as 7B+) has not been verified. The distribution of selfish parameters and the merging behavior of the action expert may differ for larger models.
路由器依赖任务子空间可区分性（inferred）The router depends on task subspaces being distinguishable (inferred)
测试时任务路由器通过 V projection 的 SVD 主成分子空间区分任务，
当任务在视觉观测上高度相似（如同一物体的不同操作）时，子空间区分度可能下降，影响路由准确率。The test-time task router distinguishes tasks through the SVD principal subspace of the V projection; when tasks are highly similar in their visual observations (for instance different manipulations of the same object), subspace separability may drop and routing accuracy suffers.
