What Matters in Building Vision-Language-Action Models for Generalist Robots

01 Motivation

大型视觉语言模型（VLM）为通用机器人策略带来了新的可能，但如何将其转化为高效的 VLA 模型，业界缺乏系统性的实证研究——现有工作各自选择不同的 backbone、架构和数据策略，难以进行公平比较，也无法给出可推广的设计原则。Large vision-language models (VLMs) open up new possibilities for generalist robot policies, but the field lacks a systematic empirical study of how to turn them into effective VLA models—existing works each pick their own backbone, architecture and data strategy, which makes fair comparison hard and yields no generalizable design principles.

"We observe a significant gap between the performance of VLAs and expected performance of generalist robots, while the community lacks a systematic study covering all key design factors."

teaser — 图 1：学习型机器人策略的分类与比较。左侧展示了从 imitation learning 到 VLA 的演进路径；右侧展示了本文研究覆盖的三大设计维度——VLM backbone、架构制定方式（formulation）、跨机体数据（cross-embodiment data），并标注各 design choice 对最终性能的影响程度。Figure 1: Taxonomy and comparison of learned robot policies. The left side shows the evolution from imitation learning to VLA; the right side shows the three design dimensions covered by this study—VLM backbone, architecture formulation, and cross-embodiment data—annotated with how strongly each design choice affects final performance.

600+精心设计的对照实验carefully designed controlled experiments

8+VLM backbone 横向对比VLM backbones compared side by side

4策略架构类型policy architecture types

+30.3%5-task 连续执行绝对提升（vs. GR-1）absolute gain on 5-task sequential execution (vs. GR-1)

研究聚焦三个核心问题：The study focuses on three core questions:

哪种 VLM backbone 最适合机器人策略？——预训练质量与规模的影响。Which VLM backbone is best suited to robot policies?—the impact of pre-training quality and scale.
如何制定 VLA 架构？——one-step / interleaved / policy-head，离散 vs. 连续动作，训练目标（MSE+BCE、Flow Matching），以及 MoE 结构。How should the VLA architecture be formulated?—one-step / interleaved / policy-head, discrete vs. continuous actions, training objectives (MSE+BCE, Flow Matching), and the MoE structure.
何时、如何利用跨机体数据？——co-training 与 post-training 的区别。When and how should cross-embodiment data be used?—the difference between co-training and post-training.

图 2：构建 VLA 的三大关键要素。研究系统性地覆盖 backbone 选择、架构制定与跨机体数据整合三个维度，每个维度均设有多个对比变量。Figure 2: The three key ingredients of building a VLA. The study systematically covers three dimensions—backbone choice, architecture formulation and cross-embodiment data integration—each with several comparison variables.

02 Method

RoboVLMs 是一个高度模块化的实验框架，支持自由组合各类 VLM backbone 与策略架构。研究在统一的数据集和评测基准（CALVIN 仿真 + SimplerEnv + Kinova Gen3 真机）下进行所有对比实验，确保结论可靠。RoboVLMs is a highly modular experimental framework that supports free combination of VLM backbones and policy architectures. All comparative experiments are run under a unified dataset and evaluation benchmark (CALVIN simulation + SimplerEnv + a real Kinova Gen3 arm), which keeps the conclusions reliable.

图 3：VLA 架构的四种制定方式。(a) One-step Continuous：单步预测连续动作序列；(b) One-step Discrete：将动作离散化为 token，以文本生成方式预测；(c) Interleaved：在 observation token 与动作 token 之间交替编码历史信息；(d) Policy-head：VLM 提供单步特征表示，额外的 policy head（RNN/Transformer/Diffusion）建模历史动作依赖。其中 MoE（Mixture of Experts）结构可集成到各类架构中。Figure 3: Four ways of formulating a VLA architecture. (a) One-step Continuous: predict a continuous action sequence in a single step; (b) One-step Discrete: discretize actions into tokens and predict them as text generation; (c) Interleaved: encode history by alternating observation tokens and action tokens; (d) Policy-head: the VLM provides single-step feature representations and an extra policy head (RNN/Transformer/Diffusion) models the dependence on action history. The MoE (Mixture of Experts) structure can be integrated into any of these architectures.

VLM Backbone 选择Choice of VLM Backbone

实验涵盖 LLaVA、Flamingo、KosMos（8B）、PaliGemma（3B）等 8 种以上 backbone。研究发现，在大规模视觉-语言数据上充分预训练的模型（KosMos、PaliGemma）在机器人任务中表现"distinctively better"，而参数规模并不是决定性因素——3B 的 PaliGemma 与 8B 的 KosMos 表现相当。The experiments cover more than 8 backbones, including LLaVA, Flamingo, KosMos (8B) and PaliGemma (3B). The study finds that models thoroughly pre-trained on large-scale vision-language data (KosMos, PaliGemma) perform "distinctively better" on robot tasks, while parameter scale is not the decisive factor—the 3B PaliGemma performs on par with the 8B KosMos.

策略架构制定Policy Architecture Formulation

动作空间使用 7 维向量（6-DoF gripper pose + open/close 状态），连续动作归一化至 [-1,1]，离散动作均匀划分为 256 bins。四种架构中，policy-head + continuous action 在 CALVIN 上达到最高平均完成长度（Avg. Len. 4.49），同时在 zero-shot 泛化中展现出最强鲁棒性。训练目标方面，Flow Matching "slightly outperforms MSE+BCE in all experiments"，但差距不显著。MoE 结构在 zero-shot 设置下有助于泛化，但在已见场景中无额外增益。The action space is a 7-dimensional vector (6-DoF gripper pose + open/close state); continuous actions are normalized to [-1,1] and discrete actions are uniformly split into 256 bins. Among the four architectures, policy-head + continuous action reaches the highest average completed length on CALVIN (Avg. Len. 4.49) and is also the most robust under zero-shot generalization. As for training objectives, Flow Matching "slightly outperforms MSE+BCE in all experiments", but the gap is not significant. The MoE structure helps generalization in the zero-shot setting, yet brings no extra gain on seen scenarios.

跨机体数据整合策略Cross-Embodiment Data Integration Strategies

研究区分了 co-training（同时使用跨机体数据和领域内数据）与 post-training（先在跨机体数据上预训练，再用领域内数据微调）两种策略。实验表明，co-training 单独使用收益有限，而 post-training 在少样本（few-shot）场景中带来显著提升：单任务成功率提高 17.2%，平均完成任务数多 0.25。The study distinguishes two strategies: co-training (using cross-embodiment data and in-domain data at the same time) and post-training (pre-training on cross-embodiment data first, then fine-tuning with in-domain data). Experiments show that co-training used alone brings limited benefit, while post-training gives a clear boost in few-shot settings: the single-task success rate improves by 17.2% and the average number of completed tasks increases by 0.25.

03 Experiments

实验在 CALVIN 仿真基准（ABC→D zero-shot 泛化，最多 5 步连续执行）、SimplerEnv（WidowX+Bridge、Google Robot）以及真实 Kinova Gen3 机械臂（105 个操作任务，74K 轨迹）上全面评测。The evaluation is carried out on the CALVIN simulation benchmark (ABC→D zero-shot generalization, up to 5 consecutive steps), SimplerEnv (WidowX+Bridge, Google Robot) and a real Kinova Gen3 arm (105 manipulation tasks, 74K trajectories).

CALVIN 基准——Backbone 对比（Policy-head + Continuous Action）CALVIN Benchmark—Backbone Comparison (Policy-head + Continuous Action)

Backbone	1-task	2-task	3-task	4-task	5-task	Avg. Len.
LLaVA	0.873	0.678	0.506	0.376	0.275	2.71
Flamingo	0.964	0.896	0.824	0.740	0.662	4.09
PaliGemma (3B)	0.984	0.933	0.888	0.835	0.779	4.42
KosMos (8B)	0.967	0.930	0.899	0.865	0.826	4.49

SimplerEnv 对比——Google RobotSimplerEnv Comparison—Google Robot

方法Method	Pick Coke	Move Near	Open/Close	Overall
RT-1 (Converged)	0.960	0.900	0.730	0.630
OpenVLA-7b	0.270	0.030	0.356	0.219
RoboVLMs (Ours)	1.000	0.910	0.544	0.818

训练目标与 MoE 消融（PaliGemma，CALVIN Split ABC→D）Training Objective and MoE Ablation (PaliGemma, CALVIN Split ABC→D)

训练目标Training Objective	执行范式Execution Paradigm	MoE	1-task	5-task	Avg. Len.
Flow Matching	Chunk	✓	0.940	0.597	3.84
Flow Matching	Chunk	✗	0.910	0.573	3.68
MSE+BCE	Chunk	✗	0.933	0.688	4.04
Flow Matching	First	✗	0.898	0.544	3.56

图 4：RoboVLMs 在仿真与真实场景的整体实验结果，涵盖 CALVIN 多步执行、SimplerEnv 多任务以及真实 Kinova 机械臂的五类评测设置。最优 VLA 配置在所有真实世界评测中均达到最强表现，并展现了基线模型所不具备的自我纠错能力（self-correction）。Figure 4: Overall experimental results of RoboVLMs in simulation and the real world, covering CALVIN multi-step execution, SimplerEnv multi-task settings and five evaluation setups on the real Kinova arm. The best VLA configuration achieves the strongest performance in all real-world evaluations and exhibits a self-correction ability that the baseline models lack.

消融：数据规模效应Ablation: Effect of Data Scale

ablation cross-embodiment — 图 5：跨机体训练配置的消融。不同的 co-training / post-training 组合对 Bridge 环境少样本学习的影响：post-training 使单任务成功率从 44% 提升至 50%（Bridge），少样本成功率提升 17.2%。Figure 5: Ablation of cross-embodiment training configurations. Effect of different co-training / post-training combinations on few-shot learning in the Bridge environment: post-training raises the single-task success rate from 44% to 50% (Bridge) and improves the few-shot success rate by 17.2%.

数据规模消融（KosMos Policy-head）显示：从 0.1× 扩展至 1× CALVIN 数据，5-task 成功率从 17.6% 跃升至 82.6%；进一步扩展至 5× 仅带来边际增益（82.6% → 83.0%），说明更大的 VLM 具备更强的数据效率。The data-scale ablation (KosMos Policy-head) shows that scaling from 0.1× to 1× of the CALVIN data lifts the 5-task success rate from 17.6% to 82.6%, while scaling further to 5× brings only marginal gains (82.6% → 83.0%), indicating that a larger VLM has stronger data efficiency.

04 Limitations

Note: 本文没有专门的 Limitations 章节。以下各点均为从设计与实验中推断（inferred from the design），非作者明确陈述。This paper has no dedicated Limitations section. All points below are inferred from the design and the experiments, not explicitly stated by the authors.

Interleaved 架构仅适用于 decoder-only 模型The interleaved architecture applies only to decoder-only models

由于 attention masking 机制的限制，interleaved 历史建模只能应用于 decoder-only 结构的 VLM，不适用于 encoder-decoder 类模型，限制了该架构与更多 backbone 的组合可能。Because of the constraints of the attention masking mechanism, interleaved history modeling can only be applied to VLMs with a decoder-only structure and not to encoder-decoder models, which limits the backbones this architecture can be combined with.

跨机体数据收益在 full fine-tune 场景有限The benefit of cross-embodiment data is limited under full fine-tuning

实验显示，cross-embodiment pre-training 的增益主要体现在 few-shot 场景；在充足的领域内数据下，co-training 单独使用"does not consistently yield significant improvements in final performance"，说明跨机体数据的价值主要体现在数据稀缺时。Experiments show that the gain of cross-embodiment pre-training shows up mainly in few-shot settings; with sufficient in-domain data, co-training used alone "does not consistently yield significant improvements in final performance", indicating that the value of cross-embodiment data lies mainly in data-scarce regimes.

真实机器人测试局限于单一平台Real-robot testing is limited to a single platform

真实世界验证仅在 7-DoF Kinova Gen3（配 Robotiq 夹爪）上进行，结论能否迁移至其他机械臂、移动底盘或双臂系统尚未验证。Real-world validation is conducted only on a 7-DoF Kinova Gen3 (with a Robotiq gripper); whether the conclusions transfer to other arms, mobile bases or bimanual systems has not been verified.

计算成本随历史长度增长Computation cost grows with history length

Policy-head 与 interleaved 架构在处理长历史序列时计算开销显著增加，实际部署中的实时性约束未作充分讨论。The policy-head and interleaved architectures incur significantly higher computation when handling long history sequences, and the real-time constraints of actual deployment are not discussed in depth.