Moo Jin Kim · Karl Pertsch · Siddharth Karamcheti · Ted Xiao · Ashwin Balakrishna · Suraj Nair · Rafael Rafailov · Chelsea Finn · Sergey Levine · Percy Liang 等 18 人（Stanford / Google DeepMind / MIT / TRI）and 18 others (Stanford / Google DeepMind / MIT / TRI)
OpenVLA 是一个 70 亿参数的开源视觉-语言-动作模型，在 970k 真实机器人演示上训练。它在 BridgeData V2 基准上以 71.3% 的任务成功率超越闭源 RT-2-X（55B 参数）16.5%，同时支持在消费级 GPU 上通过 LoRA 进行参数高效微调，将显存需求从 163 GB 降至 60 GB。OpenVLA is an open-source vision-language-action model with seven billion parameters, trained on 970k real-world robot demonstrations. On the BridgeData V2 benchmark it reaches a 71.3% task success rate, outperforming the closed-source RT-2-X (55B parameters) by 16.5%, while supporting parameter-efficient fine-tuning with LoRA on consumer-grade GPUs, cutting memory requirements from 163 GB to 60 GB.
当前最强的视觉-语言-动作（VLA）模型（如 RT-2-X）均为闭源，架构、训练数据、部署方法均不公开；而开源社区的替代方案又缺乏在消费级硬件上高效适配新机器人和任务的实用手段。OpenVLA 旨在填补这一空白。The strongest current vision-language-action (VLA) models, such as RT-2-X, are all closed-source: their architectures, training data and deployment methods are undisclosed; meanwhile, open-source alternatives lack practical means for efficiently adapting to new robots and tasks on consumer-grade hardware. OpenVLA aims to fill this gap.
"We present OpenVLA, a 7B-parameter open-source VLA trained on 970k real-world robot demonstrations from the Open X-Embodiment dataset, and show that it outperforms RT-2-X (55B parameters) by 16.5% on manipulation tasks while requiring 7× fewer parameters."
左：OpenVLA 总览图，展示输入（图像 + 语言指令）→ 双视觉编码器（DINOv2 ∥ SigLIP）→ MLP 投影→ Llama 2 → 动作 token 输出的完整流程。右：在 BridgeData V2 和 Google Robot 平台上的成功率对比。Left: OpenVLA overview, showing the full pipeline from inputs (image + language instruction) → dual vision encoders (DINOv2 ∥ SigLIP) → MLP projection → Llama 2 → action token outputs. Right: success-rate comparison on the BridgeData V2 and Google Robot platforms.
970kOpen X-Embodiment 真实演示轨迹Real-world demonstration trajectories from Open X-Embodiment
+16.5%超越 RT-2-X（BridgeData V2 成功率）Improvement over RT-2-X (BridgeData V2 success rate)
7B参数量（RT-2-X 参数的 1/7）Parameter count (1/7 of RT-2-X's parameters)
1.4%LoRA 仅需微调该比例参数即可达全量效果Share of parameters LoRA must tune to match full fine-tuning
现有方法的两大痛点Two Pain Points of Existing Approaches
封闭生态Closed ecosystem
RT-2-X 等顶尖 VLA 不公开权重与训练代码，研究者无法复现、改进或深入理解其决策机制。学术界和工业界被迫从头开发，资源浪费严重。Top-tier VLAs such as RT-2-X release neither weights nor training code, so researchers cannot reproduce, improve or deeply understand their decision-making mechanisms. Academia and industry are forced to build from scratch, at a heavy cost in wasted resources.
部署门槛高High deployment barrier
现有 VLA 论文几乎没有讨论如何高效地将模型适配到新机器人、新场景、新任务。全量微调需要多台 A100，普通实验室无法承受。量化和 LoRA 等方法在 VLA 场景下的效果尚未系统研究。Existing VLA papers hardly discuss how to efficiently adapt a model to new robots, new scenes and new tasks. Full fine-tuning requires several A100 GPUs, which is out of reach for an ordinary lab. The effectiveness of methods such as quantization and LoRA has not been systematically studied in the VLA setting.
02方法Method
OpenVLA 将机器人控制重新表述为视觉-语言任务：给定图像观测和语言指令，模型以自回归 token 序列方式预测连续动作。核心创新在于双视觉编码器融合、动作 token 化方案，以及基于 Open X-Embodiment 的大规模多机器人训练。OpenVLA reformulates robot control as a vision-language task: given an image observation and a language instruction, the model predicts continuous actions as an autoregressive token sequence. The core innovations are the fusion of dual vision encoders, the action tokenization scheme, and large-scale multi-robot training on Open X-Embodiment.
OpenVLA 三段式架构：① 视觉编码器（DINOv2 + SigLIP 特征拼接）捕获空间细节与语义理解；② 2 层 MLP 投影器将视觉特征映射至语言空间；③ Llama 2 7B 语言主干自回归生成动作 token。OpenVLA's three-stage architecture: ① the vision encoders (concatenated DINOv2 + SigLIP features) capture spatial detail and semantic understanding; ② a 2-layer MLP projector maps visual features into the language space; ③ the Llama 2 7B language backbone autoregressively generates action tokens.
双视觉编码器融合Dual Vision Encoder Fusion
OpenVLA 将 DINOv2（擅长空间推理与细粒度定位）和 SigLIP（擅长语义理解与语言对齐）的特征在 token 维度直接拼接，输入 224×224 像素图像。实验证明：更高分辨率（如 384×384）在机器人操作任务上没有带来性能提升，却使训练时间增加了 3 倍。OpenVLA directly concatenates, along the token dimension, the features of DINOv2 (strong at spatial reasoning and fine-grained localization) and SigLIP (strong at semantic understanding and language alignment), taking 224×224 pixel images as input. Experiments show that higher resolution (e.g. 384×384) brings no performance gain on robot manipulation tasks, yet increases training time threefold.
动作离散化与 Token 化Action Discretization and Tokenization
对每个动作维度，使用训练数据的 1st–99th 百分位范围均匀离散化为 256 个 bin。为最小化对语言知识的干扰，将 Llama 2 词表中使用频率最低的 256 个 token 直接覆写为动作 token，并在交叉熵损失中只计算动作 token 的梯度。For each action dimension, the 1st–99th percentile range of the training data is uniformly discretized into 256 bins. To minimize interference with linguistic knowledge, the 256 least frequently used tokens of the Llama 2 vocabulary are directly overwritten as action tokens, and the cross-entropy loss takes gradients only on action tokens.
训练数据与课程Training Data and Curriculum
数据源：Open X-Embodiment 数据集，经严格过滤后保留 970k 条轨迹Data source: the Open X-Embodiment dataset, leaving 970k trajectories after strict filtering
过滤准则：第三视角相机、单臂操作、非全零动作序列Filtering criteria: third-person camera view, single-arm manipulation, action sequences that are not all zeros
混合权重沿用 Octo 的 embodiment/task/scene 均衡策略；DROID 数据集因动作 token 准确率过低在训练后期移除Mixture weights follow Octo's embodiment/task/scene balancing strategy; the DROID dataset was dropped late in training because its action token accuracy was too low
训练基础：Prismatic VLM（预训练于视觉-语言理解任务）Training base: Prismatic VLM (pre-trained on vision-language understanding tasks)
超参数：固定学习率 2e-5（无 warmup），batch size 2048，共 27 个 epoch；64 块 A100 训练 14 天（21,500 A100-hours）Hyperparameters: constant learning rate 2e-5 (no warmup), batch size 2048, 27 epochs in total; trained on 64 A100 GPUs for 14 days (21,500 A100-hours)
针对将 OpenVLA 适配至新机器人设置（如 Franka Emika Panda），作者系统评估了 LoRA、Sandwich adapter 及全量微调方案。LoRA r=32 仅需训练 97.6M 参数（全量的 1.4%），在单块 A100 上完成微调（10–15 小时），显存需求从 163.3 GB 降至 59.7 GB，且性能与全量微调持平。For adapting OpenVLA to a new robot setup (e.g. Franka Emika Panda), the authors systematically evaluate LoRA, a Sandwich adapter and full fine-tuning. LoRA r=32 trains only 97.6M parameters (1.4% of the full model), completes fine-tuning on a single A100 (10–15 hours) and lowers memory demand from 163.3 GB to 59.7 GB, while matching the performance of full fine-tuning.
量化推理Quantized Inference
使用 bitsandbytes 库的 int4 量化，在保持 71.9% 成功率（与 bfloat16 的 71.3% 相当）的同时，将推理显存从 16.8 GB 压缩至 7.0 GB，使 OpenVLA 可在单张消费级 GPU 上运行。With int4 quantization from the bitsandbytes library, the success rate is kept at 71.9% (on par with 71.3% at bfloat16) while inference memory is compressed from 16.8 GB to 7.0 GB, letting OpenVLA run on a single consumer-grade GPU.
03实验Experiments
在三个真实机器人平台（WidowX、Google Robot、Franka）上评估，对比基线包括 RT-2-X、RT-1-X、Octo 和 Diffusion Policy。评测维度涵盖视觉泛化（新背景、干扰物）、运动泛化（新位置）、物理泛化（新大小/形状）和语义泛化（新物体、新指令）。Evaluation is carried out on three real robot platforms (WidowX, Google Robot, Franka), with baselines including RT-2-X, RT-1-X, Octo and Diffusion Policy. The evaluated dimensions cover visual generalization (new backgrounds, distractors), motion generalization (new positions), physical generalization (new sizes/shapes) and semantic generalization (new objects, new instructions).
BridgeData V2（WidowX 机器人）上 29 个任务的成功率对比。OpenVLA 在视觉、运动、物理、语义四类泛化场景下均优于 RT-2-X（55B），且仅使用 7B 参数。Success-rate comparison over 29 tasks on BridgeData V2 (WidowX robot). OpenVLA beats RT-2-X (55B) in all four generalization categories — visual, motion, physical and semantic — while using only 7B parameters.
模型Model
BridgeData V2 成功率BridgeData V2 Success Rate
Google Robot 成功率Google Robot Success Rate
参数量Parameters
RT-2-X（闭源）RT-2-X (closed-source)
54.8%
~75%
55B
RT-1-X
—
<75%
35M
Octo
<54.8%
<75%
93M
OpenVLA（本文）OpenVLA (ours)
71.3% ±4.8%
75.0%
7B
Google Robot 平台评估结果，OpenVLA 在多个任务类别下与 RT-2-X 表现相当，并在语言泛化任务中展现出更强的指令跟随能力。Evaluation results on the Google Robot platform: OpenVLA is on par with RT-2-X across several task categories and shows stronger instruction-following ability on language generalization tasks.
数据高效微调：适配新机器人设置Data-Efficient Fine-Tuning: Adapting to New Robot Setups
在 Franka Emika Panda 机器人 7 个任务（10–150 条演示）上的微调对比。OpenVLA 微调后在所有任务上均达到 ≥50% 成功率，而竞争基线方法在部分任务上接近 0%。Fine-tuning comparison on 7 tasks (10–150 demonstrations) with the Franka Emika Panda robot. After fine-tuning, OpenVLA reaches ≥50% success on every task, while competing baselines fall to nearly 0% on some of them.
视觉编码器必须微调：冻结视觉编码器使性能显著下降，因为预训练 VLM 的特征缺乏机器人控制所需的细粒度空间信息。The vision encoder must be fine-tuned: freezing it degrades performance markedly, because the features of the pre-trained VLM lack the fine-grained spatial information required for robot control.
Prismatic 优于 LLaVA ~10%：在多物体语言定位任务上，基于 Prismatic 的 VLM 主干优于 LLaVA-1.5 约 10%。Prismatic beats LLaVA by ~10%: on multi-object language grounding tasks, the Prismatic-based VLM backbone outperforms LLaVA-1.5 by about 10%.
数据量是关键：OpenVLA 的 970k 演示（含精心过滤）远超 RT-2-X 的 350k，这是性能领先的重要因素之一。Data scale is decisive: OpenVLA's 970k demonstrations (carefully filtered) far exceed the 350k of RT-2-X, one of the important factors behind its performance lead.
推理吞吐：bfloat16 精度下在 RTX 4090 上约 6 Hz；int4 量化在 Ada Lovelace 架构 GPU 上因 kernel 优化实现更高吞吐。Inference throughput: about 6 Hz on an RTX 4090 at bfloat16 precision; int4 quantization reaches higher throughput on Ada Lovelace GPUs thanks to kernel optimizations.
不同 GPU 和量化精度下 OpenVLA 的推理吞吐（Hz）对比。int4 量化在支持的 GPU 上可显著提升推理速度。Comparison of OpenVLA inference throughput (Hz) across different GPUs and quantization precisions. int4 quantization noticeably speeds up inference on supported GPUs.
04局限性Limitations
Note：以下局限性均由作者在论文中明确指出（stated by the authors）。Note: all limitations below are explicitly stated by the authors in the paper.
单帧图像输入，缺乏时序历史Single-frame image input, no temporal history
OpenVLA 当前仅支持单张图像作为观测输入，不支持观测历史序列，也不支持 proprioceptive（本体感知）信息（如关节角度）。这使其无法处理需要时序推理的任务（如 ALOHA 双臂操作），也与 Octo 等支持多模态输入的策略存在差距。OpenVLA currently accepts only a single image as observation input; it supports neither observation history sequences nor proprioceptive information (such as joint angles). This makes it unable to handle tasks requiring temporal reasoning (e.g. bimanual ALOHA manipulation) and leaves a gap to policies such as Octo that accept multimodal inputs.
受 Llama 2 自回归解码的限制，OpenVLA 的推理速度约为 6 Hz，远低于 ALOHA 等高频控制系统所需的 50 Hz。这限制了其在需要快速响应的灵巧操作任务中的应用。Constrained by the autoregressive decoding of Llama 2, OpenVLA runs at about 6 Hz, far below the 50 Hz required by high-frequency control systems such as ALOHA. This limits its use in dexterous manipulation tasks that demand fast responses.
任务成功率通常低于 90%Task success rates are usually below 90%
尽管在多个基准上超越现有方法，OpenVLA 在大多数任务上的成功率仍低于 90%，离真实部署所需的鲁棒性存在差距。Although it surpasses existing methods on several benchmarks, OpenVLA's success rate on most tasks is still below 90%, short of the robustness needed for real-world deployment.
多项设计问题尚待探索Several design questions remain to be explored
作者指出以下问题未被充分研究：① 基础 VLM 规模（7B 是否最优）；② 与 Internet 数据联合训练的效果；③ 视觉特征选择的最优方案（DINOv2 + SigLIP 组合是否是最优选择）。这些问题留待未来工作解答。The authors point out the following insufficiently studied questions: ① the scale of the base VLM (is 7B optimal?); ② the effect of co-training with Internet data; ③ the best scheme for choosing visual features (is the DINOv2 + SigLIP combination the optimal choice?). These are left to future work.
