RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

01 动机 MotivationMotivation

现有机器人控制方法往往将视觉-语言模型（VLM）仅用于高层规划，底层动作执行依赖独立的控制器，无法享受互联网规模预训练带来的丰富语义知识。另一方面，直接用海量机器人数据追赶 VLM 的训练量几乎不可能实现。那么，能否将大规模预训练 VLM 直接整合进闭环机器人控制，从而提升泛化能力并使突现式语义推理成为可能？Existing robotic control methods typically use vision-language models (VLM) only for high-level planning, while low-level action execution relies on a separate controller, so they cannot benefit from the rich semantic knowledge of internet-scale pre-training. On the other hand, matching the training scale of VLMs with massive robot data alone is close to infeasible. Can large-scale pre-trained VLMs therefore be integrated directly into closed-loop robotic control, so as to improve generalization and make emergent semantic reasoning possible?

"can large pretrained vision-language models be integrated directly into low-level robotic control to boost generalization and enable emergent semantic reasoning?"

**图 1：RT-2 总览。**机器人动作被表示为另一种"语言"，编码为文本 token 后与互联网规模的视觉-语言数据集联合训练。推理时，文本 token 被解码为机器人动作，实现闭环控制。这一设计允许视觉-语言主干的预训练权重直接用于策略学习，将 VLM 的泛化、语义理解与推理能力迁移到机器人控制中。**Figure 1: RT-2 overview.** Robot actions are represented as another "language", encoded as text tokens and trained jointly with internet-scale vision-language datasets. At inference time the text tokens are decoded back into robot actions, achieving closed-loop control. This design lets the pre-trained weights of the vision-language backbone be used directly for policy learning, transferring the generalization, semantic understanding and reasoning abilities of the VLM into robotic control.

6k+真实机器人评估轨迹Real-robot evaluation trajectories

~2×泛化任务相对 RT-1/MOO 的提升Improvement on generalization tasks over RT-1/MOO

>3×突现能力相对最优基线的提升Improvement in emergent capabilities over the best baseline

55B最大模型参数量（PaLI-X-55B）Parameter count of the largest model (PaLI-X-55B)

02 方法 MethodMethod

RT-2 的核心思路极为简洁：将机器人动作表示为文本 token，并将其以与自然语言 token 完全相同的方式加入 VLM 的训练集，实现"动作即语言"的统一输出空间。在此基础上，通过联合微调（co-fine-tuning）同时使用机器人轨迹数据与原始互联网视觉-语言数据，防止灾难性遗忘并保留 VLM 预训练知识。The core idea of RT-2 is remarkably simple: represent robot actions as text tokens and add them to the VLM training set in exactly the same way as natural-language tokens, yielding a unified output space in which "action is language". On top of this, co-fine-tuning uses robot trajectory data together with the original internet vision-language data, preventing catastrophic forgetting and preserving the pre-trained knowledge of the VLM.

动作表示为文本 Token（Action Tokenization）Actions as Text Tokens (Action Tokenization)

机器人动作空间包含 6-DoF 末端执行器位移/旋转、夹爪开合和终止命令，共 8 个维度。连续维度被均匀离散化为 256 个 bin，整个动作向量拼接为一串整数字符串，例如："1 128 91 241 5 101 127"。对于 PaLI-X，直接复用表示整数的已有 token；对于 PaLM-E，则将 256 个使用频率最低的 token 覆写为动作词表——这本质上是一种 symbol tuning。VLM 以标准 VQA 格式接受输入："Q: what action should the robot take to [task instruction]? A:"，输出对应动作 token 串。The robot action space covers 6-DoF end-effector translation/rotation, gripper opening and a termination command, 8 dimensions in total. The continuous dimensions are uniformly discretized into 256 bins, and the whole action vector is concatenated into a string of integers, for example "1 128 91 241 5 101 127". For PaLI-X the existing tokens that represent integers are reused directly; for PaLM-E the 256 least frequently used tokens are overwritten to form the action vocabulary — essentially a form of symbol tuning. The VLM takes its input in the standard VQA format "Q: what action should the robot take to [task instruction]? A:" and outputs the corresponding action token string.

联合微调（Co-Fine-Tuning）Co-Fine-Tuning

实验表明，仅用机器人数据微调（naive fine-tuning）会导致 VLM 遗忘预训练语义知识，泛化能力下降。RT-2 的关键训练细节是在每个 batch 中混合机器人数据与原始互联网视觉-语言数据，并通过上采样机器人数据集来平衡比例。这使模型在低层次机器人动作与抽象视觉概念之间保持双向兼容。Experiments show that fine-tuning on robot data alone (naive fine-tuning) makes the VLM forget its pre-trained semantic knowledge and degrades generalization. The key training detail of RT-2 is to mix robot data with the original internet vision-language data within every batch, balancing the ratio by up-sampling the robot dataset. This keeps the model compatible in both directions, between low-level robot actions and abstract visual concepts.

实时推理（Real-Time Inference）Real-Time Inference

55B 参数的 RT-2-PaLI-X 无法在桌面 GPU 上实时运行。论文采用多 TPU 云服务部署模型，机器人通过网络查询该服务。RT-2-PaLI-X-55B 控制频率为 1–3 Hz；5B 版本约 5 Hz。据作者所述，这是目前直接用于闭环机器人控制的最大模型，超过此前同类方法达一个数量级以上。The 55B-parameter RT-2-PaLI-X cannot run in real time on a desktop GPU. The paper deploys the model on a multi-TPU cloud service that the robot queries over the network. RT-2-PaLI-X-55B reaches a control frequency of 1–3 Hz; the 5B version about 5 Hz. According to the authors this is the largest model so far used directly for closed-loop robotic control, exceeding earlier comparable methods by more than an order of magnitude.

思维链推理扩展（Chain-of-Thought）Chain-of-Thought Reasoning Extension

在 RT-2-PaLM-E 上，仅用数百步梯度更新的数据增强，即可让模型先以自然语言生成"Plan"步骤，再输出动作 token，例如："Instruction: I'm hungry. Plan: pick rxbar chocolate. Action: 1 128 124 136 121 158 111 255."这一扩展使模型能应对更复杂的多步语义推理指令。On RT-2-PaLM-E, data augmentation with only a few hundred gradient steps is enough to let the model first generate a "Plan" step in natural language and then output action tokens, for example "Instruction: I'm hungry. Plan: pick rxbar chocolate. Action: 1 128 124 136 121 158 111 255." This extension enables the model to handle more complex multi-step semantic reasoning instructions.

03 实验 ExperimentsExperiments

实验在真实 7-DoF 移动机械臂上进行，共完成约 6,000 条评估轨迹。基线包括：RT-1（35M 参数 Transformer）、VC-1（预训练表征 + RT-1 骨干）、R3M（预训练表征）、MOO（VLM 生成语义地图后接 RT-1）。评估分为两大类：(1) 在见过/未见过的物体、背景、环境中的泛化；(2) 突现能力（symbol understanding / reasoning / human recognition）。Experiments are carried out on a real 7-DoF mobile manipulator, completing about 6,000 evaluation trajectories in total. Baselines include RT-1 (a 35M-parameter Transformer), VC-1 (pre-trained representation + RT-1 backbone), R3M (pre-trained representation) and MOO (a VLM that produces a semantic map feeding RT-1). The evaluation falls into two categories: (1) generalization to seen/unseen objects, backgrounds and environments; (2) emergent capabilities (symbol understanding / reasoning / human recognition).

generalization results — **图 4：泛化性能对比。**在已见任务上，RT-2 与 RT-1 表现相当；在未见物体、背景、环境的泛化评估中，RT-2（PaLM-E 和 PaLI-X 两版本）平均约为 RT-1 和 MOO 的 **~2×**，约为 VC-1/R3M 的 **~6×**。较难场景中 RT-2-PaLM-E 优于 RT-2-PaLI-X，较易场景则相反，平均相近。**Figure 4: Generalization comparison.** On seen tasks RT-2 performs on par with RT-1; on generalization evaluations over unseen objects, backgrounds and environments, RT-2 (both the PaLM-E and PaLI-X versions) averages roughly **~2×** RT-1 and MOO, and about **~6×** VC-1/R3M. RT-2-PaLM-E outperforms RT-2-PaLI-X in harder scenarios and the reverse holds in easier ones, with comparable averages.

Language-Table 仿真基准Language-Table Simulation Benchmark

模型Model	Language-Table 成功率Language-Table success rate
BC-Zero (Jang et al., 2021)	72 ± 3
RT-1 (Brohan et al., 2022)	74 ± 13
LAVA (Lynch et al., 2022)	77 ± 4
RT-2-PaLI-3B（本文）RT-2-PaLI-3B (ours)	90 ± 10

在开源 Language-Table 仿真环境中，RT-2-PaLI-3B 达到 90 ± 10% 成功率，远超此前最佳的 LAVA（77 ± 4%）。In the open-source Language-Table simulation environment, RT-2-PaLI-3B reaches a 90 ± 10% success rate, far beyond the previous best LAVA (77 ± 4%).

emergent capabilities — **图 6：突现能力与消融实验。**左图：在 symbol understanding、reasoning、human recognition 三类突现任务上，RT-2-PaLI-X 平均成功率超过最优基线 RT-1 的 3×。右图：消融实验显示，从零开始训练性能极差；co-fine-tuning 优于仅用机器人数据微调；模型规模越大泛化越好（55B > 5B）。**Figure 6: Emergent capabilities and ablations.** Left: on the three categories of emergent tasks — symbol understanding, reasoning and human recognition — the average success rate of RT-2-PaLI-X exceeds that of the best baseline RT-1 by 3×. Right: the ablations show that training from scratch performs very poorly; co-fine-tuning beats fine-tuning on robot data alone; and the larger the model, the better the generalization (55B > 5B).

Chain-of-Thought 推理示例Chain-of-Thought Reasoning Examples

chain of thought rollouts — **图 7：思维链推理展示。**RT-2 先生成自然语言"Plan"（如"pick rxbar chocolate"、"Rocks"），再输出动作 token 执行。示例任务包括"我饿了"→拿出能量棒、"帮我找个能敲钉子的东西"→选择石块。这些能力均未在机器人演示数据中出现，来源于 VLM 的网络知识迁移。**Figure 7: Chain-of-thought rollouts.** RT-2 first generates a natural-language "Plan" (such as "pick rxbar chocolate" or "Rocks") and then outputs action tokens to execute it. Example tasks include "I'm hungry" → fetch an energy bar, and "find me something to hammer a nail with" → pick up a rock. None of these abilities appear in the robot demonstration data; they come from the transfer of web knowledge held by the VLM.

消融分析Ablation Analysis

针对 RT-2-PaLI-X，论文比较了三种训练方案：从零开始训练（scratch）、仅机器人数据微调（fine-tune）、联合微调（co-fine-tune）；以及两种规模：5B 和 55B。主要结论：For RT-2-PaLI-X the paper compares three training schemes — training from scratch, fine-tuning on robot data only, and co-fine-tuning — as well as two scales, 5B and 55B. The main conclusions:

从零训练即便是 5B 规模也泛化极差，因此跳过 55B scratch 评估。Training from scratch generalizes very poorly even at the 5B scale, so the 55B scratch evaluation was skipped.
Co-fine-tuning 在所有规模上均优于 fine-tuning，证明保留原始 VLM 数据防止遗忘的重要性。Co-fine-tuning outperforms fine-tuning at every scale, demonstrating how important it is to retain the original VLM data in order to prevent forgetting.
模型规模增大（5B → 55B）始终带来更好的泛化性能。Increasing model scale (5B → 55B) consistently brings better generalization performance.
PaLM-E 在数学推理子任务上优于 PaLI-X，归因于 PaLM-E 的预训练数据混合中数学内容更多。PaLM-E outperforms PaLI-X on the mathematical reasoning subtasks, attributed to the larger amount of mathematical content in the pre-training data mixture of PaLM-E.

04 局限性 LimitationsLimitations

说明：以下局限性均为论文第 5 节（Limitations）中作者明确陈述的内容（stated）。Note: All the limitations below are explicitly stated by the authors in Section 5 (Limitations) of the paper.

物理技能受限于机器人演示数据分布Physical skills are bounded by the distribution of the robot demonstration data

RT-2 虽然能以新方式部署已有操作技能，但不会从 VLM 预训练中习得全新的运动能力。模型的物理技能仍局限于机器人数据所覆盖的技能分布（如拾取、放置等）。作者指出，一个有前景的方向是通过人类视频等新数据范式扩展技能多样性。Although RT-2 can deploy existing manipulation skills in new ways, it does not acquire entirely new motor capabilities from VLM pre-training. The physical skills of the model remain confined to the skill distribution covered by the robot data (picking, placing and so on). The authors point out that a promising direction is to expand skill diversity through new data paradigms such as human videos.

推理计算成本高，难以满足高频控制需求Inference is computationally expensive and hard to reconcile with high-frequency control

55B 参数模型需要多 TPU 云服务支撑，控制频率仅 1–3 Hz，无法满足要求高频控制的任务。论文指出，量化（quantization）和蒸馏（distillation）是值得探索的方向，以期在更低成本硬件上实现更高频率推理。The 55B-parameter model needs a multi-TPU cloud service to support it, and its control frequency of only 1–3 Hz cannot meet the needs of tasks that require high-frequency control. The paper notes that quantization and distillation are directions worth exploring, in the hope of higher-frequency inference on lower-cost hardware.

可用 VLM 基座数量有限Few VLM backbones are available

构建 VLA 模型需要对 VLM 进行微调，而目前可公开微调的大型 VLM 数量非常有限。论文表达了对更多开源 VLM（如 LLaVA）出现以及商业模型开放微调 API 的期待，认为这是推广 VLA 方法的必要前提。Building a VLA model requires fine-tuning a VLM, yet the number of large VLMs that can currently be fine-tuned publicly is very limited. The paper expresses its hope for more open-source VLMs (such as LLaVA) to appear and for commercial models to open fine-tuning APIs, regarding this as a necessary precondition for spreading the VLA approach.