机器人学习长期受限于单一机器人、单一场景的小规模数据集。该工作汇聚来自21个机构、22种机器人的60个数据集（超过100万条真实轨迹），构建 Open X-Embodiment (OXE) 数据集，并在其上训练 RT-1-X 和 RT-2-X 两个模型，验证了跨机器人的正向迁移效果，其中 RT-2-X 在泛化任务上相比单一机器人训练取得约3倍提升。Robot learning has long been constrained by small-scale datasets collected on a single robot in a single scene. This work aggregates 60 datasets from 21 institutions covering 22 robot embodiments (more than 1000000 real-world trajectories) into the Open X-Embodiment (OXE) dataset, and trains two models on it, RT-1-X and RT-2-X, verifying positive cross-robot transfer; on generalization tasks RT-2-X attains roughly three times the performance of single-robot training.
Open X-EmbodimentRT-Xcross-embodiment transfer机器人学习robot learningvision-language-action model多机器人数据集multi-robot datasetRT-2-X迁移学习transfer learning
01动机Motivation
NLP 和计算机视觉领域通过在大规模、多样化数据集上预训练，实现了显著的泛化能力。而机器人学习领域仍停滞于"各自为政"的小规模数据集——每个实验室只训练自己机器人上的任务，无法从他人数据中受益。核心问题是：机器人学习能否像 NLP 一样，围绕通用模型整合，从而实现正向迁移？In NLP and computer vision, pretraining on large-scale, diverse datasets has produced remarkable generalization. Robot learning, by contrast, is still stuck with small, siloed datasets — every lab trains only on tasks for its own robot and cannot benefit from other people's data. The central question is: can robot learning, like NLP, consolidate around general-purpose models and thereby achieve positive transfer?
"Robotic learning datasets are often still narrow along some axes of variation, either focusing on a single environment, a single set of objects, or a narrow range of tasks."
图1：Open X-Embodiment 数据集概览。(a) 60个数据集在22种机器人上的分布；(b) 每种机器人的场景多样性；(c) 各机器人的轨迹数量（xArm 和 Google Robot 占主导）；(d)(e) 通过 PaLM 分析提取的技能和物体多样性。数据集涵盖527种技能和160,266个任务。Figure 1: Overview of the Open X-Embodiment dataset. (a) Distribution of the 60 datasets over 22 robot embodiments; (b) scene diversity for each robot; (c) trajectory counts per robot (xArm and Google Robot dominate); (d)(e) skill and object diversity extracted through PaLM analysis. The dataset covers 527 skills and 160,266 tasks.
22机器人种类robot embodiments
60聚合数据集aggregated datasets
1M+真实轨迹real-world trajectories
527技能类型skill types
02方法Method
该工作提出两个核心贡献：(1) 构建 Open X-Embodiment (OXE) 数据集，将60个异构机器人数据集统一转换为 RLDS 格式；(2) 在 OXE 上训练 RT-1-X 和 RT-2-X 两个模型，分别代表中等容量专用网络和大容量 vision-language-action 模型。This work makes two core contributions: (1) it builds the Open X-Embodiment (OXE) dataset, converting 60 heterogeneous robot datasets into a unified RLDS format; (2) it trains two models on OXE, RT-1-X and RT-2-X, representing a medium-capacity specialized network and a high-capacity vision-language-action model respectively.
图2：RT-1-X（左）和 RT-2-X（右）架构。两者均以图像序列和自然语言指令为输入，输出离散化的7自由度末端执行器动作。RT-1-X 使用 EfficientNet + FiLM + Transformer Decoder（35M参数）；RT-2-X 基于 PaLI-X 骨干（55B参数），将动作表示为文本 token 输出。Figure 2: Architectures of RT-1-X (left) and RT-2-X (right). Both take an image sequence and a natural language instruction as input and output discretized 7-DoF end-effector actions. RT-1-X uses EfficientNet + FiLM + Transformer Decoder (35M parameters); RT-2-X builds on the PaLI-X backbone (55B parameters) and emits actions as text tokens.
RT-1-X：中等容量专用模型RT-1-X: a medium-capacity specialized model
35M 参数 Transformer 网络，专为机器人控制设计35M-parameter Transformer network designed specifically for robot control
输入：15帧图像历史 + 自然语言指令Input: a 15-frame image history + a natural language instruction
与机器人数据和网络数据联合微调（co-fine-tuning）Co-fine-tuning on robot data together with web data
OXE 数据集构建Construction of the OXE dataset
所有60个数据集统一转换为 RLDS（serialized tfrecord）格式。数据来自21个机构，涵盖22种机器人平台，包含160,266个任务、527种技能，超过100万条真实机器人轨迹。各数据集按比例混合（混合权重基于数据集规模和质量评估），以避免大数据集完全主导训练。All 60 datasets are converted into a unified RLDS (serialized tfrecord) format. The data comes from 21 institutions and spans 22 robot platforms, containing 160,266 tasks, 527 skills and more than 1000000 real robot trajectories. The datasets are mixed proportionally (mixture weights based on dataset size and a quality assessment) so that large datasets do not completely dominate training.
03实验Experiments
在6种机器人上共评估3,600次试验，涵盖小规模和大规模数据集场景，同时评估 RT-2-X 的 emergent skills（跨机器人平台迁移能力）和泛化能力。A total of 3,600 trials were evaluated on 6 robots, covering both small-scale and large-scale dataset regimes, together with an evaluation of the emergent skills of RT-2-X (skill transfer across robot platforms) and of its generalization ability.
小规模数据集场景中，RT-1-X 相比原始方法或单机器人 RT-1 基线取得 50% 更高的平均成功率。大规模数据集场景下，RT-1-X 出现欠拟合（underfitting），而高容量的 RT-2-X (55B) 能有效学习并优于基线。In the small-scale dataset regime, RT-1-X achieves a 50% higher average success rate than the original methods or the single-robot RT-1 baseline. In the large-scale dataset regime RT-1-X underfits, whereas the high-capacity RT-2-X (55B) learns effectively and outperforms the baselines.
图3：各机器人平台上 RT-1-X 与原始方法和 RT-1 基线的成功率对比（小规模数据集场景）。RT-1-X 在多个机器人平台上均展现出正向迁移，平均成功率提升约50%。Figure 3: Success rates of RT-1-X against the original methods and the RT-1 baseline on each robot platform (small-scale dataset regime). RT-1-X exhibits positive transfer on multiple robot platforms, raising the average success rate by roughly 50%.
Emergent Skills（涌现技能）评估Emergent skills evaluation
在 Google Robot 上执行来自 Bridge 数据集（WidowX 机器人）的任务，用于评估跨机器人平台的技能迁移能力：Tasks drawn from the Bridge dataset (WidowX robot) are executed on the Google Robot in order to evaluate skill transfer across robot platforms:
27.3%RT-2（单机器人）RT-2 (single robot)
75.8%RT-2-X（多机器人）RT-2-X (multi-robot)
~3×涌现技能提升倍数Emergent-skill improvement factor
42.8%去除 Bridge 数据后After removing Bridge data
图4：RT-2-X emergent skills 评估结果。RT-2-X 在 Google Robot 上执行 Bridge 数据集任务，成功率75.8%，相比单机器人训练的 RT-2（27.3%）提升约3倍。去除 Bridge 训练数据后性能下降至42.8%，确认 WidowX 数据的迁移贡献。Figure 4: RT-2-X emergent skills evaluation. Executing Bridge dataset tasks on the Google Robot, RT-2-X reaches a success rate of 75.8%, roughly three times that of the single-robot-trained RT-2 (27.3%). Removing the Bridge training data drops performance to 42.8%, confirming the transfer contribution of the WidowX data.
消融实验（Table II）Ablation study (Table II)
配置Configuration
模型规模Model scale
Web 预训练Web pretraining
Emergent Skills
泛化能力Generalization
RT-2（单机器人）RT-2 (single robot)
55B
✓
27.3%
62%
RT-2-X（完整）RT-2-X (full)
55B
✓
75.8%
61%
RT-2-X（去除 Bridge）RT-2-X (Bridge removed)
55B
✓
42.8%
54%
RT-2-X（5B）RT-2-X (5B)
5B
✓
44.4%
52%
RT-2-X（5B，无图像历史）RT-2-X (5B, no image history)
5B
✓
14.5%
30%
RT-2-X（5B，无 Web 预训练）RT-2-X (5B, no web pretraining)
5B
✗
0%
1%
消融实验揭示三个关键因素：(1) Web 预训练至关重要——去除后 emergent skills 从44.4%骤降至0%；(2) 模型容量影响迁移效果——55B vs 5B 在 emergent skills 上相差约31个百分点（75.8% vs 44.4%）；(3) 图像历史帮助性能——去除后 emergent skills 下降约30个百分点。The ablations reveal three key factors: (1) web pretraining is essential — removing it makes emergent skills plunge from 44.4% to 0%; (2) model capacity affects transfer — 55B vs. 5B differ by about 31 percentage points on emergent skills (75.8% vs. 44.4%); (3) image history helps performance — removing it lowers emergent skills by about 30 percentage points.
04局限性Limitations
Note:以下局限性均来自论文正文作者明确陈述（stated by authors）。All limitations below are explicitly stated by the authors in the body of the paper.
不涵盖感知与驱动模态差异极大的机器人Robots with very different sensing and actuation modalities are not covered
论文原文："Our experiments do not consider robots with very different sensing and actuation modalities."
当前实验主要集中在使用相机视觉输入和标准末端执行器的机械臂，未探索触觉传感、力传感器、腿式机器人等差异更大的模态。From the paper: "Our experiments do not consider robots with very different sensing and actuation modalities." The current experiments focus on manipulator arms using camera vision input and standard end-effectors; more divergent modalities such as tactile sensing, force sensors and legged robots are not explored.
未研究向全新机器人的泛化（zero-shot embodiment transfer）Generalization to entirely new robots (zero-shot embodiment transfer) is not studied
论文原文："They do not study generalization to new robots, and provide a decision criterion for when positive transfer does or does not happen."
所有评估机器人均在训练阶段出现过，未测试能否将策略迁移到从未见过的机器人平台。From the paper: "They do not study generalization to new robots, and provide a decision criterion for when positive transfer does or does not happen." Every evaluated robot appeared during training; whether a policy can be transferred to a robot platform never seen before was not tested.
RT-1-X 在大规模数据集上欠拟合RT-1-X underfits on large-scale datasets
RT-1-X（35M参数）在大规模数据集场景下出现欠拟合现象，无法充分利用 OXE 数据的多样性。需要更高容量的模型（如 RT-2-X 55B）才能有效实现正向迁移，这对计算资源提出了较高要求。RT-1-X (35M parameters) underfits in the large-scale dataset regime and cannot fully exploit the diversity of the OXE data. A higher-capacity model (such as RT-2-X 55B) is needed to realize positive transfer effectively, which imposes considerable demands on compute resources.
缺乏正向迁移发生条件的理论判据No theoretical criterion for when positive transfer occurs
论文原文提到该工作未能提供一个明确的判断准则来预测何时会发生正向迁移、何时会出现负向迁移，这一理论问题留待未来工作解决。The paper notes that this work does not provide a clear decision criterion for predicting when positive transfer will happen and when negative transfer will appear, leaving this theoretical question to future work.
