RT-1: Robotics Transformer for Real-World Control at Scale
用大规模真实演示训练的通用机器人 Transformer，实现多任务操控与零样本泛化A general-purpose robotics Transformer trained on large-scale real-world demonstrations, enabling multi-task manipulation and zero-shot generalization
Anthony Brohan, Noah Brown, Chelsea Finn, Sergey Levine 等 (共 50+ 位作者) · Google Research & Everyday RoboticsAnthony Brohan, Noah Brown, Chelsea Finn, Sergey Levine et al. (50+ authors in total) · Google Research & Everyday Robotics
RT-1 是一个专为真实世界机器人控制设计的 Transformer 模型：以 17 个月、13 台机器人收集的 ~130k 次演示进行训练，覆盖 700+ 条任务指令，实现了在已见任务上 97% 的成功率，以及对未见新任务的强泛化能力（76%），远超此前最强的 imitation learning 和多模态大模型基线。RT-1 is a Transformer model designed for real-world robotic control: trained on ~130k demonstrations collected over 17 months with 13 robots and covering 700+ task instructions, it attains a 97% success rate on seen tasks and strong generalization to unseen new tasks (76%), far surpassing the strongest prior imitation learning and multimodal large-model baselines.
机器人学习robot learningtransformer模仿学习imitation learning多任务机器人multi-task roboticszero-shot 泛化zero-shot generalizationaction tokenizationEfficientNetTokenLearner大规模数据large-scale data真实世界控制real-world control
01动机Motivation
视觉、语言领域已出现能力强大的通用预训练模型，但机器人领域却迟迟未能复制这一成功。
核心瓶颈在于：高质量机器人演示数据稀缺且昂贵，而现有方法往往依赖工程繁琐的自主采集流程或代价高昂的人工演示。
那么，能否在大规模、多样化的机器人数据上训练单一多任务模型，使其对新任务、新环境、新物体实现零样本泛化？Vision and language already have powerful general-purpose pretrained models, yet robotics has been slow to replicate that success. The core bottleneck: high-quality robot demonstration data is scarce and expensive, while existing methods usually rely on engineering-heavy autonomous collection pipelines or costly human demonstrations. So, can a single multi-task model be trained on large-scale, diverse robot data so that it generalizes zero-shot to new tasks, new environments, and new objects?
"Open-ended task-agnostic training, combined with high-capacity architectures" enables models to
effectively learn from varied robotic data — just as large language and vision models benefit from
scale and diversity.
图 1：RT-1 总览。以 ~130k 次演示（覆盖 700+ 任务）训练、经 3000 次真实测试验证：在已见任务上达到 97% 成功率，并对全新任务、环境、物体展现出强泛化能力。右侧示意了 RT-1 可完成的多样化操控技能。Figure 1: RT-1 overview. Trained on ~130k demonstrations (covering 700+ tasks) and validated over 3000 real-world trials: it reaches a 97% success rate on seen tasks and shows strong generalization to entirely new tasks, environments, and objects. The right side illustrates the diverse manipulation skills RT-1 can perform.
97%已见任务成功率Success rate on seen tasks
76%未见任务泛化成功率Generalization success on unseen tasks
130k真实演示样本Real-world demonstrations
700+覆盖任务指令数Task instructions covered
图 2：硬件与场景设置。左：机器人"教室"训练环境；中：两个真实办公厨房用于评估（包含一个训练时从未见过的厨房）；右：可扩展技能与物体多样性的移动操控平台。Figure 2: Hardware and scene setup. Left: the robot "classroom" training environment. Middle: two real office kitchens used for evaluation (including one kitchen never seen during training). Right: a mobile manipulation platform that scales up skill and object diversity.
02方法Method
RT-1 将语言指令和历史图像帧作为输入，输出离散化的机器人动作 token。整个架构由三个模块串联组成：
基于 FiLM 调制的 EfficientNet 视觉编码器、压缩 token 的 TokenLearner，以及解码动作的 decoder-only Transformer。
模型仅有 35M 参数，却能在 100ms 预算内以 3Hz 频率实时控制机器人。RT-1 takes a language instruction and a history of image frames as input and outputs discretized robot action tokens. The architecture chains three modules: a FiLM-modulated EfficientNet visual encoder, a TokenLearner that compresses tokens, and a decoder-only Transformer that decodes actions. With only 35M parameters, the model still controls the robot in real time at 3Hz within a 100ms budget.
图 3：RT-1 架构。语言指令经 Universal Sentence Encoder (USE) 嵌入，通过 FiLM 层条件化预训练的 EfficientNet-B3 提取 81 个视觉 token；TokenLearner 将其压缩为 8 个 token；历史 6 帧共 48 个 token 送入 8 层 decoder-only Transformer，输出 11 维离散动作（256 bins/维）。Figure 3: RT-1 architecture. The language instruction is embedded by a Universal Sentence Encoder (USE) and, through FiLM layers, conditions a pretrained EfficientNet-B3 that extracts 81 visual tokens; TokenLearner compresses them into 8 tokens; 6 history frames give 48 tokens in total, which are fed into an 8-layer decoder-only Transformer that outputs an 11-dimensional discrete action (256 bins per dimension).
图像以预训练的 EfficientNet-B3（ImageNet 初始化）提取特征，生成 9×9×512 的特征图（共 81 个视觉 token）。
语言指令通过 Universal Sentence Encoder 转化为嵌入向量，再以 FiLM（Feature-wise Linear Modulation） 层将语义条件注入视觉编码器，实现指令对视觉特征的逐层调制。
FiLM 层采用恒等初始化，总参数量仅约 16M。Images are encoded by a pretrained EfficientNet-B3 (ImageNet initialization), producing a 9×9×512 feature map (81 visual tokens in total). The language instruction is turned into an embedding vector by a Universal Sentence Encoder, and a FiLM (Feature-wise Linear Modulation) layer injects that semantic conditioning into the visual encoder, modulating visual features layer by layer. The FiLM layers use identity initialization and add only about 16M parameters in total.
TokenLearner 压缩TokenLearner Compression
81 个视觉 token 通过 TokenLearner（基于 element-wise 注意力）压缩为 8 个 token，显著降低后续 Transformer 的计算量。
6 帧历史图像产生 48 个 token（8×6），送入 Transformer 处理。
TokenLearner 和 token 复用分别带来 2.4× 和 1.7× 的推理加速，保证实时控制（3 Hz，100ms/步）。The 81 visual tokens are compressed into 8 tokens by TokenLearner (based on element-wise attention), sharply reducing the compute of the downstream Transformer. 6 history frames yield 48 tokens (8×6), which are passed to the Transformer. TokenLearner and token reuse bring 2.4× and 1.7× inference speedups respectively, keeping control in real time (3 Hz, 100ms per step).
动作离散化（Action Discretization）Action Discretization
机器人动作包含 11 个维度（7 维手臂关节 + 3 维底座移动 + 1 维模式切换），每维离散为 256 个 bin，使用 categorical cross-entropy loss 和因果掩码训练。
这一设计将连续控制问题转化为分类问题，使 Transformer 架构可以直接被复用于动作预测。A robot action has 11 dimensions (7 for arm joints + 3 for base movement + 1 for mode switching), each discretized into 256 bins, trained with a categorical cross-entropy loss and a causal mask. This design turns continuous control into a classification problem, so the Transformer architecture can be reused directly for action prediction.
训练数据Training Data
数据采集历时 17 个月，使用 13 台机器人收集了 ~130k 次演示，覆盖 744 项技能（包括拾放、抽屉操作、容器交互等），指令多样性超过 700 条。
数据多样性（任务种类）被实验验证比数据数量更关键：消融研究显示，去除 25% 的任务种类（但保留 97% 的数据量）对泛化能力的损害，等效于将数据集规模缩减 49%。Data collection ran for 17 months and used 13 robots to gather ~130k demonstrations covering 744 skills (including pick-and-place, drawer operations, container interactions, and more), with instruction diversity exceeding 700. Experiments confirm that data diversity (the variety of tasks) matters more than data quantity: an ablation shows that removing 25% of the task types (while keeping 97% of the data volume) hurts generalization as much as shrinking the dataset by 49%.
03实验Experiments
所有实验均在真实机器人上进行，共计 3000 次评测。基线方法包括 BC-Z（视觉-语言 imitation learning）、Gato（通用多模态大模型）及 SayCan 框架。
评估维度覆盖：已见/未见任务成功率、干扰物鲁棒性、背景鲁棒性、长时序任务执行，以及异构数据融合收益。All experiments were run on real robots, for 3000 evaluation trials in total. Baselines include BC-Z (vision-language imitation learning), Gato (a general-purpose multimodal large model), and the SayCan framework. Evaluation covers success rates on seen/unseen tasks, robustness to distractors, robustness to backgrounds, long-horizon task execution, and the gains from fusing heterogeneous data.
图 4：与基线的全面对比。RT-1 在已见任务（Seen）和未见任务（Unseen）上均大幅超越 BC-Z 和 Gato，尤其在需要泛化的场景优势显著。SayCan 长时序任务中，Gato 几乎为 0%，BC-Z 为 13%，而 RT-1 达到 67%。Figure 4: Full comparison with the baselines. RT-1 substantially outperforms BC-Z and Gato on both seen and unseen tasks, with an especially large margin wherever generalization is required. On the SayCan long-horizon tasks Gato is near 0%, BC-Z reaches 13%, while RT-1 attains 67%.图 5：鲁棒性评测。干扰物鲁棒性测试中，场景分为 0–5 个简单干扰（easy）、9 个中等干扰（medium）、9 个含遮挡的硬干扰（hard）；背景鲁棒性测试中，包括原始场景、花纹背景、全新厨房。RT-1 在所有条件下均显著超越基线。Figure 5: Robustness evaluation. In the distractor robustness test, scenes are split into 0–5 simple distractors (easy), 9 medium distractors (medium), and 9 hard distractors with occlusion (hard); the background robustness test covers the original scene, patterned backgrounds, and an entirely new kitchen. RT-1 clearly beats the baselines under every condition.
消融：数据规模与多样性Ablation: Data Scale and Diversity
图 6：数据消融实验。任务多样性（task diversity）比数据数量更关键——去除 25% 的任务种类（保留 97% 数据量）对泛化的损害，等效于将数据集规模缩减 49%。右图展示了从仿真数据融合和多机器人混训的收益。Figure 6: Data ablation. Task diversity matters more than data quantity — removing 25% of the task types (while keeping 97% of the data volume) hurts generalization as much as shrinking the dataset by 49%. The right panel shows the gains from fusing simulation data and from co-training across multiple robots.
异构数据融合Fusing Heterogeneous Data
仿真数据融合：加入仿真数据后，未见仿真物体的成功率从 23% 提升至 87%，而真实物体性能仅从 92% 微降至 90%（几乎无损）。Simulation data fusion: after simulation data is added, the success rate on unseen simulated objects rises from 23% to 87%, while performance on real objects drops only slightly from 92% to 90% (essentially lossless).
多机器人混训：将 Everyday Robots 与 Kuka 数据混训后，bin-picking 任务成功率从 22%（单机器人训练）提升至 39%（提升 77%）。Multi-robot co-training: co-training on Everyday Robots and Kuka data raises the bin-picking success rate from 22% (single-robot training) to 39% (a 77% improvement).
04局限性Limitations
Note:以下局限性均由论文作者明确陈述（stated），非推断。All limitations below are explicitly stated by the authors of the paper, not inferred.
受限于 Imitation Learning 的上限Bounded by the ceiling of imitation learning
作者明确指出："It may not be able to surpass the performance of the demonstrators."
模型本质上是在复刻人类演示者的行为，难以超越演示者的操作水平，也无法通过探索发现更优策略。The authors state plainly: "It may not be able to surpass the performance of the demonstrators." The model essentially replicates the behavior of human demonstrators, so it can hardly exceed their level of skill, nor discover better policies through exploration.
泛化仅限于已见概念的新组合Generalization limited to new combinations of seen concepts
"Generalization to new instructions is limited to combinations of previously seen concepts" — 模型能够组合已学的技能与物体，但无法泛化到训练中从未出现过的全新动作或物体类别。"Generalization to new instructions is limited to combinations of previously seen concepts" — the model can recombine learned skills and objects, but cannot generalize to entirely new actions or object categories that never appeared during training.
仅适用于非精细操控任务Only suited to non-dexterous manipulation tasks
RT-1 适用于 "a large but not very dexterous set of manipulation tasks"，对精细化抓取（如插针、装配）等高灵巧度任务的适用性有限。RT-1 targets "a large but not very dexterous set of manipulation tasks"; its applicability to high-dexterity tasks such as fine-grained grasping (peg insertion, assembly) is limited.
背景与环境鲁棒性有待提升Robustness to backgrounds and environments needs improvement
作者承认："robustness to backgrounds and environments could be further improved by greatly increasing the environment diversity."
当前数据来自相对固定的厨房场景，对更广泛环境的适应性仍受限。The authors concede: "robustness to backgrounds and environments could be further improved by greatly increasing the environment diversity." The current data comes from relatively fixed kitchen scenes, so adaptability to a wider range of environments remains limited.
实时控制约束限制模型容量Real-time control constraints cap model capacity
3 Hz 的控制频率（100ms 预算）约束了可部署的模型规模。相比无实时约束的架构，这在模型容量与推理速度之间形成了权衡。The 3 Hz control frequency (a 100ms budget) constrains the size of model that can be deployed. Compared with architectures free of real-time constraints, this creates a trade-off between model capacity and inference speed.
