Octo: An Open-Source Generalist Robot Policy

01 动机Motivation

大规模预训练已彻底改变了 NLP 与计算机视觉领域，但机器人学习为何仍难以复制这一成功？核心障碍在于机器人的多样性：不同的本体（embodiment）、传感器配置、动作空间和任务规范，使得统一的通用策略难以构建。现有的通用机器人策略（Generalist Robot Policies, GRP）通常将用户锁定在固定的观测输入格式上，且对新平台的微调支持十分有限。Large-scale pretraining has transformed NLP and computer vision, so why has robot learning struggled to replicate that success? The core obstacle is the diversity of robots: differing embodiments, sensor configurations, action spaces and task specifications make a unified generalist policy hard to build. Existing Generalist Robot Policies (GRPs) typically lock users into a fixed observation input format and offer very limited support for finetuning to new platforms.

"Octo is the first GRP that can be effectively finetuned to new observations and action spaces."

Octo 系统概览 — Octo 的整体架构：将任务（语言/目标图像）和观测（相机图像）tokenize 后输入 Transformer 主干，通过 block-wise attention 结构和 diffusion action head 输出动作序列。模块化设计使微调时可灵活添加或去除输入/输出通道，无需重新训练核心参数。The overall architecture of Octo: the task (language / goal image) and the observations (camera images) are tokenized and fed into a Transformer backbone, which outputs an action sequence through a block-wise attention structure and a diffusion action head. The modular design allows input/output channels to be added or removed freely at finetuning time, without retraining the core parameters.

800k预训练轨迹数（Open X-Embodiment）Pretraining trajectories (Open X-Embodiment)

25训练数据集来源Source training datasets

9零样本评估机器人平台数Robot platforms in zero-shot evaluation

72%微调后六项任务平均成功率Average success rate across six finetuning tasks

九台机器人评估平台 — 来自四个机构的九套真实机器人评估场景，涵盖多样化物体交互、长时序任务与精细操作，用于评估 Octo 的零样本跨平台泛化能力。Nine real-robot evaluation setups across four institutions, covering diverse object interactions, long-horizon tasks and fine manipulation, used to assess Octo's zero-shot cross-platform generalization.

02 方法Method

Octo 由三个核心模块组成：输入 tokenizer、Transformer 主干、以及基于 diffusion 的动作解码器。整体设计强调模块化与可扩展性——新的观测类型（如力矩传感器）或动作空间（如关节位置控制）可在微调阶段轻松接入，核心预训练权重保持不变。Octo consists of three core modules: an input tokenizer, a Transformer backbone, and a diffusion-based action decoder. The design emphasizes modularity and extensibility — new observation types (such as force-torque sensors) or action spaces (such as joint position control) can be attached easily during finetuning while the core pretrained weights stay unchanged.

Octo 架构图 — Octo 的详细架构。语言指令经 pretrained T5-base（111M 参数）编码；图像观测经浅层卷积后 patch tokenization；所有 token 送入 Transformer，经 block-wise masked attention 处理后，由 readout token 汇聚信息，最终输入 diffusion head 预测动作 chunk。The detailed architecture of Octo. Language instructions are encoded by a pretrained T5-base (111M parameters); image observations go through a shallow convolution followed by patch tokenization; all tokens enter the Transformer, are processed by block-wise masked attention, aggregated by a readout token, and finally fed to the diffusion head to predict an action chunk.

Block-wise Masked Attention

Transformer 主干采用 block-wise masked attention 结构：观测 token 在时序上以因果（causal）方式关注更早时间步的观测与任务 token；专门设计的 "readout token" 被动汇聚信息，不影响观测 token 的注意力计算。这种结构使得不同类型的输入/输出可以在微调时灵活插拔，而不破坏预训练阶段学到的表征。The Transformer backbone uses a block-wise masked attention structure: observation tokens attend causally over time to earlier observation and task tokens, while a purpose-built "readout token" passively aggregates information without affecting the attention computation of the observation tokens. This structure lets different input/output types be plugged in and out at finetuning time without disturbing the representations learned during pretraining.

Diffusion Action Head

动作解码采用 diffusion-based policy head，预测动作 chunk（多步动作序列），而非单步 MSE 或离散动作。消融实验表明，这一设计对模型性能至关重要：在 WidowX 任务上，离散动作预测（18%）和连续 MSE 预测（35%）均远低于 diffusion head 对应的 Octo-Small 基线（83%）。Action decoding uses a diffusion-based policy head that predicts an action chunk (a multi-step action sequence) rather than a single-step MSE or discrete action. Ablations show this design is critical to performance: on the WidowX tasks, discrete action prediction (18%) and continuous MSE prediction (35%) both fall far short of the Octo-Small baseline with a diffusion head (83%).

大规模多数据集预训练Large-scale multi-dataset pretraining

Octo 在来自 Open X-Embodiment 的 25 个数据集、共计 80 万条轨迹上预训练。训练数据混合包含多种机器人本体、相机视角和任务类型，是迄今为止规模最大的开源机器人预训练数据集之一。消融实验表明，使用全部 25 个数据集的混合（83%）显著优于仅用 RT-X 的 11 个数据集子集（60%），以及单一 Bridge Data（43%）。Octo is pretrained on 25 datasets from Open X-Embodiment, totalling 800000 trajectories. The training mixture spans multiple robot embodiments, camera viewpoints and task types, making it one of the largest open-source robot pretraining datasets to date. Ablations show that the full 25-dataset mixture (83%) clearly outperforms the RT-X subset of 11 datasets (60%) and single-source Bridge Data (43%).

模型规模消融 — 模型规模对性能的影响：Tiny（10M）< Small（27M）< Base（93M）。更大的模型在场景变化时表现出更强的鲁棒性和更谨慎的抓取行为。Effect of model scale on performance: Tiny (10M) < Small (27M) < Base (93M). Larger models are more robust to scene changes and grasp more cautiously.

03 实验Experiments

实验在九个真实机器人平台上进行，分零样本（zero-shot）评估和微调（finetuning）评估两个维度。零样本测试对比 RT-1-X 和 RT-2-X；微调测试对比从头训练的 ResNet+Transformer baseline 和 VC-1 预训练特征。每类任务约 100 条目标域演示数据。Experiments are run on nine real robot platforms along two axes: zero-shot evaluation and finetuning evaluation. Zero-shot tests compare against RT-1-X and RT-2-X; finetuning tests compare against a ResNet+Transformer baseline trained from scratch and against VC-1 pretrained features. Each task category uses roughly 100 target-domain demonstrations.

零样本多机器人控制Zero-shot multi-robot control

在零样本多机器人评估中，Octo 的成功率比 RT-1-X 高出 29%，尽管参数量远小于 RT-2-X（55B），Octo 与其性能相当。在 WidowX 任务上，目标图像条件（goal-conditioned）的成功率比语言条件（language-conditioned）高出 25%。In zero-shot multi-robot evaluation, Octo's success rate is 29% higher than RT-1-X, and despite having far fewer parameters than RT-2-X (55B), Octo matches its performance. On the WidowX tasks, goal-image conditioning yields a success rate 25% higher than language conditioning.

零样本评估结果 — 零样本多机器人评估结果：对比 Octo、RT-1-X 和 RT-2-X 在多个机器人本体上的成功率。Octo 在大多数任务上优于 RT-1-X，并与参数量大得多的 RT-2-X 持平。Zero-shot multi-robot evaluation results: success rates of Octo, RT-1-X and RT-2-X across several robot embodiments. Octo beats RT-1-X on most tasks and matches the much larger RT-2-X.

微调评估（Table I）Finetuning evaluation (Table I)

在六项微调评估任务中，Octo 平均成功率达 72%，比次优 baseline（VC-1）高出 52%。其中包含两项特殊挑战：Berkeley Insertion（新观测输入：力矩传感器）和 Berkeley Pick-Up / Berkeley Bimanual（新动作空间：关节位置控制），Octo 均表现出色。Across six finetuning evaluation tasks, Octo reaches an average success rate of 72%, 52% higher than the next-best baseline (VC-1). Two of these pose special challenges: Berkeley Insertion (new observation input: force-torque sensor) and Berkeley Pick-Up / Berkeley Bimanual (new action space: joint position control), and Octo performs well on both.

任务Task	ResNet+Transformer（从头）ResNet+Transformer (from scratch)	VC-1	Octo（ours）
Berkeley Insertion*	10%	5%	70%
Stanford Coffee	45%	0%	75%
CMU Baking	25%	30%	50%
Berkeley Pick-Up†	0%	0%	60%
Berkeley Coke	20%	10%	100%
Berkeley Bimanual†	20%	50%	80%
平均Average	20%	15%	72%

*新观测输入（force-torque sensor）；†新动作空间（joint position control）*New observation input (force-torque sensor); †New action space (joint position control)

消融实验（Table II，WidowX 平台）Ablations (Table II, WidowX platform)

以下消融均在 WidowX 平台 40 次试验、四项任务上进行：All ablations below are run on the WidowX platform over 40 trials across four tasks:

组件/变体Component / variant	成功率Success rate
Octo-Small（完整版）Octo-Small (full)	83%
仅用 RT-X 数据子集（11 个数据集）RT-X data subset only (11 datasets)	60%
仅用 Bridge Data（单一数据集）Bridge Data only (single dataset)	43%
离散化动作预测（Discretized）Discretized action prediction	18%
连续 MSE 动作预测Continuous MSE action prediction	35%
ResNet-50 + Transformer（替换 ViT）ResNet-50 + Transformer (replacing ViT)	70%

消融结果表明：多样化数据混合、diffusion action head、以及 ViT 图像编码器是 Octo 成功的关键三要素。The ablations show that a diverse data mixture, the diffusion action head, and the ViT image encoder are the three keys to Octo's success.

04 局限性Limitations

Note: 以下局限性均为论文作者明确陈述（stated），非推断。Note: All limitations below are explicitly stated by the authors, not inferred.

腕部相机信息处理能力不足Inadequate handling of wrist camera information

"The current Octo model struggles with adequately processing wrist camera information. Often finetuning results were stronger when using only a third person camera instead of combining third person and wrist camera."预训练数据中仅有 27% 包含腕部相机信息，导致模型对该输入模态的利用不充分。"The current Octo model struggles with adequately processing wrist camera information. Often finetuning results were stronger when using only a third person camera instead of combining third person and wrist camera." Only 27% of the pretraining data includes wrist camera information, so the model underuses that input modality.

语言条件与目标图像条件之间存在较大性能差距A large performance gap between language conditioning and goal-image conditioning

"Only 56% of the pretraining data contains language annotations"，导致语言条件策略性能系统性弱于目标图像条件策略。数据模态分布不均是根本原因。"Only 56% of the pretraining data contains language annotations", which makes language-conditioned policies systematically weaker than goal-image-conditioned ones. The uneven distribution of data modalities is the root cause.

仅能从最优演示中学习，无法利用次优或在线交互数据Learns only from optimal demonstrations; cannot exploit sub-optimal or online interaction data

模型完全依赖 imitation learning，训练于人类专家的最优演示。论文指出，"future work may consider learning from sub-optimal or online interaction data that require alternative objectives."The model relies entirely on imitation learning and is trained on optimal human expert demonstrations. The paper notes that "future work may consider learning from sub-optimal or online interaction data that require alternative objectives."

仅覆盖单臂与双臂机械臂，不涉及导航与移动操作Covers only single-arm and bimanual arms; no navigation or mobile manipulation

当前 Octo 仅在固定基座的单臂/双臂机器人上验证，论文明确指出未探索移动机器人（navigation and mobile manipulation）场景。Octo is currently validated only on fixed-base single-arm / bimanual robots, and the paper states explicitly that mobile robot settings (navigation and mobile manipulation) were not explored.

零样本泛化在新场景与新行为上表现下降Zero-shot generalization degrades on new scenes and new behaviors

Zero-shot 性能"degrades in a new scene, and high degradation for novel behaviors like flipping or precise insertion"，说明预训练策略对分布外场景的鲁棒性有限。Zero-shot performance "degrades in a new scene, and high degradation for novel behaviors like flipping or precise insertion", indicating limited robustness of the pretrained policy to out-of-distribution scenes.