AgiBot World Colosseo: 面向可扩展智能具身系统的大规模操作平台

01 动机Motivation

机器人学习在大规模基础模型建设上远落后于 NLP 与计算机视觉，根本原因在于高质量数据采集的困难。现有数据集往往受限于实验室受控环境、短时序任务和异构硬件，导致策略难以迁移至真实世界的多样场景。Robot learning lags far behind NLP and computer vision in building large-scale foundation models, and the root cause lies in the difficulty of collecting high-quality data. Existing datasets are often confined to controlled laboratory environments, short-horizon tasks and heterogeneous hardware, which makes policies hard to transfer to the diverse scenes of the real world.

"existing robot learning datasets remain constrained by their reliance on short-horizon tasks in highly controlled laboratory environments"

AgiBot World 平台总览 — **图 1：AgiBot World Colosseo 平台总览。** 左侧展示 100 台双臂人形机器人组成的采集集群；中部对比 AgiBot World 与已有数据集在规模与任务覆盖上的差距；右侧展示 GO-1 策略在复杂操作任务中的成功率优势（较 RDT 提升 32%）。**Figure 1: Overview of the AgiBot World Colosseo platform.** Left: the collection fleet formed by 100 dual-arm humanoid robots; middle: a comparison between AgiBot World and existing datasets in scale and task coverage; right: the success-rate advantage of the GO-1 policy on complex manipulation tasks (a 32% improvement over RDT).

1M+操作轨迹总量Total manipulation trajectories

217覆盖任务数量Tasks covered

+30%vs. Open X-Embodiment 平均提升Average gain vs. Open X-Embodiment

+32%GO-1 vs. RDT 提升GO-1 gain vs. RDT

现有数据集的三大瓶颈Three bottlenecks of existing datasets

规模不足：已有数据集（如 Open X-Embodiment）在轨迹量和任务多样性上均大幅落后，难以支撑 scaling law 研究。Insufficient scale: existing datasets (such as Open X-Embodiment) lag far behind in both trajectory count and task diversity, and can hardly support the study of scaling law.
任务简单：现有基准多为 5–20 秒的短时序抓取任务，无法覆盖现实中需要 30–60+ 秒的长时序操作需求。Simplistic tasks: existing benchmarks mostly consist of 5–20 second short-horizon grasping tasks, and fail to cover the real-world demand for long-horizon manipulation that lasts 30–60+ seconds.
质量参差：采集流程缺乏系统性质量控制，数据标注不一致，限制了下游策略的泛化能力。Uneven quality: the collection pipeline lacks systematic quality control and the annotations are inconsistent, which limits the generalization ability of downstream policies.

02 方法Method

AgiBot World 从数据采集与策略建模两个维度同时发力：前者通过人在环路（human-in-the-loop）的三阶段流程保证数据质量；后者提出 Vision-Language-Latent-Action（ViLLA）三阶段框架，将网络规模视觉语言预训练与高频扩散控制解耦。AgiBot World pushes forward along two dimensions at once, data collection and policy modelling: the former guarantees data quality through a three-stage human-in-the-loop pipeline; the latter proposes the three-stage Vision-Language-Latent-Action (ViLLA) framework, which decouples web-scale vision-language pre-training from high-frequency diffusion control.

数据采集：三阶段人在环路流程Data collection: a three-stage human-in-the-loop pipeline

**图 2：三阶段数据采集流程。** 第一阶段：可行性验证与任务设计；第二阶段：熟练远程操控员执行示范；第三阶段：后处理标注与质量审核。整个流程形成"采集→训练→部署→反馈"的迭代闭环，并收录约 1% 的失败恢复轨迹（failure recovery data）。**Figure 2: The three-stage data collection pipeline.** Stage one: feasibility validation and task design; stage two: demonstrations performed by skilled teleoperators; stage three: post-processing annotation and quality review. The pipeline forms an iterative closed loop of "collection → training → deployment → feedback", and includes about 1% failure recovery data.

规模：100 台双臂人形机器人，配备灵巧手与视触觉传感器，覆盖家庭、零售、工业、餐厅、办公五大场景。Scale: 100 dual-arm humanoid robots equipped with dexterous hands and visuo-tactile sensors, covering five scenarios: home, retail, industry, restaurant and office.
质量：人工验证数据（528 条）相比未验证数据（482 条）策略得分提升 0.18。Quality: human-verified data (528 trajectories) raises the policy score by 0.18 compared with unverified data (482 trajectories).
长时序：轨迹时长主要分布在 30–60+ 秒，远超已有数据集的 5–20 秒。Long horizon: trajectory durations are mainly distributed in the 30–60+ second range, far beyond the 5–20 seconds of existing datasets.

GO-1：ViLLA 三阶段策略框架GO-1: the three-stage ViLLA policy framework

GO-1 架构图 — **图 3：GO-1（Genie Operator-1）架构。** 三个阶段依次为：Latent Action Model（LAM）学习通用动作表征 → Latent Planner 利用预训练 VLM 进行高层规划 → Action Expert 通过扩散目标解码高频低层动作。**Figure 3: The GO-1 (Genie Operator-1) architecture.** The three stages are, in order: the Latent Action Model (LAM) learns general-purpose action representations → the Latent Planner performs high-level planning with a pre-trained VLM → the Action Expert decodes high-frequency low-level actions through a diffusion objective.

Stage 1 · Latent Action Model（LAM）

在互联网规模的异构视频数据上训练编码器-解码器式潜在动作模型：编码器基于 inverse dynamics model 将相邻帧映射为潜在动作向量；解码器基于 forward dynamics model 预测未来帧，从而学习与具身形式无关的通用动作表征。An encoder-decoder style latent action model is trained on internet-scale heterogeneous video data: the encoder, built on an inverse dynamics model, maps adjacent frames into a latent action vector; the decoder, built on a forward dynamics model, predicts future frames, so that embodiment-agnostic general-purpose action representations are learned.

Stage 2 · Latent Planner

以 InternVL2.5-2B 作为骨干（24 个 transformer 层），通过"逐层条件注入" （layer-by-layer conditioning）将视觉语言理解能力迁移至机器人规划，输出高层潜在规划信号，实现跨具身的通用性。InternVL2.5-2B serves as the backbone (24 transformer layers); through "layer-by-layer conditioning" the vision-language understanding capability is transferred to robot planning, producing high-level latent planning signals and achieving cross-embodiment generality.

Stage 3 · Action Expert

利用扩散目标（diffusion objective）对低层连续动作分布进行建模，以 action chunks（H = 30）输出高频关节控制指令，支持灵巧操作任务。A diffusion objective models the low-level continuous action distribution, emitting high-frequency joint control commands as action chunks (H = 30) to support dexterous manipulation tasks.

开源资源Open-source resources

数据集、采集工具链、GO-1 模型权重及评测协议均已公开发布，旨在推动机器人基础模型研究的可复现与可扩展性。The dataset, the collection toolchain, the GO-1 model weights and the evaluation protocol have all been publicly released, with the aim of advancing the reproducibility and scalability of research on robot foundation models.

03 实验Experiments

所有评估均在真实世界场景中进行，覆盖域内（in-domain）和分布外（out-of-distribution）两类设置，并与 Open X-Embodiment 预训练基线及 RDT 进行对比。All evaluations are carried out in real-world scenarios, covering both in-domain and out-of-distribution settings, and are compared against the Open X-Embodiment pre-training baseline and RDT.

数据集对比：AgiBot World vs. Open X-EmbodimentDataset comparison: AgiBot World vs. Open X-Embodiment

评估设置Evaluation setting	Open X-Embodiment 预训练Open X-Embodiment pre-training	AgiBot World 预训练AgiBot World pre-training	提升Gain
域内（in-domain）In-domain	0.47	0.77	+0.30
分布外（out-of-distribution）Out-of-distribution	0.38	0.67	+0.29

基于 AgiBot World 预训练的策略在两类场景下均实现 "average performance improvement of 30% over those trained on Open X-Embodiment"。Policies pre-trained on AgiBot World achieve, under both settings, an "average performance improvement of 30% over those trained on Open X-Embodiment".

GO-1 vs. RDT 对比GO-1 vs. RDT comparison

GO-1 与 RDT 性能对比 — **图 4：GO-1 与 RDT 在不同复杂操作任务中的成功率对比。** GO-1 在 "Fold Shorts"、"Restock Beverage" 等长时序灵巧任务中均优于 RDT，平均提升 **0.12** 任务完成得分； Latent Planner 的加入贡献了显著增益。**Figure 4: Success-rate comparison between GO-1 and RDT on different complex manipulation tasks.** GO-1 outperforms RDT on long-horizon dexterous tasks such as "Fold Shorts" and "Restock Beverage", raising the task completion score by **0.12** on average; adding the Latent Planner contributes a notable gain.

模型Model	复杂任务成功率Success rate on complex tasks	相对 RDT 提升Gain over RDT
RDT（prior best）	—	—
GO-1（ours）	> 60%	+32%

Scaling Law 与数据质量消融Ablations on scaling law and data quality

Scaling law 与质量消融 — **图 5：消融实验结果。** （a）数据集规模（9.2k → 1M 轨迹）与策略性能呈幂律关系，Pearson r = 0.97，验证了 scaling law 的存在。（b）人工验证数据（528 条）相比未验证数据（482 条）提升策略得分 **0.18**，说明数据质量对策略性能的关键影响。**Figure 5: Ablation results.** (a) Dataset scale (9.2k → 1M trajectories) follows a power-law relation with policy performance, with Pearson r = 0.97, confirming the existence of a scaling law. (b) Human-verified data (528 trajectories) raises the policy score by **0.18** compared with unverified data (482 trajectories), showing the critical impact of data quality on policy performance.

Scaling law：数据量从 9.2k 扩展至 1M 条轨迹，策略性能持续提升，幂律拟合 Pearson r = 0.97。Scaling law: scaling the data from 9.2k to 1M trajectories keeps improving policy performance, with a power-law fit of Pearson r = 0.97.
数据质量：人在环路验证使策略得分净提升 0.18，仅靠增加未验证数据无法等价替代。Data quality: human-in-the-loop verification yields a net policy score gain of 0.18, which cannot be matched by simply adding more unverified data.
Latent Planner 贡献：去掉 Latent Planner 后复杂任务完成得分平均下降 0.12。Contribution of the Latent Planner: removing the Latent Planner lowers the completion score on complex tasks by 0.12 on average.

04 局限性Limitations

注：以下第 1 点为作者在论文中明确陈述（stated）；第 2、3 点为基于方法设计推断（inferred）。Note: item 1 below is explicitly stated by the authors in the paper; items 2 and 3 are inferred from the design of the method.

缺乏仿真评估环境（明确陈述）No simulation evaluation environment (stated)

论文明确指出："All evaluations are conducted in real-world scenarios. We are currently developing the simulation environment to facilitate fast and reproducible evaluation." 目前所有指标均来自真实机器人测试，缺乏可快速重现的仿真基准，限制了社区对结果的独立验证。The paper states explicitly: "All evaluations are conducted in real-world scenarios. We are currently developing the simulation environment to facilitate fast and reproducible evaluation." All current metrics come from tests on real robots, and the lack of a quickly reproducible simulation benchmark limits independent verification of the results by the community.

硬件依赖性（推断）Hardware dependence (inferred)

整套系统依赖 100 台定制双臂人形机器人与专用灵巧手、视触觉传感器，数据采集与策略复现的硬件门槛较高，难以被一般研究机构直接复制。The whole system relies on 100 customized dual-arm humanoid robots together with dedicated dexterous hands and visuo-tactile sensors; the hardware barrier for data collection and policy reproduction is high, and it is hard for ordinary research institutions to replicate it directly.

跨具身泛化的隐含假设（推断）Implicit assumptions of cross-embodiment generalization (inferred)

Latent Action Model 虽声称学习"具身无关"的表征，但预训练视频数据与目标机器人之间的域差异（embodiment gap）尚未被系统分析；在形态差异较大的机器人平台上的迁移效果有待验证。Although the Latent Action Model claims to learn "embodiment-agnostic" representations, the domain difference between the pre-training video data and the target robot (the embodiment gap) has not been systematically analyzed; how well it transfers to robot platforms with markedly different morphologies remains to be verified.