π₀：面向通用机器人控制的视觉-语言-动作流模型

01 动机Motivation

机器人学习要达到人类般的灵活性与泛化能力，面临三大核心障碍：数据规模不足、模型泛化能力弱以及训练方法不成熟。现有 VLA 模型（如 OpenVLA、Octo）大多采用自回归离散化动作表示，在高频连续控制和精细操作任务上能力受限。For robot learning to reach human-like dexterity and generalization, three core obstacles stand in the way: insufficient data scale, weak model generalization, and immature training recipes. Existing VLA models (such as OpenVLA and Octo) mostly adopt autoregressive discretized action representations, which limits them on high-frequency continuous control and dexterous manipulation tasks.

"We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge… Our results cover a wide variety of tasks, such as laundry folding, table cleaning, and assembling boxes."

**图 1：π₀ 系统总览。**模型以预训练 VLM 为骨干，结合来自 7 种机器人、68 种任务的跨机器人数据集进行预训练。右侧展示三类应用场景：开箱即用零样本评估（zero-shot in-distribution）、针对高难度任务的专项 post-training（如空洗衣机、批量叠衬衫），以及针对全新任务的高效 post-training（如将物品放入抽屉、更换纸巾卷）。**Figure 1: Overview of the π₀ system.** The model takes a pre-trained VLM as its backbone and is pre-trained on a cross-embodiment dataset drawn from 7 robots and 68 task types. The right side shows three classes of application: out-of-the-box zero-shot in-distribution evaluation, task-specific post-training for difficult tasks (such as emptying the washing machine and folding shirts in bulk), and efficient post-training for entirely new tasks (such as putting items into a drawer and replacing a paper towel roll).

3.3B总参数量（VLM 3B + action expert 300M）Total parameters (VLM 3B + action expert 300M)

10,000+小时自有机器人操作数据（+ OXE 开源数据）hours of in-house robot manipulation data (+ open-source OXE data)

7 种7机器人平台（单臂、双臂、移动操作机器人）robot platforms (single-arm, bimanual, and mobile manipulators)

68 个68任务类别，覆盖叠衣、清桌、装箱等高精操作task categories, covering laundry folding, table clearing, box packing and other high-precision manipulation

为什么现有方法不够用？Why are existing methods insufficient?

OpenVLA（7B 参数）：自回归离散化动作，不支持 action chunking，无法处理高频连续控制。OpenVLA (7B parameters): autoregressive discretized actions, no support for action chunking, unable to handle high-frequency continuous control.
Octo（93M 参数）：使用 diffusion 生成动作，表达能力有限；相比 π₀ 的流匹配 VLA 架构，在精细操作任务上差距显著。Octo (93M parameters): generates actions with diffusion and has limited expressive capacity; compared with the flow-matching VLA architecture of π₀, the gap on dexterous manipulation tasks is substantial.
数据稀缺：大多数先前工作仅有 10s 或 100s 条训练轨迹（相当于 10 小时以内），难以支撑复杂任务的泛化。Data scarcity: most prior work has only 10s or 100s of training trajectories (equivalent to under 10 hours), too little to support generalization on complex tasks.

02 方法Method

π₀ 的核心设计理念是"预训练 + 后训练"两阶段范式：首先在大规模多样化数据上训练一个广泛泛化的 base model，再用高质量任务数据进行 fine-tuning，使模型既能开箱即用，又能高效掌握精细技能。The core design idea of π₀ is a two-stage "pre-training + post-training" paradigm: first train a broadly generalizing base model on large-scale, diverse data, then fine-tune it on high-quality task data, so that the model both works out of the box and efficiently acquires dexterous skills.

π₀ 模型架构 — **图 3：π₀ 框架概览。**预训练混合数据（π 自有操作数据集 + OXE 开源数据集）输入统一的流匹配 VLA 模型。该模型由两部分组成：（1）*预训练 VLM 骨干*（PaliGemma，SigLIP 400M + Gemma 2.6B），处理图像与语言指令；（2）*action expert*（约 300M 参数），处理机器人状态和动作 token，并通过流匹配输出高频连续动作块。输出支持 14-DoF 双臂操作机器人、18-DoF 移动操作机器人及 7-8 DoF 单臂机器人。**Figure 3: Overview of the π₀ framework.** The pre-training data mixture (the in-house π manipulation datasets + the open-source OXE datasets) is fed into a unified flow-matching VLA model. The model has two parts: (1) a *pre-trained VLM backbone* (PaliGemma, SigLIP 400M + Gemma 2.6B) that processes images and language instructions; (2) an *action expert* (about 300M parameters) that processes robot state and action tokens and emits high-frequency continuous action chunks through flow matching. The outputs support 14-DoF bimanual manipulators, 18-DoF mobile manipulators, and 7-8 DoF single-arm robots.

跨机器人训练（Cross-Embodiment Training）Cross-Embodiment Training

将来自 7 种机器人平台（Bimanual UR5e、Bimanual Trossen、Bimanual ARX、UR5e、Franka、Mobile Trossen、Mobile Fibocom）的数据统一编码到同一模型中。不同机器人的 configuration 维度不同（最大 18-DoF），通过 zero-padding 对齐；多余的图像槽位通过 mask 处理。Data from 7 robot platforms (Bimanual UR5e, Bimanual Trossen, Bimanual ARX, UR5e, Franka, Mobile Trossen, Mobile Fibocom) is encoded uniformly into a single model. Different robots have different configuration dimensionalities (at most 18-DoF), which are aligned by zero-padding; surplus image slots are handled with masks.

流匹配动作生成（Flow Matching Action Expert）Flow Matching Action Expert

π₀ 采用条件流匹配（conditional flow matching）建模动作的连续分布：训练目标为 L(θ) = E[‖v_θ(A_t^τ, o_t) − u(A_t^τ | A_t)‖²]，通过强调低 τ（噪声大）时步的 beta 分布采样，使模型专注于精细动作去噪。推理时用 10 步前向欧拉积分从随机噪声 A_t^0 ~ N(0,I) 生成完整的 H=50 步动作块（action chunk），支持最高 50 Hz 的高频控制。π₀ uses conditional flow matching to model the continuous distribution of actions; the training objective is L(θ) = E[‖v_θ(A_t^τ, o_t) − u(A_t^τ | A_t)‖²]. Beta-distribution sampling that emphasizes low-τ (high-noise) timesteps makes the model concentrate on denoising fine-grained actions. At inference, 10 forward Euler integration steps turn random noise A_t^0 ~ N(0,I) into a complete H=50 action chunk, supporting high-frequency control at up to 50 Hz.

混合专家架构（Mixture of Experts）Mixture-of-Experts Architecture

模型实现为单一 Transformer，但使用两套权重（两个"专家"）：图像和语言 token 被路由到 VLM 骨干（PaliGemma，基于 Gemma 2B），而机器人状态 q_t 和动作 token A_t^τ 被路由到独立的 action expert（width=1024，mlp_dim=4096，约 300M 参数）。两套权重仅在 Transformer 自注意力层中交互，确保 VLM 预训练权重不被机器人特化数据破坏，同时保留高精度连续动作建模能力。The model is implemented as a single Transformer but uses two sets of weights (two "experts"): image and language tokens are routed to the VLM backbone (PaliGemma, based on Gemma 2B), while the robot state q_t and the action tokens A_t^τ are routed to a separate action expert (width=1024, mlp_dim=4096, about 300M parameters). The two weight sets interact only in the self-attention layers of the Transformer, which keeps the pre-trained VLM weights from being destroyed by robot-specific data while preserving high-precision continuous action modeling.

注意力掩码设计（Blockwise Causal Attention）Attention Mask Design (Blockwise Causal Attention)

π₀ 使用分块因果注意力掩码：块 1 为图像/文本输入（VLM 预训练来的前缀，不可前向关注），块 2 为机器人状态 q_t（独立缓存键值对，不随流匹配步骤变化），块 3 为噪声动作 A_t^τ（全双向注意力，可关注完整输入序列）。此设计保证推理效率：o_t 的键值对可缓存，每次只需重算 action token 的前向传播。π₀ uses a blockwise causal attention mask: block 1 is the image/text input (the prefix inherited from VLM pre-training, which may not attend forward); block 2 is the robot state q_t (its key-value pairs are cached separately and do not change with the flow-matching step); block 3 is the noisy action A_t^τ (fully bidirectional attention, able to attend to the complete input sequence). This design keeps inference efficient: the key-value pairs of o_t can be cached, so each step only needs to recompute the forward pass of the action tokens.

**图 4：预训练数据集构成。**左图按步数统计各子集相对大小：π 自有数据集约占 51%，OXE Magic Soup 等开源数据集约占 9.1%。右图展示各机器人类型的权重：Bimanual ARX 占 34.2%，Mobile Trossen 与 Mobile Fibocom 各 13.7%，Bimanual Trossen 16.3%。共 903M 步自有数据（其中 106M 来自单臂，797M 来自双臂）涵盖 68 个任务。**Figure 4: Composition of the pre-training dataset.** The left plot gives the relative size of each subset by number of steps: the in-house π datasets account for about 51%, and open-source datasets such as OXE Magic Soup for about 9.1%. The right plot shows the weight of each robot type: Bimanual ARX accounts for 34.2%, Mobile Trossen and Mobile Fibocom for 13.7% each, and Bimanual Trossen for 16.3%. In total, 903M steps of in-house data (of which 106M come from single-arm and 797M from bimanual robots) cover 68 tasks.

03 实验Experiments

实验设计围绕四个核心问题：（A）π₀ 预训练后的开箱即用能力；（B）语言指令跟随能力；（C）学习新精细任务的效率；（D）掌握复杂多阶段长时序任务的能力。所有定量结果均为 10 轮评测的平均任务完成率（normalized score，满分 1.0）。The experiments are organized around four core questions: (A) the out-of-the-box capability of π₀ after pre-training; (B) its ability to follow language instructions; (C) its efficiency at learning new dexterous tasks; (D) its ability to master complex multi-stage, long-horizon tasks. All quantitative results are average task completion rates over 10 evaluation rounds (normalized score, maximum 1.0).

A. 开箱即用评测（Out-of-Box Evaluation）A. Out-of-Box Evaluation

在 5 个任务上对比 π₀（700k 步）与 3 条基线（OpenVLA、Octo、π₀-small），每个方法均以语言指令直接驱动。On 5 tasks, π₀ (700k steps) is compared against 3 baselines (OpenVLA, Octo, π₀-small); every method is driven directly by language instructions.

开箱即用评测结果柱状图 — **图 7：开箱即用评测结果。**π₀ 在所有任务和所有对比上均取得最佳成绩，在折叠衬衫和较简单的 bussing 任务上接近满分；即便是仅训练 160k 步的"parity"版本也超越所有基线。OpenVLA 因不支持 action chunking 难以处理高频控制，在精细任务上明显落后。Octo 支持 action chunk，但模型容量有限，整体表现也弱于 π₀。**Figure 7: Out-of-box evaluation results.** π₀ obtains the best score on every task and in every comparison, coming close to a perfect score on shirt folding and on the easier bussing task; even the "parity" version trained for only 160k steps surpasses all baselines. OpenVLA, lacking action chunking, struggles with high-frequency control and falls clearly behind on dexterous tasks. Octo does support action chunks, but its model capacity is limited and its overall performance is also weaker than π₀.

方法Method	Shirt Folding (Bi-ARX)	Bussing Easy (UR5e)	Bussing Hard (UR5e)	Grocery Bagging (UR5e)	Toast (Bi-Trossen)
OpenVLA	—	低Low	低Low	低Low	—
Octo	—	低Low	低Low	低Low	—
π₀-small (non-VLM)	低Low	中Medium	低Low	低Low	低Low
π₀ parity (160k)	中高Med-High	高High	中Medium	中Medium	中Medium
π₀ (700k 步)π₀ (700k steps)	≈1.0	≈1.0	最高Best	最高Best	最高Best

注：论文 Figure 7 以柱状图展示归一化得分，上表为定性描述；具体数值详见原文图表。Note: Figure 7 of the paper presents normalized scores as a bar chart; the table above is a qualitative description, and exact values can be found in the original figures.

B. 语言指令跟随B. Following Language Instructions

在 3 个任务（bussing、grocery bagging、table setting）的语言跟随实验中，π₀ 的语言跟随准确率显著优于 π₀-small（非 VLM 初始化），表明大规模 VLM 预训练对语言理解能力至关重要。In the language-following experiments on 3 tasks (bussing, grocery bagging, table setting), the language-following accuracy of π₀ is markedly better than that of π₀-small (not initialized from a VLM), showing that large-scale VLM pre-training is crucial for language understanding.

语言跟随评测结果 — **图 9：语言指令跟随评测。**左图对比 π₀ 与 π₀-small 的语言跟随准确率（flat 条件），π₀ 在三个任务上均显著领先。右图展示任务完成率：π₀ 在人类专家中间指令（-human）和高层 VLM 规划（-HL）两种条件下均优于 π₀-small；而 π₀-small 因语言能力有限，即使加入高层指导也无法有效受益。**Figure 9: Language-instruction-following evaluation.** The left plot compares the language-following accuracy of π₀ and π₀-small (flat condition); π₀ leads clearly on all three tasks. The right plot shows task completion rates: π₀ outperforms π₀-small both under intermediate instructions from a human expert (-human) and under high-level VLM planning (-HL), whereas π₀-small, with its limited language ability, cannot benefit effectively even when high-level guidance is provided.

C. 精细新任务学习（Fine-Tuning to New Dexterous Tasks）C. Fine-Tuning to New Dexterous Tasks

在 5 个全新或半新任务上，用不同数量（1h、5h、10h）的 fine-tuning 数据对比 π₀（预训练后微调）、π₀-from-scratch、OpenVLA、Octo、ACT、Diffusion Policy。关键结论：On 5 entirely new or partly new tasks, different amounts of fine-tuning data (1h, 5h, 10h) are used to compare π₀ (fine-tuned after pre-training), π₀-from-scratch, OpenVLA, Octo, ACT, and Diffusion Policy. Key findings:

对于"较难"任务（如 Paper Towel Replacement、Franka Items in Drawer），π₀ 预训练微调版本远优于从零训练，差距有时高达 2× 以上。On the "harder" tasks (such as Paper Towel Replacement and Franka Items in Drawer), the pre-trained and fine-tuned π₀ is far better than training from scratch, with a gap that sometimes exceeds 2×.
对于"容易"任务（如 Stack Bowls、Towel Folding），π₀ 预训练版本同样领先，但差距相对较小。On the "easier" tasks (such as Stack Bowls and Towel Folding), the pre-trained π₀ likewise leads, but by a relatively smaller margin.
有趣的是，从目标任务数据完全从零训练的 ACT 和 Diffusion Policy 有时能匹敌乃至超越 OpenVLA 和 Octo 的预训练+微调版本，显示预训练对这些基线模型的增益有限，而 π₀ 架构本身的预训练更有效地迁移。Interestingly, ACT and Diffusion Policy trained entirely from scratch on the target-task data can sometimes match or even surpass the pre-trained and fine-tuned versions of OpenVLA and Octo, showing that pre-training brings limited gains to those baseline models, whereas pre-training of the π₀ architecture itself transfers more effectively.

D. 复杂多阶段任务（Complex Multi-Stage Tasks）

**图 13：复杂任务 post-training 结果。**上方图组为"预训练中已有任务"（Laundry Folding、Table Bussing、Mobile Laundry、Dryer Unloading），下方图组为"预训练中未见任务"（Box Building、To-Go Box、Packing Eggs）。各组均对比 π₀-fine-tuned（深蓝）、π₀-scratch（深蓝虚线）和 π₀-out-of-box（浅灰）三条曲线。预训练版本在大多数任务上都取得超过 50% 的满分，在最难任务上改进尤为显著。

这些任务耗时 5–20 分钟，需要结合数十种子行为（抓取、折叠、展平、放置等）才能完成。论文作者表示："These tasks are very difficult, and we were not able to solve them with other methods." π₀ 是目前端到端机器人学习文献中所展示的最长灵巧操作任务。

04 局限性Limitations

说明：论文第七节（"Discussion, Limitations, and Future Work"）包含明确的局限性讨论，以下各点均为作者明确陈述（stated）的内容，并附原文引用。Note: Section VII of the paper ("Discussion, Limitations, and Future Work") contains an explicit discussion of limitations; every point below is stated explicitly by the authors and is accompanied by a quotation from the original text.

预训练数据组成的理解仍不充分Understanding of pre-training data composition is still incomplete

"First, our experiments do not yet provide a comprehensive understanding of how the pre-training datasets should be composed: we combined all data available to us, but understanding what type of data is more helpful to add and how it should be weighted remains an open problem."（原文第 VII 节）——尚不清楚哪些数据对模型最有帮助，以及如何合理权衡。"First, our experiments do not yet provide a comprehensive understanding of how the pre-training datasets should be composed: we combined all data available to us, but understanding what type of data is more helpful to add and how it should be weighted remains an open problem." (Section VII of the paper) — it is not yet clear which data helps the model most, nor how to weight it sensibly.

并非所有任务都能可靠完成，难以预测所需数据量Not all tasks succeed reliably, and the required amount of data is hard to predict

"Not all tasks in our evaluation work reliably, and it remains unclear how to predict how much and what kind of data is needed to attain near-perfect performance."（原文第 VII 节）——当前模型并不能在所有评测任务上稳定达到高分，如何根据任务规划数据收集仍是开放问题。"Not all tasks in our evaluation work reliably, and it remains unclear how to predict how much and what kind of data is needed to attain near-perfect performance." (Section VII of the paper) — the current model does not reach high scores stably on all evaluation tasks, and how to plan data collection for a given task remains an open problem.

跨任务与跨机器人的正迁移程度尚不明朗The degree of positive transfer across tasks and across robots remains unclear

"Finally, it remains to be seen how much positive transfer there is in combining highly diverse data, particularly from different tasks and different robots: although our results suggest that universal pre-trained robot foundation models might become a reality, it is left for future work to understand whether this universality extends to much more distinct domains, such as autonomous driving, navigation, and legged locomotion."（原文第 VII 节）——迁移效果是否能推广到差异更大的领域（自动驾驶、腿足运动等）仍不确定。"Finally, it remains to be seen how much positive transfer there is in combining highly diverse data, particularly from different tasks and different robots: although our results suggest that universal pre-trained robot foundation models might become a reality, it is left for future work to understand whether this universality extends to much more distinct domains, such as autonomous driving, navigation, and legged locomotion." (Section VII of the paper) — whether the transfer effect generalizes to far more distant domains (autonomous driving, legged locomotion, and so on) is still uncertain.

仅依赖高质量数据进行 post-training 会导致模型脆弱Post-training on high-quality data alone makes the model brittle

"Training on only this high-quality data results in a brittle model that does not reliably recover from mistakes, while running the pre-trained model in zero shot does not always exhibit the fluent strategies demonstrated in the post-training data."（原文第 VII 节）——post-training 数据质量高但缺乏失败恢复样本，会导致 fine-tuned 模型在出错时缺乏鲁棒性。"Training on only this high-quality data results in a brittle model that does not reliably recover from mistakes, while running the pre-trained model in zero shot does not always exhibit the fluent strategies demonstrated in the post-training data." (Section VII of the paper) — the post-training data is high in quality but lacks failure-recovery examples, which leaves the fine-tuned model non-robust when it makes mistakes.

推断（inferred）：推理速度受流匹配多步积分限制Inferred: inference speed is limited by multi-step flow-matching integration

流匹配推理需要 10 步前向传播（每步 27 ms），总推理延迟约 73–86 ms（Table I），在 50 Hz 控制下每 0.5 秒重新推理一次。对于需要极低延迟反应的场景（如抓取高速运动物体），这一频率可能仍不够快。此点为推断（inferred from the design），作者在附录 D 中讨论了推理时间但未明确标注为局限性。Flow-matching inference requires 10 forward passes (27 ms each), for a total inference latency of about 73–86 ms (Table I); under 50 Hz control, inference is re-run every 0.5 seconds. For scenarios that demand extremely low-latency reactions (such as grasping fast-moving objects), this rate may still not be fast enough. This point is inferred from the design: the authors discuss inference time in Appendix D but do not explicitly label it as a limitation.