π₀.₅: a Vision-Language-Action Model with Open-World Generalization

01 动机Motivation

当前 VLA 模型在实验室受控场景中表现亮眼，但如何在真实的、从未见过的家庭环境中可靠执行长时域任务，仍是开放性挑战。简单堆砌机器人数据规模是不够的——泛化需要来自多个抽象层次的知识。Current VLA models look impressive in controlled laboratory settings, yet reliably executing long-horizon tasks in real, never-before-seen home environments remains an open challenge. Simply piling up robot data is not enough — generalization requires knowledge from several levels of abstraction.

"In order for robots to be useful, they must perform practically relevant tasks in the real world, outside of the lab... achieving broad coverage of plausible scenarios via brute-force scaling of robot data collection is infeasible."

问题背景Background

Vision-Language-Action (VLA) 模型将大语言模型的语义理解能力与机器人控制相结合，展现了强大的指令跟随能力。然而，现有方法的评测大多在与训练数据分布相近的环境中进行，真正的"野外"泛化能力尚未被验证。Vision-Language-Action (VLA) models combine the semantic understanding of large language models with robot control, and show strong instruction-following ability. However, existing methods are mostly evaluated in environments close to the training distribution, so genuine "in-the-wild" generalization has not yet been demonstrated.

π₀.₅ 的核心主张是：有效的开放世界泛化需要来自多个信息源的知识迁移，包括多机器人平台的数据、网页视觉数据，以及对任务语义结构的显式建模。The core claim of π₀.₅ is that effective open-world generalization requires knowledge transferred from multiple sources of information, including data from multiple robot platforms, web visual data, and explicit modeling of the semantic structure of tasks.

02 方法Method

π₀.₅ 采用两阶段训练框架：首先在异构数据上进行大规模预训练，学习多抽象层次的知识；然后通过后训练阶段专门化为移动操作能力，同时引入 flow matching 实现连续动作的精细控制。π₀.₅ adopts a two-stage training framework: first large-scale pre-training on heterogeneous data to learn knowledge at multiple levels of abstraction, then a post-training stage that specializes the model for mobile manipulation while introducing flow matching for fine-grained control of continuous actions.

模型架构：高低层级分解推理Model architecture: decomposed high- and low-level inference

π₀.₅ 将联合分布分解为两个子问题：π₀.₅ decomposes the joint distribution into two sub-problems:

• 高层推理（High-level inference）： π_θ(ℓ̂ | o_t, ℓ) — 给定当前观测和高层指令，预测当前应执行的语义子任务（如"pick up the plate"）High-level inference: π_θ(ℓ̂ | o_t, ℓ) — given the current observation and the high-level instruction, predict the semantic sub-task to be executed now (e.g. "pick up the plate")
• 低层推理（Low-level inference）： π_θ(a_t:t+H | o_t, ℓ̂) — 以预测到的子任务为条件，生成机器人的具体动作序列Low-level inference: π_θ(a_t:t+H | o_t, ℓ̂) — conditioned on the predicted sub-task, generate the robot's concrete action sequence

底层架构为多模态 Transformer：图像 patch、文本 token 和连续动作值统一输入；图像/提示词使用双向注意力，动作输出使用因果注意力；本征状态（proprioceptive state）被离散化为文本 token 输入。The backbone is a multimodal Transformer: image patches, text tokens and continuous action values are fed in as one stream; images and prompts use bidirectional attention while action outputs use causal attention; the proprioceptive state is discretized into text tokens.

混合动作表示与训练目标Hybrid action representation and training objective

为了同时支持大规模预训练的效率与推理时动作精度，π₀.₅ 引入混合损失：To support both the efficiency of large-scale pre-training and action precision at inference time, π₀.₅ introduces a hybrid loss:

Loss = Cross-entropy(text tokens) + α · ‖ω − a_t − f^a_θ(...)‖²

其中第一项为 FAST 离散动作 token 的交叉熵损失，第二项为 flow matching 连续动作的 L2 损失（后训练阶段 α=10.0）。这种设计使模型"既能进行快速可扩展的离散 token 预训练，又能通过 flow matching 实现高效精细的动作推理"。The first term is the cross-entropy loss over FAST discrete action tokens and the second is the L2 loss of flow-matching continuous actions (α=10.0 in the post-training stage). This design lets the model "perform fast, scalable discrete-token pre-training while achieving efficient and fine-grained action inference through flow matching".

五类异构训练数据Five heterogeneous training data sources

03 实验Experiments

评测分为两类：受控的"模拟房间"（mock homes）用于可复现的定量比较，以及全新真实家庭（real homes）进行最终验证。核心指标为任务完成进度（task progress）和语言跟随成功率。Evaluation comes in two kinds: controlled "mock homes" for reproducible quantitative comparison, and entirely new real homes for final validation. The core metrics are task progress and language-following success rate.

与基线的对比Comparison with baselines

模型Model	平均任务进度Average task progress	相对提升Relative gain
π₀（基线）π₀ (baseline)	~35%	—
π₀-FAST+Flow	~42%	+20%
π₀.₅（本文）π₀.₅ (ours)	~60%	+71%
在测试环境上训练的模型（上限参考）Model trained on the test environments (upper-bound reference)	~62%	—

训练环境数量的影响（Scaling）Effect of the number of training environments (scaling)

训练环境数量Number of training environments	平均任务进度Average task progress
3 个3	~25%
53 个53	~45%
104 个104	~60%
在测试家庭中训练（参考上限）Trained in the test homes (reference upper bound)	~62%

消融实验Ablations

配置Configuration	平均任务进度Average task progress	说明Notes
完整模型Full model	~60%	所有数据源All data sources
去除 WD（网页数据）Remove WD (web data)	~58%	对整体任务影响较小Minor effect on the overall task
去除 MERemove ME	~45%	性能显著下降Marked performance drop
去除 CE（跨平台数据）Remove CE (cross-embodiment data)	~40%	性能大幅下降Large performance drop
去除 ME + CERemove ME + CE	~25%	严重退化Severe degradation

高层推理的重要性Why high-level inference matters

实验进一步验证了显式高层推理的价值：The experiments further validate the value of explicit high-level inference:

• 完整 π₀.₅（显式高层推理）： ~63% 任务进度Full π₀.₅ (explicit high-level inference): ~63% task progress
• 隐式高层（有 HL 训练数据但不做显式推理）： ~58%Implicit high level (HL training data but no explicit inference): ~58%
• 无高层（无推理，无 HL 训练数据）： ~40%No high level (no inference, no HL training data): ~40%
• GPT-4 作为高层策略（oracle）： ~30%（低于预期，说明仅有语义推理不够）GPT-4 as the high-level policy (oracle): ~30% (below expectation, showing that semantic reasoning alone is not enough)
• 人类 oracle 高层策略： ~55%Human oracle high-level policy: ~55%

此外，移除 VI（语言指令，仅占训练数据 11%）会使性能下降到 ~50%，说明少量的专家语言指令对强性能"至关重要"。Moreover, removing VI (verbal instructions, only 11% of the training data) drops performance to ~50%, showing that a small amount of expert verbal instruction is "critical" to strong performance.

04 局限性Limitations

特定场景中的持续性错误Persistent failures in specific environments

作者指出："Some environments present persistent challenges (e.g., unfamiliar handles on drawers, or cabinets that are physically hard for the robot to open)." 陌生的物理结构（如特殊把手、难以打开的橱柜）仍会造成机器人持续失败，说明低层操作技能的泛化仍有瓶颈。The authors note: "Some environments present persistent challenges (e.g., unfamiliar handles on drawers, or cabinets that are physically hard for the robot to open)." Unfamiliar physical structures (unusual handles, cabinets that are hard to open) still make the robot fail repeatedly, showing that the generalization of low-level manipulation skills remains a bottleneck.

部分可观测性（Partial Observability）Partial observability

"Some behaviors present challenges with partial observability (e.g., the robot arm occluding a spill that should be wiped)." 当机械臂遮挡了需要处理的区域时，模型难以感知和规划，揭示了视觉输入受限时的决策盲区。"Some behaviors present challenges with partial observability (e.g., the robot arm occluding a spill that should be wiped)." When the arm occludes the region that has to be handled, the model struggles to perceive and plan, revealing a decision blind spot under limited visual input.

高层子任务推理容易被干扰High-level sub-task inference is easily distracted

"In some cases the high-level sub-task inference is easily distracted (e.g., closing and opening a drawer multiple times while putting away items)." 高层语义推理模块在复杂场景下可能出现循环或无效行为，说明长时域规划的稳定性仍需提升。"In some cases the high-level sub-task inference is easily distracted (e.g., closing and opening a drawer multiple times while putting away items)." The high-level semantic reasoning module can fall into loops or ineffective behavior in complex scenes, showing that the stability of long-horizon planning still needs improvement.

提示词复杂度和上下文窗口受限Limited prompt complexity and context window

模型"can accommodate relatively simple prompts"，且使用"relatively modest context"，限制了其在需要复杂多步骤规划或跨房间任务中的表现。未来工作需要探索更丰富的上下文建模能力。The model "can accommodate relatively simple prompts" and uses a "relatively modest context", which limits its performance on tasks that need complex multi-step planning or that span rooms. Future work needs to explore richer context modeling.