π*₀.₆：能从经验中学习的视觉-语言-动作模型

01 动机Motivation

机器人基础模型能否像人类一样"熟能生巧"——通过实际操作积累经验来持续提升技能？现有 VLA 模型依赖大量人工示范进行模仿学习，但"示范数据永远无法覆盖真实环境的所有变化"，导致模型在部署时仍会遭遇大量失败。如何高效地利用机器人自主采集的轨迹数据（包括失败、成功和人工干预），是迈向实用级机器人自主性的关键障碍。Can robot foundation models get better with practice the way people do — accumulating experience through actual operation and steadily improving their skills? Existing VLA models rely on large volumes of human demonstrations for imitation learning, but demonstration data can never cover every variation of a real environment, so deployed models still run into plenty of failures. Efficiently exploiting the trajectory data a robot gathers on its own (failures, successes and human interventions alike) is a key obstacle on the road to practical robot autonomy.

"Practice makes perfect — humans need many attempts at complex tasks to achieve mastery… We need methods that can learn from autonomous experience, can correct actual deployment mistakes, and can improve speed beyond human teleoperation."

任务展示胶卷 — **任务展示：**π*₀.₆ 在洗衣折叠（家庭场景）、咖啡馆制作浓缩咖啡、工厂流水线盒子组装等真实场景中的操作画面。这些任务均在 Recap 自我改进迭代后得到显著提升。**Task showcase:** π*₀.₆ operating in real settings — laundry folding at home, pulling espresso in a café, box assembly on a factory line. All of these tasks improve markedly after the self-improvement iterations of Recap.

>2×多样化洗衣与咖啡任务
吞吐量提升倍数Diverse laundry and coffee tasks
throughput gain factor

~50%失败率下降幅度
（最难任务）Drop in failure rate
(hardest task)

13 小时13 hours咖啡制作连续无间断
实际部署记录Uninterrupted espresso run
real deployment record

97%领口朝向校正子任务
严格标准下成功率Collar-orientation correction subtask
success rate under the strict criterion

任务类别示意图 — **四类评估任务：**洗衣折叠（T恤/短裤）、多样洗衣（最难物品）、制作浓缩咖啡、盒子组装。每个任务均在多轮 Recap 迭代后测量成功率与每小时完成任务数（throughput）。**Four evaluation tasks:** laundry folding (T-shirts / shorts), diverse laundry (hardest items), pulling espresso, and box assembly. For each task the success rate and the number of tasks completed per hour (throughput) are measured after several rounds of Recap iteration.

02 方法Method

Recap 将异构数据（人工示范、自主采集轨迹、专家干预）统一纳入 VLA 的训练管线，通过优势条件化策略（advantage-conditioned policy）实现策略提升—— 无需显式策略梯度，只需对成功动作与失败动作分别建模即可从经验中学习。Recap brings heterogeneous data (human demonstrations, autonomously collected trajectories, expert interventions) into a single VLA training pipeline and obtains policy improvement through an advantage-conditioned policy — no explicit policy gradient is required; modelling successful and failing actions separately is enough to learn from experience.

模型架构图 — **π*₀.₆ 系统架构：**以 Gemma 3 4B 语言骨干网络 + 860M 参数流匹配（flow-matching）动作专家为核心。训练时在提示中附加二值化优势指示符 I_t（advantage indicator），使模型同时学习无条件行为 π(a|o,ℓ) 和优势条件化行为 π(a|I,o,ℓ)，推断时用高优势分数引导策略输出。**π*₀.₆ system architecture:** built around a Gemma 3 4B language backbone plus an 860M-parameter flow-matching action expert. During training a binarised advantage indicator I_t is appended to the prompt, so the model learns the unconditional behaviour π(a|o,ℓ) and the advantage-conditioned behaviour π(a|I,o,ℓ) at the same time; at inference a high advantage value steers the policy output.

Recap 三步循环流程The three-step Recap loop

数据采集Data collection

机器人自主执行任务，标注员在必要时远程干预并提供示范修正。干预轨迹的优势指示符强制置为 True，为模型提供"改进性动作"样例。每次迭代采集约 300–600 条自主轨迹和 280–380 条干预轨迹。The robot executes the task autonomously while annotators teleoperate in when necessary and supply demonstrated corrections. The advantage indicator of an intervention trajectory is forced to True, giving the model examples of improving actions. Each iteration collects roughly 300–600 autonomous trajectories and 280–380 intervention trajectories.

价值函数训练Value function training

训练一个分布式价值函数（distributional value function），以负剩余步数（归一化至 (-1,0)）为目标。优化目标：最小化 H(R^B_t(τ), p_φ(V|o_t,ℓ))，即预测分布与真实返回的交叉熵。任务阈值 ε_ℓ 设为预训练阶段价值预测的第 30 百分位数。A distributional value function is trained with the negative number of remaining steps (normalised to (-1,0)) as its target. The objective minimises H(R^B_t(τ), p_φ(V|o_t,ℓ)), i.e. the cross-entropy between the predicted distribution and the true return. The task threshold ε_ℓ is set to the 30th percentile of the value predictions from the pre-training stage.

优势条件化策略提取Advantage-conditioned policy extraction

预训练 + 任务特化两阶段设计A two-stage design: pre-training plus task specialisation

离线 RL 预训练Offline RL pre-training Pre-training

在数万小时的多机器人数据上用 Recap 进行大规模离线 RL 预训练，使 π*₀.₆ 获得广泛的"何时需要改进"先验知识，为下游任务特化提供更好的初始化。Large-scale offline RL pre-training with Recap on tens of thousands of hours of multi-robot data gives π*₀.₆ broad prior knowledge about when improvement is needed, and a better initialisation for downstream task specialisation.

任务迭代精调Iterative task-specific fine-tuning Task-specific

针对具体部署任务，循环执行上述三步流程（通常 2 轮迭代），利用任务特定的成功/失败奖励信号逐步消除失败模式并提升操作速度。For a concrete deployment task, the three-step loop above is run repeatedly (typically 2 iterations), using task-specific success/failure reward signals to progressively eliminate failure modes and raise operating speed.

稀疏奖励定义Sparse reward definition

任务奖励函数定义为：r_t = 0（成功）、−C_fail（失败）、−1（每步惩罚）。该稀疏奖励仅依赖人工标记的片段级成功/失败标签，无需设计复杂的密集奖励函数。The task reward is defined as r_t = 0 (success), −C_fail (failure), −1 (per-step penalty). This sparse reward depends only on human-labelled episode-level success/failure flags, so no elaborate dense reward function has to be designed.

03 实验Experiments

在 4 个真实机器人任务上对比 π*₀.₆（有/无 Recap）及多个基线方法，主要指标为每小时成功完成任务数（throughput）和成功率（success rate）。π*₀.₆ (with and without Recap) is compared against several baseline methods on 4 real robot tasks; the main metrics are the number of tasks completed successfully per hour (throughput) and the success rate.

主要量化结果Main quantitative results

任务Task	基线 π₀.₆（无RL）Baseline π₀.₆ (no RL)	π*₀.₆ + Recap	提升Gain
洗衣折叠（T恤/短裤）Laundry folding (T-shirt / shorts)	~5.5 任务/时, ~90%~5.5 tasks/h, ~90%	~8.5 任务/时, ~95%~8.5 tasks/h, ~95%	吞吐量 +55%+55% throughput
洗衣折叠（多样·最难物品）Laundry folding (diverse, hardest items)	~3 任务/时, ~50%~3 tasks/h, ~50%	~7 任务/时, ~75%~7 tasks/h, ~75%	>2× 吞吐量>2× throughput
制作浓缩咖啡Pulling espresso	~2 任务/时, ~45%~2 tasks/h, ~45%	~5 任务/时, ~90%~5 tasks/h, ~90%	>2× 吞吐量>2× throughput
盒子组装Box assembly	~5 任务/时, ~75%~5 tasks/h, ~75%	~9.5 任务/时, ~90%~9.5 tasks/h, ~90%	>2× 吞吐量>2× throughput

多样洗衣实验结果 — **多样洗衣（最难物品）：**Recap 每轮迭代持续提升吞吐量，最终超过 2× 的基线水平，而对比基线（π₀.₅、纯SFT、AWR、PPO）均未达到同等效果。**Diverse laundry (hardest items):** Recap raises throughput at every iteration, ending above 2× the baseline level, whereas none of the comparison baselines (π₀.₅, plain SFT, AWR, PPO) reaches the same result.

咖啡制作实验结果 — **制作浓缩咖啡：**π*₀.₆+Recap 在成功率和吞吐量上均超过 2× 基线，并完成了连续 **13 小时**无间断自动制作浓缩咖啡的实际部署测试。**Pulling espresso:** π*₀.₆+Recap exceeds 2× the baseline in both success rate and throughput, and completed a real deployment test making espresso automatically for **13 hours** without interruption.

T恤折叠实验结果 — **洗衣折叠（T恤/短裤）：**经过两轮 Recap 迭代后吞吐量提升约 50%，成功率从 ~90% 提升至 ~95%。每轮采集约 300 条轨迹（4台机器人并行）。**Laundry folding (T-shirt / shorts):** after two Recap iterations throughput rises by about 50% and the success rate goes from ~90% to ~95%. Each round collects about 300 trajectories (4 robots in parallel).

盒子组装实验结果 — **盒子组装：**两轮迭代后吞吐量提升约 2×，在工厂部署场景下各子任务（取件 90%、组装 95%、贴标 85%、放置 85%）均达到高成功率。**Box assembly:** after two iterations throughput improves by about 2×, and in the factory deployment every subtask (picking 90%, assembly 95%, labelling 85%, placing 85%) reaches a high success rate.

基线方法对比Comparison of baseline methods

方法Method	类型Type	表现Performance
π₀.₅	上一代通用模型，无RLPrevious-generation generalist model, no RL	最低基线Lowest baseline
π₀.₆（SL baseline）π₀.₆ (SL baseline)	监督学习，无优势条件化Supervised learning, no advantage conditioning	中等Middling
Offline RL + SFT	仅使用示范精调Fine-tuning on demonstrations only	略高于SLSlightly above SL
AWR（优势加权回归）AWR (advantage-weighted regression)	替代RL算法Alternative RL algorithm	低于RecapBelow Recap
PPO（策略梯度）PPO (policy gradient)	在线RLOnline RL	显著低于RecapMarkedly below Recap
*π₀.₆ + Recap（本文）π₀.₆ + Recap (this paper)*	离线优势条件化RLOffline advantage-conditioned RL	最优Best

消融实验与迭代改进Ablations and iterative improvement

消融实验验证了三个关键设计选择的必要性：The ablations confirm that three key design choices are necessary:

RL 预训练的必要性：跳过大规模预训练直接进行任务特化的效果明显更差，说明广泛先验知识对适应具体任务至关重要。Necessity of RL pre-training: skipping large-scale pre-training and going straight to task specialisation is clearly worse, showing that broad prior knowledge is essential for adapting to a specific task.
优势条件化 vs AWR：与优势加权回归相比，优势条件化在多任务场景下更稳定，能更有效地利用失败轨迹。Advantage conditioning vs AWR: compared with advantage-weighted regression, advantage conditioning is more stable in multi-task settings and makes more effective use of failed trajectories.
干预数据的价值：专家远程干预（intervention）的轨迹标记为高优势，为模型提供了"如何在当前状态下做得更好"的直接示例，显著加速收敛。Value of intervention data: trajectories from expert remote interventions are marked as high-advantage, giving the model direct examples of how to do better in the current state, which markedly accelerates convergence.

领口朝向校正专项任务（collar-orientation）：经过 2 轮 Recap 迭代（共采集 1,200 条轨迹），在严格标准下成功率达到 97%，在家庭环境中折叠新款洗衣连续运行 超过 2 小时无需人工干预。Dedicated collar-orientation correction task: after 2 rounds of Recap iteration (1,200 trajectories collected in total), the success rate under the strict criterion reaches 97%, and the system folds unseen laundry in a home environment for more than 2 hours continuously with no human intervention.

04 局限性Limitations

说明：以下局限性均为论文作者在 Limitations 章节中明确陈述（stated），非推断（inferred）。原文引用如下。Note: every limitation below is explicitly stated by the authors in the Limitations section of the paper rather than inferred. The original quotations follow.

依赖人工标注与人工干预Reliance on human labelling and human intervention

"Our system is not fully autonomous: it relies on human labeling and effort for reward feedback, interventions, and episode resets." ——系统并非完全自主，每轮迭代需要标注员提供片段级成功/失败标签，并在机器人卡死时进行重置或干预，限制了大规模自动化部署的可行性。— The system is not fully autonomous: every iteration needs annotators to supply episode-level success/failure labels and to reset or intervene when the robot gets stuck, which limits the feasibility of large-scale automated deployment.

探索策略较为朴素Exploration is relatively naïve

"Our system is relatively naïve in how it approaches exploration." ——Recap 的探索依赖策略自身的随机性和人工干预来访问新状态，缺乏主动探索（active exploration）机制，对于需要大幅偏离当前策略才能改进的任务可能效率较低。— Recap's exploration relies on the stochasticity of the policy itself and on human intervention to visit new states; it has no active-exploration mechanism, so it may be inefficient on tasks that require a large departure from the current policy in order to improve.

批量离线 RL，非完全在线学习Batched offline RL rather than fully online learning

"Recap performs iterated 'offline' updates rather than running a fully online RL loop." ——每次策略更新需要先收集一批数据再离线更新，而非实时在线强化学习，导致样本效率低于理论最优的在线方法，且难以快速响应分布偏移。— Each policy update first collects a batch of data and then updates offline instead of running real-time online reinforcement learning, which makes sample efficiency lower than that of theoretically optimal online methods and makes it hard to react quickly to distribution shift.

规模化与推广性尚待验证Scaling and generality remain to be validated

推断（inferred）：目前评估任务数量有限（4个），且均在 Physical Intelligence 内部场景测试，对更广泛的任务类别（如动态环境、多物体复杂操作）的泛化能力尚未系统验证。此外，预训练阶段需要"数万小时"多机器人数据，数据获取成本对外部研究者构成较大门槛。Inferred: the number of evaluation tasks is currently limited (4), and all of them are tested in Physical Intelligence's own settings, so generalisation to broader task categories (dynamic environments, complex multi-object manipulation) has not been systematically verified. In addition, the pre-training stage requires tens of thousands of hours of multi-robot data, and the cost of acquiring it is a considerable barrier for outside researchers.