Learning while Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies

01 Motivation

近年的 VLA 策略（RT-1/RT-2、Octo、OpenVLA、π0、π0.5 等）靠大规模离线预训练获得广泛能力，但离线数据本身不足以支撑稳健的真实世界部署。论文指出："offline pretraining alone does not make a policy deployment-ready" —— 部署环境不是固定的测试分布，机器人会不断遇到分布偏移、长尾失败、任务变体，以及人工纠正的机会，而固定的示教数据集无法覆盖这些情况。单台机器人只能采样部署分布的很小一部分，而一个车队能覆盖多样的任务、环境、物体和用户指令，产生包含成功、失败、恢复、部分进展和人工干预的异质经验。Recent VLA policies (RT-1/RT-2, Octo, OpenVLA, π0, π0.5 and others) gain broad capabilities from large-scale offline pretraining, but offline data alone is not enough to support robust real-world deployment. The paper argues: "offline pretraining alone does not make a policy deployment-ready" — a deployment environment is not a fixed test distribution; a robot keeps running into distribution shift, long-tail failures, task variants and opportunities for human correction, none of which a fixed demonstration dataset can cover. A single robot can only sample a tiny slice of the deployment distribution, whereas a fleet covers diverse tasks, environments, objects and user instructions, producing heterogeneous experience that contains successes, failures, recoveries, partial progress and human interventions.

"We refer to this setting as Learning While Deploying (LWD): continual policy improvement driven by the accumulated real-world autonomous experience of a deployed robot fleet."

此前的交互式模仿学习方法（如 HG-DAgger）只把部署数据当作监督学习的动作标签来源，无法利用包含成功/失败/部分进展/任务奖励的完整自主试错经验；而现有机器人 RL 方法大多局限于小规模、短时序或单任务场景，很少能在保持通才能力的同时，从车队部署经验里对端到端 VLA 策略做规模化后训练——这正是 LWD 要填补的空白。Earlier interactive imitation learning methods (such as HG-DAgger) treat deployment data only as a source of action labels for supervised learning, and cannot exploit the full autonomous trial-and-error experience that contains successes, failures, partial progress and task rewards; existing robot RL methods, meanwhile, are mostly confined to small-scale, short-horizon or single-task settings, and rarely manage scalable post-training of an end-to-end VLA policy from fleet deployment experience while preserving generalist capability — exactly the gap LWD sets out to fill.

02 Method

LWD 分两个阶段：Stage 1 离线预训练在静态离线缓冲区 B_off（示教 + 历史策略 rollout + 人工引导的失败探索 play data）上训练策略、critic 和分布式价值模型，为部署提供强初始化；Stage 2 在线后训练把当前策略部署到机器人车队做自主 rollout，把 policy transition 与人工干预汇入不断更新的在线缓冲区 B_on，learner 在 B_off ∪ B_on 混合回放上持续更新价值函数与策略，并周期性把新 checkpoint 重新分发给车队——形成一个数据飞轮。两个阶段共用同一套 RL 目标，缓解了 offline-to-online 常见的价值失配问题。LWD proceeds in two stages. Stage 1, offline pretraining, trains the policy, the critic and the distributional value model on a static offline buffer B_off (demonstrations + rollouts of earlier policies + human-guided failure-exploration play data), providing a strong initialization for deployment. Stage 2, online post-training, deploys the current policy on the robot fleet for autonomous rollouts, funnels policy transitions and human interventions into a continually updated online buffer B_on, and has the learner keep updating the value functions and the policy on mixed replay over B_off ∪ B_on, periodically redistributing new checkpoints to the fleet — forming a data flywheel. Both stages share the same RL objective, which alleviates the value-mismatch problem that commonly afflicts offline-to-online transitions.

Distributional Implicit Value Learning (DIVL)

DIVL 是 Implicit Q-Learning (IQL) 的分布式改造：IQL 用标量 expectile 回归把价值向数据集中较高价值的动作偏置，从而在不做显式 max_a Q(s,a) backup 的情况下给出隐式的策略提升目标；但车队异步采集数据下，同一 state-action 对应的回报可能是多峰、重尾的，标量 critic 会把这些结果压成一个平均值，抹掉罕见但可复现的高回报模式。DIVL 转而学习一个 categorical 分布式价值模型 p_ψ(v∣s_t)，用其 τ-quantile 作为 TD bootstrap 目标；论文证明（Proposition 1）对同一族非对称损失，直接标量回归与"先拟合分布再取分位数"这一两步过程收敛到相同的最优标量价值。此外，τ 会依据价值分布的归一化熵自适应调整：分布越发散（越不确定），τ 越保守，反之越乐观。DIVL is a distributional reworking of Implicit Q-Learning (IQL). IQL uses scalar expectile regression to bias the value toward the higher-value actions in the dataset, yielding an implicit policy-improvement target without an explicit max_a Q(s,a) backup; but when data is collected asynchronously by a fleet, the returns for one and the same state-action pair can be multi-modal and heavy-tailed, and a scalar critic squashes those outcomes into a single mean, erasing rare but reproducible high-return modes. DIVL instead learns a categorical distributional value model p_ψ(v∣s_t) and uses its τ-quantile as the TD bootstrap target; the paper proves (Proposition 1) that for the same family of asymmetric losses, direct scalar regression and the two-step procedure of "fit the distribution first, then take the quantile" converge to the same optimal scalar value. In addition, τ adapts to the normalized entropy of the value distribution: the more dispersed (the more uncertain) the distribution, the more conservative τ becomes, and the other way round the more optimistic.

Q-learning via Adjoint Matching (QAM) 策略抽取Policy extraction via Q-learning through Adjoint Matching (QAM)

Critic 的动作梯度 ∇_a Q_φ(s,a) 本可直接指导策略改进，但 flow-based VLA 的动作生成是多步去噪过程，直接对整条去噪轨迹反传梯度"is computationally expensive and numerically unstable"。QAM 把轨迹级的 KL 正则化策略提升目标，转化为沿参考 flow 轨迹的局部回归目标：用 DIVL critic 在去噪终点的梯度设定 adjoint 终端条件，再逐步反解 adjoint 动态，逼近参考策略 f_β 与待优化策略 f_θ 之间的局部差异。整个训练中 f_β 固定为离线 RL 前的行为克隆策略，f_θ 在离线和在线阶段都持续优化。The critic's action gradient ∇_a Q_φ(s,a) could in principle guide policy improvement directly, but action generation in a flow-based VLA is a multi-step denoising process, and backpropagating gradients through the entire denoising trajectory "is computationally expensive and numerically unstable". QAM turns the trajectory-level KL-regularized policy-improvement objective into a local regression objective along the reference flow trajectory: the gradient of the DIVL critic at the denoising endpoint sets the adjoint terminal condition, the adjoint dynamics are then solved backwards step by step, approximating the local difference between the reference policy f_β and the policy being optimized f_θ. Throughout training f_β stays fixed to the behavior-cloning policy from before offline RL, while f_θ keeps being optimized in both the offline and the online stage.

网络架构Network architecture

价值/critic 网络与策略网络分离：actor 采用 π0.5 flow-based VLA 架构（PaliGemma 视觉语言骨干 + Gemma-2B 语言模型 + SigLIP 视觉编码器 + Gemma-300M action expert）；价值 V_ψ 与 critic Q_φ 共享一个 Gemma3-SigLIP 骨干（Gemma 3-270M-IT + SigLIP-So400M 初始化），critic 采用 clipped double-Q 设计以缓解高估。只有策略 checkpoint 会异步分发到车队用于推理，价值/critic 网络始终留在中心 learner 上。在线 QAM 更新时，策略的 VLM 骨干被冻结、只更新 action expert，以兼顾更新效率与预训练视觉语言表征的稳定性。The value/critic networks are kept separate from the policy network: the actor uses the π0.5 flow-based VLA architecture (PaliGemma vision-language backbone + Gemma-2B language model + SigLIP vision encoder + Gemma-300M action expert); the value V_ψ and the critic Q_φ share a Gemma3-SigLIP backbone (initialized from Gemma 3-270M-IT + SigLIP-So400M), and the critic adopts a clipped double-Q design to mitigate overestimation. Only policy checkpoints are distributed asynchronously to the fleet for inference; the value/critic networks always stay on the central learner. During online QAM updates the policy's VLM backbone is frozen and only the action expert is updated, balancing update efficiency against the stability of the pretrained vision-language representations.

03 Experiments

实验平台为 Agibot G1 双臂机器人（两条 7-DoF 手臂 + 平行夹爪 + 3 路 RGB 相机，30 Hz 关节位控），在线训练时部署 16 台机器人车队（4 台跑生鲜/超市补货任务，其余每个长时序任务各配 3 台）。8 项真实任务分两组：4 项超市补货任务（flat-shelf restocking、misplaced-item correction、freezer restocking、open-cooler restocking，二值成功率）；4 项 3–5 分钟长时序任务（Gongfu Tea、Fruit Juice、Cocktail、Shoebox，按 5–8 个标注子步骤打分，1/0.5/0 三档评分取平均）。对比对象：reference 策略、SFT、以及两个代表性后训练基线 RECAP 和 HG-DAgger。每个在线实验分配 4 小时墙钟预算，对应车队约 60 小时在线数据；learner 每 50 个训练步向车队广播一次新策略。The experimental platform is the Agibot G1 bimanual robot (two 7-DoF arms + parallel grippers + 3 RGB cameras, 30 Hz joint position control); online training deploys a fleet of 16 robots (4 running the grocery/supermarket restocking tasks, with 3 assigned to each of the remaining long-horizon tasks). The 8 real-world tasks split into two groups: 4 supermarket restocking tasks (flat-shelf restocking, misplaced-item correction, freezer restocking, open-cooler restocking, scored by binary success); and 4 long-horizon tasks of 3–5 minutes (Gongfu Tea, Fruit Juice, Cocktail, Shoebox, scored over 5–8 annotated sub-steps on a 1/0.5/0 three-level scale and averaged). The comparisons are the reference policy, SFT, and two representative post-training baselines, RECAP and HG-DAgger. Each online experiment is granted a 4-hour wall-clock budget, corresponding to roughly 60 hours of online fleet data; the learner broadcasts a new policy to the fleet every 50 training steps.

Main Results

LWD (Online) 在全部 8 项任务上取得最优平均分 0.95，全面超过所有基线，且在四项长时序任务上均拿到最高分。LWD (Online) attains the best average score, 0.95, across all 8 tasks, surpassing every baseline, and takes the top score on each of the four long-horizon tasks.

长时序任务上的优势最明显：LWD (Online) 长时序平均分 0.91，高于 SFT (0.68)、RECAP (0.77)、HG-DAgger (0.73) 和 LWD (Offline) (0.79)。论文分析 HG-DAgger 在长时序任务上收益有限、有时甚至掉分，原因可能是"DAgger-style training relies on human correction data, whose variability can introduce inconsistencies and provide limited exploration of the broader state space"；而 RL 能通过 TD backup 把终端成功信号传播到更早的决策步骤，为长时序任务提供更强的学习信号。在已经趋于饱和的超市补货任务上，LWD (Online) 依然保持在最优或接近最优水平，说明它在带来长时序收益的同时没有牺牲通才行为。The advantage is clearest on long-horizon tasks: LWD (Online) reaches a long-horizon average of 0.91, above SFT (0.68), RECAP (0.77), HG-DAgger (0.73) and LWD (Offline) (0.79). The paper suggests that HG-DAgger gains little on long-horizon tasks and sometimes even loses ground because "DAgger-style training relies on human correction data, whose variability can introduce inconsistencies and provide limited exploration of the broader state space"; RL, by contrast, can propagate the terminal success signal to much earlier decision steps through TD backup, providing a stronger learning signal for long-horizon tasks. On the already near-saturated supermarket restocking tasks LWD (Online) still stays at or close to the best level, showing that its long-horizon gains do not come at the cost of generalist behavior.

Ablations

价值学习设计（DIVL vs 标量 expectile 回归）：在保持其他组件不变的情况下，DIVL 在全部任务上优于标量基线，长时序任务上收益更大——离线阶段 +9.7%，在线阶段 +16.7%。Value learning design (DIVL vs scalar expectile regression): with all other components held fixed, DIVL beats the scalar baseline on every task, with larger gains on long-horizon tasks — +9.7% in the offline stage and +16.7% in the online stage.

自适应 τ 策略：相较把 τ 固定为训练统计平均值（τ=0.52）的常量基线，自适应 τ 把离线平均分从 0.84 提升到 0.88，在 Restocking、Correction、Cocktail 等任务上收益更一致——这说明基于分布熵调节 bootstrap 乐观程度确实有帮助：不确定性高时更保守，价值估计确信时更乐观。Adaptive τ schedule: compared with a constant baseline that fixes τ at the mean of the training statistics (τ=0.52), adaptive τ lifts the offline average score from 0.84 to 0.88, with more consistent gains on tasks such as Restocking, Correction and Cocktail — evidence that modulating bootstrap optimism by the entropy of the distribution really does help: more conservative when uncertainty is high, more optimistic when the value estimate is confident.

论文还可视化了 Gongfu Tea 成功/失败两条 episode 的价值曲线（Fig. 5，未收录本页）：成功轨迹中价值随子步骤完成逐步上升，失败轨迹中价值在停止推进后维持在较低水平——作者将其定性描述为"the learned value estimate tracking task-progress differences in these examples"，用于诊断稀疏奖励下价值函数是否捕捉到了任务进展信号，而非作为额外的定量结果。The paper also visualizes the value curves of a successful and a failed Gongfu Tea episode (Fig. 5, not reproduced on this page): on the successful trajectory the value rises step by step as sub-steps are completed, while on the failed trajectory it stays at a low level once progress halts — the authors describe this qualitatively as "the learned value estimate tracking task-progress differences in these examples", using it to diagnose whether the value function captures the task-progress signal under sparse rewards, rather than as an additional quantitative result.

04 Limitations