LifeLong-RFT：面向长效机器人的持续学习 VLA 强化微调

01 Motivation

在大规模多样数据上预训练的 VLA 模型展现出很强的泛化与适应能力，但把它们适配到新任务时，主流做法 Supervised Fine-Tuning（SFT）存在两个纠缠在一起的问题：需要大量任务专属数据才能学好，而且学新技能时会侵蚀已有能力，即 catastrophic forgetting。作者指出这两个问题并非独立：“improving data-efficient adaptation often exacerbates forgetting, while preserving prior knowledge restricts effective learning from limited new data”——要让机器人成长为能够不断学习新技能的 long-lived agent，必须在 plasticity 与 stability 之间取得平衡。VLA models pre-trained on large-scale, diverse data exhibit strong generalization and adaptability, but when they are adapted to new tasks the mainstream approach, Supervised Fine-Tuning (SFT), suffers from two entangled problems: it needs large amounts of task-specific data to learn well, and learning new skills erodes existing capabilities, i.e. catastrophic forgetting. The authors point out that these two problems are not independent: “improving data-efficient adaptation often exacerbates forgetting, while preserving prior knowledge restricts effective learning from limited new data” — for a robot to grow into a long-lived agent that keeps learning new skills, a balance must be struck between plasticity and stability.

“can on-policy reinforcement learning be leveraged to enable continual adaptation of VLA foundation models, supporting their evolution into long-lived agents?”

受 LLM 领域“on-policy RL 对遗忘更鲁棒”这一观察启发，本文用不依赖环境交互、不依赖额外奖励模型的 RFT 策略回答这个问题——核心难点在于如何为 VLA 的动作序列设计一套高效、可靠、可扩展的奖励机制。Inspired by the observation in the LLM literature that “on-policy RL is more robust to forgetting”, this paper answers the question with an RFT strategy that depends on neither environment interaction nor an extra reward model — the core difficulty being how to design an efficient, reliable and scalable reward mechanism for the action sequences of a VLA.

LifeLong-RFT 概览图：SFT 与持续学习中的数据依赖和遗忘问题 — Figure 1（论文原图）：VLA post-training 的两个阶段——单阶段多任务适配与增量式持续学习。SFT 在这两个阶段都面临数据依赖重、易遗忘的问题，LifeLong-RFT 结合 on-policy RL 与多维过程奖励机制来应对。Figure 1 (from the paper): the two stages of VLA post-training — single-stage multi-task adaptation and incremental continual learning. SFT faces heavy data dependence and forgetting in both stages, which LifeLong-RFT addresses by combining on-policy RL with a multi-dimensional process reward mechanism.

+22%LIBERO 持续学习平均成功率相对 SFT 的提升（论文摘要原话）Gain in average success rate on LIBERO continual learning over SFT (verbatim from the paper abstract)

20%适配新任务仅需的训练数据比例Fraction of training data needed to adapt to a new task

+8.7真实世界多任务学习 overall 成功率相对 SFT baseline 的提升（百分点）Gain in real-world multi-task learning overall success rate over the SFT baseline (percentage points)

+31.7真实世界持续学习 AUC 指标相对 SFT baseline 的提升Gain in the real-world continual learning AUC metric over the SFT baseline

02 Method

LifeLong-RFT 在离散动作 VLA（以 NORA-Long 为 backbone）上，把 chunking-level on-policy reinforcement learning 与一个 multi-dimensional process reward 机制结合起来：对策略生成的中间 action chunk 逐块打分，量化它们对最终任务成功的“异质贡献”，从而更精细地引导策略优化，而不需要真实环境的在线反馈，也不需要额外训练一个奖励模型。On discrete-action VLAs (with NORA-Long as the backbone), LifeLong-RFT combines chunking-level on-policy reinforcement learning with a multi-dimensional process reward mechanism: it scores the intermediate action chunks generated by the policy chunk by chunk, quantifying their “heterogeneous contributions” to final task success, so as to guide policy optimization more finely without needing online feedback from a real environment or training an extra reward model.

LifeLong-RFT 方法总览：chunking-level on-policy RL + 三个过程奖励 — Figure 2（论文原图）：LifeLong-RFT 总览。该策略将 chunking-level on-policy RL 算法与多维过程奖励机制结合，共同引导策略优化。Figure 2 (from the paper): overview of LifeLong-RFT. The strategy combines a chunking-level on-policy RL algorithm with a multi-dimensional process reward mechanism to jointly guide policy optimization.

Multi-Dimensional Process Reward：三个互补的奖励维度Multi-Dimensional Process Reward: Three Complementary Reward Dimensions

三个奖励分别作用于动作预测的不同环节：The three rewards act on different stages of action prediction:

Quantized Action Consistency Reward (QACR) —— “ensures accurate action prediction within the discrete action space”，约束离散 token 层面的动作预测准确性。Quantized Action Consistency Reward (QACR) — “ensures accurate action prediction within the discrete action space”, constraining the accuracy of action prediction at the discrete token level.
Continuous Trajectory Alignment Reward (CTAR) —— “aligns decoded continuous action chunks with reference trajectories to ensure precise control”，把离散 token 解码回连续动作后与参考轨迹对齐，保证控制精度。消融实验显示这是三者中最关键的一项：去掉它后平均成功率从 95.6 骤降到 4.7（Δ −90.9）。Continuous Trajectory Alignment Reward (CTAR) — “aligns decoded continuous action chunks with reference trajectories to ensure precise control”, decoding discrete tokens back into continuous actions and aligning them with the reference trajectory to guarantee control precision. Ablations show this is the most critical of the three: removing it makes the average success rate plunge from 95.6 to 4.7 (Δ −90.9).
Format Compliance Reward (FCR) —— “guarantees the structural validity of outputs”，用 Fast+ tokenizer 校验生成 token 序列是否符合规定的动作维度与 chunk size。Format Compliance Reward (FCR) — “guarantees the structural validity of outputs”, using the Fast+ tokenizer to check whether the generated token sequence conforms to the prescribed action dimensionality and chunk size.

Chunking-Level On-Policy RL

该策略以 action chunk（而非单步 action）为最小优化单元，用当前策略自身采样出的样本来更新参数（on-policy），这使其相较依赖标注数据集的 SFT，对遗忘更加鲁棒——这也是论文选择 RFT 而非继续沿用 SFT-based continual learning 技术的核心动机。The strategy takes the action chunk (rather than a single-step action) as the smallest unit of optimization and updates parameters with samples drawn by the current policy itself (on-policy), which makes it more robust to forgetting than SFT on annotated datasets — this is also the paper’s core motivation for choosing RFT over continuing with SFT-based continual learning techniques.

03 Experiments

实验覆盖 SimplerEnv、LIBERO 两个仿真 benchmark 与真实机器人任务，分为 multi-task learning（单阶段多任务）与 continual learning（增量持续学习）两条主线，backbone 统一为 NORA-Long（discrete action VLA），对比对象是同 backbone 下的 SFT baseline，以及 Octo、π0、OpenVLA、SpatialVLA、GR00T 等一系列连续/离散动作 VLA。Experiments span the two simulation benchmarks SimplerEnv and LIBERO plus real-robot tasks, along two main lines — multi-task learning (single-stage multi-task) and continual learning (incremental) — with NORA-Long (discrete action VLA) as the unified backbone, compared against the SFT baseline on the same backbone as well as a series of continuous/discrete action VLAs such as Octo, π0, OpenVLA, SpatialVLA and GR00T.

Multi-Task Learning（单阶段多任务适配）Multi-Task Learning (Single-Stage Multi-Task Adaptation)

Benchmark	NORA-Long SFT (baseline)	NORA-Long RFT (Ours)	Δ
SimplerEnv · WidowX avg	65.5	69.0	+3.5
SimplerEnv · Google Robot avg	74.7	79.1	+4.4
LIBERO avg	91.8	95.6	+3.8
Real-world overall SR	78.8	87.5	+8.7

Continual Learning（LIBERO，AUC ↑，同一 backbone NORA-Long 下 RFT vs SFT）Continual Learning (LIBERO, AUC ↑, RFT vs SFT on the same NORA-Long backbone)

Task Split	NORA-Long SFT (baseline)	NORA-Long RFT (Ours)	Δ
LIBERO-Object	79.7	94.8	+15.1
LIBERO-Spatial	71.7	91.2	+19.5
LIBERO-Goal	54.4	90.3	+35.9
LIBERO-Long	47.3	64.5	+17.2

四个 split 的 AUC 提升均值为 (15.1+19.5+35.9+17.2)/4 ≈ +21.9，对应摘要所述 “a 22% gain in average success rate over SFT”。真实世界持续学习上，AUC 从 44.2 → 75.9（Δ +31.7），NBT（遗忘指标，越低越好）从 18.3 → 6.1（Δ −12.2）。The mean AUC gain across the four splits is (15.1+19.5+35.9+17.2)/4 ≈ +21.9, matching the “a 22% gain in average success rate over SFT” stated in the abstract. In real-world continual learning, AUC rises from 44.2 → 75.9 (Δ +31.7) and NBT (a forgetting metric, lower is better) drops from 18.3 → 6.1 (Δ −12.2).

LifeLong-RFT 在新任务上的数据适应效率曲线 — Figure 4（论文原图）：Adaptation efficiency on representative new tasks —— LifeLong-RFT 用远少于 SFT 的训练数据即可达到相近或更高的成功率，论文据此得出“仅需 20% 训练数据即可有效适配新任务”的结论。Figure 4 (from the paper): Adaptation efficiency on representative new tasks — LifeLong-RFT reaches comparable or higher success rates with far less training data than SFT, from which the paper concludes that “only 20% of the training data is needed to adapt effectively to new tasks”.

Ablations

奖励组合权重的消融实验结果 — Figure 5（论文原图）：Ablation study on the reward combination weights，展示三个奖励项权重变化对最终性能的影响。Figure 5 (from the paper): Ablation study on the reward combination weights, showing how varying the weights of the three reward terms affects final performance.

TABLE VI 的消融进一步量化了各奖励项的贡献（LIBERO 四个 split 平均 SR，Ours = 95.6）：w/o QACR 降至 92.8（Δ −2.8）；w/o FCR 降至 93.0（Δ −2.6）；w/o CTAR 骤降至 4.7（Δ −90.9）——说明 CTAR 对齐连续动作轨迹这一项是整个 reward 设计中最不可或缺的部分，没有它策略几乎无法学习有效控制。The ablation in TABLE VI further quantifies the contribution of each reward term (average SR over the four LIBERO splits, Ours = 95.6): w/o QACR drops to 92.8 (Δ −2.8); w/o FCR drops to 93.0 (Δ −2.6); w/o CTAR plunges to 4.7 (Δ −90.9) — showing that aligning continuous action trajectories via CTAR is the most indispensable part of the entire reward design, without which the policy can barely learn effective control.

04 Limitations

Note：第一条为论文 “Limitations & Future Work” 段落原话；后两条为根据方法设计与消融结果推断（inferred），非作者明确列出。Note: The first item is verbatim from the paper’s “Limitations & Future Work” paragraph; the latter two are inferred from the method design and the ablation results, and are not explicitly listed by the authors.

仅验证于离散动作模型（stated）Validated only on discrete action models (stated)

论文原话：“This work primarily focuses on discrete action models, yet their performance falls short of the levels achieved by continuous action models. Future research extending the LifeLong-RFT training strategy to continuous action models will significantly accelerate the transition of VLAs from laboratory research to industrial applications.” 即当前方法基于 NORA-Long 等离散动作 VLA，其绝对性能上限仍低于 π0 等连续动作模型，尚未验证能否直接迁移到连续动作范式。Verbatim from the paper: “This work primarily focuses on discrete action models, yet their performance falls short of the levels achieved by continuous action models. Future research extending the LifeLong-RFT training strategy to continuous action models will significantly accelerate the transition of VLAs from laboratory research to industrial applications.” That is, the current method is built on discrete action VLAs such as NORA-Long, whose absolute performance ceiling is still below that of continuous action models such as π0, and it remains unverified whether it can transfer directly to the continuous action paradigm.

对 CTAR 及其参考轨迹的依赖较重（inferred）Heavy reliance on CTAR and its reference trajectories (inferred)

消融实验显示去掉 CTAR 后平均成功率从 95.6 暴跌到 4.7，说明多维过程奖励的有效性高度集中在“将解码后的连续动作与参考轨迹对齐”这一项上——而 reference trajectory 本身仍来自演示数据，这意味着该方法虽然不依赖在线环境反馈或预训练奖励模型，但并未完全摆脱对离线示范轨迹的依赖，且奖励设计对该单一分量较为脆弱。Ablations show that removing CTAR makes the average success rate collapse from 95.6 to 4.7, indicating that the effectiveness of the multi-dimensional process reward is highly concentrated in “aligning decoded continuous actions with reference trajectories” — yet the reference trajectory itself still comes from demonstration data, which means that although the method depends on neither online environment feedback nor a pre-trained reward model, it has not fully escaped its reliance on offline demonstration trajectories, and the reward design is rather fragile with respect to this single component.

评测任务规模有限（inferred）Limited scale of the evaluation tasks (inferred)

真实世界实验只覆盖 4 个任务（Pick Banana / Pick Bread / Pull Drawer / Hang Chinese Knot），持续学习设置也主要围绕 LIBERO 四个 task suite 展开，尚未在更大规模、更多样化的任务集合或跨具身（cross-embodiment）场景下验证 LifeLong-RFT 的持续学习能力是否能同等程度地保持。Real-world experiments cover only 4 tasks (Pick Banana / Pick Bread / Pull Drawer / Hang Chinese Knot), and the continual learning setting mainly revolves around the four LIBERO task suites; it has not yet been verified on larger, more diverse task collections or in cross-embodiment scenarios whether the continual learning ability of LifeLong-RFT is preserved to the same degree.