Interactive Post-Training for Vision-Language-Action Models
RIPT-VLA：用稀疏二元奖励对 VLA 模型进行强化交互式后训练RIPT-VLA: Reinforcement Interactive Post-Training of VLA Models with Sparse Binary Rewards
Shuhan Tan · Kairan Dou · Yue Zhao · Philipp Krähenbühl | UT Austin · Nankai University | arXiv 2025
现有 VLA 模型依赖大量离线专家演示进行监督微调，模型在训练中从不感知自身行动的后果。RIPT-VLA 引入第三个训练阶段——在真实或仿真环境中与场景交互，只需稀疏的二元成功/失败信号，即可将模型性能大幅提升，甚至在仅有 1 条演示的情况下，将成功率从 4% 提升至 97%。Existing VLA models rely on large amounts of offline expert demonstrations for supervised fine-tuning, and the model never perceives the consequences of its own actions during training. RIPT-VLA introduces a third training stage — interacting with the scene in a real or simulated environment — that needs only a sparse binary success/failure signal, yet substantially improves model performance, raising the success rate from 4% to 97% even with only 1 demonstration.
大型预训练 VLA 模型（如 OpenVLA、QueST）在离线监督微调（SFT）之后仍存在两大根本缺陷：模型从未体验过自身决策的环境反馈；任务特化微调又需要昂贵的大量专家演示。强化学习理论上可以弥补这两点，但将 RL 扩展到大规模多任务 VLA 时，面临奖励稀疏、信用分配模糊、多任务难度不均衡等挑战。Large pre-trained VLA models (such as OpenVLA and QueST) still suffer from two fundamental deficiencies after offline supervised fine-tuning (SFT): the model has never experienced environmental feedback on its own decisions; and task-specific fine-tuning again requires large quantities of expensive expert demonstrations. Reinforcement learning can in principle remedy both, but scaling RL to large multi-task VLA models faces challenges such as sparse rewards, ambiguous credit assignment, and imbalanced difficulty across tasks.
"Current VLA training paradigms operate on static offline datasets...models never experience the consequences of their own actions during training, and task-specific fine-tuning requires large quantities of expensive human demonstrations."
图 1：传统两阶段（预训练 + SFT）与 RIPT-VLA 三阶段训练流程对比。第三阶段利用环境交互和二元成功信号，对 SFT 模型进行强化交互式后训练，无需额外的专家演示或稠密奖励。Figure 1: Comparison between the conventional two-stage pipeline (pre-training + SFT) and the three-stage RIPT-VLA training pipeline. The third stage uses environment interaction and a binary success signal to perform reinforcement interactive post-training on the SFT model, requiring no additional expert demonstrations and no dense reward.
97%1-shot 场景下从 4% 到 97% 成功率（15 轮内）Success rate from 4% to 97% in the 1-shot setting (within 15 iterations)
+21.2%LIBERO-LONG 5-shot 场景提升幅度Improvement on LIBERO-LONG in the 5-shot setting
97.5%OpenVLA-OFT + RIPT 在 LIBERO 标准多任务成功率OpenVLA-OFT + RIPT success rate on the standard LIBERO multi-task setting
+10.9%QueST 多任务平均成功率提升Improvement in QueST multi-task average success rate
02方法Method
RIPT-VLA 在标准预训练和 SFT 之后，引入基于环境交互的强化学习后训练阶段。其核心是将 Leave-One-Out 优势估计（RLOO）与 PPO 优化相结合，并通过 dynamic sampling 机制过滤零优势样本组，从而在多任务不均衡场景下稳定策略梯度更新。整个框架仅需稀疏的二元成功/失败奖励，无需稠密奖励设计或额外的 critic 模型。After standard pre-training and SFT, RIPT-VLA introduces a reinforcement learning post-training stage based on environment interaction. Its core is to combine Leave-One-Out advantage estimation (RLOO) with PPO optimization and to filter out zero-advantage sample groups through a dynamic sampling mechanism, thereby stabilizing policy-gradient updates under imbalanced multi-task settings. The whole framework needs only a sparse binary success/failure reward, with no dense reward design and no additional critic model.
对每个上下文 c，模型在相同上下文下采样 K 条轨迹。第 k 条轨迹的 baseline 为其他 K−1 条轨迹奖励的均值：For each context c, the model samples K trajectories under the same context. The baseline of the k-th trajectory is the mean reward of the other K−1 trajectories:
bk = 1/(K−1) · ∑j≠k Rj ， Ak = Rk − bk
这提供了"group-normalized advantage"，无需训练与 VLA 等规模的 value function，大幅降低显存开销。PPO 目标使用标准 clipped 策略比率（ε=0.2）。This provides a "group-normalized advantage" without training a value function on the same scale as the VLA, greatly reducing memory overhead. The PPO objective uses the standard clipped policy ratio (ε=0.2).
多任务场景中，部分任务已被 SFT 解决（全部成功），另一些任务则始终失败——两者都产生全零优势，对梯度更新毫无贡献。RIPT-VLA 的核心创新之一是：In multi-task settings, some tasks have already been solved by SFT (all rollouts succeed) while others always fail — both yield all-zero advantages and contribute nothing to the gradient update. One of the core innovations of RIPT-VLA is:
"We discard any sampled context for which all K rollouts receive the same reward and resample."
这使得每个训练 batch 都由有效的非零优势样本构成，保持梯度信号的稳定性。This makes every training batch consist of effective non-zero-advantage samples, keeping the gradient signal stable.
连续动作头的扩展（OpenVLA-OFT）Extension to Continuous Action Heads (OpenVLA-OFT)
对于使用回归头（而非 token 分类）的模型，log-probability 无法直接获取。RIPT-VLA 为其增加一个轻量 scale prediction head 来估计 σθ，将连续动作输出建模为因子化高斯分布，从而对接 PPO 目标。For models that use a regression head (rather than token classification), the log-probability is not directly available. RIPT-VLA adds a lightweight scale prediction head to estimate σθ, modeling the continuous action output as a factorized Gaussian distribution so that it can be connected to the PPO objective.
图 5：Dynamic sampling 对训练稳定性的影响。带 dynamic sampling 的 RIPT-VLA（蓝色）收敛更快、波动更小，最终成功率从 90.2%（无 dynamic sampling）提升至 93.5%。Figure 5: Effect of dynamic sampling on training stability. RIPT-VLA with dynamic sampling (blue) converges faster with smaller fluctuations, and the final success rate rises from 90.2% (without dynamic sampling) to 93.5%.
03实验Experiments
实验在 LIBERO（Goal / Spatial / Object / Long / 90）和 MetaWorld-45（ML45）仿真环境中展开，覆盖标准多任务、few-shot 泛化、跨场景/跨目标泛化四类设置。基线模型包括 QueST（离散 token 动作）和 OpenVLA-OFT（连续回归动作）。奖励信号仅为任务结束时的二元成功信号。Experiments are conducted in the LIBERO (Goal / Spatial / Object / Long / 90) and MetaWorld-45 (ML45) simulation environments, covering four categories of settings: standard multi-task, few-shot generalization, and cross-scene / cross-goal generalization. Baseline models include QueST (discrete token actions) and OpenVLA-OFT (continuous regression actions). The reward signal is only the binary success signal at the end of a task.
Table 1：LIBERO 标准多任务成功率（%）Table 1: Success Rate on Standard LIBERO Multi-Task Settings (%)
图 2（few-shot 学习曲线）：LIBERO-LONG 上，随演示条数增加，QueST + RIPT-VLA 的成功率显著高于纯 SFT 基线。5-shot 下从 50.2% 提升至 71.4%（+21.2 个百分点）。Figure 2 (few-shot learning curves): On LIBERO-LONG, as the number of demonstrations increases, the success rate of QueST + RIPT-VLA is markedly higher than that of the pure SFT baseline. Under 5-shot it rises from 50.2% to 71.4% (+21.2 percentage points).图 3（跨场景泛化）：1-shot 设置下，RIPT-VLA 在跨场景任务中成功率高达 97.2%，而 SFT 基线仅约 5%。强化交互式后训练赋予模型从极少演示中泛化的能力。Figure 3 (cross-scene generalization): Under the 1-shot setting, RIPT-VLA reaches a success rate as high as 97.2% on cross-scene tasks, while the SFT baseline is only about 5%. Reinforcement interactive post-training endows the model with the ability to generalize from very few demonstrations.图 4（跨目标泛化）：不同任务对之间的 cross-goal 泛化成功率对比，RIPT-VLA 在多个任务组合上均优于 SFT 基线，体现了策略的跨目标适应能力。Figure 4 (cross-goal generalization): Comparison of cross-goal generalization success rates across different task pairs. RIPT-VLA outperforms the SFT baseline on multiple task combinations, reflecting the cross-goal adaptability of the policy.
消融实验Ablation Studies
Table 3 展示 dynamic sampling 的贡献：QueST SFT 基线为 85.0%，去掉 dynamic sampling 的 RIPT-VLA 达到 90.2%（+5.2），完整 RIPT-VLA 达到 93.5%（+8.5）。Dynamic sampling 在 QueST SFT 基础上带来额外 +3.3 个百分点的提升，作者指出这"significantly boosts performance across all task categories"。Table 3 shows the contribution of dynamic sampling: the QueST SFT baseline is 85.0%, RIPT-VLA without dynamic sampling reaches 90.2% (+5.2), and the full RIPT-VLA reaches 93.5% (+8.5). Dynamic sampling brings an extra +3.3 percentage points on top of QueST SFT, and the authors note that it "significantly boosts performance across all task categories".
04局限性Limitations
Note：论文未设独立 Limitations 章节。以下前两条为作者在正文/讨论中明确提及的限制；第三、四条为根据方法设计推断（inferred）。Note: The paper has no separate Limitations section. The first two items below are limitations explicitly mentioned by the authors in the main text / discussion; the third and fourth are inferred from the method design (inferred).
需要可交互的仿真/真实环境Requires an interactive simulated / real environment
RIPT-VLA 需要在训练期间执行 rollout 并获取环境反馈。对于无法重置或成本极高的真实环境，这一需求会带来显著的部署障碍。作者仅在仿真 benchmark（LIBERO、MetaWorld）上验证，未报告真实机器人实验。RIPT-VLA needs to execute rollouts and obtain environment feedback during training. For real environments that cannot be reset or are extremely costly, this requirement brings a significant deployment barrier. The authors validate only on simulation benchmarks (LIBERO, MetaWorld) and report no real-robot experiments.
对初始状态方差敏感（inferred）Sensitive to initial-state variance (inferred)
Figure 7 显示，当初始状态标准差超过自然方差的 2× 时，性能开始下降。这意味着 RIPT-VLA 学到的策略对物体初始摆放位置的扰动容忍度有限，在高随机性真实场景中鲁棒性存疑。Figure 7 shows that performance starts to degrade once the initial-state standard deviation exceeds 2× the natural variance. This means the policy learned by RIPT-VLA has limited tolerance to perturbations of the initial object placement, and its robustness in highly stochastic real-world scenes is questionable.
依赖足够多样的 context 数据集（inferred）Depends on a sufficiently diverse context dataset (inferred)
Figure 6 表明，context 数据集规模影响 RIPT-VLA 的泛化性能。当可用 context（演示）数量极少时，策略的泛化能力受限。Figure 6 indicates that the size of the context dataset affects the generalization performance of RIPT-VLA. When the number of available contexts (demonstrations) is extremely small, the generalization ability of the policy is limited.
连续动作头需额外 scale head（inferred）Continuous action heads need an extra scale head (inferred)
对于 OpenVLA-OFT 等使用连续回归头的模型，RIPT-VLA 需训练一个额外的轻量 scale prediction head 来估计高斯分布的方差，增加了架构改动成本；该 head 的质量对策略优化也有潜在影响。For models with continuous regression heads such as OpenVLA-OFT, RIPT-VLA must train an additional lightweight scale prediction head to estimate the variance of the Gaussian distribution, which increases the cost of architectural modification; the quality of this head also has a potential impact on policy optimization.
未结合推理与规划（作者展望）No integration with reasoning and planning (author outlook)
作者在讨论中指出，未来应"combine RIPT-VLA with reasoning and planning in VLA models to enable more sophisticated and generalizable behaviors"，暗示当前方法在长时程复杂推理任务上仍有提升空间。In the discussion the authors point out that future work should "combine RIPT-VLA with reasoning and planning in VLA models to enable more sophisticated and generalizable behaviors", suggesting that the current method still has room for improvement on long-horizon, complex reasoning tasks.
