VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning

01 动机Motivation

模仿学习驱动的 VLA 模型在遇到训练数据未覆盖的状态时会出现执行失败，根本原因在于其固有的"利用（exploitation）"策略：模型只知道如何重现已见过的轨迹，缺乏在线探索与自我修正能力。Imitation-learning-driven VLA models fail to execute when they encounter states not covered by the training data. The root cause is their inherent "exploitation" strategy: the model only knows how to reproduce trajectories it has already seen, and lacks the ability to explore online and self-correct.

"exploiting offline data with limited visited states will cause execution failure in out-of-distribution scenarios."

作者提出将 VLA 的训练范式从"剥削式"（exploitation-based）转向"探索式"（exploration-based），通过强化学习在测试时收集的在线数据上持续改进。传统 RL 面临数据低效与繁重的奖励工程挑战，但从大型基础模型出发进行微调可以显著缓解这两个问题。The authors propose shifting the VLA training paradigm from exploitation-based to exploration-based, improving continually through reinforcement learning on online data collected at test time. Classical RL suffers from data inefficiency and heavy reward engineering, but fine-tuning from a large foundation model substantially alleviates both problems.

VLA-RL 与模仿学习对比 — **Figure 1.** 左图对比模仿学习（利用离线示范）与 VLA-RL（通过可扩展 RL 进行在线探索）的范式差异；右图展示 VLA-RL 在 LIBERO 40 项操作任务上相对基线的一致性提升。**Figure 1.** Left: the paradigm contrast between imitation learning (exploiting offline demonstrations) and VLA-RL (online exploration through scalable RL). Right: the consistent improvement of VLA-RL over the baseline across the 40 LIBERO manipulation tasks.

+4.5%OpenVLA-7B 在 LIBERO 上的平均成功率提升Average success-rate gain of OpenVLA-7B on LIBERO

+1.8%相比 DPO 基线（GRAPE）的提升Improvement over the DPO baseline (GRAPE)

48 hRL 训练所需 GPU 小时数GPU hours required for RL training

40LIBERO 评测任务数量Number of LIBERO evaluation tasks

02 方法Method

VLA-RL 框架将机器人操作建模为"多模态多轮对话"（multi-modal multi-turn conversation），通过轨迹级 PPO 优化自回归策略，并配备机器人过程奖励模型（RPRM）提供密集奖励信号。The VLA-RL framework casts robotic manipulation as a "multi-modal multi-turn conversation", optimizes the auto-regressive policy with trajectory-level PPO, and equips it with a robotic process reward model (RPRM) that supplies dense reward signals.

**Figure 2.** VLA-RL 系统流水线：Transformer 策略网络（actor）、价值网络（critic）、冻结的 RPRM（奖励生成）与向量化并行环境共同构成在线 RL 训练回路。策略根据图像观测和语言指令自回归生成动作 token 序列，RPRM 对每个里程碑子任务提供过程级奖励。**Figure 2.** The VLA-RL system pipeline: a Transformer policy network (actor), a value network (critic), a frozen RPRM (reward generation) and vectorized parallel environments together form the online RL training loop. The policy auto-regressively generates action token sequences from image observations and language instructions, and the RPRM provides process-level rewards for every milestone sub-task.

轨迹级 RL 公式（Trajectory-Level RL Formulation）Trajectory-Level RL Formulation

标准 PPO 难以直接应用于自回归 VLA，因为动作空间维度高且每步由多个 token 组成。VLA-RL 的关键推导是："the log-probability of an action sequence can be decomposed into the summation of token-level log probabilities"，从而将轨迹级策略梯度转化为对每个动作 token 的 PPO clip 目标，实现了端到端的在线优化。Standard PPO cannot be applied directly to auto-regressive VLAs, because the action space is high-dimensional and each step consists of several tokens. The key derivation of VLA-RL is that "the log-probability of an action sequence can be decomposed into the summation of token-level log probabilities", which turns the trajectory-level policy gradient into a PPO clip objective over every action token and enables end-to-end online optimization.

机器人过程奖励模型（Robotic Process Reward Model, RPRM）Robotic Process Reward Model (RPRM)

稀疏的任务成功奖励难以提供有效学习信号。RPRM 自动从演示轨迹中提取伪标签，分两步构建密集奖励：Sparse task-success rewards provide little effective learning signal. The RPRM automatically extracts pseudo labels from demonstration trajectories and builds a dense reward in two steps:

里程碑分割（Milestone Segmentation）：根据夹爪状态变化将任务分解为子任务阶段。Milestone Segmentation: the task is decomposed into sub-task stages according to changes in gripper state.
进度标注（Progress Labeling）：在末端执行器速度趋近于零时识别关键帧作为子任务完成标志。Progress Labeling: key frames are identified as sub-task completion markers when the end-effector velocity approaches zero.

最终奖励为稀疏成功奖励与 RPRM 预测的过程奖励之直接求和："the direct summation of the golden sparse reward and the predicted reward from robotic process reward model"。The final reward is the direct sum of the sparse success reward and the process reward predicted by the RPRM: "the direct summation of the golden sparse reward and the predicted reward from robotic process reward model".

训练稳定化技术Training Stabilization Techniques

课程采样（Curriculum Selection）Curriculum Selection

任务采样概率正比于当前难度，优先采样成功率约为 50% 的任务，即"the frontier of the agent's capabilities"，避免过易（无梯度）或过难（无奖励）的极端情况。The task sampling probability is proportional to current difficulty, favouring tasks whose success rate is around 50%, i.e. "the frontier of the agent's capabilities", so as to avoid the extremes of being too easy (no gradient) or too hard (no reward).

Critic 预热（Critic Warmup）Critic Warmup

在联合优化开始前，先用模仿预训练策略收集初始轨迹来稳定价值估计，防止 critic 误导 actor 造成早期崩溃。实验表明去除此步骤成功率从 90.2% 降至 80.0%（−10.2%）。Before joint optimization begins, initial trajectories are collected with the imitation-pretrained policy to stabilize the value estimate, preventing the critic from misleading the actor and causing early collapse. Experiments show that removing this step drops the success rate from 90.2% to 80.0% (−10.2%).

此外，框架采用跨 GPU 均衡向量化环境以管理并行 rollout 的显存消耗，并以较低学习率（2e-5）防止灾难性遗忘。In addition, the framework adopts GPU-balanced vectorized environments to manage the memory consumption of parallel rollouts, and uses a low learning rate (2e-5) to prevent catastrophic forgetting.

03 实验Experiments

在 LIBERO 四个任务套件（Spatial / Object / Goal / Long）上各评测 500 个 episode，与 Diffusion Policy、Octo、OpenVLA (SFT)、GRAPE (DPO)、π₀-FAST 进行比较。Each of the four LIBERO task suites (Spatial / Object / Goal / Long) is evaluated over 500 episodes and compared against Diffusion Policy, Octo, OpenVLA (SFT), GRAPE (DPO) and π₀-FAST.

方法Method	Spatial	Object	Goal	Long	平均Average
Diffusion Policy	78.3%	92.5%	68.3%	50.5%	72.4%
Octo (SFT)	78.9%	85.7%	84.6%	51.1%	75.1%
OpenVLA (SFT)	84.7%	88.4%	79.2%	53.7%	76.5%
GRAPE (DPO)	87.6%	91.2%	82.2%	55.8%	79.2%
π₀-FAST	96.4%	96.8%	88.6%	60.2%	85.5%
VLA-RL（本文）VLA-RL (ours)	90.2%	91.8%	82.2%	59.8%	81.0%

VLA-RL 在 LIBERO-Long 上与商业闭源模型 π₀-FAST 的差距仅为 0.4%（59.8% vs 60.2%），在 LIBERO-Spatial 与 LIBERO-Long 上排名第一，综合平均排名第 1.5，超越所有开源方法。On LIBERO-Long, VLA-RL trails the commercial closed-source model π₀-FAST by only 0.4% (59.8% vs 60.2%); it ranks first on LIBERO-Spatial and LIBERO-Long, and attains an overall average rank of 1.5, surpassing all open-source methods.

推理阶段缩放（Test-Time Scaling）Test-Time Scaling

Test-time scaling curves — **Figure 4.** 在四个 LIBERO 任务套件上，增加测试时优化次数后性能持续提升，作者称之为机器人操作领域"an early spark of inference scaling laws"。**Figure 4.** Across the four LIBERO task suites, performance keeps improving as the number of test-time optimization steps grows; the authors call this "an early spark of inference scaling laws" for robotic manipulation.

训练动态Training Dynamics

消融实验（LIBERO-Spatial）Ablation Study (LIBERO-Spatial)

配置Configuration	成功率Success rate	差值Δ
VLA-RL（完整）VLA-RL (full)	90.2%	—
去除 RPRMw/o RPRM	85.8%	−4.4%
去除 Curriculumw/o Curriculum	88.0%	−2.2%
Temperature 1.5→1.0	85.8%	−4.4%
去除 Critic Warmupw/o Critic Warmup	80.0%	−10.2%
学习率 2e-5→2e-4Learning rate 2e-5→2e-4	0.2%	−90.0%

消融结果表明，"eliminating any individual stabilizing technique leads to rapid collapse"。学习率过高导致几乎完全失败，Critic Warmup 去除导致 10.2% 的大幅下降，RPRM 与温度系数均带来约 4.4% 的稳定收益。The ablations show that "eliminating any individual stabilizing technique leads to rapid collapse". Too high a learning rate causes almost complete failure, removing Critic Warmup produces a large 10.2% drop, and both the RPRM and the temperature coefficient bring a steady gain of about 4.4%.

定性分析：接触丰富操作Qualitative Analysis: Contact-Rich Manipulation

Qualitative case study — **Figure 7.** 案例研究对比 VLA-RL 与 SFT 基线在接触丰富抓取任务上的表现。SFT 策略在未见的初始姿态下失败，而 VLA-RL 通过探索学习到的多样化动作分布成功完成抓取。**Figure 7.** A case study comparing VLA-RL with the SFT baseline on a contact-rich grasping task. The SFT policy fails under an unseen initial pose, whereas VLA-RL succeeds thanks to the diverse action distribution learned through exploration.

04 局限性Limitations

Note: 以下局限性部分为作者在论文中明确陈述，部分为从设计层面推断（已标注）。Some of the limitations below are explicitly stated by the authors in the paper, while others are inferred at the design level (marked as such).

伪奖励标签难以捕捉灵巧操作的细粒度（作者明确陈述）Pseudo reward labels struggle to capture the fine granularity of dexterous manipulation (explicitly stated by the authors)

"The proposed heuristics for extracting pseudo reward labels may not fully capture the nuances of more dexterous manipulation tasks, potentially leading to inefficient policy optimization." 里程碑分割依赖夹爪状态变化，对于需要精细协调的灵巧手任务可能出现标注不准确。"The proposed heuristics for extracting pseudo reward labels may not fully capture the nuances of more dexterous manipulation tasks, potentially leading to inefficient policy optimization." Milestone segmentation relies on changes in gripper state, so labelling may be inaccurate for dexterous-hand tasks that require fine-grained coordination.

仅在仿真环境（LIBERO）中验证，缺乏真实世界实验（作者明确陈述）Validated only in simulation (LIBERO), with no real-world experiments (explicitly stated by the authors)

所有实验在 LIBERO 模拟基准上进行，尚未验证 sim-to-real 迁移。作者将"exploring online self-improvement with large-scale real-world experience"列为未来工作。All experiments are conducted on the LIBERO simulation benchmark, and sim-to-real transfer has not been validated. The authors list "exploring online self-improvement with large-scale real-world experience" as future work.

仅支持自回归 VLA，不适用于扩散策略（设计层面推断）Supports auto-regressive VLAs only, not diffusion policies (inferred at the design level)

轨迹级 RL 公式依赖 token-level log-probability 分解，天然适配自回归架构（如 OpenVLA），但无法直接应用于扩散策略（如 π₀）。作者明确将"training diffusion-based policies with reinforcement learning beyond auto-regressive VLAs"列为未来工作方向。The trajectory-level RL formulation relies on a token-level log-probability decomposition, which naturally fits auto-regressive architectures (such as OpenVLA) but cannot be applied directly to diffusion policies (such as π₀). The authors explicitly list "training diffusion-based policies with reinforcement learning beyond auto-regressive VLAs" as a future direction.

对超参数高度敏感（设计层面推断，消融实验佐证）Highly sensitive to hyper-parameters (inferred at the design level, corroborated by the ablations)

消融实验显示，仅将学习率从 2e-5 调整为 2e-4 即导致成功率从 90.2% 崩溃至 0.2%；温度系数与 Critic Warmup 步数同样对性能影响显著，提示方法对超参数搜索要求较高。The ablations show that merely changing the learning rate from 2e-5 to 2e-4 collapses the success rate from 90.2% to 0.2%; the temperature coefficient and the number of Critic Warmup steps likewise affect performance markedly, indicating that the method demands a careful hyper-parameter search.