SimpleVLA-RL：通过强化学习扩展 VLA 训练

01 动机Motivation

VLA 模型在机器人操控领域展现出强大潜力，但面临两大核心挑战：一是人工操作轨迹数据稀缺——大量高质量示范数据难以获取；二是分布偏移下泛化能力有限——在监督微调（SFT）范式下，模型面对未见场景时表现急剧下降。已有工作尝试将强化学习（RL）引入 VLA，但受限于在线训练效率低下和缺乏可扩展的并行化框架，实际应用受阻。VLA models show strong potential in robot manipulation, yet face two core challenges. First, human teleoperation trajectory data are scarce—large volumes of high-quality demonstrations are hard to obtain. Second, generalization under distribution shift is limited—under the supervised fine-tuning (SFT) paradigm, performance degrades sharply on unseen scenarios. Prior work has tried to bring reinforcement learning (RL) to VLA, but low online-training efficiency and the lack of a scalable parallelization framework have blocked practical use.

"我们提出 SimpleVLA-RL，通过 VLA 专用的轨迹采样、可扩展并行化、多环境渲染和优化的损失计算，将 veRL 框架扩展至 VLA 模型，实现高效在线 RL 训练。""We propose SimpleVLA-RL, which extends the veRL framework to VLA models through VLA-specific trajectory sampling, scalable parallelization, multi-environment rendering and optimized loss computation, enabling efficient online RL training."

SimpleVLA-RL 概览 — **图1：SimpleVLA-RL 总体概览。** 框架将 VLA 模型的闭环环境交互与 veRL 的分布式训练基础设施结合，支持多环境并行渲染与在线策略优化。**Figure 1: Overview of SimpleVLA-RL.** The framework couples the closed-loop environment interaction of VLA models with the distributed training infrastructure of veRL, supporting multi-environment parallel rendering and online policy optimization.

99.1%LIBERO 平均成功率
（原 SFT 基线：91.0%）LIBERO average success rate
(original SFT baseline: 91.0%)

68.8%RoboTwin 2.0 平均成功率
（原 SFT 基线：38.3%）RoboTwin 2.0 average success rate
(original SFT baseline: 38.3%)

38.5%Sim-to-Real 迁移成功率
（无真实机器人数据）Sim-to-Real transfer success rate
(no real-robot data)

+74.4%LIBERO-Goal 单轨迹数据场景
RL 相对 SFT 的提升幅度LIBERO-Goal single-trajectory setting
gain of RL over SFT

02 方法Method

SimpleVLA-RL 以 OpenVLA-OFT 为骨干，将 veRL（专为 LLM RL 训练设计的分布式框架）扩展至具有闭环环境交互需求的 VLA 场景。核心创新在于三个 VLA 专属机制：交互式轨迹采样、结果奖励建模和探索增强策略。SimpleVLA-RL takes OpenVLA-OFT as its backbone and extends veRL, a distributed framework designed for LLM RL training, to VLA settings that require closed-loop environment interaction. The core innovation lies in three VLA-specific mechanisms: interactive trajectory sampling, outcome reward modeling and exploration enhancement strategies.

训练框架示意图 — **图2：SimpleVLA-RL 训练框架。** 左侧为 Actor Worker（负责与模拟环境交互生成轨迹），右侧为 Critic/Reference Worker（负责策略更新）。通过分离 rollout 与训练进程实现高效并行。**Figure 2: The SimpleVLA-RL training framework.** On the left is the Actor Worker, which interacts with the simulated environment to generate trajectories; on the right is the Critic/Reference Worker, responsible for policy updates. Separating rollout from the training process yields efficient parallelism.

交互式 VLA RolloutInteractive VLA Rollout

与 LLM 生成文本不同，VLA 需要在环境中持续闭环执行动作并获取新观测。SimpleVLA-RL 通过温度采样（T=1.6）对动作 token 进行多样化采样，每组生成 G 条轨迹并使用二值奖励（成功=1，失败=0）统一标注，避免了繁琐的过程奖励设计。Unlike LLM text generation, a VLA must keep executing actions in a closed loop within the environment and acquire new observations. SimpleVLA-RL diversifies action-token sampling with temperature sampling (T=1.6), generating G trajectories per group and labeling them uniformly with a binary reward (success=1, failure=0), thereby avoiding cumbersome process-reward design.

结果奖励建模Outcome Reward Modeling

采用简单二值结果奖励（outcome reward），成功轨迹奖励为 1，失败轨迹奖励为 0，奖励在整条轨迹的所有 token 上均匀分配。该设计无需人工设计复杂的中间奖励信号，显著降低了奖励工程成本，同时仍能提供有效的学习信号。A simple binary outcome reward is adopted: a successful trajectory receives a reward of 1 and a failed one 0, and the reward is spread uniformly over all tokens of the whole trajectory. This design needs no hand-crafted intermediate reward signal, markedly lowering reward-engineering cost while still supplying an effective learning signal.

探索增强策略（三项改进）Exploration Enhancement Strategies (Three Improvements)

**图3：探索增强消融实验结果。** 三项增强措施均对性能提升有正向贡献：动态采样、Clip Higher 和更高 rollout 温度。图中展示了各组件在 RoboTwin 任务上的消融对比。**Figure 3: Ablation results for the exploration enhancements.** All three enhancements contribute positively to performance: dynamic sampling, Clip Higher and a higher rollout temperature. The figure shows the ablation comparison of each component on RoboTwin tasks.

动态采样（Dynamic Sampling）：排除所有轨迹全部成功或全部失败的分组，仅对具有混合结果的分组进行梯度更新，避免了无信息样本对训练的干扰。Dynamic Sampling: Groups whose trajectories all succeed or all fail are discarded, and gradient updates are applied only to groups with mixed outcomes, avoiding interference from uninformative samples.
Clip Higher：将 PPO 的裁剪范围从标准的 [0.8, 1.2] 调整为 [0.8, 1.28]，允许策略在成功方向上做出更大幅度的更新，加速学习进程。Clip Higher: The PPO clipping range is changed from the standard [0.8, 1.2] to [0.8, 1.28], letting the policy make larger updates in the direction of success and speeding up learning.
更高 Rollout 温度：提升采样温度（T=1.6）以增加动作多样性，扩大策略的探索空间，有助于发现更优轨迹。Higher Rollout Temperature: Raising the sampling temperature (T=1.6) increases action diversity and enlarges the exploration space of the policy, helping it discover better trajectories.

03 实验Experiments

实验在三个主要基准上进行：LIBERO（4 个子任务）、RoboTwin 1.0（4 个任务）和 RoboTwin 2.0（12 个任务，按操作时长分为短/中/长三类），并另设真实机器人 sim-to-real 迁移实验（4 个任务）。基线包括 OpenVLA-OFT（SFT）、π₀ 和 RDT。Experiments are run on three main benchmarks: LIBERO (4 sub-tasks), RoboTwin 1.0 (4 tasks) and RoboTwin 2.0 (12 tasks, grouped into short, medium and long by manipulation horizon), plus an additional real-robot sim-to-real transfer experiment (4 tasks). Baselines include OpenVLA-OFT (SFT), π₀ and RDT.

LIBERO 基准结果LIBERO Benchmark Results

模型Model	Spatial	Object	Goal	Long	平均Average
OpenVLA-OFT（SFT 基线）OpenVLA-OFT (SFT baseline)	95.3	90.6	86.5	91.0	91.0
w/ SimpleVLA-RL	99.1	99.2	98.5	99.1	99.1
提升 (Δ)Gain (Δ)	+3.8	+8.6	+12.0	+8.1	+8.1

RoboTwin 2.0 综合结果（按时长分类）RoboTwin 2.0 Overall Results (Grouped by Horizon)

模型Model	短时长均值Short-horizon mean	中时长均值Medium-horizon mean	长时长均值Long-horizon mean	总体均值Overall mean
RDT	—	—	—	33.3
π₀	45.5	58.8	43.3	49.2
OpenVLA-OFT（SFT）OpenVLA-OFT (SFT)	21.3	47.1	46.5	38.3
w/ SimpleVLA-RL	64.9	72.5	69.0	68.8
提升 (Δ)Gain (Δ)	+43.6	+25.4	+22.4	+30.5

Sim-to-Real 迁移（无真实机器人数据）Sim-to-Real Transfer (No Real-Robot Data)

模型Model	Stack Bowls	Place Empty Cup	Pick Bottle	Click Bell	平均Average
RDT	60.0	4.0	10.0	20.0	23.5
OpenVLA-OFT	38.0	2.0	0.0	30.0	17.5
w/ SimpleVLA-RL	70.0	10.0	14.0	60.0	38.5

数据稀缺场景：单轨迹数据 + RLData-Scarce Setting: Single-Trajectory Data + RL

在每个任务仅提供 一条示范轨迹 的极端稀缺条件下，RL 训练仍能将 LIBERO-Long 成功率从 17.3%（SFT）提升至 91.7%，接近全量数据 SFT 的 96.9%，展示出强大的数据效率优势。Under the extremely scarce condition of only one demonstration trajectory per task, RL training still lifts the LIBERO-Long success rate from 17.3% (SFT) to 91.7%, close to the 96.9% of full-data SFT, showing a strong data-efficiency advantage.

设置Setting	Spatial	Object	Goal	Long
单轨迹 SFTSingle-trajectory SFT	54.9	59.6	17.3	48.9
单轨迹 SFT + RLSingle-trajectory SFT + RL	98.7	98.8	91.7	96.9
提升 (Δ)Gain (Δ)	+43.8	+39.2	+74.4	+48.0

新现象：「推切」行为（"Pushcut" Phenomenon）A New Phenomenon: the "Pushcut" Behavior

推切行为示例 — **图4："Pushcut" 行为演示。** 在 RL 训练过程中，策略自发涌现出示范数据中从未出现的行为模式：在 RoboTwin 任务中，策略学会直接将物体推至目标位置，而非沿示范轨迹执行抓取-移动-放置的完整序列，同样实现了任务成功，但路径更为简洁高效。**Figure 4: Demonstration of the "Pushcut" behavior.** During RL training the policy spontaneously develops a behavior pattern that never appears in the demonstration data: on RoboTwin tasks it learns to push an object directly to the target position instead of executing the full grasp-move-place sequence of the demonstrated trajectory, achieving task success along a more concise and efficient path.

消融实验结果Ablation Results

消融实验验证了三项探索增强措施均对性能有正向贡献：去除任一组件均导致成功率下降。此外，模型先验能力是 RL 奏效的前提——当基础模型初始成功率为 0% 时，RL 训练无法产生有效梯度，无法收敛；基础模型需具备约 7–10% 的初始成功率，RL 才能有效发挥作用（见 Table 7）。The ablations confirm that all three exploration enhancements contribute positively to performance: removing any single component lowers the success rate. Moreover, the prior capability of the model is a precondition for RL to work—when the initial success rate of the base model is 0%, RL training produces no effective gradient and fails to converge; the base model needs an initial success rate of roughly 7–10% before RL can take effect (see Table 7).

**图5：LIBERO 上的泛化分析。** 在空间配置、物体类型、任务场景等多维度对比 SFT 与 RL 的泛化能力，RL 在未见场景下的成功率显著高于 SFT，而 SFT 表现出明显的灾难性遗忘现象。**Figure 5: Generalization analysis on LIBERO.** Comparing the generalization of SFT and RL along dimensions such as spatial configuration, object type and task scene, RL attains a markedly higher success rate on unseen scenarios than SFT, whereas SFT exhibits pronounced catastrophic forgetting.

04 局限性Limitations

说明：论文结论部分提及部分局限，下列条目中「作者明确指出」的内容均来自论文原文，「推断（inferred）」的内容为基于设计分析所得。Note: The conclusion of the paper mentions some limitations. Items below marked "stated by the authors" are taken from the original text, while those marked "inferred" are derived from an analysis of the design.

依赖足够的初始模型能力（作者明确指出）Depends on sufficient initial model capability (stated by the authors)

RL 训练的有效性高度依赖基础模型的初始能力。当基础模型对某任务的初始成功率为 0% 时，无法生成任何成功轨迹，梯度流完全中断，RL 训练无法带来任何提升。实验显示，模型需具备约 7–10% 的基线成功率，RL 才开始奏效（Table 7）。The effectiveness of RL training depends heavily on the initial capability of the base model. When the initial success rate of the base model on a task is 0%, no successful trajectory can be produced, the gradient flow is entirely cut off, and RL training brings no improvement at all. Experiments show that the model needs a baseline success rate of about 7–10% before RL begins to work (Table 7).

依赖结果奖励，需成功轨迹支撑梯度（作者明确指出）Relies on outcome rewards and needs successful trajectories to sustain gradients (stated by the authors)

当前框架仅使用二值结果奖励（outcome-only rewards），这要求每批 rollout 中必须包含至少部分成功轨迹，方能产生有效的学习信号。对于极难任务，初始成功率极低，导致 RL 收益有限，甚至无效。The current framework uses only binary outcome-only rewards, which requires every rollout batch to contain at least some successful trajectories in order to yield an effective learning signal. For extremely hard tasks the initial success rate is very low, so the benefit of RL is limited or even absent.

仿真环境与真实世界之间的感知差异（推断，inferred）Perceptual discrepancy between simulation and the real world (inferred)

尽管框架已实现无真实机器人数据的 sim-to-real 迁移（平均 38.5%），但仿真训练中视觉渲染、物理特性与真实环境的差距（sim-to-real gap）仍不可忽视，可能限制迁移性能上限，尤其对感知精度要求较高的任务（如 Pick Bottle 仅 14.0%）。Although the framework achieves sim-to-real transfer without real-robot data (38.5% on average), the gap in visual rendering and physical properties between simulated training and the real environment (the sim-to-real gap) is still not negligible and may cap transfer performance, especially for tasks that demand high perceptual precision (e.g. Pick Bottle at only 14.0%).

计算开销与训练成本（推断，inferred）Compute overhead and training cost (inferred)

在线 RL 训练要求持续与模拟环境交互生成轨迹，相比 SFT 的离线训练，计算资源消耗显著增加。论文未报告具体训练时间或 GPU 成本，实际部署的计算可行性尚不明确。Online RL training requires continuous interaction with the simulated environment to generate trajectories, consuming far more compute than the offline training of SFT. The paper reports neither concrete training time nor GPU cost, so the computational feasibility of real deployment remains unclear.

技术报告阶段，代码/模型尚未公开（推断，inferred）Technical-report stage; code and models not yet released (inferred)

本文以 Technical Report 形式发布，目前未提及开源代码仓库或预训练模型权重，社区复现和后续研究的便利性有待提升。This work is released as a Technical Report; no open-source code repository or pre-trained model weights are mentioned so far, leaving room for improvement in the ease of community reproduction and follow-up research.