RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models
一套统一接口 + 混合细粒度流水线 GPU 调度，让 VLA 的强化学习训练又快又可复现A unified interface plus hybrid fine-grained pipelined GPU scheduling, making reinforcement-learning training of VLA models fast and reproducible
Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo, Yuanqing Wang, et al. · Tsinghua University / Zhongguancun Academy / Infinigence AI / Peking University / UC Berkeley / HIT / CASIA
RLinf-VLA 是一个面向 vision-language-action（VLA）模型强化学习后训练的统一系统：一套接口即可切换 ManiSkill / LIBERO / RoboTwin 三种模拟器、OpenVLA / OpenVLA-OFT 两种架构、PPO / GRPO 两种算法；针对 GPU 并行化模拟器提出的 hybrid fine-grained pipelining GPU 分配策略带来 1.61×–1.88× 的训练加速。用该系统训练的策略在三个基准上取得约 20%–85% 的性能提升。RLinf-VLA is a unified system for reinforcement-learning post-training of vision-language-action (VLA) models: a single interface switches among the three simulators ManiSkill / LIBERO / RoboTwin, the two architectures OpenVLA / OpenVLA-OFT, and the two algorithms PPO / GRPO; the hybrid fine-grained pipelining GPU allocation strategy proposed for GPU-parallelized simulators delivers a 1.61×–1.88× training speedup. Policies trained with this system achieve roughly 20%–85% performance gains on three benchmarks.
vision-language-actionreinforcement learningPPOGRPOGPU allocationfine-grained pipeliningLIBEROManiSkillRoboTwin训练系统training systems
01 Motivation
VLA 模型部署前通常需要后训练来弥合训练数据与部署环境之间的分布差异；相比 SFT，RL 能让策略走出专家演示的窄分布、获得纠正性与自适应行为，"recent studies indicate that RL fine-tuning can yield stronger out-of-distribution generalization than SFT"。但把 RL 用到 VLA 上目前仍然零散：SimpleVLA-RL 虽借助 VeRL 实现了大规模 RL 训练，却"lacks system-level optimizations tailored to embodied settings, where simulators compete with model inference and learning for GPU resources"，而在线 RL 又要求模型与环境紧密耦合的重复交互，缺乏针对性系统设计会造成大量 GPU 空闲与流水线气泡。VLA models usually need post-training before deployment to bridge the distribution gap between the training data and the deployment environment; compared with SFT, RL lets a policy escape the narrow distribution of expert demonstrations and acquire corrective and adaptive behaviors — "recent studies indicate that RL fine-tuning can yield stronger out-of-distribution generalization than SFT". Yet applying RL to VLA remains fragmented: although SimpleVLA-RL realizes large-scale RL training on top of VeRL, it "lacks system-level optimizations tailored to embodied settings, where simulators compete with model inference and learning for GPU resources", while online RL further demands repeated, tightly coupled model-environment interaction, so the absence of purpose-built system design causes large amounts of GPU idle time and pipeline bubbles.
"existing methods remain fragmented, lacking both a unified platform for fair comparison across architectures and algorithms and an efficient system design for scalable training."
Figure 1（论文原图）：VLA 模型强化学习的整体训练流水线（Training / Generation / Simulator 三大组件及其数据流转）。Figure 1 (from the paper): the overall training pipeline for reinforcement learning of VLA models (the three components Training / Generation / Simulator and the data flow between them).
98.11%LIBERO 130 任务统一模型平均成功率Average success rate of the unified model over LIBERO's 130 tasks
1.61×–2.27×相较基线 / SimpleVLA-RL 的训练吞吐加速Training-throughput speedup over the baseline / SimpleVLA-RL
02 Method
RLinf-VLA 的设计分两条主线：（1）针对 Training / Generation / Simulator 三类工作负载的 GPU 分配策略，解决 GPU 并行化模拟器下渲染、推理、训练争抢显存与算力的问题；（2）覆盖模拟器 / 模型 / 算法三个维度的 统一接口，让新模拟器、新 VLA 架构、新 RL 算法都能以最小改动接入系统。The design of RLinf-VLA follows two main threads: (1) a GPU allocation strategy for the three workload types Training / Generation / Simulator, which addresses the contention for memory and compute among rendering, inference and training under GPU-parallelized simulators; (2) a unified interface spanning the three dimensions of simulator / model / algorithm, so that new simulators, new VLA architectures and new RL algorithms can be plugged into the system with minimal changes.
Figure 5（论文原图）：Hybrid 分配模式 + fine-grained pipelining——单个 GPU 上的模拟器实例被切分为多个子模拟器 S(1), S(2), …, S(k)，Simulator 与 Generation 交替流水执行，从而在避免 collocated 模式频繁 offload/onload 开销的同时，消除 disaggregated 模式下的 GPU 空闲气泡。Figure 5 (from the paper): the Hybrid allocation mode + fine-grained pipelining — the simulator instance on a single GPU is split into several sub-simulators S(1), S(2), ..., S(k), and Simulator and Generation execute in an alternating pipeline, thereby eliminating the GPU idle bubbles of the disaggregated mode while avoiding the frequent offload/onload overhead of the collocated mode.
Collocated（同位）模式下所有组件共享同一批 GPU；论文实现了改进版——仅在 rollout 阶段的开始和结束做 offload/onload，避免每次交互都做迁移，但 Generation 与 Simulator 仍需相互等待，"leading to resource wastage and limited scalability"。Disaggregated（分离）模式把每个组件固定分配到独立 GPU 分区，"simple to implement, but it lead to GPU underutilization due to inter-component dependencies"（例如某些 GPU 在 rollout 阶段完全空闲）。论文提出的 Hybrid 模式让组件可灵活选择 GPU 分区（Training 可用全部 GPU），并在此基础上引入 fine-grained pipelining：模拟器被拆成 k 个子实例交替产生观测与执行动作，"allows Simulator and Generation to run concurrently, reducing idle time while preserving correctness"。In the Collocated mode all components share the same set of GPUs; the paper implements an improved version that performs offload/onload only at the beginning and the end of the rollout phase, avoiding a migration at every interaction, but Generation and Simulator still have to wait for each other, "leading to resource wastage and limited scalability". The Disaggregated mode pins each component to a dedicated GPU partition, which is "simple to implement, but it lead to GPU underutilization due to inter-component dependencies" (for instance, some GPUs are entirely idle during the rollout phase). The Hybrid mode proposed in the paper lets components choose GPU partitions flexibly (Training may use all GPUs), and on top of it introduces fine-grained pipelining: the simulator is split into k sub-instances that alternately produce observations and execute actions, which "allows Simulator and Generation to run concurrently, reducing idle time while preserving correctness".
系统用统一的 Gym 风格接口标准化 ManiSkill、LIBERO、RoboTwin 三种模拟器（涵盖 GPU 并行与 CPU 并行两类后端），并原生支持 action chunking 与灵活的 episode 终止；模型侧接入 OpenVLA（单步动作）与 OpenVLA-OFT（多步 action chunking）并集成 LoRA；算法侧仅需配置 advantage 函数与 loss 函数即可扩展 PPO / GRPO。此外，框架在 advantage 估计与 log-probability 计算上支持 chunk-level / action-level / token-level 三种粒度，论文的消融实验发现 "action-level formulation consistently leads to better p[erformance]"（对 PPO 的 critic 设计而言，action-level value 估计优于 chunk-level）。The system standardizes the three simulators ManiSkill, LIBERO and RoboTwin behind a unified Gym-style interface (covering both GPU-parallel and CPU-parallel backends), and natively supports action chunking and flexible episode termination; on the model side it plugs in OpenVLA (single-step actions) and OpenVLA-OFT (multi-step action chunking) with LoRA integrated; on the algorithm side, extending to PPO / GRPO only requires configuring the advantage function and the loss function. In addition, the framework supports chunk-level / action-level / token-level granularity for both advantage estimation and log-probability computation, and the paper's ablations find that the "action-level formulation consistently leads to better p[erformance]" (for PPO's critic design, action-level value estimation beats chunk-level).
03 Experiments
评测覆盖三个基准：ManiSkill（25 个 pick-and-place 任务，OOD 评测涵盖 vision / language / action 三类泛化，每子设置 256 个随机 episode）、LIBERO（130 个任务，5 组，训练单个统一模型而非分组各自训练，50 episode/任务、3 个随机种子）、RoboTwin（6 个双臂操作任务，1000 个固定随机场景种子训练、128 个未见种子评测 OOD 泛化）。The evaluation covers three benchmarks: ManiSkill (25 pick-and-place tasks, whose OOD evaluation spans vision / language / action generalization with 256 random episodes per sub-setting), LIBERO (130 tasks in 5 suites, training one single unified model instead of one model per suite, with 50 episodes per task and 3 random seeds), and RoboTwin (6 bimanual manipulation tasks, trained on 1000 fixed scene seeds and evaluated on 128 unseen seeds for OOD generalization).
+63.75 pt（论文所述"average improvement"）+63.75 pt (the "average improvement" stated in the paper)
论文强调："Direct OOD comparison between OpenVLA and OpenVLA-OFT is not strictly fair because their base performance differs."；同时 PPO 在 ManiSkill 上 "consistently outperforms GRPO and exhibits greater stability" for both backbones。The paper stresses: "Direct OOD comparison between OpenVLA and OpenVLA-OFT is not strictly fair because their base performance differs."; meanwhile, on ManiSkill PPO "consistently outperforms GRPO and exhibits greater stability" for both backbones.
Figure 7(c)（论文原图）：OpenVLA-OFT + GRPO 在 LIBERO 130 任务上的训练曲线——"the success rate improves substantially from approximately 73% to 98%, yielding an overall performance gain of about 30%"。Figure 7(c) (from the paper): the training curve of OpenVLA-OFT + GRPO on LIBERO's 130 tasks — "the success rate improves substantially from approximately 73% to 98%, yielding an overall performance gain of about 30%".Figure 8(a)（论文原图）：不同集群规模 / GPU 分配策略下的端到端吞吐对比——ManiSkill 上 Hybrid（pipe=2）模式相较 disaggregated 基线在 8 GPU 下达到 1.88× 加速，扩展到更多 GPU 时仍维持 1.61×–1.69× 优势；LIBERO / RoboTwin 上 collocated 设置相较 SimpleVLA-RL 达到 1.34×–2.27× 加速。Figure 8(a) (from the paper): end-to-end throughput under different cluster sizes / GPU allocation strategies — on ManiSkill the Hybrid mode (pipe=2) reaches a 1.88× speedup over the disaggregated baseline at 8 GPUs, and still retains a 1.61×–1.69× advantage when scaling to more GPUs; on LIBERO / RoboTwin the collocated setting reaches a 1.34×–2.27× speedup over SimpleVLA-RL.
Ablations
PPO 侧：(a) action-level value 估计在 ManiSkill 与 LIBERO-Goal 上都 "consistently yields higher success rates and lower value loss" 优于 chunk-level；(b) Partial Reset（子环境一旦终止立即单独重置）相较 Fixed Episode Length 模式 "leads to a significantly higher success rate"，因为优化目标是 "success_once"。GRPO 侧：(a) trajectory length normalization 在 LIBERO-Goal 上 "can lead to substantially higher performance"；(b) valid action mask 在 LIBERO-Goal 上同样带来一致提升，但"in the ManiSkill setting, we do not observe clear benefits"，效果 task-dependent；(c) success-rate filtering 能缓解 GRPO 在 OpenVLA + ManiSkill 设置下约 step 400 出现的训练崩溃，"however, the benefit of the filter is not universal"，在 OpenVLA-OFT 的 ManiSkill / LIBERO-Goal 设置下效果明显减弱。On the PPO side: (a) action-level value estimation "consistently yields higher success rates and lower value loss" than chunk-level on both ManiSkill and LIBERO-Goal; (b) Partial Reset (a sub-environment is individually reset as soon as it terminates) "leads to a significantly higher success rate" than the Fixed Episode Length mode, because the optimization target is "success_once". On the GRPO side: (a) trajectory length normalization "can lead to substantially higher performance" on LIBERO-Goal; (b) a valid action mask likewise brings consistent gains on LIBERO-Goal, but "in the ManiSkill setting, we do not observe clear benefits", so the effect is task-dependent; (c) success-rate filtering mitigates the training collapse that GRPO exhibits around step 400 under the OpenVLA + ManiSkill setting, "however, the benefit of the filter is not universal", and its effect is markedly weaker under the ManiSkill / LIBERO-Goal settings of OpenVLA-OFT.
04 Limitations
Note：论文正文没有单独的 "Limitations" 小节，以下要点均来自消融实验/系统评测中论文明确写出的现象（标注为 stated），或从系统设计与实验范围推断（标注为 inferred）。Note: The body of the paper has no dedicated "Limitations" section; the points below are either phenomena explicitly stated by the paper in its ablations / system evaluation (marked stated), or inferred from the system design and the scope of the experiments (marked inferred).
GRPO 的算法技巧效果因任务而异（stated）The gains from GRPO's algorithmic tricks vary across tasks (stated)
valid action masking 与 trajectory length normalization 在 LIBERO-Goal 上有明确收益，但论文写道 "in the ManiSkill setting, we do not observe clear benefits from either valid action masking or trajectory length normalization"；success-rate filter 对 OpenVLA + ManiSkill 训练崩溃有效，但 "its effectiveness is much less pronounced" 在 OpenVLA-OFT 的 ManiSkill / LIBERO-Goal 设置下。这些"最佳实践"并非普适配置，需要按任务调参。valid action masking and trajectory length normalization bring clear benefits on LIBERO-Goal, but the paper writes that "in the ManiSkill setting, we do not observe clear benefits from either valid action masking or trajectory length normalization"; the success-rate filter is effective against the training collapse of OpenVLA + ManiSkill, yet "its effectiveness is much less pronounced" under the ManiSkill / LIBERO-Goal settings of OpenVLA-OFT. These "best practices" are not universal configurations and have to be tuned per task.
Hybrid fine-grained pipelining 的加速比随 GPU 规模衰减，且对模拟器类型敏感（stated）The speedup of Hybrid fine-grained pipelining decays as GPU scale grows, and is sensitive to the simulator type (stated)
论文指出 "scaling to more GPUs introduces overheads from model loading, offloading, and state switching"，pipe=2 相较 disaggregated 基线的优势从 1.88× 降到 1.61×–1.69×；而对于 OpenVLA-OFT 在 LIBERO 上，因生成/执行比接近 1:15 且 CPU 并行模拟器成为瓶颈，"this imbalance diminishes the benefit of pipelining, effectively reverting execution to a near-sequential process"——效率增益并非在所有 模拟器×模型 组合下都成立。The paper points out that "scaling to more GPUs introduces overheads from model loading, offloading, and state switching": the advantage of pipe=2 over the disaggregated baseline falls from 1.88× to 1.61×–1.69×. For OpenVLA-OFT on LIBERO, because the generation/execution ratio is close to 1:15 and the CPU-parallel simulator becomes the bottleneck, "this imbalance diminishes the benefit of pipelining, effectively reverting execution to a near-sequential process" — the efficiency gain does not hold for every simulator × model combination.
实证验证的算法与骨干模型范围有限（inferred）The empirically validated range of algorithms and backbone models is limited (inferred)
系统架构宣称可扩展支持任意模拟器/模型/算法，但论文当前实证结果仅覆盖两种 RL 算法（PPO、GRPO）与两种 VLA 骨干（OpenVLA、OpenVLA-OFT），均在仿真基准（ManiSkill、LIBERO、RoboTwin）而非真实机器人上完成；框架在更多算法族或真实世界部署上的表现未在本文中给出数据。The system architecture claims extensible support for arbitrary simulators/models/algorithms, but the paper's current empirical results only cover two RL algorithms (PPO, GRPO) and two VLA backbones (OpenVLA, OpenVLA-OFT), all obtained on simulation benchmarks (ManiSkill, LIBERO, RoboTwin) rather than on real robots; the framework's behavior on further algorithm families or in real-world deployment is not backed by data in this paper.
