系统 · RL Training System · arXiv 2025Systems · RL Training System · arXiv 2025
RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation
通过宏到微流变换实现灵活高效的大规模强化学习训练Flexible and Efficient Large-scale RL Training via Macro-to-Micro Flow Transformation
Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, 等 29 位作者 · Tsinghua University & 合作机构Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, et al., 29 authors in total · Tsinghua University & collaborating institutions
RLinf 是一个高性能 RL 训练系统，提出 macro-to-micro flow transformation（M2Flow）范式，将高层易于组合的 RL 工作流自动分解为优化后的执行计划。系统通过 elastic pipelining 和 context switching 两种调度原语，以及 profiling-guided 调度策略，在推理型与具身 RL 任务上均取得显著吞吐量提升。RLinf is a high-performance RL training system that proposes the macro-to-micro flow transformation (M2Flow) paradigm, automatically decomposing high-level, easily composable RL workflows into optimized execution plans. Through the two scheduling primitives elastic pipelining and context switching, together with a profiling-guided scheduling policy, the system achieves substantial throughput gains on both reasoning and embodied RL tasks.
强化学习系统RL training systemM2Flowelastic pipeliningcontext switchingdistributed RL训练吞吐量training throughput具身智能embodied intelligence调度策略scheduling policyGRPOPPO
01动机Motivation
现代 RL 工作流高度异构且动态变化，现有训练系统受制于单一执行模式，GPU 利用率低、训练效率差。RLinf 发现系统灵活性不足是高效 RL 训练的核心障碍。Modern RL workflows are highly heterogeneous and dynamic, while existing training systems are locked into a single execution mode, which yields low GPU utilization and poor training efficiency. RLinf identifies insufficient system flexibility as the central obstacle to efficient RL training.
"The major roadblock to efficient RL training lies in system flexibility. Single execution mode of existing RL training systems fails to capture this diversity, leading to suboptimal efficiency."
图 1：多样化的 RL 工作流场景，涵盖 GRPO、RLHF with PPO、具身智能（embodied intelligence）以及 Deep Research 等应用。不同场景的组件特性差异显著，单一执行模式难以兼顾全部场景的效率需求。Figure 1: Diverse RL workflow scenarios, covering GRPO, RLHF with PPO, embodied intelligence and Deep Research applications. Component characteristics differ markedly across scenarios, so a single execution mode can hardly meet the efficiency requirements of all of them.
论文归纳了现有系统面临的三大核心瓶颈：The paper summarizes three core bottlenecks faced by existing systems:
动态 rollout 长尾问题：生成阶段中响应长度高度可变，少量慢查询会阻塞整个 phase，使大量 GPU 处于空闲状态（long-tail problem）。Dynamic rollout long-tail problem: response lengths vary widely during the generation stage, and a few slow queries block the entire phase, leaving a large number of GPUs idle (long-tail problem).
执行模式不兼容：Collocated（顺序共享设备）和 disaggregated（流水线并行）两种模式各有局限，没有单一模式对所有 RL 组件都最优。Incompatible execution modes: collocated (sequential device sharing) and disaggregated (pipeline parallelism) each have their limits; no single mode is optimal for all RL components.
编排复杂度高："Finding the most suitable orchestration for a given RL workflow is challenging, as the characteristics are diverse and the dependencies are complex."High orchestration complexity: "Finding the most suitable orchestration for a given RL workflow is challenging, as the characteristics are diverse and the dependencies are complex."
1.07×–2.43×端到端训练吞吐量加速（vs. SOTA 系统）End-to-end training throughput speedup (vs. SOTA systems)
<6 秒<6 s在 1024 GPU 集群上的调度策略搜索开销Scheduling policy search overhead on a 1024-GPU cluster
02方法Method
RLinf 的核心设计范式是 macro-to-micro flow transformation（M2Flow）：将高层逻辑 RL 工作流（macro flow）与物理执行计划（micro flow）解耦，自动生成在时间和空间两个维度上的优化执行方案。The core design paradigm of RLinf is macro-to-micro flow transformation (M2Flow): it decouples the high-level logical RL workflow (macro flow) from the physical execution plan (micro flow) and automatically generates an execution plan optimized along both the temporal and the spatial dimension.
图 4：RLinf 整体架构，包含工作流构建（workflow construction）、worker 抽象（worker abstraction）、调度（scheduling）和执行管理（execution management）四大核心模块。用户只需编写 <100 行 Python 代码即可定义一个完整 RL 工作流。Figure 4: The overall RLinf architecture, with four core modules — workflow construction, worker abstraction, scheduling and execution management. A user needs fewer than 100 lines of Python code to define a complete RL workflow.
Worker 抽象与自适应通信Worker Abstraction and Adaptive Communication
每个 RL 组件被封装为一个 RLinf Worker，内置支持任意 Python 对象的 send/recv 通信原语。系统根据通信双方的位置自动选择最优后端：GPU 间使用 NCCL、同 GPU 内使用 cudaIPC、CPU 间使用 Gloo，从而实现 adaptive communication，无需用户手动配置。Every RL component is encapsulated as an RLinf Worker with built-in send/recv communication primitives for arbitrary Python objects. The system automatically picks the optimal backend from the locations of the two endpoints: NCCL across GPUs, cudaIPC within one GPU, and Gloo across CPUs, achieving adaptive communication with no manual configuration by the user.
图 6：M2Flow 如何将逻辑工作流转化为具体执行计划。系统在时间（temporal）和空间（spatial）两个维度上分解任务，并根据不同数据粒度（data granularity）重组执行流，使各组件的 GPU 利用率最大化。Figure 6: How M2Flow turns a logical workflow into a concrete execution plan. The system decomposes tasks along the temporal and the spatial dimension and reassembles the execution flow according to different data granularities, maximizing the GPU utilization of every component.
对于在同一设备上顺序执行的组件，RLinf 引入 context switching 机制：通过自动设备锁（device lock）和 offload/onload 函数实现多个 worker 对同一 GPU 的时分复用（time-division multiplexing）。分布式锁确保跨 worker 的全局一致性，避免死锁。For components that run sequentially on the same device, RLinf introduces a context switching mechanism: an automatic device lock together with offload/onload functions realizes time-division multiplexing of one GPU among several workers. A distributed lock guarantees global consistency across workers and avoids deadlock.
Elastic pipelining 允许 worker 以可变数据粒度（variable data granularity）跨设备并行处理——"output data can be forwarded once a configured size of data batch is ready"，无需等待整个 batch 处理完毕即可触发下游计算，从根本上解决 long-tail rollout 问题。Elastic pipelining lets workers process data in parallel across devices at a variable data granularity — "output data can be forwarded once a configured size of data batch is ready" — so downstream computation can be triggered without waiting for the whole batch to finish, which fundamentally resolves the long-tail rollout problem.
图 7：三种代表性执行模式：纯 temporal scheduling（组件顺序共享 GPU）、带 pipelining 的 spatial scheduling（组件并行、数据流式传输）、以及 hybrid scheduling（混合两种策略）。不同模式适用于不同的组件特性组合。Figure 7: Three representative execution modes: pure temporal scheduling (components share the GPU sequentially), spatial scheduling with pipelining (components run in parallel and data is streamed), and hybrid scheduling (a mix of the two). Different modes fit different combinations of component characteristics.
RLinf 采用递归算法对工作流做 s-t cut 划分，逐层评估 temporal 与 spatial 执行模式的运行时估算。Spatial 模式下的运行时估算公式为：RLinf partitions the workflow with a recursive s-t cut algorithm, evaluating level by level the runtime estimates of the temporal and the spatial execution mode. The runtime estimate under the spatial mode is:
T_critical + (M/m − 1) × T_bottleneck
其中 M 为总 batch size，m 为数据处理粒度，T_bottleneck 为流水线瓶颈阶段耗时。系统通过 profiling 实测各组件在不同 batch size 下的执行时间，指导最终调度决策，估算误差 temporal 模式 <2%，spatial 模式 <5%。Here M is the total batch size, m is the data-processing granularity, and T_bottleneck is the time of the bottleneck stage of the pipeline. The system profiles the measured execution time of each component under different batch sizes to guide the final scheduling decision; the estimation error is <2% for the temporal mode and <5% for the spatial mode.
03实验Experiments
RLinf 在推理型 RL（Reasoning RL）和具身 RL（Embodied RL）两大场景下与 SOTA 系统（veRL、Slime、SimpleVLA-RL 等）进行对比，评估指标涵盖训练吞吐量（throughput）和模型性能（task success rate / accuracy）。RLinf is compared against SOTA systems (veRL, Slime, SimpleVLA-RL and others) in the two major settings of reasoning RL and embodied RL, with evaluation metrics covering training throughput and model performance (task success rate / accuracy).
图 8：RLinf 与 veRL 在不同规模下的 GRPO 训练吞吐量对比（Qwen2.5 系列模型）。RLinf 在 64 GPU 上对 Qwen2.5-1.5B 实现 1.10×–1.58× 加速，对 Qwen2.5-7B PPO 在 16–256 GPU 规模下实现 35.0%–69.6% 吞吐量提升。Figure 8: GRPO training throughput of RLinf versus veRL at different scales (Qwen2.5 model family). On 64 GPUs RLinf reaches a 1.10×–1.58× speedup for Qwen2.5-1.5B, and for Qwen2.5-7B PPO it delivers a 35.0%–69.6% throughput gain at scales of 16–256 GPUs.
图 14：具身 RL 场景下 RLinf 在不同执行模式下的端到端吞吐量提升。ManiSkill 任务下混合模式（hybrid mode）实现 52.2%–87.2% 加速；LIBERO 任务下实现 37.8%–143.4% 的加速（vs. SimpleVLA-RL）。Figure 14: End-to-end throughput gains of RLinf under different execution modes in embodied RL settings. On ManiSkill tasks the hybrid mode reaches a 52.2%–87.2% speedup; on LIBERO tasks the speedup is 37.8%–143.4% (vs. SimpleVLA-RL).
调度策略的搜索开销随 GPU 规模从 8 到 1024 增长，但始终控制在 7×10⁻⁴ 至 5.98 秒以内，远小于 RL 训练时间。吞吐量预测误差在 temporal 模式下 <2%，spatial 模式下 <5%，验证了 profiling-guided 策略的有效性。Qwen3-30B-A3B 在 32–128 GPU 规模上对比 Slime 实现 7.2%–31.2% 吞吐量提升。The search overhead of the scheduling policy grows as the GPU scale goes from 8 to 1024, yet it always stays within 7×10⁻⁴ to 5.98 seconds, far below the RL training time. The throughput prediction error is <2% for the temporal mode and <5% for the spatial mode, which validates the effectiveness of the profiling-guided policy. On 32–128 GPUs, Qwen3-30B-A3B attains a 7.2%–31.2% throughput gain over Slime.
04局限性Limitations
注：论文未设独立的 Limitations 章节。以下各项为从系统设计与实验描述中推断（inferred from the design），非作者明确陈述（stated）。Note: the paper has no dedicated Limitations section. The items below are inferred from the system design and the experimental description, not stated explicitly by the authors.
空间调度模式的预测误差随 pipeline 引入而增大The prediction error of the spatial scheduling mode grows once pipelining is introduced
论文报告 spatial 模式的吞吐量预测误差为 <5%，高于 temporal 模式的 <2%。响应长度的动态性（response-length variability）是主要原因——elastic pipelining 引入的动态数据粒度使得精确建模更困难（inferred）。The paper reports a throughput prediction error of <5% for the spatial mode, higher than the <2% of the temporal mode. Response-length variability is the main cause — the dynamic data granularity brought by elastic pipelining makes accurate modeling harder (inferred).
当前设计面向典型的小型 RL 工作流（节点数 <10）The current design targets typical small RL workflows (fewer than 10 nodes)
论文中讨论的 RL 工作流通常包含少于 10 个节点（典型 GRPO/PPO 拓扑结构）。对于更复杂的工作流（如多模型协作、长链 agentic RL），调度搜索空间的指数级增长可能带来挑战（inferred）。The RL workflows discussed in the paper usually contain fewer than 10 nodes (typical GRPO/PPO topologies). For more complex workflows, such as multi-model collaboration or long-chain agentic RL, the exponential growth of the scheduling search space may become a challenge (inferred).
系统实现规模大（20K 行代码），自定义成本不低The implementation is large (20K lines of code), so customization is not cheap
RLinf 系统代码量约 20K 行（5K 核心组件 + 2K 公共 worker + 13K 算法/模型支持），尽管提供了声明式 API，但对用户自定义新 RL 算法或集成非标准模型框架，仍需理解内部 worker/调度机制（inferred）。The RLinf codebase amounts to about 20K lines (5K core components + 2K common workers + 13K algorithm/model support). Although a declarative API is provided, users who customize a new RL algorithm or integrate a non-standard model framework still need to understand the internal worker and scheduling mechanisms (inferred).
