WMPO（World Model-based Policy Optimization）通过像素级视频扩散世界模型，在无需与真实机器人交互的情况下实现 VLA 的 on-policy 强化学习。利用 GRPO（Group Relative Policy Optimization），WMPO 在仿真与真实机器人实验中均大幅超越基于 DPO/off-policy 的方法，并展现出自我纠错、鲁棒泛化和持续学习等涌现能力。WMPO (World Model-based Policy Optimization) achieves on-policy reinforcement learning of VLA models without any interaction with real robots, by means of a pixel-space video diffusion world model. Using GRPO (Group Relative Policy Optimization), WMPO substantially outperforms DPO/off-policy based methods in both simulated and real-robot experiments, and exhibits emergent capabilities such as self-correction, robust generalization and lifelong learning.
VLA 模型（Vision-Language-Action models）在通用机器人操作上展现出强大潜力，但对专家 demonstration 的依赖使其无法从失败中学习、也难以进行自我修正。强化学习（RL）虽然能解决这一问题，却在真实机器人上存在极高的 sample complexity。VLA models (Vision-Language-Action models) show strong potential for general-purpose robotic manipulation, but their dependence on expert demonstration leaves them unable to learn from failure and hard to self-correct. Reinforcement learning (RL) can address this issue, yet it suffers from extremely high sample complexity on real robots.
"RL addresses these through self-improving interactions with the physical environment, but suffers from high sample complexity on real robots."
现有三类 VLA 训练范式各有局限：Imitation Learning（IL）依赖人类 demonstration，策略脆弱易出错；Off-policy RL（如 DPO）利用历史数据但性能受限；On-policy RL（如 GRPO）需要大量真实机器人交互，成本高昂。WMPO 通过像素级世界模型在"想象"空间中生成 on-policy 轨迹，从根本上解决了这一矛盾。The three existing paradigms of VLA training each have their limits: Imitation Learning (IL) relies on human demonstration and yields brittle, error-prone policies; Off-policy RL (e.g. DPO) exploits historical data but is limited in performance; On-policy RL (e.g. GRPO) requires massive real-robot interaction at high cost. By using a pixel-space world model to generate on-policy trajectories in an "imagined" space, WMPO resolves this tension at its root.
图 1：三种 VLA 训练范式对比。(a) Imitation learning 从人类 demonstration 学习；(b) Off-policy RL 利用历史收集的轨迹；(c) WMPO 利用世界模型生成 on-policy 的"想象"轨迹，无需真实机器人交互。Figure 1: Comparison of three VLA training paradigms. (a) Imitation learning learns from human demonstration; (b) Off-policy RL exploits historically collected trajectories; (c) WMPO uses a world model to generate on-policy "imagined" trajectories, requiring no real-robot interaction.
57.6%WMPO 仿真平均成功率（P=1280）WMPO mean success rate in simulation (P=1280)
+15.2pp相对 DPO 的提升（P=1280）Improvement over DPO (P=1280)
0.95+Reward model F1 score（全任务）Reward model F1 score (all tasks)
02方法Method
WMPO 框架由三个核心模块组成：像素级视频世界模型、轻量级 reward model 和基于 GRPO 的 on-policy 策略优化。整体思路是让 policy 在世界模型生成的"想象轨迹"中进行 on-policy 更新，从而规避真实机器人交互的高成本。The WMPO framework comprises three core modules: a pixel-space video world model, a lightweight reward model, and GRPO-based on-policy policy optimization. The overall idea is to let the policy perform on-policy updates inside the "imagined trajectories" generated by the world model, thereby avoiding the high cost of real-robot interaction.
图 2：WMPO 整体训练流程。从初始状态 s₀ 出发，世界模型生成 G 条想象轨迹；reward model 对每条轨迹评分；GRPO 计算 normalized advantage 并更新 VLA policy。三个组件交替迭代，实现 on-policy 的自我提升。Figure 2: The overall WMPO training pipeline. Starting from an initial state s₀, the world model generates G imagined trajectories; the reward model scores each trajectory; GRPO computes the normalized advantage and updates the VLA policy. The three components iterate alternately, achieving on-policy self-improvement.
像素级视频世界模型（Pixel-Space World Model）Pixel-Space Video World Model
与常见的 latent world model 不同，WMPO 采用像素级预测，使"想象"轨迹与 VLA 预训练特征（基于海量网络图像）保持对齐。世界模型基于 video diffusion 架构：将 OpenSora 的 3D VAE 替换为来自 SDXL 的 2D VAE，以更好地保留运动细节。模型首先在 Open X-Embodiment（OXE）数据集上预训练，再通过 Policy Behavior Alignment 在策略自身采集的真实轨迹上微调，以解决 state-distribution mismatch 问题——确保模型能准确模拟当前 policy 产生的失败模式。两项技术增强保证了质量：(1) Noisy-frame conditioning：条件帧加入 50/1000 步扩散噪声，增强鲁棒性；(2) Frame-level action control：通过扩展的 AdaLN block 实现精确的动作-帧对齐，支持 clip-level 自回归视频生成。Unlike the common latent world model, WMPO adopts pixel-space prediction, keeping the "imagined" trajectories aligned with the VLA pretraining features (learned from massive web images). The world model builds on a video diffusion architecture: the 3D VAE of OpenSora is replaced by the 2D VAE from SDXL, so as to better preserve motion details. The model is first pretrained on the Open X-Embodiment (OXE) dataset, then finetuned through Policy Behavior Alignment on real trajectories collected by the policy itself, in order to resolve the state-distribution mismatch problem — ensuring that the model faithfully simulates the failure modes produced by the current policy. Two technical enhancements secure the quality: (1) Noisy-frame conditioning: 50/1000 steps of diffusion noise are added to the conditioning frames to improve robustness; (2) Frame-level action control: an extended AdaLN block realizes precise action-frame alignment and supports clip-level autoregressive video generation.
轻量级 Reward ModelLightweight Reward Model
Reward model 在真实轨迹上训练，以二元信号（成功/失败）预测任务完成情况，为 GRPO 提供 sparse reward 信号。在所有仿真任务上的 F1 score 均超过 0.95，确保了训练信号的可靠性。The reward model is trained on real trajectories and predicts task completion with a binary signal (success/failure), providing a sparse reward signal for GRPO. Its F1 score exceeds 0.95 on all simulation tasks, which guarantees the reliability of the training signal.
WMPO 采用 Group Relative Policy Optimization（GRPO）进行 on-policy 学习。对每个初始状态采样 G 条想象轨迹，reward model 评分后计算 normalized advantage，再通过 clipped policy gradient 更新 policy：WMPO adopts Group Relative Policy Optimization (GRPO) for on-policy learning. For each initial state, G imagined trajectories are sampled; after the reward model scores them, the normalized advantage is computed and the policy is updated through a clipped policy gradient:
为保证有效学习，采用 dynamic sampling 策略：丢弃结果完全一致（全成功或全失败）的 group，确保每个 batch 包含成功与失败轨迹的混合，从而产生有意义的梯度信号。VLA backbone 为 OpenVLA-OFT，动作空间离散化为每维度 256 个 bin。To ensure effective learning, a dynamic sampling strategy is adopted: groups whose outcomes are entirely uniform (all success or all failure) are discarded, so that every batch contains a mixture of successful and failed trajectories and therefore produces meaningful gradient signals. The VLA backbone is OpenVLA-OFT, and the action space is discretized into 256 bins per dimension.
图 3：Square 任务中 WMPO 的自我纠错能力分析。WMPO 在遭遇失败后能主动调整策略、恢复到正确轨迹，而 base policy 和 DPO 则无法做到。Figure 3: Analysis of WMPO's self-correction ability on the Square task. After encountering a failure, WMPO can actively adjust its policy and recover to the correct trajectory, which the base policy and DPO cannot do.
03实验Experiments
实验在 Mimicgen benchmark 的四个精细操作任务（Coffee_D0、StackThree_D0、ThreePieceAssembly_D0、Square_D0）上进行仿真评估，并在真实机器人上验证 square 插入任务。Baselines 包括：base policy（OpenVLA-OFT imitation learning）、GRPO（on-policy，真实机器人交互）和 DPO（off-policy）。Simulation evaluation is conducted on four fine-grained manipulation tasks of the Mimicgen benchmark (Coffee_D0, StackThree_D0, ThreePieceAssembly_D0, Square_D0), and the square insertion task is validated on a real robot. Baselines include: base policy (OpenVLA-OFT imitation learning), GRPO (on-policy, with real-robot interaction) and DPO (off-policy).
Table 1：仿真基准成功率（%）Table 1: Success Rate on the Simulation Benchmark (%)
Rollout Budget
方法Method
Coffee
StackThree
ThreePieceAssembly
Square
Mean (%)
—
Base policy
43.8
46.9
19.5
24.2
33.6
P=128
GRPO
38.3
52.3
17.2
25.0
33.2
DPO
43.8
53.9
23.4
28.1
37.3
WMPO
61.7
56.3
37.5
32.8
47.1
P=1280
GRPO
47.7
54.7
20.3
25.8
37.1
DPO
52.3
57.0
26.7
33.6
42.4
WMPO
75.0
64.1
46.1
45.3
57.6
WMPO 在小 rollout 预算（P=128）下即超越基线 +9.8pp（vs. DPO 37.3%），在大预算（P=1280）下领先 +15.2pp（vs. DPO 42.4%），充分体现出 on-policy 更新的 sample efficiency 优势。Under a small rollout budget (P=128) WMPO already surpasses the baseline by +9.8pp (vs. DPO 37.3%), and under a large budget (P=1280) it leads by +15.2pp (vs. DPO 42.4%), fully demonstrating the sample efficiency advantage of on-policy updates.
Table 2：泛化能力（%）Table 2: Generalization (%)
方法Method
Position Disruption
Background Disruption
Texture Disruption
Mean
Base policy
14.1
46.1
10.9
23.7
GRPO
15.6
47.7
10.9
24.7
DPO
16.4
34.4
7.8
19.5
WMPO
22.3
50.0
16.4
29.6
图 4a：Position Disruption — 物体位置扰动下的泛化测试。WMPO 22.3% vs. DPO 16.4%。Figure 4a: Position Disruption — generalization test under object position perturbation. WMPO 22.3% vs. DPO 16.4%.图 4b：Background Disruption — 背景替换下的泛化测试。WMPO 50.0% vs. DPO 34.4%。Figure 4b: Background Disruption — generalization test under background replacement. WMPO 50.0% vs. DPO 34.4%.
真实机器人实验Real-Robot Experiments
图 7：真实机器人上的精细操作实验（square 插入任务），共 30 次试验。Base policy：53%；DPO：60%；WMPO：70%。WMPO 同时展现出失败预测能力，能在任务失败前主动停止。Figure 7: Fine-grained manipulation experiment on a real robot (the square insertion task), 30 trials in total. Base policy: 53%; DPO: 60%; WMPO: 70%. WMPO also exhibits failure-prediction ability, stopping proactively before the task fails.
持续学习（Lifelong Learning）Lifelong Learning
图 5：不同策略在成功试验中的相对平均轨迹长度。WMPO 的自我纠错能力使其路径更高效。Figure 5: Relative mean trajectory length of different policies on successful trials. The self-correction ability of WMPO makes its paths more efficient.
持续学习实验中，迭代收集 P=128 条真实轨迹后执行 WMPO 更新，再用更新后的策略继续采集。结果表明 WMPO 实现了"稳定且显著的提升"，而 DPO 由于训练不稳定无法持续改进（StackThree 任务上验证）。In the lifelong learning experiment, P=128 real trajectories are collected iteratively, a WMPO update is then performed, and the updated policy continues collecting. The results show that WMPO achieves "stable and significant improvement", whereas DPO cannot improve continually because of unstable training (verified on the StackThree task).
局限性消融：世界模型失败案例Limitation Ablation: World Model Failure Cases
图 9：世界模型预测失败示例。在某些复杂场景下，世界模型生成的视频与真实物理动态存在偏差，导致 reward 信号失真。Figure 9: Examples of world model prediction failure. In some complex scenes, the video generated by the world model deviates from the true physical dynamics, distorting the reward signal.
04局限性Limitations
说明：以下局限性部分来自论文明确陈述，部分为从方法设计推断（已标注）。Note: Some of the limitations below are explicitly stated in the paper, others are inferred from the method design (marked accordingly).
仅支持离散化动作表示（论文明确陈述）Only discretized action representations are supported (explicitly stated in the paper)
"While the WMPO framework can in principle support flow-based policies, this work focuses on discretized action representations."论文将 flow-matching based policies 的扩展列为未来工作，当前版本每维度离散为 256 个 bin，限制了连续动作空间场景的适用性。"While the WMPO framework can in principle support flow-based policies, this work focuses on discretized action representations." The paper lists the extension to flow-matching based policies as future work; the current version discretizes each dimension into 256 bins, which limits its applicability to continuous action space scenarios.
世界模型预测质量受限（论文明确陈述）Limited prediction quality of the world model (explicitly stated in the paper)
论文在 Figure 9 中展示了世界模型预测失败的案例：在某些复杂操作场景下，生成的视频与真实物理动态存在偏差，导致 reward model 收到失真信号，影响策略优化质量。In Figure 9 the paper shows cases of world model prediction failure: in some complex manipulation scenes the generated video deviates from the true physical dynamics, so the reward model receives a distorted signal, which degrades the quality of policy optimization.
Policy Behavior Alignment 带来额外训练成本（从设计推断）Policy Behavior Alignment incurs extra training cost (inferred from the design)
为缓解 state-distribution mismatch，世界模型需要在每轮迭代中用当前策略的真实轨迹进行微调（Policy Behavior Alignment），这意味着每次策略更新后世界模型也需要同步更新，增加了整体训练复杂度和计算开销。To alleviate the state-distribution mismatch, the world model has to be finetuned at every iteration on real trajectories from the current policy (Policy Behavior Alignment); this means the world model must be updated in step after each policy update, increasing overall training complexity and computational overhead.
仅在有限任务集上验证（从设计推断）Validated only on a limited set of tasks (inferred from the design)
实验仅在 Mimicgen 的四个操作任务和单一真实机器人场景上验证，尚未扩展到更多元的任务类型（如灵巧手操作、移动操作等），泛化能力有待进一步验证。The experiments are validated only on four manipulation tasks of Mimicgen and a single real-robot scenario, and have not been extended to more diverse task types (such as dexterous hand manipulation or mobile manipulation); the generalization ability remains to be further verified.
