World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy
通过视频世界模型与VLA策略的闭环迭代联合优化，低成本实现机器人操作能力跃升Closed-loop iterative joint optimization of a video world model and a VLA policy, achieving a low-cost leap in robot manipulation capability
Xiaokang Liu, Zechen Bai, Hai Ci, Kevin Yuchen Ma, Mike Zheng Shou · Show Lab, National University of Singapore
物理机器人强化学习代价高昂，而现有视频世界模型作为虚拟环境时存在两大缺陷：对细粒度动作不敏感（尤其在"近成功"失败情形下大量幻觉成功），以及缺乏原生奖励信号。
本文提出 World-VLA-Loop，将 Success And Near-Success（SANS）数据集、状态感知奖励头、GRPO 策略优化以及迭代闭环增广四个模块整合为统一框架，策略与世界模型互相促进，持续提升。Reinforcement learning on physical robots is prohibitively expensive, while existing video world models used as virtual environments suffer from two major flaws: insensitivity to fine-grained actions (in particular they hallucinate success extensively in "near-success" failure cases), and the absence of a native reward signal. This paper proposes World-VLA-Loop, which integrates four modules — the Success And Near-Success (SANS) dataset, a state-aware reward head, GRPO policy optimization, and iterative closed-loop augmentation — into a unified framework, so that the policy and the world model reinforce each other and keep improving.
VLA policyvideo world modelreinforcement learningrobot manipulationclosed-loop learningnear-success trajectorystate-aware rewardGRPO机器人操作robot manipulation视频世界模型video world model
01动机Motivation
在真实物理环境中对VLA（Vision-Language-Action）策略进行强化学习，需要大量机器人交互，代价极高且存在安全风险。
视频世界模型作为虚拟环境是一条有吸引力的替代路径，但现有方案存在两个关键瓶颈，导致实际效果受限。Applying reinforcement learning to VLA (Vision-Language-Action) policies in the real physical world requires a large amount of robot interaction, which is extremely costly and carries safety risks. Using a video world model as a virtual environment is an attractive alternative, but existing solutions have two key bottlenecks that limit their practical effectiveness.
问题一：动作感知不精准Problem 1: imprecise action following
当机器人动作存在微小误差（"近成功"情形，例如差一点就能抓到物体）时，现有视频世界模型
"frequently hallucinate successful outcomes even under erroneous actions, reflecting weak grounding in fine-grained physical dynamics"（论文原话）。
这意味着用于RL训练的虚拟rollout质量低，导致策略无法从失败中学习。When the robot's action contains a small error (the "near-success" case, e.g. just barely missing a grasp), existing video world models "frequently hallucinate successful outcomes even under erroneous actions, reflecting weak grounding in fine-grained physical dynamics" (the paper's own wording). This means the virtual rollouts used for RL training are of low quality, so the policy cannot learn from failure.
问题二：缺乏原生奖励信号Problem 2: no native reward signal
奖励需要通过另一个模块（如VLM）从生成的视频帧中提取。由于视频质量本身就存在幻觉，
计算出来的奖励信号不可靠，策略优化方向失真。
此外，随着VLA策略在RL训练中不断改进，其失败模式也随之改变，固定的世界模型无法跟上，造成分布偏移。The reward has to be extracted from the generated video frames by another module (such as a VLM). Because the generated video itself hallucinates, the computed reward signal is unreliable and the direction of policy optimization is distorted. Moreover, as the VLA policy keeps improving during RL training, its failure modes shift as well; a fixed world model cannot keep up, which causes distribution shift.
"Current video generation-based world models, when used as RL environments for VLA policies, struggle with two critical limitations: imprecise action-following, especially in near-success failure cases, and the absence of a reliable native reward signal."
Figure 1：三种机器人VLA强化学习范式对比：手工构建的数字孪生环境、基于3D重建的仿真器、视频世界模型。
World-VLA-Loop 属于第三类，通过闭环迭代解决动作感知不精准的核心问题。
右图显示两轮联合优化后，真实机器人实验中两项任务的成功率分别提升了 36.7% 和 26.6%。Figure 1: Comparison of three paradigms for robot VLA reinforcement learning: hand-built digital-twin environments, simulators based on 3D reconstruction, and video world models. World-VLA-Loop belongs to the third category and tackles the core problem of imprecise action following through closed-loop iteration. The right panel shows that after two rounds of joint optimization, the success rates of two real-robot tasks improve by 36.7% and 26.6% respectively.
+24.0%LIBERO-Object 成功率提升LIBERO-Object success-rate gain
+10.0%LIBERO-Spatial 成功率提升LIBERO-Spatial success-rate gain
+23.4%真实场景 Pick Cup 提升Real-world Pick Cup gain
+13.3%第二次迭代额外相对提升Extra relative gain of the second iteration
02方法Method
World-VLA-Loop 由四个相互配合的模块构成：SANS 数据集构建 → 状态感知视频世界模型训练 → VLA策略GRPO强化学习 → 迭代闭环数据增广。
四个模块首尾相接，形成闭环：策略进化产生新的rollout数据，新数据再次微调世界模型，从而实现"策略—世界模型"的持续共同进化。World-VLA-Loop is built from four mutually reinforcing modules: SANS dataset construction → state-aware video world model training → GRPO reinforcement learning of the VLA policy → iterative closed-loop data augmentation. The four modules are chained end to end into a closed loop: policy evolution produces new rollout data, the new data fine-tunes the world model again, and the "policy-world model" pair thus keeps co-evolving.
Figure 3：完整流程。
（1）从真实机器人收集"成功 + 近成功"轨迹，构建 SANS 数据集；
（2）基于 Cosmos-Predict 2 预训练状态感知视频世界模型（含联合奖励预测头）；
（3）以世界模型为虚拟环境，用 GRPO 对 OpenVLA-OFT 进行强化学习后训练；
（4）将改进策略产生的新成功/近成功轨迹追加到 SANS，微调世界模型，开始下一轮迭代。Figure 3: the full pipeline. (1) Collect "success + near-success" trajectories from a real robot and build the SANS dataset; (2) pre-train a state-aware video world model (with a joint reward prediction head) on top of Cosmos-Predict 2; (3) use the world model as a virtual environment and post-train OpenVLA-OFT with GRPO; (4) append the new success / near-success trajectories produced by the improved policy to SANS, fine-tune the world model, and start the next iteration.
模块一：SANS 数据集（Success And Near-Success）Module 1: the SANS dataset (Success And Near-Success)
传统数据集只保留成功轨迹，导致世界模型看不到"差一点就失败"的细粒度物理动态。
SANS 数据集刻意混入"近成功"失败轨迹——即"the robot fails to achieve a specific goal due to minor action errors"——
迫使模型"focus on fine-grained nuances in spatial dynamics"。
在 ManiSkill 预训练阶段收集 35k 视频-动作对；在任务特定阶段，每个任务收集约 50 条成功轨迹和 50 条近成功失败轨迹。Conventional datasets keep only successful trajectories, so the world model never sees the fine-grained physical dynamics of "failing by a hair". The SANS dataset deliberately mixes in "near-success" failure trajectories — cases where "the robot fails to achieve a specific goal due to minor action errors" — forcing the model to "focus on fine-grained nuances in spatial dynamics". 35k video-action pairs are collected in the ManiSkill pre-training stage; in the task-specific stage, about 50 successful and 50 near-success failure trajectories are collected per task.
Figure 2：近成功失败样例。
透明叠加层显示机器人夹爪真实轨迹（ground-truth），与世界模型生成帧中的位置存在明显偏差。
现有视频世界模型在这类情形下常常"幻觉"出抓取成功的画面，而 SANS 训练后的模型能更准确区分成功与失败。Figure 2: near-success failure examples. The transparent overlay shows the ground-truth trajectory of the robot gripper, which deviates noticeably from its position in the frames generated by the world model. Existing video world models often "hallucinate" a successful grasp in such cases, whereas the model trained on SANS distinguishes success from failure far more accurately.
模块二：状态感知视频世界模型Module 2: the state-aware video world model
以 Cosmos-Predict 2 为基础，输入观测帧序列和机器人动作（6-DoF 末端执行器位姿 + 夹爪状态），
自回归预测未来帧。关键创新是在扩散 Transformer 中增加奖励预测头（reward prediction head），
直接作用于扩散隐变量而非后处理步骤。联合训练损失为：Built on Cosmos-Predict 2, the model takes a sequence of observation frames and robot actions (6-DoF end-effector pose + gripper state) as input and autoregressively predicts future frames. The key innovation is adding a reward prediction head inside the diffusion Transformer, acting directly on the diffusion latents rather than as a post-processing step. The joint training loss is:
ℒ = ℒ_flow + λ · Σᵢ₌₁ᵀ ‖r̂ₜ − rₜ‖²
联合训练带来双重好处：（1）奖励与视觉结果自然对齐；（2）生成器受奖励监督约束，被迫
"better distinguish successful versus failed execution outcomes under different action conditions"。
对比实验显示，集成奖励头的准确率（88–94%）优于独立VLM奖励（Qwen3-VL，84–93.9%），且推理效率更高。Joint training brings two benefits: (1) the reward is naturally aligned with the visual outcome; (2) the generator, constrained by reward supervision, is forced to "better distinguish successful versus failed execution outcomes under different action conditions". Comparative experiments show that the integrated reward head is more accurate (88–94%) than a standalone VLM reward (Qwen3-VL, 84–93.9%), and more efficient at inference as well.
模块三：VLA策略GRPO强化学习Module 3: GRPO reinforcement learning of the VLA policy
以 OpenVLA-OFT 为基础策略，世界模型作为虚拟环境提供多步观测和二值奖励信号。
对步骤级奖励设阈值，转化为任务成功信号，驱动 GRPO 优化。
chunk size 统一设为 24 帧。OpenVLA-OFT serves as the base policy, and the world model, acting as a virtual environment, provides multi-step observations and a binary reward signal. Step-level rewards are thresholded into a task-success signal that drives GRPO optimization. The chunk size is uniformly set to 24 frames.
每轮 RL 训练后，将改进策略在真实机器人上产生的新成功和近成功 rollout 追加至 SANS 数据集，
再次微调世界模型，进入下一轮迭代，实现
"a comprehensive, iterative joint-optimization framework for both the world model and the VLA policy"。After each round of RL training, the new success and near-success rollouts that the improved policy produces on the real robot are appended to the SANS dataset; the world model is fine-tuned again and the next iteration begins, realizing "a comprehensive, iterative joint-optimization framework for both the world model and the VLA policy".
03实验Experiments
在仿真（LIBERO benchmark：Object / Goal / Spatial 三个任务套件）和真实机器人（Franka 机械臂 + RealSense D435）上进行评估。
基线为 OpenVLA-OFT SFT（监督微调版）；上界为在 LIBERO 物理仿真器中进行 RL 的 Oracle 系统。Evaluation is carried out both in simulation (the LIBERO benchmark: the Object / Goal / Spatial task suites) and on a real robot (a Franka arm + a RealSense D435). The baseline is OpenVLA-OFT SFT (the supervised fine-tuned version); the upper bound is an Oracle system trained with RL inside the LIBERO physics simulator.
视频生成质量Video generation quality
指标Metric
数值Value
说明Notes
SSIM
0.91
结构相似度Structural similarity
PSNR
28.09
峰值信噪比 (dB)Peak signal-to-noise ratio (dB)
LPIPS
0.045
感知相似度（越低越好）Perceptual similarity (lower is better)
Visual Outcome Alignment
90%
视觉结果与真实一致率（平均）Agreement of the visual outcome with the ground truth (average)
~98.5%（在真实物理仿真器RL）~98.5% (RL in the real physics simulator)
—
真实机器人成功率对比Real-robot success-rate comparison
任务Task
OpenVLA-OFT SFT（基线）OpenVLA-OFT SFT (baseline)
World-VLA-Loop（本文）World-VLA-Loop (ours)
提升Gain
Pick and Place Cup
13.3%
36.7%
+23.4%
Pushing Cube
26.7%
40.0%
+13.3%
Figure 4：RL训练步数对应的成功率曲线，涵盖 LIBERO Object / Goal / Spatial 和真实机器人两项任务。
可见在所有场景下，World-VLA-Loop 均随训练步数持续提升，最终接近或超过基线水平的天花板。Figure 4: Success-rate curves against RL training steps, covering LIBERO Object / Goal / Spatial and the two real-robot tasks. In every setting World-VLA-Loop keeps improving as training proceeds, eventually approaching or exceeding the ceiling of the baseline.Figure 5：定性对比。上方为世界模型生成的rollout，下方为真实机器人执行结果。
第一列为SFT基线（夹爪轨迹不精准），第二列为第一轮RL后（动作精准度提升），第三列为第二轮迭代RL后（进一步精细化）。
迭代闭环带来的累积改进清晰可见。Figure 5: qualitative comparison. The top row shows rollouts generated by the world model, the bottom row the real-robot executions. The first column is the SFT baseline (imprecise gripper trajectory), the second column is after the first round of RL (improved action precision), and the third column is after the second iteration of RL (further refinement). The cumulative improvement brought by the closed loop is clearly visible.
消融实验Ablation studies
去除近成功数据：视觉对齐准确率下降约 30%，证明 SANS 数据集对模型精细物理建模能力至关重要。Removing near-success data: visual alignment accuracy drops by about 30%, showing that the SANS dataset is critical for the model's fine-grained physical modeling ability.
去除集成奖励头（改用 VLM 奖励）：奖励准确率从 88–94% 降至 84–93.9%，
且推理效率显著下降（VLM 推理需要额外计算步骤）。Removing the integrated reward head (using a VLM reward instead): reward accuracy drops from 88–94% to 84–93.9%, and inference efficiency degrades markedly (VLM inference requires extra computation steps).
迭代增益：第二轮 RL 迭代相对第一轮额外带来 +13.3% 相对提升，
验证闭环增广策略的有效性。Iteration gain: the second round of RL iteration brings an extra +13.3% relative improvement over the first round, validating the effectiveness of the closed-loop augmentation strategy.
04局限性Limitations
说明：以下局限性均为论文作者明确陈述（stated by the authors）。Note: all limitations below are explicitly stated by the authors.
长时域任务（>300 帧）不适用Long-horizon tasks (>300 frames) are out of reach
论文明确指出："Severe quality degradation typically occurs only after the 300-frame mark"，
因此 LIBERO-100 等需要 400+ 帧的长时域任务在当前框架下无法支持。
自回归视频模型存在"limited context memory and quality drift"的固有缺陷，是该限制的根本原因。The paper states explicitly that "Severe quality degradation typically occurs only after the 300-frame mark", so long-horizon tasks requiring 400+ frames, such as LIBERO-100, cannot be supported by the current framework. The root cause is the inherent "limited context memory and quality drift" of autoregressive video models.
奖励信号稀疏，收敛慢Sparse reward signal, slow convergence
目前使用的是稀疏终态奖励（任务最终是否成功），而非步骤级中间子目标奖励。
作者指出未来需要"transitioning from sparse final-state rewards to step-wise intermediate sub-goals for improved RL convergence"。What is currently used is a sparse final-state reward (whether the task ultimately succeeds), not step-level intermediate sub-goal rewards. The authors note that future work needs "transitioning from sparse final-state rewards to step-wise intermediate sub-goals for improved RL convergence".
视频骨干网络的长期稳定性有待提升Long-term stability of the video backbone needs improvement
当前基于 Cosmos-Predict 2 的视频模型在帧数超过阈值后存在质量漂移（quality drift）。
作者指出需要"exploring video backbones with enhanced long-term stability"和
更新的自回归视频生成技术来突破 300 帧限制，以支持更复杂的操作任务。The current video model, based on Cosmos-Predict 2, exhibits quality drift once the frame count exceeds a threshold. The authors note the need for "exploring video backbones with enhanced long-term stability" and newer autoregressive video generation techniques in order to break the 300-frame limit and support more complex manipulation tasks.
