World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training
以世界模型为虚拟环境的 VLA 强化学习后训练框架A reinforcement-learning post-training framework for VLA that uses a world model as the virtual environment
Junjin Xiao, Yandan Yang, Xinyuan Chang, Ronghan Chen, Feng Xiong, Mu Xu, Wei-Shi Zheng, Qing Zhang
VLA（Vision-Language-Action）模型依赖大规模示例学习，在数据稀缺场景下性能严重下降。
World-Env 提出以扩散式世界模型替代物理机器人环境，配合 VLM 奖励模块和 Leave-One-Out PPO，
仅需每任务 5 条专家示例即可在 LIBERO 基准上取得显著的策略性能提升，同时规避了真实机器人交互的安全风险与高成本。VLA (Vision-Language-Action) models rely on large-scale demonstration learning, and their performance degrades severely in data-scarce scenarios. World-Env proposes replacing the physical robot environment with a diffusion-based world model; combined with a VLM reward module and Leave-One-Out PPO, it attains significant policy performance gains on the LIBERO benchmark with only 5 expert demonstrations per task, while avoiding the safety risks and high cost of real-robot interaction.
VLA 模型通过模仿学习训练，在数据充足时表现优异，但在示例稀缺时性能大幅退化。
虽然强化学习（RL）后训练可缓解数据稀缺问题，但直接在真实机器人上应用 RL 面临两大核心障碍：
真实环境的不可重置性，以及在工业自动化等高风险领域中的安全与成本约束。
此外，现有 VLA 方法缺乏可靠的任务完成检测机制，导致"冗余动作降低整体任务成功率"。VLA models are trained by imitation learning and perform excellently when data is abundant, but degrade sharply when demonstrations are scarce. Although reinforcement learning (RL) post-training can alleviate data scarcity, applying RL directly on real robots faces two core obstacles: the non-resettability of real environments, and the safety and cost constraints of high-risk domains such as industrial automation. Moreover, existing VLA methods lack a reliable task-completion detection mechanism, so that "redundant actions lower the overall task success rate".
"video-based world model offers a promising solution" — 通过对动作结果进行安全、低成本的仿真，
实现策略的探索与优化，无需与物理环境交互。"video-based world model offers a promising solution" — by simulating the outcomes of actions safely and at low cost, policy exploration and optimization become possible without interacting with the physical environment.
图 1：左：VLA 策略在世界模型环境中进行探索的示意图；
右：World-Env 在 LIBERO 四个子任务套件（Spatial、Object、Goal、Long）上与基线方法的性能对比，
在仅使用 5 条示例的数据稀缺设置下实现了当前最优或持平表现。Figure 1: Left: illustration of a VLA policy exploring inside the world-model environment; Right: performance comparison between World-Env and baseline methods on the four LIBERO task suites (Spatial, Object, Goal, Long), reaching state-of-the-art or on-par results under the data-scarce setting that uses only 5 demonstrations.图 2：三种 VLA 训练范式的对比：（a）纯 SFT（监督微调），依赖大量示例；
（b）基于真实机器人的 RL 后训练，成本高且存在安全风险；
（c）本文提出的 World-Env，以世界模型作为虚拟环境进行 RL 后训练，兼顾安全性与数据效率。Figure 2: Comparison of three VLA training paradigms: (a) pure SFT (supervised fine-tuning), which relies on large numbers of demonstrations; (b) RL post-training on real robots, which is costly and carries safety risks; (c) the proposed World-Env, which performs RL post-training with a world model as the virtual environment, balancing safety against data efficiency.
5每任务专家示例数Expert demonstrations per task
79.6%LIBERO 平均成功率（本文）LIBERO average success rate (ours)
+4.75%相对 OpenVLA-OFT 基线提升Gain over the OpenVLA-OFT baseline
4任务套件全面提升Task suites improved across the board
02方法Method
World-Env 框架由三个核心模块构成：物理一致性世界模拟器（生成视觉预测）、
VLM 引导的 Instant Reflector（提供连续奖励与终止信号），
以及基于 Leave-One-Out PPO 的后训练优化循环。
三者协同，使 VLA 模型能够在世界模型环境中安全探索并超越初始训练分布。The World-Env framework consists of three core modules: a physically-consistent world simulator (which generates visual predictions), a VLM-guided Instant Reflector (which provides a continuous reward and an end signal), and a post-training optimization loop based on Leave-One-Out PPO. Working together, they let the VLA model explore safely inside the world-model environment and go beyond its initial training distribution.
图 3：World-Env 整体框架，包含三个子模块：
训练数据策略（Training Data Strategy）通过自主探索收集成功与失败轨迹；
优化循环（Optimization Loop）使用 Leave-One-Out PPO 更新策略；
奖励与终止信号（Reward & End Signal）由 Instant Reflector 在线提供。Figure 3: The overall World-Env framework, comprising three sub-modules: the Training Data Strategy collects successful and failed trajectories through autonomous exploration; the Optimization Loop updates the policy with Leave-One-Out PPO; the Reward & End Signal is provided online by the Instant Reflector.
物理一致性世界模拟器（Physically-Consistent World Simulator）Physically-Consistent World Simulator
世界模拟器采用扩散模型架构，根据当前观测和动作序列生成时序一致的未来视觉帧。
关键创新在于"geometry-aware feature injection strategy"：
通过 cross-attention 将 VGGT 提取的几何结构特征和 CLIP 提取的语义特征注入 U-Net 去噪层，
使生成帧在物理上保持连贯。
为增强数据多样性，将已 SFT 的 OpenVLA-OFT 策略部署在模拟器中自主探索，
收集成功轨迹和失败轨迹；同时通过 Laplace 分布的 scale head 增强动作多样性。The world simulator adopts a diffusion-model architecture and generates temporally consistent future visual frames from the current observation and the action sequence. The key innovation lies in a "geometry-aware feature injection strategy": geometric structure features extracted by VGGT and semantic features extracted by CLIP are injected into the U-Net denoising layers through cross-attention, keeping the generated frames physically coherent. To enrich data diversity, the SFT-trained OpenVLA-OFT policy is deployed in the simulator to explore autonomously, collecting both successful and failed trajectories; action diversity is further enhanced by a scale head with a Laplace distribution.
图 4：物理一致性世界模拟器架构详图。VGGT 提供几何特征，CLIP 提供语义特征，
两者通过 cross-attention 注入扩散模型的 U-Net 去噪层，生成物理上合理的视觉预测。Figure 4: Detailed architecture of the physically-consistent world simulator. VGGT supplies geometric features and CLIP supplies semantic features; both are injected through cross-attention into the U-Net denoising layers of the diffusion model to produce physically plausible visual predictions.
VLM 引导的 Instant Reflector（奖励与终止信号）VLM-Guided Instant Reflector (Reward and End Signal)
Instant Reflector 是一个 VLM 模块，为每一帧预测 [0, 1] 范围内的任务完成概率作为连续奖励信号。
架构采用冻结视觉编码器和 LLM，配合可训练的奖励头，使用二元交叉熵损失在专家和策略生成的轨迹上进行训练。
连续奖励解决了"prior VLA post-training approaches rely on binary rewards (e.g., 1 for success, 0 for failure),
which lead to degenerate advantage estimates when rollout trajectories are homogeneous"的问题，
确保 advantage 估计始终非平凡，无需人工平衡成功与失败轨迹的比例。
当 Reflector 分数超过阈值 η=0.5 或达到最大时间步时，轨迹终止。The Instant Reflector is a VLM module that predicts, for every frame, a task-completion probability in [0, 1] as a continuous reward signal. Its architecture uses a frozen vision encoder and LLM together with a trainable reward head, trained with a binary cross-entropy loss on expert and policy-generated trajectories. The continuous reward resolves the problem that "prior VLA post-training approaches rely on binary rewards (e.g., 1 for success, 0 for failure), which lead to degenerate advantage estimates when rollout trajectories are homogeneous", ensuring that advantage estimates are always non-trivial and removing the need to manually balance the ratio of successful to failed trajectories. A trajectory terminates once the Reflector score exceeds the threshold η=0.5 or the maximum time step is reached.
后训练阶段结合 RLOO（Leave-One-Out）advantage 估计与 PPO 策略更新，
对 VLA 模型进行端到端强化学习优化。
世界模拟器生成的多样化轨迹（含成功与失败样本）作为 rollout 数据，
Instant Reflector 实时提供奖励，驱动策略持续改进并安全探索训练分布之外的状态空间。The post-training stage combines RLOO (Leave-One-Out) advantage estimation with PPO policy updates to optimize the VLA model end-to-end by reinforcement learning. The diverse trajectories generated by the world simulator (including both successful and failed samples) serve as rollout data, while the Instant Reflector supplies rewards in real time, driving continual policy improvement and safe exploration of the state space outside the training distribution.
03实验Experiments
在 LIBERO 基准的四个任务套件（Spatial、Object、Goal、Long）上评估，
每个任务仅使用训练集中的 5 条示例，在完整测试集上评测。
基线包括 π0、UniVLA、OpenVLA-OFT 及同期 RIPT-VLA 等。Evaluation is carried out on the four task suites of the LIBERO benchmark (Spatial, Object, Goal, Long), using only 5 demonstrations from the training set per task and testing on the full test set. Baselines include π0, UniVLA, OpenVLA-OFT and the concurrent RIPT-VLA.
图 7（局部）："clean table"任务真实机器人执行的采样帧序列。
World-Env 后训练后，机器人能更准确地感知任务完成时机并及时停止，避免冗余动作。Figure 7 (detail): Sampled frame sequence of a real robot executing the "clean table" task. After World-Env post-training, the robot perceives the moment of task completion more accurately and stops in time, avoiding redundant actions.
消融实验（Table 5）Ablation Study (Table 5)
消融分析揭示各模块的贡献：额外训练数据（世界模拟器自主探索轨迹）贡献 +11.4% 平均提升；
奖励头（Instant Reflector）贡献 +10.8% 平均提升。
终止信号分析（Table 4）表明，强制策略执行完整时序而不提前停止会导致性能退化至 54.9–65.4%，
而 World-Env 通过 Instant Reflector 检测任务完成时机，实现 74.9% 的成功率。The ablation analysis reveals the contribution of each module: the additional training data (trajectories from the world simulator’s autonomous exploration) contributes a +11.4% average gain; the reward head (Instant Reflector) contributes a +10.8% average gain. The end-signal analysis (Table 4) shows that forcing the policy to run the full horizon without stopping early degrades performance to 54.9–65.4%, whereas World-Env detects the moment of task completion through the Instant Reflector and achieves a 74.9% success rate.
04局限性Limitations
Note：以下局限性均为论文作者在论文中明确陈述（stated）。Note: All limitations below are explicitly stated by the authors in the paper.
数据依赖性（Data Dependency）Data Dependency
论文指出："Both the world simulator and the instant reflector rely on diverse training data
to achieve high-fidelity simulation and accurate task evaluation."
当前框架仍依赖针对特定任务域收集的数据来训练模拟器和奖励模型；
作者预期"future advances in general-purpose world models will alleviate this dependency"。The paper notes: "Both the world simulator and the instant reflector rely on diverse training data to achieve high-fidelity simulation and accurate task evaluation." The current framework still depends on data collected for a specific task domain in order to train the simulator and the reward model; the authors expect that "future advances in general-purpose world models will alleviate this dependency".
论文承认："Policy optimization in our framework is currently slower than in concurrent methods
due to computational bottlenecks in simulator-based trajectory generation."
基于世界模拟器的轨迹生成速度慢于直接在仿真器（如 IsaacGym）中运行策略的方法，
作者将其列为"a key focus of our future work"。The paper acknowledges: "Policy optimization in our framework is currently slower than in concurrent methods due to computational bottlenecks in simulator-based trajectory generation." Trajectory generation based on the world simulator is slower than methods that run the policy directly in a simulator (such as IsaacGym), and the authors list this as "a key focus of our future work".
与仿真器 RL 方法的对比（inferred）Comparison with Simulator-Based RL Methods (inferred)
（从设计推断）论文将 World-Env 与基于物理仿真器的 RL 方法（如 RIPT-VLA）并列比较，
指出两者在 LIBERO 上性能相当，但 World-Env 的优势在于"readily deployable in real-world settings"——
即当任务缺乏对应物理仿真器时仍可使用。
然而，当物理仿真器可用时，World-Env 的模拟保真度与速度仍不及直接仿真方案。(inferred from the design) The paper places World-Env side by side with RL methods built on physics simulators (such as RIPT-VLA), noting that the two perform comparably on LIBERO, while the advantage of World-Env is being "readily deployable in real-world settings" — that is, it remains usable when a task has no corresponding physics simulator. When a physics simulator is available, however, World-Env still falls short of direct simulation in fidelity and speed.
