VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model
用世界模型生成合成数据，迭代协同提升 VLA 策略与世界模型的物理真实性Generating synthetic data with a world model to iteratively co-improve the VLA policy and the physical fidelity of the world model
Yanjiang Guo, Tony Lee, Lucy Xiaoyang Shi, Jianyu Chen, Percy Liang, Chelsea Finn · Stanford University / Tsinghua University
VLAW 提出一种迭代框架：通过真实机器人回滚数据接地预训练世界模型，再让接地后的世界模型生成大量合成轨迹以优化 VLA 策略；两者相互增益，循环迭代，在 DROID 平台五类接触密集任务中平均成功率提升 39.2%。VLAW proposes an iterative framework: real robot rollout data grounds a pretrained world model, and the grounded world model in turn generates large amounts of synthetic trajectories to optimize the VLA policy; the two benefit each other and are improved in a loop, raising the mean success rate by 39.2% across five contact-rich task categories on the DROID platform.
大规模预训练 VLA 策略（如 π₀.₅）在泛化上表现出色，但在新任务上直接部署时需要大量真实示范或代价高昂的物理回滚才能完成微调。现有方法要么依赖人工重置与监督（成本高），要么在合成数据与真实物理之间存在较大 sim-to-real gap。Large-scale pretrained VLA policies (such as π₀.₅) generalize remarkably well, but deploying them directly on a new task still requires abundant real demonstrations or costly physical rollouts in order to fine-tune. Existing approaches either rely on manual resets and supervision (expensive), or suffer from a substantial sim-to-real gap between synthetic data and real physics.
"Although the learned world model achieves high fidelity on the downstream tasks from which online data are collected, our current evaluation is limited to five task categories."
世界模型（video generation model）是生成合成机器人数据的理想工具，但预训练世界模型往往缺乏对目标任务物理动态的精确建模——尤其是接触密集型操作（如叠放物体、擦黑板、翻书）。VLAW 的核心洞察是：少量真实回滚数据足以将预训练世界模型接地，使其能忠实模拟目标任务的物理过程；接地后的世界模型又能批量生成高质量合成轨迹来训练 VLA 策略，从而减少对昂贵真实交互的依赖。A world model (video generation model) is an ideal tool for producing synthetic robot data, yet pretrained world models often lack an accurate model of the physical dynamics of the target task — especially for contact-rich manipulation (such as stacking objects, wiping a board, or opening a book). The core insight of VLAW is: a small amount of real rollout data suffices to ground a pretrained world model so that it faithfully simulates the physical process of the target task; the grounded world model can then generate high-quality synthetic trajectories in bulk to train the VLA policy, reducing the reliance on expensive real interaction.
VLAW 总体思路。传统 VLA 在线回滚需要人工重置与持续监督，成本极高。VLAW 利用有限的真实回滚数据（每次迭代仅 50 条）训练接地世界模型，再由世界模型生成大量合成训练数据（每任务 500 条），大幅降低对真实物理交互的依赖，同时实现显著的策略性能提升。Overall idea of VLAW. Conventional VLA online rollouts require manual resets and continuous supervision, which is extremely costly. VLAW uses a limited amount of real rollout data (only 50 rollouts per iteration) to train a grounded world model, and then has the world model generate a large volume of synthetic training data (500 trajectories per task), greatly reducing the reliance on real physical interaction while achieving a substantial gain in policy performance.
39.2%平均绝对成功率提升（vs 基础策略）mean absolute success-rate gain (vs base policy)
11.6%合成数据带来的额外提升（vs 仅用真实数据）additional gain from synthetic data (vs real data only)
50每次迭代所需真实回滚条数real rollouts required per iteration
2迭代轮数即达最优iterations suffice to reach the best result
02方法Method
VLAW 是一个四步迭代循环：①收集真实回滚轨迹；②用回滚数据微调世界模型（同时与 DROID 示范数据联合训练以防 catastrophic forgetting）；③让 VLA 策略在世界模型中进行 closed-loop 推演，生成合成轨迹并由奖励模型过滤；④用过滤后的合成轨迹以加权 flow-matching 损失更新 VLA 策略。VLAW is a four-step iterative loop: (i) collect real rollout trajectories; (ii) fine-tune the world model on the rollout data (co-training it with DROID demonstration data to prevent catastrophic forgetting); (iii) let the VLA policy perform closed-loop rollouts inside the world model, generating synthetic trajectories that are filtered by a reward model; (iv) update the VLA policy on the filtered synthetic trajectories with a weighted flow-matching loss.
VLAW 四步迭代流程。（1）在目标任务上收集真实机器人回滚轨迹；（2）以回滚数据对预训练 Ctrl-World 模型进行接地微调，使用扩散目标 ℒ = 𝔼‖x̂₀(x_{t′},t′,c)−x₀‖²，并与 DROID 演示数据联合训练（正则化系数 λ）；（3）策略在接地世界模型中并行推演，生成 N 条合成轨迹，经 Qwen3-VL 奖励模型按成功率阈值过滤；（4）以加权 flow-matching 损失更新 VLA 策略，成功轨迹权重更高。The four-step iterative pipeline of VLAW. (1) Collect real robot rollout trajectories on the target task; (2) fine-tune the pretrained Ctrl-World model on the rollout data for grounding, using the diffusion objective ℒ = 𝔼‖x̂₀(x_{t′},t′,c)−x₀‖², co-trained with DROID demonstration data (regularization coefficient λ); (3) roll out the policy in parallel inside the grounded world model to generate N synthetic trajectories, filtered by a Qwen3-VL reward model against a success-rate threshold; (4) update the VLA policy with a weighted flow-matching loss, giving successful trajectories a higher weight.
世界模型接地（World Model Grounding）World Model Grounding
世界模型以预训练 Ctrl-World（在 DROID 数据集上训练的动作条件视频生成模型）为起点。关键改进在于：在微调时同时使用成功与失败的真实轨迹，并与原始 DROID 数据联合训练，以正则化系数 λ 控制两者比例。这一设计防止模型对合成成功场景过度乐观（over-optimism），保证世界模型能如实模拟任务失败动态，为后续策略优化提供可靠的负样本信号。The world model starts from the pretrained Ctrl-World, an action-conditioned video generation model trained on the DROID dataset. The key improvement is that fine-tuning uses both successful and failed real trajectories, co-trained with the original DROID data, with a regularization coefficient λ controlling the ratio between the two. This design prevents the model from becoming over-optimistic about synthetically successful scenarios, ensures that the world model faithfully simulates task failure dynamics, and provides a reliable negative-sample signal for the subsequent policy optimization.
策略优化（Policy Optimization via Filtered Synthetic Data）Policy Optimization via Filtered Synthetic Data
VLA 策略（π₀.₅）在接地世界模型中进行 closed-loop 推演：当前观测输入策略，策略输出动作，世界模型生成下一帧观测，依此循环最长 20 秒。推演结束后，Qwen3-VL 奖励模型以成功率阈值对轨迹进行二值过滤。过滤通过的轨迹以加权 flow-matching 损失更新策略参数，成功轨迹获得更高权重。作者指出，这等价于一种正则化强化学习，但完全使用监督学习实现，避免了策略梯度方法的不稳定性。The VLA policy (π₀.₅) performs closed-loop rollouts inside the grounded world model: the current observation is fed to the policy, the policy outputs an action, the world model generates the next observation frame, and the loop repeats for up to 20 seconds. After the rollout, a Qwen3-VL reward model applies a binary filter to the trajectories using a success-rate threshold. Trajectories that pass the filter update the policy parameters with a weighted flow-matching loss, where successful trajectories receive a higher weight. The authors note that this is equivalent to a form of regularized reinforcement learning, yet is implemented entirely with supervised learning, avoiding the instability of policy-gradient methods.
世界模型接地效果对比。从左到右：预训练 Ctrl-World（未接地）、仅用专家数据微调、加入在线回滚数据后微调。接地后，世界模型对接触密集型交互（如擦除痕迹、翻书翘起）的渲染精度显著提升，PSNR 从 16.32 提升至 21.77，FVD 从 225.13 降至 64.12。Comparison of world model grounding. From left to right: pretrained Ctrl-World (ungrounded), fine-tuned on expert data only, and fine-tuned with online rollout data added. After grounding, the rendering fidelity of the world model on contact-rich interactions (such as erasing marks or a book page lifting up) improves markedly: PSNR rises from 16.32 to 21.77 and FVD drops from 225.13 to 64.12.
03实验Experiments
实验在 DROID 平台（Franka Panda + Robotiq 双指爪）上进行，评估五类接触密集型任务：Stacking（叠放）、Wiping（擦除）、Open Book（翻书）、Scooping（勺取）、Drawing（绘图）。每类任务各进行 50 次评估。基线方法包括：Filtered BC（仅在真实成功轨迹上做监督微调）和 DSRL（在潜在噪声空间做扩散策略优化）。Experiments are conducted on the DROID platform (Franka Panda with a Robotiq two-finger gripper), evaluating five contact-rich task categories: Stacking, Wiping, Open Book, Scooping and Drawing. Each task category is evaluated over 50 trials. The baselines include Filtered BC (supervised fine-tuning on real successful trajectories only) and DSRL (diffusion policy optimization in the latent noise space).
世界模型精度（Action Replay 评估）World Model Fidelity (Action Replay Evaluation)
指标Metric
预训练（无接地）Pretrained (no grounding)
+专家数据微调+ expert-data fine-tuning
+在线回滚（完整接地）+ online rollouts (full grounding)
PSNR ↑
16.32
19.87
21.77
SSIM ↑
0.634
0.748
0.784
LPIPS ↓
0.347
0.189
0.136
FID ↓
41.03
12.76
9.58
FVD ↓
225.13
99.98
64.12
交互事件混淆矩阵（50 条 clip）：TP=26，FN=4，TN=19，FP=1，显示世界模型对接触事件的识别具有高准确性。Confusion matrix for interaction events (50 clips): TP=26, FN=4, TN=19, FP=1, showing that the world model recognizes contact events with high accuracy.
各方法迭代提升曲线。VLAW 在两次迭代后成功率（0.868）显著高于 Filtered BC（0.752）和 DSRL（0.500）。第一迭代 VLAW 与 Filtered BC Iter 2 相当，但 VLAW 第二迭代借助更高质量的合成数据进一步拉开差距，尤其在 Drawing（难任务）上从 0.22 提升至 0.78。Iterative improvement curves for each method. After two iterations, the success rate of VLAW (0.868) is markedly higher than that of Filtered BC (0.752) and DSRL (0.500). At the first iteration VLAW is comparable to Filtered BC Iter 2, but at the second iteration VLAW widens the gap further thanks to higher-quality synthetic data, especially on Drawing (the hard task), which rises from 0.22 to 0.78.
消融实验（Ablation Studies）Ablation Studies
Drawing 任务消融实验。将合成轨迹数量从 500 条减至 250 条，或去掉真实数据联合训练，均导致性能下降，验证了合成数据规模与真实数据正则化两者的必要性。Ablation studies on the Drawing task. Reducing the number of synthetic trajectories from 500 to 250, or removing co-training with real data, both degrade performance, confirming that the scale of synthetic data and the regularization from real data are each necessary.
消融结果表明：① 合成数据量越大，策略性能越好；② 在世界模型微调阶段去掉原始 DROID 数据会导致 catastrophic forgetting，使合成轨迹质量下降；③ 策略在线回滚数据对世界模型接地至关重要，仅用专家演示数据无法达到相同精度。The ablations show that: (i) the more synthetic data, the better the policy performs; (ii) removing the original DROID data during world model fine-tuning causes catastrophic forgetting and degrades the quality of the synthetic trajectories; (iii) online policy rollout data is essential for grounding the world model, and expert demonstration data alone cannot reach the same fidelity.
04局限性Limitations
Note:以下局限性部分为作者在论文结论部分明确陈述（标注 stated），部分为从方法设计中推断（标注 inferred）。Some of the limitations below are explicitly stated by the authors in the conclusion of the paper (marked stated), while others are inferred from the method design (marked inferred).
评估任务类别有限（stated）Limited range of evaluated task categories (stated)
作者明确指出："Although the learned world model achieves high fidelity on the downstream tasks from which online data are collected, our current evaluation is limited to five task categories." 目前仅在 DROID 平台五类任务上验证，对更大规模、更多样化任务集的泛化性尚未得到系统评估。The authors state explicitly: "Although the learned world model achieves high fidelity on the downstream tasks from which online data are collected, our current evaluation is limited to five task categories." So far the method has only been validated on five task categories on the DROID platform, and its generalization to larger, more diverse task sets has not been systematically evaluated.
世界模型分布偏移问题（inferred）Distribution shift in the world model (inferred)
世界模型在目标任务数据上接地，若 VLA 策略在推演时产生分布外动作，世界模型可能生成失真帧，进而影响合成轨迹质量。论文中对跨任务泛化的世界模型稳定性讨论有限。The world model is grounded on target-task data, so if the VLA policy produces out-of-distribution actions during rollouts, the world model may generate distorted frames and thereby degrade the quality of the synthetic trajectories. The paper offers limited discussion of world model stability under cross-task generalization.
真实回滚仍有一定成本（inferred）Real rollouts still carry a cost (inferred)
虽然每次迭代仅需 50 条真实回滚，但接触密集型任务仍需人工监督和场景重置。与完全无需真实交互的纯仿真方法相比，VLAW 在部署便利性上仍存在一定门槛。Although only 50 real rollouts are needed per iteration, contact-rich tasks still require human supervision and scene resets. Compared with purely simulation-based methods that need no real interaction at all, VLAW still presents a certain barrier to convenient deployment.
奖励模型过滤的可靠性（inferred）Reliability of reward-model filtering (inferred)
使用 Qwen3-VL 作为奖励模型对合成轨迹做二值过滤，VLM 对复杂接触状态（如勺取成功与否）的判断可能存在误判，错误过滤可能引入噪声训练数据。Qwen3-VL is used as the reward model to binary-filter the synthetic trajectories, but a VLM may misjudge complex contact states (such as whether a scooping attempt succeeded), and erroneous filtering can introduce noisy training data.
