Interactive World Simulator for Robot Policy Training and Evaluation
用隐空间一致性模型（consistency model）构建可交互的动作条件世界模型，服务策略训练数据生成与可复现评估An interactive action-conditioned world model built with a latent-space consistency model, serving policy-training data generation and reproducible evaluation
Yixuan Wang, Rhythm Syed, Fangyu Wu, Mengchao Zhang, Aykut Onol, Jose Barreiros, Hooshang Nayyeri, Tony Dear, Huan Zhang, Yunzhu Li ｜ Columbia University · Toyota Research Institute · Amazon · UIUC
论文提出 Interactive World Simulator：从中等规模的机器人交互数据集中训练一个动作条件视频预测模型（world model）。核心是用一致性模型同时做图像解码和隐空间动态预测，使模型在单张 RTX 4090 上以 15 FPS 稳定运行超过 10 分钟，并可作为可扩展的示教数据生成器和可复现的策略评估环境使用。The paper introduces Interactive World Simulator: an action-conditioned video prediction model (world model) trained from a moderate-sized robot interaction dataset. Its core idea is to use consistency models for both image decoding and latent dynamics prediction, letting the model run stably at 15 FPS for over 10 minutes on a single RTX 4090, and serve both as a scalable demonstration-data generator and as a reproducible policy evaluation environment.
动作条件视频预测模型（action-conditioned video prediction models，常被称为 world models）在机器人规划、控制、策略引导与评估上展现出潜力，但现有方法要么推理很慢，要么在长时程 rollout 中难以保持物理一致的交互。论文希望用一个既快又稳的世界模型，同时充当可扩展的示教数据生成器和可复现的策略评估环境。Action-conditioned video prediction models (commonly called world models) have shown potential for robot planning, control, policy guidance and evaluation, but existing approaches are either slow at inference or struggle to maintain physically consistent interactions over long-horizon rollouts. The paper aims for a world model that is both fast and stable, and that serves simultaneously as a scalable demonstration-data generator and a reproducible policy evaluation environment.
“Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation.”
Fig. 1：给定真实机器人交互数据集（左），训练出能捕捉复杂物理动态、支持长时程稳定交互的动作条件视频模型（中）——在单张 RTX 4090 上实现 10 分钟以上、15 FPS 的稳定预测，且在 realism (PSNR) 上明显优于 Cosmos、UVA、Dreamer4、DINO-WM。由此得到的 Interactive World Simulator 支持可扩展的策略训练数据生成，以及与真实世界高度相关的策略评估（右）。Fig. 1: Given a real-robot interaction dataset (left), an action-conditioned video model is trained that captures complex physical dynamics and supports stable long-horizon interaction (middle) — achieving stable prediction at 15 FPS for over 10 minutes on a single RTX 4090, and clearly outperforming Cosmos, UVA, Dreamer4 and DINO-WM in realism (PSNR). The resulting Interactive World Simulator supports scalable policy-training data generation as well as policy evaluation that correlates strongly with the real world (right).
25.82 dBPSNR（长时程预测，全部指标最优）PSNR (long-horizon prediction; best on every metric)
15 FPS单张 RTX 4090 上交互速度Interactive speed on a single RTX 4090
10+ min稳定长时程 rolloutStable long-horizon rollout
r=0.85~0.99仿真内策略评分 vs 真实世界评分相关性Correlation between in-simulator policy scores and real-world scores
02 Method
模型训练分两阶段：先把高维图像压缩到紧凑的 2D 隐空间，再在隐空间中做未来预测，从而实现高效的长时程推理。两个阶段都用一致性模型（consistency model），因为它既能高效表示，又能建模机器人交互中多模态的未来分布。Training proceeds in two stages: first compress high-dimensional images into a compact 2D latent space, then predict the future within that latent space, achieving efficient long-horizon inference. Both stages use consistency models, because they provide efficient representation while also modelling the multimodal future distribution of robot interaction.
Fig. 2：Stage 1 训练一个自编码器，用 CNN encoder Eφ 把 RGB 观测映射到紧凑的 2D 隐表示，并用一致性模型解码器 Dθ 高保真重建；Stage 2 冻结自编码器，训练动作条件一致性模型 Fψ，在隐空间中用 next-frame supervision 学习去噪，网络由 3D 卷积块 + FiLM 调制 + 时空注意力组成。推理时 Fψ 自回归预测未来隐变量，再由 Dθ 解码为长时程视频。Fig. 2: Stage 1 trains an autoencoder that maps RGB observations into a compact 2D latent representation with a CNN encoder Eφ and reconstructs them at high fidelity with a consistency-model decoder Dθ; Stage 2 freezes the autoencoder and trains an action-conditioned consistency model Fψ that learns to denoise in latent space under next-frame supervision, with a network built from 3D convolutional blocks + FiLM modulation + spatio-temporal attention. At inference, Fψ autoregressively predicts future latents, which Dθ then decodes into long-horizon video.
受 Consistency Trajectory Model (CTM) 启发，编码器 Eφ 把图像 o 编码为隐表示 z，解码器 Dθ 被训练为把两个不同噪声尺度 σt > σs 下加噪的隐变量互相映射（一致性轨迹训练），以获得高效、高保真的图像重建，避免多步扩散的开销。Inspired by the Consistency Trajectory Model (CTM), the encoder Eφ encodes an image o into a latent z, and the decoder Dθ is trained to map between latents noised at two different noise scales σt > σs (consistency-trajectory training), so as to obtain efficient, high-fidelity image reconstruction while avoiding the cost of multi-step diffusion.
Stage 2：动作条件隐空间动态模型Stage 2: Action-conditioned latent dynamics model
冻结自编码器后，动态模型 Fψ 在隐空间中学习给定动作序列和历史上下文条件下预测下一帧隐变量。推理阶段一个关键技巧是向历史上下文注入小噪声：由于自回归预测的误差会累积并作为后续步骤的上下文，模型对噪声上下文的鲁棒性对稳定长时程预测至关重要。上下文窗口自回归滑动，丢弃过旧的隐变量以避免计算量随时程增长。With the autoencoder frozen, the dynamics model Fψ learns in latent space to predict the next-frame latent given an action sequence and historical context. A key trick at inference is to inject small noise into the historical context: since errors of autoregressive prediction accumulate and serve as context for later steps, robustness to noisy context is critical for stable long-horizon prediction. The context window slides autoregressively, discarding overly old latents so that computation does not grow with the horizon.
03 Experiments
在 MuJoCo 一个仿真任务与 ALOHA 双臂机器人上的六个真实任务（Mug Grasping、Rope Routing、Rope Collecting、T Pushing、Box Packing、Pile Sweeping，涉及刚体、可变形物体、物体堆和多物体交互）上评估。视频预测基线包括 Cosmos、UVA、Dreamer4、DINO-WM，rollout horizon 为 192 步（19.2 秒）。Evaluation is carried out on one simulated MuJoCo task and six real-world tasks on the ALOHA bimanual robot (Mug Grasping, Rope Routing, Rope Collecting, T Pushing, Box Packing, Pile Sweeping, involving rigid bodies, deformable objects, object piles and multi-object interaction). Video prediction baselines include Cosmos, UVA, Dreamer4 and DINO-WM, with a rollout horizon of 192 steps (19.2 seconds).
Metric（聚合全部任务）Metric (aggregated over all tasks)
最优基线Best baseline
Ours
MSE ↓
0.012 ± 0.009（Dreamer4）
0.005 ± 0.005
LPIPS ↓
0.163 ± 0.052（Dreamer4）
0.051 ± 0.019
PSNR ↑
20.81 ± 2.21（Dreamer4）
25.82 ± 2.72
SSIM ↑
0.693 ± 0.045（Dreamer4）
0.831 ± 0.019
FVD ↓
799.34 ± 220.07（Cosmos）
243.20 ± 103.58
（完整六项指标 MSE/LPIPS/FID/PSNR/SSIM/UIQI/FVD 见论文 Table I；上表仅摘取具代表性的几项，Interactive World Simulator 在全部七项指标上均优于全部四个基线。）(The complete six metrics MSE/LPIPS/FID/PSNR/SSIM/UIQI/FVD are given in Table I of the paper; the table above extracts only a few representative ones. Interactive World Simulator outperforms all four baselines on all seven metrics.)
Fig. 3：Pile Sweeping 任务的长时程中间帧对比（左）显示基线方法出现机器人位姿漂移、动态不准确、长时程伪影或细节丢失，而本文方法保持物理合理的交互；右侧展示其余五个任务在第 191 步（终止帧）上的对比，本文方法始终维持更高的视觉保真度和交互一致性。Fig. 3: The comparison of long-horizon intermediate frames on the Pile Sweeping task (left) shows that baseline methods exhibit robot pose drift, inaccurate dynamics, long-horizon artifacts or loss of detail, whereas our method preserves physically plausible interaction; the right side shows the comparison on the remaining five tasks at step 191 (the terminal frame), where our method consistently maintains higher visual fidelity and interaction consistency.
数据生成与仿真—真实相关性Data generation and simulator–real correlation
用世界模拟器生成的数据训练 DP、ACT、π₀、π0.5 等模仿学习策略，结果显示：“policies trained on 100% world simulator data perform comparably to those trained on an equivalent volume of real-robot expert data.” 例如 ACT 用模拟器数据达到 76.2% 成功率，用真实数据为 73.6%；DP 分别为 87.9%（100% 模拟器数据）和 90.3%（100% 真实数据）。数据规模从 5 到 100 条示教递增时，模拟器数据训练出的策略与 MuJoCo 真实数据训练的策略呈现相似的 scaling 趋势。Imitation-learning policies such as DP, ACT, π₀ and π0.5 were trained on data generated by the world simulator, and the results show: “policies trained on 100% world simulator data perform comparably to those trained on an equivalent volume of real-robot expert data.” For instance, ACT reaches a 76.2% success rate with simulator data versus 73.6% with real data; DP reaches 87.9% (100% simulator data) and 90.3% (100% real data) respectively. As the data scale grows from 5 to 100 demonstrations, policies trained on simulator data exhibit a scaling trend similar to those trained on MuJoCo real data.
Fig. 7：对 T Pushing、Rope Routing、Mug Grasping、Pile Sweeping 四个任务，把同一策略在世界模拟器和真实世界中的评分对比，观测到强正相关（Pearson r 分别为 0.8553 / 0.8455 / 0.8869 / 0.9908）。除 T Pushing 外，其余任务的拟合线存在轻微正偏（仿真评分略高于真实评分），但作者指出这一偏差不影响相对排序的可靠性：“if one policy substantially outperforms another in the world simulator, our results suggest that it is likely to do so in the real world as well.”Fig. 7: For the four tasks T Pushing, Rope Routing, Mug Grasping and Pile Sweeping, scores of the same policy in the world simulator and in the real world are compared, and a strong positive correlation is observed (Pearson r of 0.8553 / 0.8455 / 0.8869 / 0.9908 respectively). Except for T Pushing, the fitted lines of the remaining tasks show a slight positive bias (simulator scores are slightly higher than real-world scores), but the authors note that this bias does not compromise the reliability of relative ranking: “if one policy substantially outperforms another in the world simulator, our results suggest that it is likely to do so in the real world as well.”
04 Limitations
Note:论文正文没有单独的 "Limitations" 小节；以下第 1、3 条为作者在实验/结论中明确陈述的局限，第 2 条为根据实验设置推断得到。The body of the paper has no separate "Limitations" section; items 1 and 3 below are limitations explicitly stated by the authors in the experiments/conclusion, while item 2 is inferred from the experimental setup.
仿真—真实评分存在轻微正偏（stated）Slight positive bias between simulator and real-world scores (stated)
“For tasks other than T pushing, the fitted lines exhibit a slight positive bias, indicating that policies tend to achieve slightly higher scores in the world simulator than in the real world.” 作者认为这不影响策略间的相对排序，但绝对分数并非完全一致。“For tasks other than T pushing, the fitted lines exhibit a slight positive bias, indicating that policies tend to achieve slightly higher scores in the world simulator than in the real world.” The authors argue that this does not affect the relative ranking among policies, but the absolute scores are not fully consistent.
图像分辨率与任务规模有限（inferred from design）Limited image resolution and task scale (inferred from design)
实验默认图像分辨率为 128×128（“our default image resolution is 128 × 128, which is sufficient for most of our tasks”），仅在 ALOHA 双臂机器人上评估 6 个真实任务 + 1 个 MuJoCo 仿真任务，训练数据规模也是“moderate-sized”（每个真实任务约 600 条 episode），尚未验证在更高分辨率、更大规模数据或更多样化机器人平台上的表现。The default image resolution in the experiments is 128×128 (“our default image resolution is 128 × 128, which is sufficient for most of our tasks”), and evaluation covers only 6 real-world tasks + 1 MuJoCo simulated task on the ALOHA bimanual robot, with a training data scale that is likewise “moderate-sized” (about 600 episodes per real task); performance at higher resolution, larger data scale or on more diverse robot platforms has not yet been verified.
尚未研究模型随数据/算力的 scaling 规律（stated as future work）Scaling behaviour of the model with data/compute not yet studied (stated as future work)
结论部分写道：“We plan to extend this framework to more diverse environments and increasingly complex manipulation tasks... An important direction for future work is to study how the performance of world models scales with increasing amounts of interaction data and computational resources.” 这意味着当前工作尚未系统评估更大数据/算力规模下的表现。The conclusion states: “We plan to extend this framework to more diverse environments and increasingly complex manipulation tasks... An important direction for future work is to study how the performance of world models scales with increasing amounts of interaction data and computational resources.” This implies that the current work has not systematically evaluated performance at larger data or compute scales.
