UniSim：用视频扩散模型构建可交互的真实世界通用仿真器UniSim: an interactive universal simulator of the real world built with video diffusion models
Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, Pieter Abbeel · UC Berkeley / Google DeepMind / MIT / University of Alberta
生成模型已在文本、图像和视频创作上取得突破。UniSim 迈向下一个里程碑：学习一个通用真实世界仿真器，
能够根据人类、机器人及其他智能体的动作，生成逼真的交互式观测序列。
通过统一整合多源异构数据集，UniSim 支持从高层语言指令到底层连续控制的全部动作类型，
并能以零样本方式将在仿真中训练的策略迁移至真实机器人。Generative models have achieved breakthroughs in text, image and video creation. UniSim takes the next step: learning a universal real-world simulator that generates realistic interactive observation sequences in response to the actions of humans, robots and other agents. By unifying heterogeneous multi-source datasets, UniSim supports the full range of action types, from high-level language instructions to low-level continuous control, and can transfer policies trained in simulation to real robots in a zero-shot manner.
5.6B 参数 Video U-Net5.6B-parameter Video U-Net512 TPU-v3 · 20天训练512 TPU-v3 · 20 days of training多源数据统一训练Unified training on multi-source data📄 arXiv:2310.06114🌐 Project Page
world modelvideo diffusioninteractive simulator机器人策略robot policyaction-conditioned generation真实世界仿真real-world simulationreinforcement learning视频生成video generation
01动机 MotivationMotivation
训练现实的交互式仿真器是机器人和智能体研究的核心挑战。
传统仿真器（MuJoCo、Isaac Gym 等）依赖手工设计的物理引擎，
难以捕捉真实世界的视觉多样性与复杂交互；
而已有的视频预测方法通常局限于单一场景或特定动作类型。
能否用互联网规模的多源数据训练出一个通用的、
支持多种动作接口的真实世界仿真器？Training realistic interactive simulators is a core challenge in robotics and agent research. Traditional simulators (MuJoCo, Isaac Gym and the like) rely on hand-crafted physics engines and struggle to capture the visual diversity and complex interactions of the real world, while existing video prediction methods are usually confined to a single scene or a specific action type. Can internet-scale multi-source data be used to train a universal real-world simulator that supports multiple action interfaces?
"We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling."
图1：UniSim 总览。
通过整合机器人操作数据、人类活动视频、室内导航、全景扫描和互联网图文数据等多源异构数据集，
UniSim 学习一个统一的交互式仿真器。该仿真器支持高层语言指令（"open the drawer"）
和底层连续控制（末端执行器坐标）等多种动作输入，并能生成逼真的交互式视频帧序列。Figure 1: UniSim overview. By combining heterogeneous multi-source datasets — robot manipulation data, human activity videos, indoor navigation, panoramic scans and internet image-text data — UniSim learns a unified interactive simulator. The simulator accepts diverse action inputs, from high-level language instructions ("open the drawer") to low-level continuous control (end-effector coordinates), and generates realistic interactive video frame sequences.
5.6B模型参数量Model parameters
0.34RDG（仿真策略，语言桌面任务）RDG (simulator-trained policy, Language Table task)
81%RL 训练后成功率（vs BC baseline 58%）Success rate after RL training (vs BC baseline 58%)
46.23CIDEr（仿真数据微调后，ActivityNet）CIDEr (after fine-tuning on simulated data, ActivityNet)
02方法 MethodMethod
UniSim 将真实世界仿真器形式化为一个观测预测模型：
给定历史观测帧 ht-1 和动作 at-1，
预测下一帧观测 ot。
核心创新在于通过数据编排策略，将多源异构数据统一到同一视频扩散框架下联合训练。UniSim formulates the real-world simulator as an observation prediction model: given the observation history ht-1 and the action at-1, it predicts the next observation ot. The key innovation is a dataset orchestration strategy that unifies heterogeneous multi-source data for joint training within a single video diffusion framework.
图2：训练与推理框架。
所有数据集的观测被统一表示为视频帧；动作被统一为条件信号（高层语言用 T5 embedding，
底层连续控制离散化后与语言 embedding 拼接）。
Video U-Net 在交织的时序与空间 attention 层中同时处理历史帧与噪声预测帧。
推理时以自回归方式滚动生成长时序交互视频。Figure 2: Training and inference framework. Observations from all datasets are represented uniformly as video frames; actions are unified into conditioning signals (high-level language via T5 embeddings, low-level continuous control discretized and concatenated with the language embedding). The Video U-Net processes history frames and noisy prediction frames jointly in interleaved temporal and spatial attention layers. At inference time, long-horizon interactive videos are produced by autoregressive rollout.
不同数据集的动作空间差异巨大——机器人数据有连续控制量，人类活动视频仅有文字标签，
全景扫描只有相机移动参数。UniSim 对每类数据采用专门的动作表示方式：Action spaces differ enormously across datasets: robot data carries continuous control values, human activity videos carry only text labels, and panoramic scans carry only camera motion parameters. UniSim adopts a dedicated action representation for each type of data:
仿真渲染 / 机器人操作：提取任务描述作为高层动作；连续控制量离散化后与语言 embedding 拼接Simulated rendering / robot manipulation: the task description is extracted as the high-level action; continuous control values are discretized and concatenated with the language embedding
人类活动视频（Ego4D 等）：将活动标签转换为文本动作Human activity videos (Ego4D and others): activity labels are converted into textual actions
全景扫描：构建相机平移/旋转序列作为导航动作Panoramic scans: camera translation/rotation sequences are constructed as navigation actions
互联网图文数据："treat individual images as single-frame videos and image captions as actions"Internet image-text data: "treat individual images as single-frame videos and image captions as actions"
视频扩散模型（Video Diffusion Model）Video Diffusion Model
模型为 5.6B 参数 Video U-Net，包含交织的时序与空间 attention 层及卷积层。
训练目标为 MSE 去噪损失，采用 classifier-free guidance：The model is a 5.6B-parameter Video U-Net containing interleaved temporal and spatial attention layers together with convolutional layers. The training objective is an MSE denoising loss with classifier-free guidance:
εθ = (1 + η) εθ(conditional) − η εθ(unconditional)
推理阶段通过自回归滚动（autoregressive rollout）生成长时序视频：
将已生成帧作为下一步历史条件，反复迭代，支持数十步以上的连续交互仿真。
历史帧采用最近 4 帧（而非单帧或远期帧），在消融实验中取得最优 FVD。At inference time, long-horizon videos are generated by autoregressive rollout: the already generated frames serve as the history condition for the next step, iterated repeatedly, which supports continuous interactive simulation over dozens of steps. The history condition uses the 4 most recent frames (rather than a single frame or distant frames), which attains the best FVD in the ablation study.
03实验 ExperimentsExperiments
实验从三个维度验证 UniSim 的价值：
(1) 作为视觉语言策略（VLM Policy）的训练环境；
(2) 作为强化学习（RL）训练环境；
(3) 为视频描述（Video Captioning）任务生成合成数据。
仿真质量用 FVD（Fréchet Video Distance）和 CLIP 相似度评估；
下游任务用任务特定指标评估。The experiments validate UniSim's value along three dimensions: (1) as a training environment for a vision-language policy (VLM Policy); (2) as a reinforcement learning (RL) training environment; (3) as a generator of synthetic data for video captioning. Simulation quality is evaluated with FVD (Fréchet Video Distance) and CLIP similarity; downstream tasks are evaluated with task-specific metrics.
消融：历史帧数对视频生成质量的影响（Ego4D 验证集）Ablation: effect of the number of history frames on video generation quality (Ego4D validation set)
Simulator-Hindsight（UniSim 仿真训练）Simulator-Hindsight (trained in UniSim simulation)
0.34 ± 0.13
0.34 ± 0.13
RDG（Reduction in Distance to Goal）越高越好；UniSim 仿真训练的策略在 Language Table 任务上显著优于行为克隆基线。Higher RDG (Reduction in Distance to Goal) is better; the policy trained in UniSim simulation clearly outperforms the behavioral cloning baseline on the Language Table task.
下游任务：RL Policy 训练Downstream task: RL policy training
Simulator-RL（UniSim 仿真 RL 训练）Simulator-RL (RL training in UniSim simulation)
71%
81%
通过在 UniSim 仿真环境中进行 RL 训练，策略成功率大幅超越行为克隆基线，且训练所得策略可零样本迁移至真实机器人部署。With RL training inside the UniSim simulation environment, policy success rates far exceed the behavioral cloning baseline, and the resulting policies can be transferred zero-shot to deployment on real robots.
下游任务：视频描述合成数据增强（Video Captioning）Downstream task: synthetic data augmentation for video captioning
数据来源Data source
ActivityNet CIDEr ↑
VATEX CIDEr ↑
SMIT CIDEr ↑
无微调（基线）No fine-tuning (baseline)
15.2
—
—
仿真数据微调（UniSim）Fine-tuned on simulated data (UniSim)
46.23
27.63
40.03
真实数据微调Fine-tuned on real data
54.90
—
—
使用 UniSim 生成的合成视频微调视频描述模型，ActivityNet CIDEr 从 15.2 提升至 46.23，并对未见数据集（VATEX、SMIT）展现出更好的跨域泛化能力。Fine-tuning the video captioning model on synthetic videos generated by UniSim raises ActivityNet CIDEr from 15.2 to 46.23, and yields better cross-domain generalization to unseen datasets (VATEX, SMIT).
图3：长时序交互仿真（8步序列）。
UniSim 在接受一系列语言动作指令后，能自回归地生成连贯的长时序交互视频，
保持场景一致性并正确响应"open the drawer"、"pick up the object"等动作。Figure 3: Long-horizon interactive simulation (8-step sequence). Given a series of language action instructions, UniSim autoregressively generates a coherent long-horizon interactive video, preserving scene consistency and responding correctly to actions such as "open the drawer" and "pick up the object".图4：VLM 策略长时序任务执行。
在 UniSim 仿真环境中训练的 vision-language 策略，部署到真实机器人后能完成多步操作任务，
验证了仿真到真实（sim-to-real）的零样本迁移能力。Figure 4: Long-horizon task execution by the VLM policy. The vision-language policy trained in the UniSim simulation environment completes multi-step manipulation tasks once deployed on a real robot, confirming zero-shot sim-to-real transfer.
消融：数据集组合对仿真质量的影响Ablation: effect of dataset composition on simulation quality
数据配置Data configuration
FVD ↓
CLIP ↑
仅互联网数据Internet data only
219.62
22.27
无互联网数据Without internet data
307.80
21.99
全部数据（UniSim）All data (UniSim)
211.30
22.63
消融：模型规模对仿真质量的影响Ablation: effect of model scale on simulation quality
模型规模Model scale
FVD ↓
500M 参数500M parameters
277.85
1.6B 参数1.6B parameters
224.61
5.6B 参数（UniSim）5.6B parameters (UniSim)
211.30
随模型规模增大，FVD 持续下降，但增益逐渐收窄，表明还有提升空间。FVD keeps falling as model scale grows, but the gains narrow progressively, indicating that room for improvement remains.
04局限性 LimitationsLimitations
Note:以下局限性均由论文作者在 Limitations 章节中明确陈述（stated）。All of the limitations below are explicitly stated by the authors in the paper's Limitations section.
幻觉（Hallucination）Hallucination
当给定的动作对当前场景不合理时（例如对桌面机器人发出"wash hands"指令），
仿真器会产生幻觉，生成不可能发生的视觉结果（如桌子变成水槽）。
原文：When the given action is implausible for the current scene (for example, issuing a "wash hands" instruction to a tabletop robot), the simulator hallucinates and produces visual outcomes that could not occur (such as the table turning into a sink). From the paper: "we observe hallucinations where the simulated outcomes may be unrealistic."
记忆能力有限（Limited Memory）Limited Memory
仿真器仅以最近几帧作为历史条件，无法捕捉长期物体持久性与场景状态。
例如，早期交互中移动过的物体在若干步后可能"复原"，违背物理一致性。
原文：The simulator conditions only on the most recent few frames as history and cannot capture long-term object permanence or scene state. For example, an object moved during an earlier interaction may "revert" after several steps, violating physical consistency. From the paper: "cannot capture long-term memory."
对于训练数据中未覆盖的场景（如新型机器人形态），仿真质量显著下降。
原文：For scenes not covered by the training data (such as novel robot embodiments), simulation quality degrades markedly. From the paper: "This is especially true for domains that are not represented in the training data."
仅限视觉仿真（Visual Simulation Only）Visual Simulation Only
UniSim 仅模拟视觉观测变化，无法仿真非视觉效应（力、声音、触觉反馈等）。
对于需要力控或触觉感知的任务，该仿真器不适用。
原文：UniSim simulates only changes in visual observation and cannot simulate non-visual effects (forces, sound, haptic feedback and so on). It is therefore unsuitable for tasks that require force control or tactile sensing. From the paper: "Our simulator is not suitable for environments where actions do not cause visual observation change."
