机器人学习 · NVIDIA Research · arXiv 2025Robot Learning · NVIDIA Research · arXiv 2025
DreamGen
Unlocking Generalization in Robot Learning through Video World Models
Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang 等 · NVIDIA & 合作机构Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang et al. · NVIDIA & collaborating institutions
DreamGen 提出了一套四阶段流水线，利用视频世界模型（video world models）生成"神经轨迹"（neural trajectories）——带伪动作标注的合成机器人视频——以此大幅扩充训练数据。仅凭单一 pick-and-place 任务的遥操作数据，便能训练出可在 22 种全新行为和未见过的环境中泛化的策略。DreamGen presents a four-stage pipeline that uses video world models (video world models) to generate "neural trajectories" — synthetic robot videos with pseudo action labels — and thereby greatly expand the training data. With teleoperation data from a single pick-and-place task alone, it trains a policy that generalizes to 22 entirely new behaviors and unseen environments.
arXiv 2505.12705 · 2025年5月arXiv 2505.12705 · May 2025仿真 + 三种真实机器人Simulation + three kinds of real robotsGR1 人形机器人 / Franka / SO-100GR1 humanoid robot / Franka / SO-100📄 arXiv:2505.12705🌐 项目主页Project page
video world modelneural trajectorypseudo action labelingbehavior generalizationenvironment generalizationimitation learning策略泛化policy generalization视频生成video generation机器人操作robot manipulation
01动机Motivation
机器人策略的泛化能力长期受限于数据匮乏：遥操作采集成本高昂，且单一场景的数据难以覆盖多样行为与陌生环境。如何在极少真实数据的前提下，让策略跨行为、跨环境迁移，是领域核心难题。The generalization ability of robot policies has long been constrained by data scarcity: teleoperated collection is expensive, and data from a single scene can hardly cover diverse behaviors and unfamiliar environments. How to make a policy transfer across behaviors and across environments given very little real data is a core problem of the field.
"DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories—synthetic robot data generated from video world models."
图1：DreamGen 泛化能力展示。系统利用视频世界模型生成跨越 22 种新行为和未见环境的合成轨迹，而训练所用的真实遥操作数据仅来自单一 pick-and-place 任务。左侧展示行为泛化（seen environments, novel behaviors），右侧展示环境泛化（unseen environments）。Figure 1: DreamGen's generalization ability. The system uses a video world model to generate synthetic trajectories spanning 22 new behaviors and unseen environments, while the real teleoperation data used for training comes only from a single pick-and-place task. The left side shows behavior generalization (seen environments, novel behaviors), the right side shows environment generalization (unseen environments).
10–13每任务真实轨迹数（极少数据即可显著提升）real trajectories per task (very little data already gives a marked gain)
02方法Method
DreamGen 将视频世界模型从实时规划器转变为合成数据生成器，通过四个阶段构建"神经轨迹"：视频模型微调 → 视频生成 → 伪动作标注 → 策略训练。DreamGen turns the video world model from a real-time planner into a synthetic data generator, constructing "neural trajectories" in four stages: video model fine-tuning → video generation → pseudo action labeling → policy training.
图2：DreamGen 完整流水线。 (1) 用 LoRA 将视频世界模型（如 WAN2.1、Cosmos、CogVideoX）在机器人轨迹上微调，使其捕捉机器人动态；(2) 用语言指令和初始帧引导模型在多样环境生成合成视频；(3) 用 IDM 或 LAPA 从视频帧对中提取伪动作；(4) 将视频-伪动作对（神经轨迹）与真实数据混合训练视觉运动策略。Figure 2: The complete DreamGen pipeline. (1) Fine-tune a video world model (such as WAN2.1, Cosmos, CogVideoX) on robot trajectories with LoRA so that it captures robot dynamics; (2) guide the model with language instructions and an initial frame to generate synthetic videos in diverse environments; (3) extract pseudo actions from pairs of video frames with IDM or LAPA; (4) mix the video–pseudo-action pairs (neural trajectories) with real data to train a visuomotor policy.
Stage 1 & 2：视频世界模型微调与生成Stage 1 & 2: Video World Model Fine-tuning and Generation
以 LoRA 对预训练视频扩散模型进行 embodiment-specific 微调，使模型学会特定机器人的运动动态和手臂外观。推理时，以初始帧（机器人当前状态）和语言指令（目标行为描述）为条件，生成跨越不同环境和行为的合成机器人操作视频，构成丰富的视觉观测序列。A pretrained video diffusion model is fine-tuned with LoRA in an embodiment-specific manner, so that the model learns the motion dynamics and arm appearance of a particular robot. At inference, conditioned on an initial frame (the robot's current state) and a language instruction (a description of the target behavior), it generates synthetic robot manipulation videos spanning different environments and behaviors, forming rich visual observation sequences.
扩散 transformer 以两帧图像为条件预测动作 chunk；采用滑动窗口策略逐帧推断，生成与视频对齐的完整动作序列。需要预先训练的 IDM 模型（需真实动作监督）。A diffusion transformer predicts an action chunk conditioned on two image frames; a sliding-window strategy infers frame by frame, producing a complete action sequence aligned with the video. It requires a pretrained IDM model (which needs ground-truth action supervision).
Transformer encoder-decoder 以 VQ-VAE 目标训练，捕捉帧间视觉变化（visual delta）作为潜在动作表示，无需真实机器人动作标注，可从纯视频数据学习。A transformer encoder-decoder is trained with a VQ-VAE objective to capture the visual change between frames (visual delta) as a latent action representation, requiring no ground-truth robot action labels, so it can be learned from pure video data.
Stage 4：策略训练Stage 4: Policy Training
将神经轨迹（生成视频 + 伪动作）与真实遥操作数据混合，训练视觉运动策略（visuomotor policy）。实验中每任务仅需 10–13 条真实轨迹，配合 100–300 条神经轨迹，即可取得显著提升。Neural trajectories (generated video + pseudo actions) are mixed with real teleoperation data to train a visuomotor policy. In the experiments each task needs only 10–13 real trajectories, combined with 100–300 neural trajectories, to obtain a marked improvement.
03实验Experiments
实验在仿真（RoboCasa 平台，24 个任务）和真实环境（GR1 人形机器人、Franka 机械臂、SO-100）上评估。核心指标为任务成功率（%）。Experiments are evaluated in simulation (the RoboCasa platform, 24 tasks) and in real environments (GR1 humanoid robot, Franka arm, SO-100). The core metric is task success rate (%).
个别任务中，DreamGen 在"Pour Water"、"Light Candle"、"Hit Keyboard"等行为上达到 90%–95% 成功率。On individual tasks, DreamGen reaches a 90%–95% success rate on behaviors such as "Pour Water", "Light Candle" and "Hit Keyboard".
真实世界数据增强（Figure 5）Real-World Data Augmentation (Figure 5)
在三种真实机器人平台上，以极少真实数据配合神经轨迹：On three real robot platforms, very little real data is combined with neural trajectories:
机器人平台Robot platform
任务数Tasks
Baseline
DreamGen
GR1 人形机器人GR1 humanoid robot
4
37%
46.4%
Franka 机械臂Franka arm
3
23%
37%
SO-100
2
21%
45.5%
图5：真实机器人平台上的数据增强效果。三种机器人（GR1 人形机器人、Franka 机械臂、SO-100）在各自任务上，神经轨迹数据增强均带来一致的成功率提升。每任务仅使用 10–13 条真实遥操作轨迹。Figure 5: Data augmentation results on real robot platforms. On all three robots (GR1 humanoid robot, Franka arm, SO-100), neural trajectory data augmentation brings a consistent success-rate gain on their respective tasks. Each task uses only 10–13 real teleoperation trajectories.
在 RoboCasa 24 个任务上，神经轨迹数量与策略性能呈 log-linear 正相关：240k 神经轨迹配合 7.2k 真实数据时，平均成功率达 ~57.6%；仅用 IDM 神经轨迹（无真实数据）达 20.55%（24 任务平均）。On the 24 RoboCasa tasks, the number of neural trajectories is log-linearly correlated with policy performance: with 240k neural trajectories plus 7.2k real samples the average success rate reaches ~57.6%; using IDM neural trajectories alone (no real data) reaches 20.55% (averaged over 24 tasks).
图4：RoboCasa 神经轨迹数量 vs. 策略性能。在低数据（720条）、中数据（2.4k条）、高数据（7.2k条）三种真实数据规模下，增加神经轨迹均带来 log-linear 的性能提升，验证了合成数据的可扩展性。Figure 4: Number of RoboCasa neural trajectories vs. policy performance. Under three real-data scales — low (720 trajectories), medium (2.4k) and high (7.2k) — adding neural trajectories consistently yields log-linear performance gains, confirming the scalability of synthetic data.
论文引入 DreamGen Bench 评估视频世界模型质量，包含 Instruction Following (IF) 和 Physics Alignment (PA) 两个维度（GPT 评估），并验证其与下游策略成功率的 Pearson 相关系数 >0.90。The paper introduces DreamGen Bench to assess the quality of video world models along two dimensions, Instruction Following (IF) and Physics Alignment (PA), scored by GPT, and verifies that it has a Pearson correlation coefficient of >0.90 with downstream policy success rate.
消融实验要点Key Ablation Findings
IDM 与 LAPA 均有效，LAPA 无需真实动作标注，适用范围更广。Both IDM and LAPA are effective; LAPA needs no ground-truth action labels and is therefore applicable more broadly.
混合真实数据与神经轨迹始终优于单独使用其中一种。Mixing real data with neural trajectories consistently outperforms using either one alone.
视频模型 LoRA 微调对捕捉机器人特定动态至关重要。LoRA fine-tuning of the video model is crucial for capturing robot-specific dynamics.
04局限性Limitations
Note:以下局限性均为论文原文（Limitation 章节）明确陈述，非推断。All limitations below are explicitly stated in the paper itself (Limitation section), not inferred.
"Tasks are relatively simple and cover a limited portion of the robot's full kinematic capabilities." 当前实验任务较为简单，尚未覆盖机器人全部运动学能力，对高自由度、长时程复杂任务的有效性有待验证。The current experimental tasks are relatively simple and do not yet cover the robot's full kinematic capabilities; their effectiveness on high-DoF, long-horizon complex tasks remains to be verified.
"Generating the 240k-sample RoboCasa dataset took 54 hours on 1500 NVIDIA L40 GPUs." 大规模神经轨迹生成对算力要求极高，限制了方法在资源受限场景下的可用性。Large-scale neural trajectory generation demands enormous compute, which limits the method's usability in resource-constrained settings.
初始帧需人工提供（Manual Initial Frame）Initial frames must be provided manually (Manual Initial Frame)
"Method also relies on manually providing initial frames, which introduces operational overhead." 视频生成需要手工准备初始帧，增加了部署时的操作负担，难以实现完全自动化。Video generation requires manually preparing initial frames, which adds operational burden at deployment and makes full automation hard to achieve.
自动评估器存在幻觉风险（Evaluator Hallucination）Automatic evaluator can hallucinate (Evaluator Hallucination)
"Automatic evaluator used in DreamGen Bench...can occasionally hallucinate, especially when evaluating physical realism." 基于 GPT 的视频质量评估偶尔出现幻觉，特别是在物理合理性判断上，影响基准可靠性。GPT-based video quality evaluation occasionally hallucinates, especially when judging physical plausibility, which affects the benchmark's reliability.
缺乏与相关工作的直接基准对比No direct benchmark comparison against related work
"Does not directly benchmark against" existing video-generation-for-robot-learning methods. 论文未与全部相关视频学习方法进行直接定量比较，结果的绝对优越性有待进一步验证。The paper does not directly compare quantitatively against all related video-learning methods, so the absolute superiority of the results remains to be further verified.
