DreamZero：联合预测视频与动作，实现零样本机器人策略DreamZero: Jointly Predicting Video and Action for Zero-shot Robot Policies
Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao 等 · NVIDIA & 多机构合作Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao et al. · NVIDIA & multiple institutions
DreamZero 是一个 14B 参数的 World Action Model（WAM），基于预训练视频扩散模型 Wan2.1，联合预测未来视频帧和机器人动作序列。通过继承世界物理先验，它在从未见过的任务和环境中展现出 2× 以上的零样本泛化能力，并以 7Hz 实现实时闭环控制。DreamZero is a 14B-parameter World Action Model (WAM) built on the pre-trained video diffusion model Wan2.1, jointly predicting future video frames and robot action sequences. By inheriting world physics priors, it delivers more than 2× the zero-shot generalization of prior work on never-seen tasks and environments, and runs real-time closed-loop control at 7Hz.
World Action Model机器人操作robot manipulationzero-shot generalizationvideo predictionflow matching跨平台迁移cross-embodiment transferVLA扩散模型diffusion models
01动机Motivation
当前主流机器人策略（VLA）依赖大量重复性示范数据进行训练，对未见任务和环境的泛化能力极为有限；即便经过预训练，在多样化非重复数据上也几乎学不到任何有效行为。如何让机器人策略真正"理解"物理世界、实现开放世界零样本泛化，是本文的核心问题。Mainstream robot policies (VLAs) are trained on large amounts of repetitive demonstration data, so they generalize very poorly to unseen tasks and environments; even after pre-training, they learn almost no useful behavior from diverse, non-repetitive data. How to make a robot policy genuinely “understand” the physical world and achieve open-world zero-shot generalization is the central question of this paper.
"By jointly predicting video and action, World Action Models (WAMs) inherit world physics priors that enable 1) effective learning from diverse, non-repetitive data, 2) open-world generalization, 3) cross-embodiment learning from video-only data, and 4) few-shot adaptation to new robots."
图 1：DreamZero 概览。Figure 1: DreamZero overview.WAM 通过联合预测视频与动作，继承了世界物理先验，从而支持：(1) 从多样化非重复数据中有效学习；(2) 开放世界零样本泛化；(3) 从视频专属数据实现跨平台迁移；(4) 仅凭 30 分钟游玩数据适应新型机器人。 By jointly predicting video and action, the WAM inherits world physics priors, which enable: (1) effective learning from diverse, non-repetitive data; (2) open-world zero-shot generalization; (3) cross-embodiment transfer from video-only data; (4) adaptation to a new robot from just 30 minutes of play data.
>2×零样本泛化提升zero-shot generalization gain vs. SOTA VLA
38×推理加速inference speedup （vs. 原始 DiT）(vs. the original DiT)
7 Hz实时闭环控制real-time closed-loop control （150ms 延迟）(150ms latency)
30 min少样本适配新机器人few-shot adaptation to a new robot 仅需游玩数据play data only
02方法Method
DreamZero 以 Wan2.1-I2V-14B-480P 图像到视频扩散模型为骨干，采用 flow matching 联合去噪视频帧潜变量与动作序列，训练时施加 teacher-forcing（逐块去噪），推理时将真实观测替换预测帧以避免误差累积。额外参数仅为状态编码器、动作编码器与解码器，骨干权重几乎不动。DreamZero uses the Wan2.1-I2V-14B-480P image-to-video diffusion model as its backbone and applies flow matching to jointly denoise video-frame latents and action sequences. Training uses teacher forcing (block-wise denoising), while at inference predicted frames are replaced by real observations to avoid error accumulation. The only added parameters are the state encoder, the action encoder and the action decoder; the backbone weights are left almost untouched.
图 4：DreamZero 模型架构。Figure 4: DreamZero model architecture.三路输入——视觉上下文（VAE 编码）、语言指令（文本编码器）、本体状态（状态编码器）——送入自回归 DiT 骨干，联合预测下一视频帧与动作序列。训练时对视频和动作潜变量同时去噪；推理时将预测帧异步执行，并将真实观测反馈入 KV cache，从根本上杜绝误差累积。 Three input streams — visual context (VAE encoded), language instruction (text encoder) and proprioceptive state (state encoder) — feed an autoregressive DiT backbone that jointly predicts the next video frame and the action sequence. During training, video and action latents are denoised simultaneously; at inference, predicted frames are executed asynchronously and real observations are fed back into the KV cache, eliminating error accumulation at its root.
联合视频–动作预测（Joint Video & Action Denoising）Joint Video–Action Prediction (Joint Video & Action Denoising)
模型将生成过程分解为：The model factorizes generation as: π₀(videos, actions | observations, language, state) = π₀(videos | observations, language, state) × π₀(actions | videos, state) 视频和动作在同一个 DiT 骨干中共同去噪，两者通过注意力机制深度耦合，保证动作与视频语义强对齐。相比于"先生成视频再预测动作"的两阶段方案，端到端联合训练在实验中显示出显著优势。Video and actions are denoised together inside the same DiT backbone and are deeply coupled through attention, which keeps actions strongly aligned with video semantics. Compared with the two-stage recipe of “generate video first, then predict actions”, end-to-end joint training shows a clear advantage in the experiments.
自回归架构与 KV Cache 推理加速Autoregressive Architecture and KV-Cache Inference Speedup
DreamZero 使用DreamZero uses 自回归（autoregressive）autoregressive而非双向（bidirectional）注意力。自回归架构保留了帧的原始帧率，不需要为与语言对齐而降采样视频，避免了双向 WAM 中时序错位问题（Figure 13 对比）。更重要的是，自回归结构天然支持 KV cache：历史帧的 KV 对只需计算一次，推理时只需新帧重新计算，单步推理速度提升 3–4×。 rather than bidirectional attention. The autoregressive architecture preserves the native frame rate, so video need not be downsampled for language alignment, avoiding the temporal-misalignment issue of bidirectional WAMs (compared in Figure 13). More importantly, an autoregressive structure natively supports the KV cache: KV pairs of past frames are computed once and only new frames are recomputed at inference, giving a 3–4× speedup per inference step.
为支持单步去噪（1 NFE），DreamZero-Flash 引入To enable single-step denoising (1 NFE), DreamZero-Flash introduces 解耦噪声调度（decoupled noise schedules）decoupled noise schedules：视频侧采用 Beta(7,1) 分布将噪声集中于高噪区间，而动作侧保持均匀分布。这迫使模型学会"从高度噪声的视觉上下文预测干净动作"，使得单步推理下的任务进度从 52% 恢复至 74%。结合系统级与实现级优化（CFG 并行、DiT velocity cache、Torch Compile with CUDA Graphs、NVFP4 量化），总推理速度提升: the video side draws from a Beta(7,1) distribution that concentrates noise in the high-noise regime, while the action side keeps a uniform distribution. This forces the model to learn to predict clean actions from a heavily noised visual context, raising task progress under single-step inference from 52% back to 74%. Combined with system- and implementation-level optimizations (parallel CFG, DiT velocity cache, Torch Compile with CUDA Graphs, NVFP4 quantization), total inference speed improves by 38×，延迟降至 150ms（7Hz）。, and latency drops to 150ms (7Hz).
03实验Experiments
主要评测平台：Main evaluation platforms: AgiBot G1（22 个真实场景，约 500 小时遥操数据，7.2K 轮次，平均每轮 4.4 分钟、约 42 个子任务）和 (22 real-world scenes, about 500 hours of teleoperation data, 7.2K episodes, averaging 4.4 minutes and about 42 subtasks per episode) and DROID-Franka。基线包括从零训练的 VLA 与预训练 VLA（含 π₀、RDT 等 SOTA 方法）。主要指标：平均任务进度（task progress，%）与成功率。. Baselines include VLAs trained from scratch and pre-trained VLAs (including SOTA methods such as π₀ and RDT). Main metrics: mean task progress (task progress, %) and success rate.
Q1 & Q2：从多样化数据学习 & 零样本泛化Q1 & Q2: Learning from Diverse Data & Zero-shot Generalization
评测设置Evaluation setting
从零训练 VLAVLA from scratch
预训练 VLA（最优）Pre-trained VLA (best)
DreamZero
AgiBot G1 已见任务（task progress）AgiBot G1 seen tasks (task progress)
图 8：已见任务评测。Figure 8: Seen-task evaluation.DreamZero 在 PnP-Easy、PnP-Hard 和 Contact-Rich 三类任务上均大幅领先所有 VLA 基线。从零训练的 VLA 近乎零成功率，预训练 VLA 受益于重复示范数据中的平台特定知识，仍远低于 DreamZero。 DreamZero leads every VLA baseline by a wide margin on all three task families: PnP-Easy, PnP-Hard and Contact-Rich. VLAs trained from scratch achieve nearly zero success; pre-trained VLAs benefit from embodiment-specific knowledge in repetitive demonstration data yet remain far below DreamZero.图 9：零样本泛化至未见任务。Figure 9: Zero-shot generalization to unseen tasks.DreamZero 在训练集中完全不包含的 10 项任务（熨烫、刷漆、解鞋带等）上取得了非零的任务进度，而所有 VLA 基线几乎无法完成任何子步骤。 DreamZero attains non-zero task progress on 10 tasks that never appear in the training set (ironing, painting, untying shoelaces, etc.), whereas every VLA baseline is essentially unable to complete any subtask.
Q4：跨平台迁移（Cross-Embodiment Transfer）Q4: Cross-Embodiment Transfer
图 11：跨平台迁移结果。Figure 11: Cross-embodiment transfer results.仅用 10–20 分钟的视频专属示范数据，即可将 AgiBot G1 模型迁移至 YAM 机器人（robot-to-robot）或从人体第一视角视频迁移（human-to-robot）。 With only 10–20 minutes of video-only demonstrations, the AgiBot G1 model can be transferred to the YAM robot (robot-to-robot) or adapted from egocentric human video (human-to-robot).
50%（等价任务进度，但有帧率失真问题）50% (equivalent task progress, but frame-rate distortion)
关于 DreamZero-Flash（单步去噪）：4 步去噪时任务进度为On DreamZero-Flash (single-step denoising): with 4-step denoising task progress reaches 89%，降至 1 步后 DreamZero 仅保留 52%（≈基线 83%），而 DreamZero-Flash 通过解耦噪声调度恢复至; dropping to 1 step leaves plain DreamZero at only 52% (≈83% of the baseline), while DreamZero-Flash recovers to 74%。.
04局限性Limitations
说明：Note:以下局限性均来自论文原文明确陈述（stated）或由设计可合理推断（inferred），已分别标注。 every limitation below is either explicitly stated in the paper (stated) or reasonably inferred from its design (inferred), and labeled accordingly.
视觉记忆有限（stated）Limited visual memory (stated)
DreamZero 当前视觉上下文窗口仅约 6 秒，长程推理和多步骤规划能力受限。需要更长历史窗口才能处理复杂连续任务。DreamZero currently has a visual context window of only about 6 seconds, which limits long-horizon reasoning and multi-step planning. A longer history window is needed for complex sequential tasks.
高精度任务覆盖不足（stated）Insufficient coverage of high-precision tasks (stated)
多样化预训练数据以探索性和多样性为导向，子厘米级精度任务（如精密插针）在训练集中代表性不足，影响此类任务的成功率。The diverse pre-training data emphasizes exploration and variety, so sub-centimeter precision tasks (such as fine peg insertion) are under-represented in the training set, which hurts success rates on this class of tasks.
计算开销高于 VLA（stated）Higher compute cost than VLAs (stated)
即使经过 38× 推理加速，DreamZero 仍运行在 7Hz，而典型 VLA 可达 20Hz 以上。高精度实时控制场景仍面临延迟压力。Even after the 38× inference speedup, DreamZero still runs at 7Hz, whereas typical VLAs reach 20Hz or more. High-precision real-time control scenarios therefore still face latency pressure.
Scaling law 尚未系统研究（stated）Scaling law not yet studied systematically (stated)
论文指出尚未对 WAM 特定的 scaling law 做深入探索，更大模型或更多数据的收益曲线未知。The paper notes that WAM-specific scaling laws have not been explored in depth, so the return curve of larger models or more data remains unknown.
少样本平台适配仅测试相近形态（stated）Few-shot embodiment adaptation tested only on similar morphologies (stated)
Few-shot embodiment adaptation 目前仅验证于形态相似的机器人（AgiBot G1 ↔ YAM），对形态差异极大的平台（如四足、手型机器人）的效果尚未验证。Few-shot embodiment adaptation is so far validated only on robots of similar morphology (AgiBot G1 ↔ YAM); its effectiveness on platforms with very different morphologies (such as quadrupeds or dexterous hands) remains unverified.
