DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

01 动机 MotivationMotivation

机器人操作任务面临两大核心瓶颈：现有世界模型训练数据极度匮乏，而视频中动作标注又极为稀缺。如何从海量无标注的人类日常视频中高效提取交互知识，是构建通用机器人世界模型的关键难题。Robot manipulation faces two core bottlenecks: training data for existing world models is extremely scarce, and action labels in video are scarcer still. How to efficiently extract interaction knowledge from massive unlabeled everyday human video is the key difficulty in building a generalist robot world model.

"Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels."

DreamDojo overview — 图 1：DreamDojo 通过连续潜在动作作为统一标签，从大规模人类数据集中获取全面的物理知识，并支持遥操作、策略评估和基于模型的规划等下游应用。Figure 1: Using continuous latent actions as a unified label, DreamDojo acquires comprehensive physical knowledge from large-scale human datasets and supports downstream applications such as teleoperation, policy evaluation and model-based planning.

44k小时 egocentric 人类视频（史上最大世界模型预训练数据集）hours of egocentric human video (the largest world-model pretraining dataset to date)

1,179k轨迹数量trajectories

≥6,015unique 技能种类unique skill types

10.81 FPS蒸馏后实时推理速度（较 teacher 提升近 4×）real-time inference speed after distillation (nearly 4× over the teacher)

与已有机器人世界模型数据相比，DreamDojo 数据集在规模上具有压倒性优势：视频时长长 15×，技能种类多 96×，场景数量多 2,000×。Compared with existing robot world-model data, the DreamDojo dataset holds an overwhelming advantage in scale: 15× longer video duration, 96× more skill types and 2,000× more scenes.

图 2：数据集分布分析——场景频次、每段视频的子任务数量、技能分布及语言标注可视化（技能动词与物体名称）。DreamDojo 的数据混合方案涵盖日常场景中最广泛的交互类型。Figure 2: Dataset distribution analysis — scene frequency, number of sub-tasks per video, skill distribution and visualization of language annotations (skill verbs and object names). The DreamDojo data mixture covers the widest range of interaction types in everyday scenes.

02 方法 MethodMethod

DreamDojo 以 WAN2.2 视频生成模型为骨干，在其上引入三个关键设计：连续潜在动作（解决无标注问题）、相对动作变换 + 分块注入（提升条件精度），以及时序一致性损失（增强帧间连贯性）。后处理阶段通过两阶段蒸馏管线实现实时推理。DreamDojo builds on the WAN2.2 video generation model as its backbone and adds three key designs: continuous latent actions (solving the missing-label problem), relative action transformation plus chunked injection (improving conditioning precision), and a temporal consistency loss (strengthening frame-to-frame coherence). A two-stage distillation pipeline then delivers real-time inference.

1. 连续潜在动作（Continuous Latent Actions）1. Continuous Latent Actions

核心挑战：大量以自我为中心的人类视频没有动作标注。DreamDojo 采用基于 VAE 的信息瓶颈设计，从相邻帧对中提取紧凑的潜在动作向量，作为代理标签用于条件视频生成。其训练目标同时最大化下一帧预测的似然并最小化潜在动作的 KL 散度：The core challenge: most egocentric human video carries no action labels. DreamDojo adopts a VAE-based information bottleneck that extracts a compact latent action vector from adjacent frame pairs, serving as a proxy label for conditional video generation. Its training objective jointly maximizes the likelihood of next-frame prediction and minimizes the KL divergence of the latent action:

ℒ^pred = 𝔼[log p_θ(f^t+1|â, f^t)] − β · D_KL(q_φ(â|f^t:t+1) ‖ p(â))

该设计还通过跨具身（cross-embodiment）相似帧对检索验证：不同机器人（或人手）执行相似动作时，其潜在动作向量在空间上高度对齐，表明 latent action 捕捉了真正的运动语义，具备具身无关的迁移能力。The design is further validated by cross-embodiment retrieval of similar frame pairs: when different robots (or human hands) perform similar motions, their latent action vectors align closely in space, showing that the latent action captures genuine motion semantics and transfers in an embodiment-agnostic way.

Latent action model architecture — 图 3：潜在动作模型采用信息瓶颈设计，产生连续潜在向量。右侧展示跨具身相似帧对检索结果——不同具身执行相似动作时，对应的 latent action 彼此对齐，验证了其语义一致性。Figure 3: The latent action model uses an information bottleneck design and produces continuous latent vectors. Right: cross-embodiment retrieval of similar frame pairs — when different embodiments perform similar motions, the corresponding latent actions align with one another, confirming their semantic consistency.

2. 相对动作变换（Relative Action Transformation）与分块注入（Chunked Action Injection）2. Relative Action Transformation and Chunked Action Injection

直接将动作轨迹全局注入会导致分布宽广、建模困难。DreamDojo 引入两项架构改进：Injecting the action trajectory globally yields a broad distribution that is hard to model. DreamDojo introduces two architectural refinements:

相对动作变换：将动作向量重参数化为相对于每段 latent frame 起始点的偏移量，使动作分布更集中，降低建模复杂度。Relative action transformation: the action vector is re-parameterized as an offset relative to the start of each latent frame segment, concentrating the action distribution and lowering modeling complexity.
分块动作注入（Chunked Action Injection）：与 WAN2.2 tokenizer 的时序压缩比对齐，将动作以 4 帧为单位分块注入，并严格遵循因果顺序，避免未来动作信息泄漏。Chunked action injection: aligned with the temporal compression ratio of the WAN2.2 tokenizer, actions are injected in chunks of 4 frames while strictly respecting causal order, preventing leakage of future action information.

消融实验证明，单独加入分块注入可将 Counterfactual Eval 上的 PSNR 从 19.482 提升至 20.783（+1.3 dB）。Ablations show that chunked injection alone lifts PSNR on Counterfactual Eval from 19.482 to 20.783 (+1.3 dB).

3. 时序一致性损失（Temporal Consistency Loss）与蒸馏管线（Distillation Pipeline）3. Temporal Consistency Loss and Distillation Pipeline

在标准 flow-matching 目标之外，DreamDojo 额外引入时序一致性损失以匹配帧间过渡：Beyond the standard flow-matching objective, DreamDojo adds a temporal consistency loss that matches frame-to-frame transitions:

ℒ_temporal = 𝔼[∑ ‖(zⁱ⁺¹ − zⁱ) − (vⁱ⁺¹ − vⁱ)‖²]

蒸馏管线分两阶段（warmup + distribution matching），将 teacher 模型（35 步，2.72 FPS）压缩为 student 模型（4 步，10.81 FPS），实现约 4× 加速，并通过更长的 context 窗口（1 → 12 帧）大幅提升长程一致性。The distillation pipeline runs in two stages (warmup + distribution matching), compressing the teacher model (35 steps, 2.72 FPS) into a student model (4 steps, 10.81 FPS) for roughly 4× speedup, while a longer context window (1 → 12 frames) markedly improves long-horizon consistency.

03 实验 ExperimentsExperiments

实验在多个具有挑战性的 OOD（out-of-distribution）基准上评估，包括 In-lab Eval、Counterfactual Eval 和 GR-1 Long Eval，以及三类下游应用：策略评估、基于模型的规划和实时遥操作。Evaluation spans several challenging out-of-distribution benchmarks — In-lab Eval, Counterfactual Eval and GR-1 Long Eval — together with three downstream applications: policy evaluation, model-based planning and real-time teleoperation.

架构消融（Table 5 · Counterfactual Eval）Architecture ablation (Table 5 · Counterfactual Eval)

设计组合Design combination	PSNR↑	SSIM↑	LPIPS↓
Baseline（无任何改进）Baseline (no refinements)	19.448	0.768	0.211
+ Relative actions	19.482	0.772	0.212
+ Chunked injection	20.783	0.790	0.193
+ Temporal loss（完整模型）+ Temporal loss (full model)	20.980	0.796	0.189

蒸馏效果（Table 6 · GR-1 Long Eval）Distillation results (Table 6 · GR-1 Long Eval)

方法Method	PSNR↑	SSIM↑	LPIPS↓	FPS↑	预测长度Prediction length	Context 长度Context length
Teacher（35 步）Teacher (35 steps)	14.086	0.442	0.412	2.72	12 帧12 frames	1 帧1 frame
Student（4 步）Student (4 steps)	13.146	0.379	0.485	10.81	4 帧4 frames	12 帧12 frames

Student 在速度和 context 长度上大幅领先 teacher，代价是 PSNR 小幅下降（14.086 → 13.146），在长程生成任务中表现出更好的一致性。The student leads the teacher by a wide margin in speed and context length at the cost of a small PSNR drop (14.086 → 13.146), and shows better consistency on long-horizon generation.

Qualitative ablation results — 图 9：消融对比——应用全部技术（相对动作 + 分块注入 + 时序损失）后，模型在物体建模和动作跟随上均达到最佳效果，定性质量显著优于各消融变体。Figure 9: Ablation comparison — with all techniques applied (relative actions + chunked injection + temporal loss), the model achieves the best object modeling and action following, and is qualitatively far better than every ablated variant.

下游应用Downstream applications

策略评估（Policy Evaluation）Policy Evaluation

Pearson 相关系数：r = 0.995Pearson correlation: r = 0.995
Mean Maximum Rank Violation：MMRV = 0.003Mean Maximum Rank Violation: MMRV = 0.003
DreamDojo 模拟成功率与真实成功率高度线性一致，可替代真实环境进行策略排序DreamDojo simulated success rates track real success rates almost linearly, so it can replace the real environment for ranking policies

基于模型的规划（Model-Based Planning）Model-Based Planning

使用 5 个 checkpoint 组成 ensembleAn ensemble built from 5 checkpoints
高方差策略组：成功率提升 17%High-variance policy group: 17% success-rate gain
跨策略提案：~2× 优于 uniform samplingCross-policy proposals: ~2× better than uniform sampling
支持 PICO VR 控制器输入的实时遥操作（G1 机器人）Real-time teleoperation driven by PICO VR controller input (G1 robot)

04 局限性 LimitationsLimitations

Note: 以下局限性均为作者在论文中明确陈述（stated by the authors）。All limitations below are explicitly stated by the authors in the paper.

非常见动作仿真不足Uncommon actions are poorly simulated

"While DreamDojo demonstrates significant improvements over the baseline, it is by no means perfect when simulating uncommon actions, such as slapping and fast waving."——对于不常见或快速运动，模型生成质量明显下降，反映了训练数据分布对罕见动作的覆盖不足。 — For uncommon or fast motions, generation quality degrades noticeably, reflecting insufficient coverage of rare actions in the training data distribution.

策略评估中成功率偏高（失败建模不准确）Success rates run high in policy evaluation (failures modeled inaccurately)

"When conducting policy evaluation, the absolute success rates in DreamDojo are often higher than their real counterparts, indicating a limitation in accurately generating nuanced failures."——世界模型倾向于高估策略成功率，对细微失败模式的建模能力有限，影响其作为绝对成功率指标的可靠性。 — The world model tends to overestimate policy success rates and models nuanced failure modes only weakly, which undermines its reliability as an absolute success-rate metric.

不支持多视角仿真，缺乏后训练知识保留研究No multi-view simulation; post-training knowledge retention unstudied

"Our model does not naturally support multi-view simulation, which is crucial for state-of-the-art policies. Moreover, how to retain the pretrained knowledge as much as possible has not been studied in depth."——单视角输出限制了对依赖多摄像头的先进策略的支持；后训练（post-training）阶段如何保留预训练通用知识亦有待系统研究。未来方向包括更宽泛的动作分布覆盖（如 policy rollouts）以及进一步的推理速度工程优化。 — Single-view output limits support for state-of-the-art policies that rely on multiple cameras; how to retain pretrained general knowledge during post-training also awaits systematic study. Future directions include broader coverage of the action distribution (e.g. policy rollouts) and further engineering optimization of inference speed.