通过语言条件视频生成统一具身世界建模Unifying Embodied World Modeling through Language-Conditioned Video Generation
Jie Zhang*, Xiaoyue Chen* 等 · Qwen Team (Alibaba) · 2026-06-16Jie Zhang*, Xiaoyue Chen* et al. · Qwen Team (Alibaba) · 2026-06-16
Qwen-RobotWorld 以自然语言为统一动作接口，通过一个双流 MMDiT 扩散模型，将机器人操作、自动驾驶、室内导航和人到机器人迁移统一建模为语言条件视频生成任务。模型在 EWMBench 总分第一（4.60），在 DreamGen Bench 总分第一（4.952），并在 WorldModelBench 超越所有开源模型（8.99）。Qwen-RobotWorld adopts natural language as a unified action interface and, through a single double-stream MMDiT diffusion model, casts robot manipulation, autonomous driving, indoor navigation and human-to-robot transfer as one language-conditioned video generation task. The model ranks first on EWMBench overall (4.60), first on DreamGen Bench overall (4.952), and surpasses all open-source models on WorldModelBench (8.99).
embodied world modellanguage-conditioned video generationDouble-stream MMDiT具身智能embodied intelligencehuman-to-robot transferEWK dataset机器人操作robot manipulation多视角一致性生成multi-view consistent generation自动驾驶autonomous drivingprogressive curriculum training
01动机 · MotivationMotivation
具身智能要求智能体在物理环境中感知、推理并行动——覆盖桌面机器人操作、城市自动驾驶与室内导航等场景。世界模型通过从观测数据中学习环境动力学，充当可扩展的虚拟训练平台。然而，现有方法存在根本矛盾：通用视频生成模型缺乏物理接触约束，而专用具身模型又依赖 joint angles / waypoints 等机器人特有表示，难以跨平台泛化。Embodied intelligence requires an agent to perceive, reason and act in physical environments — covering tabletop robot manipulation, urban autonomous driving, indoor navigation and similar scenarios. World models learn environment dynamics from observational data and thereby serve as scalable virtual training platforms. Existing approaches, however, suffer from a fundamental tension: general-purpose video generation models lack physical contact constraints, whereas specialized embodied models rely on robot-specific representations such as joint angles / waypoints and generalize poorly across platforms.
"natural language is the most general and accessible action representation … a single instruction such as 'pick up the red cup and place it on the shelf' implicitly encodes the complete action sequence, goal state, and physical constraints, without requiring robot-specific control interfaces."
论文首页 teaser。左侧展示 EWK 数据集的 Multi-Embodiment / Multi-Task / Multi-Scenario / Multi-View 四维多样性；中间展示 Double-stream MMDiT 架构示意，语言动作（Action）与视觉观测（Observation）分别经 Qwen2.5-VL 和 VAE 编码后融合，预测未来帧（Prediction）；右侧雷达图显示 Qwen-RobotWorld 在各评测维度的综合领先。Teaser figure from the front page of the paper. Left: the four-axis diversity of the EWK dataset — Multi-Embodiment / Multi-Task / Multi-Scenario / Multi-View. Middle: a sketch of the Double-stream MMDiT architecture, in which the language action (Action) and the visual observation (Observation) are encoded by Qwen2.5-VL and the VAE respectively and then fused to predict future frames (Prediction). Right: a radar chart showing the overall lead of Qwen-RobotWorld across the evaluation dimensions.
#1EWMBench 总分（4.60，较次优 LVP +0.55）EWMBench overall score (4.60, +0.55 over the runner-up LVP)
Figure 1：EWK（Embodied World Knowledge）训练语料总览。顶部为通用世界数据（14 平台视频 + 20 数据集图像）；中部结构化具身数据沿四个互补轴组织——Multi-Embodiment（人手、多样机械臂、移动智能体）、Multi-Task（短程原子技能、长程复合规划、特定技能）、Multi-Scenario（真实场景 + 仿真场景）、Multi-View（主视角、手腕视角、同步多视角）；底部归纳语义、几何、物理对齐与因果关系等训练目标。语料共 8.6M 视频-文本对，逾 200M 观测帧。Figure 1: Overview of the EWK (Embodied World Knowledge) training corpus.The top part is general world data (video from 14 platforms + images from 20 datasets); the middle part organizes structured embodied data along four complementary axes — Multi-Embodiment (human hands, diverse robot arms, mobile agents), Multi-Task (short-horizon atomic skills, long-horizon compositional planning, skill-specific data), Multi-Scenario (real scenes + simulated scenes), Multi-View (ego view, wrist view, synchronized multi-view); the bottom part summarizes training objectives such as semantic, geometric and physical alignment and causal relations. The corpus contains 8.6M video-text pairs and over 200M observation frames in total.
02方法 · MethodMethod
Qwen-RobotWorld 通过三大创新实现统一具身世界建模：(a) Double-Stream MMDiT with MLLM Action Encoding——将冻结的 Qwen2.5-VL 作为动作编码器，与 VAE 视觉流在每个 Transformer 层通过双向 cross-attention 融合；(b) Embodied World Knowledge (EWK)——8.6M 视频-文本语料，覆盖 20+ 机器人形态与 500+ 动作类别；(c) General + Expert Progressive Curriculum——先学通用视觉先验，再进行具身专项精调。Qwen-RobotWorld achieves unified embodied world modeling through three innovations: (a) Double-Stream MMDiT with MLLM Action Encoding — a frozen Qwen2.5-VL acts as the action encoder and is fused with the VAE visual stream at every Transformer layer via bidirectional cross-attention; (b) Embodied World Knowledge (EWK) — an 8.6M video-text corpus covering 20+ robot embodiments and 500+ action categories; (c) General + Expert Progressive Curriculum — general visual priors are learned first, followed by embodiment-specific fine-tuning.
Figure 3：60 层 Double-stream MMDiT 视频生成架构。理解流（understanding stream）接收冻结 Qwen2.5-VL 对语言动作 at 提取的 last-layer hidden states h（经可训练 connector 投影）；生成流（generation stream）接收 VAE 编码的含噪状态 latent（当前帧 observation + 目标帧 prediction），通过 Patchify 展平后送入 60 个 double-stream MMDiT block，每层两流通过 joint attention 双向交互；最终由 UnPatchify 解码为预测视频帧。总参数：MLLM 7B + VAE 127M（encoder 54M + decoder 73M）+ MMDiT 20B；支持最长 48,360 个视频 token。Figure 3: The 60-layer Double-stream MMDiT video generation architecture.The understanding stream receives the last-layer hidden states h extracted by the frozen Qwen2.5-VL from the language action at (projected through a trainable connector); the generation stream receives the noisy state latent encoded by the VAE (current-frame observation + target-frame prediction), which is flattened by Patchify and fed into 60 double-stream MMDiT blocks, where the two streams interact bidirectionally through joint attention at every layer; finally UnPatchify decodes the predicted video frames. Total parameters: MLLM 7B + VAE 127M (encoder 54M + decoder 73M) + MMDiT 20B; up to 48,360 video tokens are supported.
MLLM 作为动作编码器MLLM as Action Encoder
使用冻结的 Qwen2.5-VL 而非轻量级 T5 / CLIP 编码动作指令，具备两大优势：(1) 其深度语言理解能力可将复杂、组合式指令精确解析为控制信号；(2) 其内化的世界知识（如"机器臂关节有固定长度"）隐式约束物理可行转移空间，结合 T2I 协同训练，可在无需显式几何提示的情况下防止跨帧物体形变。Encoding action instructions with a frozen Qwen2.5-VL rather than a lightweight T5 / CLIP offers two advantages: (1) its deep language understanding can parse complex, compositional instructions precisely into control signals; (2) its internalized world knowledge (e.g. that "robot arm joints have fixed lengths") implicitly constrains the space of physically feasible transitions and, combined with T2I co-training, prevents object deformation across frames without any explicit geometric hint.
人到机器人迁移被建模为视频编辑任务：输入序列由三段构成——(1) 场景条件（原始人手操作视频，遮去手部）、(2) 机器人参考（MuJoCo 渲染的目标机器人轨迹）、(3) 待生成段（去噪为最终机器人执行视频）。三段共享同一 VAE-MMDiT pipeline，用 3D RoPE 时序索引区分。联合 attention 使生成段同时关注场景外观、机器人运动与语言动作语义，无需修改架构即可完成跨形态合成。Human-to-robot transfer is formulated as a video editing task: the input sequence consists of three segments — (1) the scene condition (the original human-hand manipulation video with the hands masked out), (2) the robot reference (the target robot trajectory rendered in MuJoCo), and (3) the segment to be generated (denoised into the final robot execution video). The three segments share one VAE-MMDiT pipeline and are distinguished by 3D RoPE temporal indices. Joint attention lets the generated segment attend simultaneously to scene appearance, robot motion and language action semantics, so cross-embodiment synthesis requires no architectural modification.
核心挑战是表示异构性：机械臂用 joint angles，驾驶用转向角，导航用方向向量——每类需要独立模型。EWK 通过 action-language mapping 框架将 20+ 机器人形态、500+ 动作类别统一映射至自然语言接口。标注采用五层分级框架：(1) Task Goal Layer，(2) Action Detail Layer（必须显式声明视角），(3) Physical Feedback Layer，(4) 50-100 词综合描述，(5) 15-30 词简洁指令。训练时以 50/50 比例采样，使模型同时具备详细轨迹执行与高层命令理解能力。The core challenge is representational heterogeneity: robot arms use joint angles, driving uses steering angles, navigation uses direction vectors — each category requiring its own model. Through an action-language mapping framework, EWK maps 20+ robot embodiments and 500+ action categories onto a unified natural-language interface. Annotation adopts a five-level hierarchical framework: (1) Task Goal Layer, (2) Action Detail Layer (the viewpoint must be stated explicitly), (3) Physical Feedback Layer, (4) a 50-100 word comprehensive description, (5) a 15-30 word concise instruction. Training samples them at a 50/50 ratio, so that the model acquires both detailed trajectory execution and high-level command understanding.
General + Expert 进阶课程训练General + Expert Progressive Curriculum Training
预训练阶段在 T2I / T2V / TI2V 三任务上联合训练，建立通用视觉先验（对象运动、光照变化、碰撞动力学）。SFT 精调阶段采用四阶段数据混合调度：单视角操作 → 多视角扩展 → 多视角拼接生成 → 复杂任务与跨领域数据。操作数据在约 90% 的采样权重下主导，确保物理接地深度；多视角拼接和导航/驾驶数据各约 5%，提供广度。Asymmetric 3D RoPE（时序维度 16 维、空间各 56 维）保证推理时跨同步相机视角的几何一致性。The pre-training stage trains jointly on the three tasks T2I / T2V / TI2V to establish general visual priors (object motion, illumination change, collision dynamics). The SFT stage adopts a four-stage data mixture schedule: single-view manipulation → multi-view extension → multi-view stitched generation → complex tasks and cross-domain data. Manipulation data dominates with roughly 90% of the sampling weight, ensuring depth of physical grounding; multi-view stitching and navigation / driving data take about 5% each, providing breadth. Asymmetric 3D RoPE (16 dimensions for the temporal axis and 56 for each spatial axis) guarantees geometric consistency across synchronized camera views at inference time.
03实验 · ExperimentsExperiments
在四个基准上综合评测，对标两类 baseline：(1) 通用视频生成模型（Sora2、Veo3、Wan2.6、Kling、LTX-2）；(2) 具身世界模型（Cosmos、GigaWorld、LVP、Vidar、Wow）。A comprehensive evaluation on four benchmarks against two families of baselines: (1) general video generation models (Sora2, Veo3, Wan2.6, Kling, LTX-2); (2) embodied world models (Cosmos, GigaWorld, LVP, Vidar, Wow).
EWMBench 在场景一致性（SceneC）、运动正确性（HSD / Dyn / nDTW）、语义对齐（Diversity / BLEU / CLIP / Logics）三维评测，共 21 样本、7 类任务。EWMBench evaluates along three dimensions — scene consistency (SceneC), motion correctness (HSD / Dyn / nDTW) and semantic alignment (Diversity / BLEU / CLIP / Logics) — over 21 samples and 7 task categories.
模型Model
SceneC
HSD ↑
Dyn
nDTW
CLIP
Logics
Overall
LVP（次优具身）LVP (runner-up embodied model)
0.8795
0.4248
0.0433
0.6226
0.8995
0.9524
4.05
GigaWorld
0.8707
0.3050
0.0849
0.2783
0.8873
0.9000
3.56
Wan2.6（最优通用）Wan2.6 (best general-purpose model)
0.6712
0.2034
0.0900
0.1715
0.8743
1.0000
3.22
Qwen-RobotWorld（Ours）Qwen-RobotWorld (Ours)
0.9142
0.5660
0.3429
0.6708
0.8834
1.0000
4.60
HSD（0.566）较次优 LVP（0.425）提升 +33%；场景一致性 SceneC（0.914）与逻辑约束满足 Logics（1.00）均居首位。HSD (0.566) improves by +33% over the runner-up LVP (0.425); scene consistency SceneC (0.914) and logical constraint satisfaction Logics (1.00) both rank first.
评测 GR1 机器人三个泛化子集（Env / Object / Behavior）的 Physics Alignment（PA）与 Instruction Following（IF）。Physics Alignment (PA) and Instruction Following (IF) are evaluated on three generalization subsets (Env / Object / Behavior) of the GR1 robot.
模型Model
GR1-Env PA
GR1-Env IF
GR1-Object PA
GR1-Object IF
GR1-Behavior PA
GR1-Behavior IF
Total
LVP
0.810
0.772
0.745
0.829
0.713
0.889
4.758
GigaWorld
0.621
0.933
0.500
0.852
0.426
0.884
4.216
Wow
0.793
0.826
0.755
0.849
0.809
0.696
4.728
Qwen-RobotWorld（Ours）Qwen-RobotWorld (Ours)
0.828
0.793
0.840
0.878
0.781
0.832
4.952
GR1-Object IF 达 0.878（第 1），体现出色的物体级组合泛化能力。GR1-Behavior IF（0.832）略低于 LVP（0.889）与 GigaWorld（0.884），长程行为泛化仍有改进空间。GR1-Object IF reaches 0.878 (rank 1), reflecting excellent object-level compositional generalization. GR1-Behavior IF (0.832) is slightly below LVP (0.889) and GigaWorld (0.884), so long-horizon behavior generalization still leaves room for improvement.
WorldModelBench 与 PBenchWorldModelBench and PBench
WorldModelBench 评测指令跟随（0-3 分）、常识和物理遵从（Newton 定律、质量守恒、流体动力学、重力——5 类物理违反）。Qwen-RobotWorld 总分 8.99，超越所有开源模型，物理遵从四项全满（1.00），指令跟随 2.33/3.0，常识分因输出分辨率较低略有差距。PBench Overall 0.804（开源最优），Domain 理解 0.857（全体第 3，超越大多数闭源模型），Motion Smoothness 0.990（开源第 2）。WorldModelBench evaluates instruction following (0-3 points), common sense, and physical adherence (Newton's laws, conservation of mass, fluid dynamics, gravity — 5 categories of physical violation). Qwen-RobotWorld scores 8.99 overall, surpassing all open-source models, with all four physical-adherence items perfect (1.00) and instruction following at 2.33/3.0; its common-sense score falls slightly short because of the lower output resolution. PBench Overall is 0.804 (best open-source), Domain understanding 0.857 (3rd overall, ahead of most closed-source models), and Motion Smoothness 0.990 (2nd among open-source models).
Table 2 & 3（原文）。EWMBench（上）和 DreamGen Bench（下）定量对比，粗体为列最优。Qwen-RobotWorld 在 EWMBench 整体排名第一（4.60）、DreamGen 整体排名第一（4.952）。Table 2 & 3 (from the paper).Quantitative comparison on EWMBench (top) and DreamGen Bench (bottom); bold marks the best entry in each column. Qwen-RobotWorld ranks first overall on EWMBench (4.60) and first overall on DreamGen (4.952).
定性分析：细粒度语言接地与跨形态泛化Qualitative Analysis: Fine-Grained Language Grounding and Cross-Embodiment Generalization
Figure 5：细粒度语言接地定性结果。(a) 对比指令跟随：每对列共享相同初始帧，仅高亮关键词不同——目标物体（红色草莓 vs. 黄色土豆）、目标位置（wooden tray vs. white paper）、动作类型（hand it to the person vs. put it in the penholder）——模型每次均精确响应判别性关键词。(b) 复杂指令跟随：多步序列任务（将两个魔方依次叠放成两层；将红色和黄色彩椒从左到右依次摆放）中，模型自动分解并生成时序连贯的执行序列，无需显式子任务提示。Figure 5: Qualitative results on fine-grained language grounding. (a) Contrastive instruction following:each pair of columns shares the same initial frame and differs only in the highlighted keyword — target object (red strawberry vs. yellow potato), target location (wooden tray vs. white paper), action type (hand it to the person vs. put it in the penholder) — and the model responds precisely to the discriminative keyword every time. (b) Complex instruction following:in multi-step sequential tasks (stacking two cubes one after another into two layers; arranging the red and yellow bell peppers in order from left to right), the model decomposes the task automatically and generates a temporally coherent execution sequence without explicit sub-task prompts.
Ablations 与零样本鲁棒性Ablations and Zero-Shot Robustness
在 RoboTwin-IF 基准零样本评测中，尽管 Qwen-RobotWorld 训练时仅混入少量开源 RoboTwin 数据，仍展现出强劲的零样本性能与稳定的多视角一致性。与 LVP 和 Cosmos2.5-14B 对比，LVP 更多出现任务未完成执行，Cosmos2.5-14B 在复杂指令下动作-结果对齐更弱，而 Qwen-RobotWorld 在保持正确物体/动作对应和目标达成方面更一致。跨领域泛化（人到机器人迁移、自动驾驶、室内导航）进一步验证语言条件状态迁移函数在跨形态与跨场景的通用性。In zero-shot evaluation on the RoboTwin-IF benchmark, Qwen-RobotWorld still exhibits strong zero-shot performance and stable multi-view consistency even though only a small amount of open-source RoboTwin data was mixed into its training. Compared with LVP and Cosmos2.5-14B, LVP more often leaves tasks unfinished and Cosmos2.5-14B aligns action and outcome more weakly under complex instructions, whereas Qwen-RobotWorld is more consistent in maintaining the correct object / action correspondence and in reaching the goal. Cross-domain generalization (human-to-robot transfer, autonomous driving, indoor navigation) further validates the generality of the language-conditioned state transition function across embodiments and scenarios.
04局限性 · LimitationsLimitations
Note：论文正文无专门的 Limitations 节。以下各点：★ 标注为论文明确陈述，☆ 标注为从设计或结果推断（inferred from design/results）。Note:The body of the paper has no dedicated Limitations section. Among the points below, ★ marks statements made explicitly in the paper and ☆ marks points inferred from the design or the results (inferred from design/results).
★ 常识评分受输出分辨率制约★ Common-sense score constrained by the output resolution
论文在 WorldModelBench 讨论中明确指出："the common-sense gap [is] attributable to our lower output resolution"——Qwen-RobotWorld 以低于通用视频生成器的输出分辨率运行，导致 VBench 像素级质量分（Aesthetic、Imaging quality）偏低，尽管此分辨率已足够下游机器人控制任务使用。In its WorldModelBench discussion the paper states explicitly: "the common-sense gap [is] attributable to our lower output resolution" — Qwen-RobotWorld runs at a lower output resolution than general-purpose video generators, which drags down its pixel-level VBench quality scores (Aesthetic, Imaging quality), even though this resolution is already sufficient for downstream robot control tasks.
★ 长程行为泛化仍有差距★ A remaining gap in long-horizon behavior generalization
在 DreamGen Bench 的 GR1-Behavior IF 子集，Qwen-RobotWorld（0.832）略逊于 LVP（0.889）与 GigaWorld（0.884）。论文明确将其列为"a direction for further improvement"，表明长时序行为泛化尚未达到最优水平。On the GR1-Behavior IF subset of DreamGen Bench, Qwen-RobotWorld (0.832) trails LVP (0.889) and GigaWorld (0.884). The paper explicitly lists this as "a direction for further improvement", indicating that long-horizon behavior generalization has not yet reached the best level.
☆ 依赖高质量语言标注（推断）☆ Dependence on high-quality language annotation (inferred)
EWK 采用五层分级标注框架，由 Qwen2.5-VL 自动生成后经人工审核和迭代提示精调，整个标注流水线对模型质量高度依赖。自动标注中偶发的视角描述不一致或物理反馈缺失，会通过质量过滤回流重标注，但仍可能引入系统性偏差（inferred from design of the annotation pipeline）。EWK adopts a five-level hierarchical annotation framework in which annotations are generated automatically by Qwen2.5-VL and then refined through human review and iterative prompt tuning, so the whole annotation pipeline depends heavily on model quality. Occasional viewpoint-description inconsistencies or missing physical feedback in the automatic annotations are routed back for re-annotation by quality filtering, yet systematic bias may still be introduced (inferred from design of the annotation pipeline).
EWK 中操作数据约 5.9M（约 68.6%），自动驾驶约 200K（2.3%），室内导航 6K+，Human-to-Robot 数据通过自动 MANO 流水线生成。导航和驾驶域的相对稀疏可能导致这些场景的生成质量不如操作场景（inferred from data statistics）。Within EWK, manipulation data accounts for about 5.9M samples (roughly 68.6%), autonomous driving about 200K (2.3%), and indoor navigation 6K+, while Human-to-Robot data is produced by an automatic MANO pipeline. The relative sparsity of the navigation and driving domains may make generation quality in those scenarios inferior to that in manipulation scenarios (inferred from data statistics).
