用统一框架整合视频生成、动作预测与语言理解的具身智能世界模型An embodied-AI world model that unifies video generation, action prediction and language understanding in a single framework
Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, Jun Zhu · 清华大学等Tsinghua University et al.
Motus 提出了一个统一的 Latent Action 世界模型框架，通过 Mixture-of-Transformers（MoT）架构将视频生成专家、动作预测专家和视觉语言理解专家融合为一体，借助 optical flow 作为跨具身的通用运动先验，在无标注视频上大规模预训练，并在多机器人平台上实现了显著的性能提升。Motus proposes a unified Latent Action world model framework that fuses a video generation expert, an action prediction expert and a vision-language understanding expert into a single model through a Mixture-of-Transformers (MoT) architecture. Using optical flow as a cross-embodiment universal motion prior, it is pretrained at scale on unlabeled video and achieves substantial performance gains on multiple robot platforms.
世界模型world modelVLAlatent actionoptical flowMixture-of-Transformersvideo generation机器人操作robot manipulation具身智能embodied AI
01动机Motivation
具身 AI 领域长期存在"能力碎片化"问题：VLA（视觉-语言-动作）模型、世界模型（WM）、逆动力学模型（IDM）、视频生成（VGM）、视频-动作联合预测五类能力分属不同模型，难以相互增益。此外，绝大多数视频数据缺乏动作标注，而机器人平台之间动作空间各异，导致大规模跨具身预训练十分困难。Embodied AI has long suffered from fragmented capabilities: VLA (vision-language-action) models, world models (WM), inverse dynamics models (IDM), video generation (VGM) and video-action joint prediction are five capabilities that reside in separate models and cannot reinforce one another. In addition, the vast majority of video data lacks action annotations, while action spaces differ from one robot platform to another, which makes large-scale cross-embodiment pretraining very difficult.
"We propose Motus, a unified world model framework that integrates five key embodied AI capabilities—VLA, WM, IDM, VGM, and video-action joint prediction—into a single model via a Mixture-of-Transformers (MoT) architecture."
图 1：Motus 整体框架。三条专家支路（视频生成专家、动作专家、视觉语言理解专家）通过 Tri-model Joint Attention 共享运动信息。输入为图像序列与语言指令，输出同时覆盖视频帧预测与机器人动作序列（at…at+k），并通过 latent action zt 连接两者。Figure 1: Overall Motus framework.Three expert branches (video generation expert, action expert, vision-language understanding expert) share motion information through Tri-model Joint Attention. The inputs are an image sequence and a language instruction; the outputs cover both video frame prediction and the robot action sequence (at…at+k), with the latent action zt linking the two.
88.66%RoboTwin 2.0 仿真平均成功率（clean）RoboTwin 2.0 simulation average success rate (clean)
+15%超越 X-VLA（仿真）over X-VLA (simulation)
+45%超越 π₀.₅（仿真）over π₀.₅ (simulation)
+11~48%真实机器人平台提升幅度gain range on real robot platforms
现有方法的核心痛点有两个：（1）如何在单一架构中统合多模态生成能力，同时不牺牲各专项功能；（2）如何在动作标注稀缺、具身差异显著的情况下利用海量异构视频数据。Motus 通过 optical flow 作为"通用运动代理"来绕过动作标注缺失的问题，用深度卷积变分自编码器（DC-AE）将光流压缩为 14 维的 latent action 向量，恰好匹配主流机器人控制空间维度，从而实现跨具身迁移。Existing methods have two core pain points: (1) how to unify multimodal generative capabilities within a single architecture without sacrificing any individual function; (2) how to exploit massive heterogeneous video data when action annotations are scarce and embodiment gaps are large. Motus uses optical flow as a "universal motion proxy" to bypass the absence of action labels: a deep convolutional variational autoencoder (DC-AE) compresses the flow into a 14-dimensional latent action vector, which happens to match the control-space dimensionality of mainstream robots and thus enables cross-embodiment transfer.
02方法Method
Motus 的核心由三部分组成：Mixture-of-Transformers 多专家架构、基于 optical flow 的 Latent Action VAE，以及 Action-Dense Video-Sparse 预测策略，配合六层具身数据金字塔完成三阶段训练。Motus consists of three core parts: a Mixture-of-Transformers multi-expert architecture, a Latent Action VAE built on optical flow, and an Action-Dense Video-Sparse prediction strategy, together with a six-layer embodiment data pyramid used for three-stage training.
Motus 集成三个专家模块：视频生成专家基于 Wan 2.2 5B（参数量 5.00B），视觉语言理解专家采用 Qwen3-VL-2B（2.13B），动作专家为自定义 Transformer 块（641.5M），理解专家（Understanding Expert）占 253.5M，总参数量约 8B。三者通过 Tri-model Joint Attention 机制共享 token，在同一前向传播中联合建模视频帧与动作序列。Motus integrates three expert modules: the video generation expert is based on Wan 2.2 5B (5.00B parameters), the vision-language understanding expert adopts Qwen3-VL-2B (2.13B), the action expert is a custom Transformer block (641.5M), and the Understanding Expert accounts for 253.5M, giving a total of about 8B parameters. The three share tokens through the Tri-model Joint Attention mechanism, jointly modeling video frames and action sequences within the same forward pass.
图 2：Action-Dense Video-Sparse Prediction。动作序列以较高频率采样，视频帧以较低频率采样，避免视频 token 过多导致动作预测过拟合，同时维持时序对齐。Figure 2: Action-Dense Video-Sparse Prediction.Action sequences are sampled at a higher frequency and video frames at a lower frequency, which prevents an excess of video tokens from causing the action prediction to overfit while preserving temporal alignment.
传统 VLA 依赖真实动作标注，而绝大多数网络视频无此标注。Motus 用 optical flow 作为运动代理：用 DC-AE（深度卷积变分自编码器）将相邻帧间光流编码为 14 维 latent action 向量 zt，该维度恰好与主流机器人（如 Aloha-2）的控制空间对齐。训练时 90% 为无标注重建损失，10% 为有标注监督，使模型在海量未标注视频上高效预训练。Conventional VLA relies on ground-truth action annotations, which the vast majority of web videos lack. Motus uses optical flow as a motion proxy: DC-AE (a deep convolutional variational autoencoder) encodes the flow between adjacent frames into a 14-dimensional latent action vector zt, a dimensionality that aligns exactly with the control space of mainstream robots such as Aloha-2. During training, 90% is an unlabeled reconstruction loss and 10% is labeled supervision, letting the model pretrain efficiently on massive unlabeled video.
图 3：Latent Action VAE 架构（DC-AE）。将连续帧对的 optical flow 编码为 14 维 latent vector，并通过轻量解码器重建。有标注样本中 latent action 直接与真实动作对齐，实现有监督微调。Figure 3: Latent Action VAE architecture (DC-AE).The optical flow of consecutive frame pairs is encoded into a 14-dimensional latent vector and reconstructed by a lightweight decoder. On annotated samples the latent action is aligned directly with the ground-truth action, enabling supervised fine-tuning.
六层具身数据金字塔与三阶段训练Six-layer embodiment data pyramid and three-stage training
训练数据按质量和规模分为六层金字塔：从底层海量网络视频（最低质量、最高数量），逐层递进至顶层目标机器人示范数据（最高质量、最少数量）。三阶段训练流程：The training data is organized by quality and scale into a six-layer pyramid: from massive web video at the bottom (lowest quality, largest quantity) up to target-robot demonstration data at the top (highest quality, smallest quantity). The three-stage training pipeline:
Stage 1（VGM 适配）：在多机器人和人类视频上微调视频生成专家，约 ~8000 GPU 小时。Stage 1 (VGM adaptation): fine-tune the video generation expert on multi-robot and human video, about ~8000 GPU hours.
Stage 2（全模型预训练）：引入 latent action，联合训练所有专家，约 ~10000 GPU 小时。Stage 2 (full-model pretraining): introduce latent action and jointly train all experts, about ~10000 GPU hours.
Stage 3（目标机器人微调）：在目标平台示范数据上端到端精调，约 ~400 GPU 小时。Stage 3 (target-robot fine-tuning): end-to-end fine-tuning on demonstration data from the target platform, about ~400 GPU hours.
图 4：六层具身数据金字塔。从网络规模视频（底层）到目标机器人示范（顶层），数量递减、质量递增，覆盖人类行为、多机器人操作和目标平台数据。Figure 4: The six-layer embodiment data pyramid.From web-scale video (bottom) to target-robot demonstrations (top), quantity decreases while quality increases, covering human behavior, multi-robot manipulation and target-platform data.
03实验Experiments
实验覆盖仿真基准（RoboTwin 2.0、LIBERO-Long、VLABench）和两个真实机器人平台（AC-One 和 Agilex-Aloha-2），基线包括 X-VLA 和 π₀.₅（Pi0.5）。The experiments cover simulation benchmarks (RoboTwin 2.0, LIBERO-Long, VLABench) and two real robot platforms (AC-One and Agilex-Aloha-2), with X-VLA and π₀.₅ (Pi0.5) as baselines.
Motus 在 RoboTwin 2.0 的 50+ 任务上平均成功率 88.66%（clean）/ 87.02%（randomized），分别比 X-VLA 提升 +15%，比 π₀.₅ 提升 +45%。On the 50+ tasks of RoboTwin 2.0, Motus reaches an average success rate of 88.66% (clean) / 87.02% (randomized), improving over X-VLA by +15% and over π₀.₅ by +45% respectively.
其他基准
基准Benchmark
π₀.₅
X-VLA
Motus（ours）Motus (ours)
LIBERO-Long
—
97.6（SoTA）97.6 (SoTA)
97.6
VLABench In-Dist.
0.43
—
0.48
VLABench Cross-Cat.
0.22
—
0.25
真实机器人实验
平台Platform
π₀.₅ 平均π₀.₅ average
Motus 平均Motus average
提升Gain
AC-One
14.79%
63.22%
+48%
Agilex-Aloha-2
48.60%
59.30%
+11%
典型任务成功率：Place Cube into Plate（AC-One）100%、Brew Coffee（AC-One）62%、Get Water from Dispenser（Agilex）96%。Representative task success rates: Place Cube into Plate (AC-One) 100%, Brew Coffee (AC-One) 62%, Get Water from Dispenser (Agilex) 96%.
逆动力学模型（IDM）性能Inverse dynamics model (IDM) performance
Latent Action VAE 在动作预测上明显优于基线：Motus action MSE 0.014，ResNet18+MLP 0.044，DINOv2+MLP 0.122。The Latent Action VAE clearly outperforms the baselines on action prediction: Motus action MSE 0.014, ResNet18+MLP 0.044, DINOv2+MLP 0.122.
视频生成质量（世界模型模式）Video generation quality (world model mode)
平台Platform
FID ↓
FVD ↓
SSIM ↑
Agilex-Aloha-2
9.4571
49.2848
0.88618
AC-One
12.9609
73.1325
0.84605
图 5：真实世界任务定义与可视化。涵盖 fold towel、brew coffee、grind beans、pour water、keyboard interaction、bread placement 等多样操作任务，在两个机器人平台（AC-One 和 Agilex-Aloha-2）上测试。Figure 5: Real-world task definitions and visualization.Covering diverse manipulation tasks such as fold towel, brew coffee, grind beans, pour water, keyboard interaction and bread placement, tested on two robot platforms (AC-One and Agilex-Aloha-2).
消融实验Ablation study
图 6：消融实验（RoboTwin 2.0 randomized setting）。对比完整 Motus、仅 Stage 1 预训练、无预训练三种变体，验证多阶段训练和 latent action 预训练的贡献。Figure 6: Ablation study (RoboTwin 2.0 randomized setting).Comparing three variants — full Motus, Stage 1 pretraining only, and no pretraining — to verify the contribution of multi-stage training and latent action pretraining.
消融结果表明，Stage 1（VGM 适配）和完整的三阶段预训练均对最终性能有显著贡献，移除预训练后性能大幅下降，验证了 latent action 共享运动先验的有效性。The ablation results show that Stage 1 (VGM adaptation) and the complete three-stage pretraining both contribute significantly to final performance; removing pretraining causes a large drop, confirming the effectiveness of the shared motion prior carried by latent action.
04局限性Limitations
Note:以下局限性部分源自论文结论与讨论章节的明确陈述（stated），部分为从设计推断（inferred）。Some of the limitations below come from explicit statements in the conclusion and discussion sections of the paper (stated); others are inferred from the design (inferred).
高昂的训练计算成本（stated）High training compute cost (stated)
三阶段训练总计约 ~18400 GPU 小时（Stage 1 ~8000 + Stage 2 ~10000 + Stage 3 ~400），对大多数研究团队而言门槛极高，限制了开放复现与迭代速度。The three training stages total about ~18400 GPU hours (Stage 1 ~8000 + Stage 2 ~10000 + Stage 3 ~400), an extremely high barrier for most research teams that limits open reproduction and iteration speed.
通用运动先验仍处于探索阶段（stated）Universal motion priors are still exploratory (stated)
论文指出"universal motion priors"是未来工作方向，当前 optical flow 压缩为 14 维 latent action 的做法依赖于目标机器人控制空间恰好与该维度匹配，对控制维度差异较大的新型具身形态泛化能力有待验证。The paper notes that "universal motion priors" is a direction for future work; the current practice of compressing optical flow into a 14-dimensional latent action relies on the target robot control space happening to match that dimensionality, and generalization to new embodiments whose control dimensionality differs substantially remains to be verified.
互联网规模视频预训练尚未充分利用（stated）Internet-scale video pretraining is not yet fully exploited (stated)
论文提到将"internet-scale video pretraining"列为未来工作，说明当前数据规模和多样性仍存在天花板，距离真正通用的具身智能尚有距离。The paper lists "internet-scale video pretraining" as future work, indicating that the current data scale and diversity still have a ceiling and that truly general embodied intelligence remains some distance away.
跨具身泛化依赖动作空间对齐（inferred）Cross-embodiment generalization depends on action-space alignment (inferred)
Latent action 维度（14 维）为针对特定机器人平台设计的，若目标平台动作空间维度差异显著，则需重新设计 DC-AE 或额外适配，通用性受限。The latent action dimensionality (14) is designed for specific robot platforms; if the action space of a target platform differs markedly in dimensionality, DC-AE must be redesigned or extra adaptation is needed, so generality is limited.
