Moto: Latent Motion Token as the Bridging Language for Robot Manipulation from Videos

01 动机Motivation

机器人操控学习面临两大核心挑战：其一，带动作标注的机器人演示数据稀缺且采集昂贵；其二，现有视觉-语言-动作（VLA）模型在视频预训练阶段与低层动作之间缺乏有效桥梁，预训练收益难以充分转化为操控能力。视频中蕴含丰富的运动语义，却因动作标签的硬件依赖性而被大量浪费。Robot manipulation learning faces two core challenges. First, robot demonstrations with action annotations are scarce and expensive to collect. Second, existing vision-language-action (VLA) models lack an effective bridge between the video pre-training stage and low-level actions, so the benefits of pre-training are hard to convert fully into manipulation ability. Videos contain rich motion semantics, yet much of it is wasted because action labels are hardware-dependent.

"effective robotic learning should emphasize motion-related knowledge, which is closely tied to low-level actions and is hardware-agnostic."

Moto 总览：三阶段流程 — **图 1 · Moto 三阶段流程概览。** 第一阶段训练 Latent Motion Tokenizer（基于 VQ-VAE），将连续帧对压缩为离散隐运动 token；第二阶段以自回归预测的方式在无标注视频上预训练 Moto-GPT，学习视觉运动先验；第三阶段通过 co-fine-tuning 加入 action query token，在带标注的机器人演示上同时优化运动预测与动作预测。**Figure 1 · Overview of the three-stage Moto pipeline.** The first stage trains the Latent Motion Tokenizer (based on VQ-VAE), compressing consecutive frame pairs into discrete latent motion tokens; the second stage pre-trains Moto-GPT on unlabeled videos via autoregressive prediction to learn visual motion priors; the third stage adds action query tokens through co-fine-tuning, jointly optimizing motion prediction and action prediction on annotated robot demonstrations.

61.4%SIMPLER 整体成功率（vs RT-2-X 60.7%，参数量仅 98M vs 55B）Overall SIMPLER success rate (vs RT-2-X 60.7%, with only 98M parameters vs 55B)

3.10CALVIN 平均任务完成链长度（vs GR-1 3.06）Average length of completed task chains on CALVIN (vs GR-1 3.06)

60%真实机器人平均成功率（vs 无 motion token 基线 23.33%）Average success rate on the real robot (vs 23.33% for the baseline without motion tokens)

52.5%仅用 1% 标注数据时的 SIMPLER 成功率（vs 未预训练变体 0%）SIMPLER success rate with only 1% of the labeled data (vs 0% for the variant without pre-training)

02 方法Method

Moto 的核心思想是将"运动"从视频中抽象为一种与硬件无关的离散语言——隐运动 token——使 GPT 式的自回归预训练能够直接在该语言空间中学习运动先验，再以统一 token 序列驱动机器人动作预测。The core idea of Moto is to abstract "motion" from videos into a hardware-agnostic discrete language — the latent motion token — so that GPT-style autoregressive pre-training can learn motion priors directly in this language space, and then drive robot action prediction with a unified token sequence.

**图 2 · Moto 详细架构。** 左：Latent Motion Tokenizer 使用 M-Former 编码器将相邻帧对编码为连续运动嵌入，再经 VQ 量化映射为离散 token（词表大小 128）；ViT 解码器负责重建后续帧以监督训练。中：Moto-GPT 以语言指令和初始观测为条件，自回归预测后续隐运动 token 序列，目标为"maximizes the likelihood of the ground-truth latent motion token sequence given the language instruction." 右：Co-fine-tuning 阶段在 Moto-GPT 的 token 序列中插入 action query token，并接入轻量 action head 预测机器人关节指令，同时保留运动预测损失作为辅助监督。**Figure 2 · Detailed architecture of Moto.** Left: the Latent Motion Tokenizer uses an M-Former encoder to encode adjacent frame pairs into continuous motion embeddings, which are then mapped to discrete tokens by VQ quantization (codebook size 128); the ViT decoder reconstructs the subsequent frame to supervise training. Middle: conditioned on the language instruction and the initial observation, Moto-GPT autoregressively predicts the subsequent latent motion token sequence, with an objective that "maximizes the likelihood of the ground-truth latent motion token sequence given the language instruction." Right: in the co-fine-tuning stage, action query tokens are inserted into Moto-GPT's token sequence and a lightweight action head predicts robot joint commands, while the motion prediction loss is retained as auxiliary supervision.

阶段一：Latent Motion Tokenizer（基于 VQ-VAE）Stage one: Latent Motion Tokenizer (based on VQ-VAE)

给定时间步 t 的观测帧 o_t 与后续帧 o_t+k，M-Former 编码器将二者拼接后提取运动嵌入，经 VQ 量化得到离散隐运动 token（词表大小 128）。ViT 解码器以初始帧和运动 token 为输入重建目标帧，重建误差提供像素级监督，确保 token 携带真实运动语义而非静态外观信息。Given the observation frame o_t at time step t and the subsequent frame o_t+k, the M-Former encoder concatenates the two and extracts a motion embedding, which VQ quantization turns into discrete latent motion tokens (codebook size 128). The ViT decoder takes the initial frame and the motion tokens as input to reconstruct the target frame; the reconstruction error provides pixel-level supervision, ensuring that the tokens carry genuine motion semantics rather than static appearance information.

阶段二：Moto-GPT 自回归预训练Stage two: autoregressive pre-training of Moto-GPT

将一段机器人（或人类）视频逐帧对量化为隐运动 token 序列后，Moto-GPT 以 GPT 架构自回归建模该序列的条件分布。预训练数据无需动作标签，只需原始视频，从而可规模化利用互联网视频或人类操作视频（实验中使用了 SSV2 数据集），学习与硬件无关的运动先验。After a robot (or human) video is quantized frame pair by frame pair into a latent motion token sequence, Moto-GPT models the conditional distribution of that sequence autoregressively with a GPT architecture. The pre-training data requires no action labels, only raw videos, so internet videos or human manipulation videos can be exploited at scale (the experiments use the SSV2 dataset) to learn hardware-agnostic motion priors.

阶段三：Co-fine-tuning 迁移到机器人控制Stage three: transfer to robot control via co-fine-tuning

在带动作标注的机器人演示数据上，将 action query token 插入 Moto-GPT 的输入序列，与运动 token 共同预测。轻量 action head 将 action query token 的隐状态映射为连续关节动作。损失函数同时包含运动 token 预测的交叉熵损失与动作预测的回归损失，运动预测起辅助正则化作用，防止预训练知识在微调中被遗忘。On robot demonstration data with action annotations, action query tokens are inserted into the input sequence of Moto-GPT and predicted jointly with the motion tokens. A lightweight action head maps the hidden states of the action query tokens to continuous joint actions. The loss function contains both the cross-entropy loss for motion token prediction and the regression loss for action prediction; motion prediction acts as auxiliary regularization, preventing the pre-trained knowledge from being forgotten during fine-tuning.

**图 3 · 隐运动 token 的可解释性验证。** 将同一 token 序列施加到不同初始帧上重建运动：结果表明相同 token 序列在不同场景下均产生语义一致的运动轨迹，说明 token 有效捕捉了与外观无关的运动模式。在 Something-Something v2 视频分类任务上，使用 7 个隐运动 token 块实现 79.7% 的分类准确率，接近使用完整 7 帧的 82.8%。**Figure 3 · Interpretability check of the latent motion tokens.** Applying the same token sequence to different initial frames to reconstruct motion: the results show that an identical token sequence produces semantically consistent motion trajectories across scenes, indicating that the tokens effectively capture appearance-agnostic motion patterns. On the Something-Something v2 video classification task, using 7 latent motion token blocks reaches 79.7% classification accuracy, close to the 82.8% obtained with the full 7 frames.

03 实验Experiments

实验在三个评测场景展开：SIMPLER（模拟器中的多任务操控基准）、CALVIN（长序列多步骤操控基准）、以及配备 FANUC 机械臂的真实机器人任务（抓取香蕉、合上笔记本、分解装置）。基线包括 RT-1-X、RT-2-X、Octo-Base、OpenVLA 和 GR-1 等主流方法。Experiments are carried out in three evaluation settings: SIMPLER (a multi-task manipulation benchmark in simulation), CALVIN (a long-horizon multi-step manipulation benchmark), and real-robot tasks on a FANUC arm (picking up a banana, closing a laptop, disassembling a device). Baselines include mainstream methods such as RT-1-X, RT-2-X, Octo-Base, OpenVLA and GR-1.

SIMPLER 基准（表 2）SIMPLER benchmark (Table 2)

方法Method	Pick Coke Can	Move Near	Open/Close Drawer	Overall
RT-1-X	0.567	0.317	0.597	0.534
RT-2-X（55B 参数）RT-2-X (55B parameters)	0.787	0.779	0.250	0.607
Octo-Base	0.170	0.042	0.227	0.169
OpenVLA	0.163	0.462	0.356	0.248
OpenVLA (fine-tuned)	0.363	0.542	0.231	0.349
Moto w/o Motion Token	0.503	0.554	0.398	0.480
Moto（98M 参数）Moto (98M parameters)	0.740	0.604	0.431	0.614

CALVIN 基准（表 3）CALVIN benchmark (Table 3)

方法Method	T=1	T=2	T=3	T=4	T=5	Avg. Length
SuSIE	0.870	0.690	0.490	0.380	0.260	2.69
RoboFlamingo	0.824	0.619	0.466	0.331	0.235	2.47
MT-R3M	0.529	0.234	0.105	0.043	0.018	0.93
GR-1	0.854	0.712	0.596	0.497	0.401	3.06
Moto w/o Motion Token	0.779	0.555	0.380	0.256	0.167	2.14
Moto	0.897	0.729	0.601	0.484	0.386	3.10

**图 4 · 真实机器人实验结果。** Moto 在三类任务（抓取香蕉 / 合上笔记本 / 分解装置）下，有无运动 token 变体的成功率对比。Moto 整体平均成功率从 23.33% 提升至 60%；在视觉干扰物条件下提升 20%，在新颖物体条件下提升 30%。**Figure 4 · Real-robot experimental results.** Success rates of the variants with and without motion tokens on three task categories (picking up a banana / closing a laptop / disassembling a device). Moto raises the overall average success rate from 23.33% to 60%; the gain is 20% under visual distractors and 30% with novel objects.

消融实验（Ablations）Ablation studies (Ablations)

核心消融在于对比"有无 motion token 预训练"：在 SIMPLER 上，去除 motion token 后整体成功率从 61.4% 降至 48.0%（降幅 13.4 个百分点）；在 CALVIN 上，平均链长从 3.10 降至 2.14（降幅约 45%）。此外，用 SSV2 人类视频辅助预训练（Move Near 子任务）进一步提升成功率，说明跨身形迁移的可行性。视频分类准确率消融（表 1）验证了隐运动 token 在语义层面的有效性：7 个 token 块达 79.7%，而仅使用初始帧仅有 29.2%。The core ablation compares "with and without motion token pre-training": on SIMPLER, removing the motion tokens lowers the overall success rate from 61.4% to 48.0% (a drop of 13.4 percentage points); on CALVIN, the average chain length falls from 3.10 to 2.14 (a drop of about 45%). In addition, using SSV2 human videos as auxiliary pre-training data (the Move Near subtask) further improves the success rate, showing the feasibility of cross-embodiment transfer. The video classification accuracy ablation (Table 1) confirms the semantic effectiveness of the latent motion tokens: 7 token blocks reach 79.7%, whereas using only the initial frame gives 29.2%.

04 局限性Limitations

Note: 以下局限性均为论文作者在结论/未来工作中明确陈述（stated），非推断。All limitations below are explicitly stated by the authors in the conclusion / future work section, not inferred.

Latent Motion Tokenizer 目前主要在机器人视频上训练，尚未充分利用互联网规模的人类视频The Latent Motion Tokenizer is currently trained mainly on robot videos and does not yet fully exploit internet-scale human videos

论文指出："While we currently mainly use robot videos to train the Latent Motion Tokenizer, the learned latent motion tokens demonstrate the potential to produce consistent visual motions across varied contexts and embodiments." 将 Tokenizer 扩展到更大规模、更多样的人类操作视频是关键的未来方向，有望显著扩大可用预训练数据量。The paper notes: "While we currently mainly use robot videos to train the Latent Motion Tokenizer, the learned latent motion tokens demonstrate the potential to produce consistent visual motions across varied contexts and embodiments." Extending the Tokenizer to larger and more diverse human manipulation videos is a key future direction, and is expected to substantially enlarge the amount of usable pre-training data.

预训练规模与微调优化仍有较大提升空间Pre-training scale and fine-tuning optimization still leave considerable room for improvement

"Future work could scale up pre-training video data and optimize fine-tuning to improve model performance on downstream robot tasks further." 当前实验受限于数据量和计算资源，预训练与微调的协同优化策略尚未充分探索。"Future work could scale up pre-training video data and optimize fine-tuning to improve model performance on downstream robot tasks further." The current experiments are limited by data volume and computational resources, and joint optimization strategies for pre-training and fine-tuning remain underexplored.

应用场景目前集中于操控（manipulation），导航与运动控制尚未验证Applications currently focus on manipulation; navigation and locomotion have not been verified

"Future research could explore Moto's use in improving the robustness of reinforcement learning agents and extending its application to a wider range of robotic tasks, such as navigation and locomotion, to develop a more versatile robot action policy." Moto 作为奖励模型和环境模拟器的潜力也尚待验证。"Future research could explore Moto's use in improving the robustness of reinforcement learning agents and extending its application to a wider range of robotic tasks, such as navigation and locomotion, to develop a more versatile robot action policy." The potential of Moto as a reward model and an environment simulator also remains to be verified.

复杂人体运动的跨身形迁移能力仍然有限Cross-embodiment transfer of complex human motions remains limited

SSV2 实验初步展示了人到机器人的运动迁移潜力，但论文承认对于更复杂的人体动作（fine-grained dexterous manipulation），当前架构需要进一步改进方能实现可靠的跨身形迁移。The SSV2 experiments give a preliminary demonstration of human-to-robot motion transfer, but the paper acknowledges that for more complex human motions (fine-grained dexterous manipulation) the current architecture needs further improvement before reliable cross-embodiment transfer is possible.