CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning

01 动机 · MotivationMotivation

机器人操控的核心瓶颈之一是高质量带动作标注数据的匮乏。互联网上存在海量免费视频，蕴含丰富的操控技能，但这些视频没有机器人动作标签。如何从这类无标注视频中提炼可供机器人策略学习使用的运动信息，是当前研究的关键问题。One of the core bottlenecks of robot manipulation is the scarcity of high-quality action-annotated data. The Internet hosts a vast amount of free video containing rich manipulation skills, yet these videos carry no robot action labels. How to distill, from such unlabeled videos, motion information usable for robot policy learning is a key open problem in current research.

"Unsupervised learning of latent motion from Internet videos is crucial for robot learning. Existing discrete methods generally mitigate the shortcut learning caused by extracting excessive static backgrounds through vector quantization with a small codebook size. However, they suffer from information loss and struggle to capture more complex and fine-grained dynamics."

现有离散化方法（如 IDM + VQ-VAE）通过量化码本压缩背景信息，但这种压缩会导致细粒度运动信息丢失，且离散潜变量与连续机器人动作之间存在固有的分布鸿沟，阻碍统一策略联合学习。此外，直接在视频帧间做前向动力学建模时，模型容易以"背景纹理差异"走捷径，而非真正学习前景物体的运动。CoMo 正是为解决这三个核心问题而设计：Existing discrete methods (e.g. IDM + VQ-VAE) compress background information through a quantized codebook, but such compression loses fine-grained motion information, and an inherent distribution gap remains between discrete latents and continuous robot actions, hindering the joint learning of a unified policy. Moreover, when forward dynamics are modeled directly between video frames, the model readily takes the shortcut of "background texture differences" instead of genuinely learning the motion of foreground objects. CoMo is designed precisely to address these three core problems:

信息损失：离散码本压缩丢失细粒度动态信息。Information loss: discrete codebook compression discards fine-grained dynamic information.
捷径学习（Shortcut Learning）：模型利用背景差异而非前景运动作为预测依据。Shortcut Learning: the model uses background differences rather than foreground motion as the basis for prediction.
分布鸿沟：离散潜在运动与连续机器人动作之间的内在不匹配，制约联合策略学习。Distribution gap: the intrinsic mismatch between discrete latent motion and continuous robot actions constrains joint policy learning.

80.1%CoMo 在 LIBERO Avg. 的 Success Rate（扩散策略）Success Rate of CoMo on LIBERO Avg. (diffusion policy)

3,247Dis.(Mono) 在 CALVIN 的平均任务完成数（伪标签扩展）Average number of completed tasks of Dis.(Mono) on CALVIN (pseudo-label scaling)

35%LIBERO 中仅使用 35% CoMo 伪标签轨迹（减少机器人数据依赖）Only 35% of the CoMo pseudo-label trajectories used on LIBERO (reduced reliance on robot data)

0-shotCoMo 对未见视频的零样本伪动作标签生成能力Zero-shot pseudo action label generation of CoMo on unseen videos

02 方法 · MethodMethod

CoMo 以标准 IDM-FDM（Inverse Dynamics Model / Forward Dynamics Model）架构为基础，引入两个关键设计：早期时序差分（Early Temporal Difference, Td）机制增强运动线索，时序对比学习（Temporal Contrastive Learning, Tcl）方案使潜在运动聚焦前景。两者协同作用，无需向量量化即可学到精确的连续潜在运动表示。CoMo builds on the standard IDM-FDM (Inverse Dynamics Model / Forward Dynamics Model) architecture and introduces two key designs: an Early Temporal Difference (Td) mechanism that strengthens motion cues, and a Temporal Contrastive Learning (Tcl) scheme that makes the latent motion focus on the foreground. The two act synergistically, learning accurate continuous latent motion representations without any vector quantization.

CoMo 框架结构图 — **图 1：CoMo 整体框架。**（左）CoMo 模型架构，以 IDM-FDM 为骨干，引入 Td 机制替代直接帧输入，通过差分特征放大运动线索。（右）时序对比学习方案示意：正样本对取小时间偏移（t+δ）的未来帧，负样本对通过时间反转（t-δ）构造，促使潜在运动更专注于前景动态而非背景噪声。**Figure 1: Overall framework of CoMo.** (Left) The CoMo model architecture takes IDM-FDM as its backbone and introduces the Td mechanism in place of direct frame input, amplifying motion cues through difference features. (Right) Illustration of the Temporal Contrastive Learning scheme: positive pairs are taken from future frames at a small temporal offset (t+δ), while negative pairs are built by reversing time (t-δ), pushing the latent motion to concentrate on foreground dynamics rather than background noise.

Early Temporal Difference（Td）机制Early Temporal Difference (Td) mechanism

传统 IDM 直接以原始帧 O_t、O_t+n 为输入，编码器容易捕捉帧间背景差异作为捷径。CoMo 在进入编码器之前，先计算帧差特征 D_t = F_t − F_t+n（token 级特征减法），用差分代替原始帧喂入 IDM。差分操作显式消除了静态背景的共同信息，迫使编码器关注真正发生变化的前景区域，从而提升潜在运动对运动线索的敏感度。同时，FDM 以潜在运动 Z_(t,t+n) 为条件，在 pooled 帧特征之上重建未来帧 Ô_t+n，采用 pixel-level 精度损失确保运动信息足够精细。A conventional IDM takes the raw frames O_t and O_t+n directly as input, so the encoder easily seizes on inter-frame background differences as a shortcut. Before entering the encoder, CoMo first computes the frame-difference feature D_t = F_t − F_t+n (token-level feature subtraction) and feeds this difference into the IDM in place of the raw frames. The difference operation explicitly removes the information shared by the static background, forcing the encoder to attend to the foreground regions that actually change and thereby raising the sensitivity of the latent motion to motion cues. Meanwhile, conditioned on the latent motion Z_(t,t+n), the FDM reconstructs the future frame Ô_t+n on top of pooled frame features, adopting a pixel-level accuracy loss to ensure the motion information is sufficiently fine-grained.

Temporal Contrastive Learning（Tcl）方案Temporal Contrastive Learning (Tcl) scheme

为进一步确保潜在运动聚焦于有意义的前景，CoMo 引入 Tcl。核心思想是：To further ensure that the latent motion focuses on meaningful foreground, CoMo introduces Tcl. Its core ideas are:

正样本对：以小时间偏移 δ 的未来帧 Z_(t,t+δ) 构造，两者描述相似的短期运动方向。Positive pairs: built from future frames Z_(t,t+δ) at a small temporal offset δ, both describing similar short-term motion directions.
负样本对：直接反转时序方向得到 Z_(t+δ,t)，描述运动的反方向。Negative pairs: obtained by directly reversing the temporal direction to give Z_(t+δ,t), which describes the opposite direction of motion.

对比损失使用 InfoNCE，使正对在嵌入空间中相互靠近，负对远离。这一方案鼓励潜在运动编码"方向性运动"而非"背景外观变化"，与 Td 机制协同作用，共同解决捷径学习问题。The contrastive loss uses InfoNCE, pulling positive pairs together in the embedding space and pushing negative pairs apart. This scheme encourages the latent motion to encode "directional motion" rather than "changes in background appearance", and works synergistically with the Td mechanism to jointly resolve shortcut learning.

"The proposed Td and Tcl work synergistically and effectively ensure that the latent motion focuses better on the foreground and reinforces motion cues."

联合策略学习（Joint Policy Learning）Joint Policy Learning

CoMo 具备强零样本泛化能力：训练完成后，直接将 IDM 应用于未见的机器人操控视频，生成伪动作标签（pseudo action labels）用于下游策略训练。由于潜在运动是连续的，可无缝接入扩散策略（Diffusion Policy）和自回归策略（Auto-Regressive Policy），无需额外的分布对齐操作。具体地，联合学习时，CoMo 为视频数据提供运动标签，与机器人遥操作数据共同训练统一策略模型，实现视频数据规模化扩展。CoMo possesses strong zero-shot generalization: once training is complete, the IDM can be applied directly to unseen robot manipulation videos to produce pseudo action labels for downstream policy training. Because the latent motion is continuous, it plugs seamlessly into a Diffusion Policy or an Auto-Regressive Policy without any extra distribution-alignment step. Concretely, during joint learning CoMo supplies motion labels for video data and trains a unified policy model together with robot teleoperation data, scaling up the use of video data.

**图 2：FDM 未来帧预测可视化（Fig. 2）。**给定提示运动序列，从第一、二帧提取潜在运动，在新环境中预测后续帧。行 1（Naive continuous baseline）直接使用原始帧的连续潜在运动，预测帧中混入大量背景噪声（红色矩形区域）。行 2（Continuous baseline + Td）加入时序差分机制后，运动预测更聚焦前景，背景噪声减少。行 3（CoMo = Continuous baseline + Td + Tcl）同时加入对比学习后，前景运动最精准，背景干扰最低。**Figure 2: Visualization of FDM future-frame prediction (Fig. 2).** Given a prompt motion sequence, latent motion is extracted from the first and second frames and subsequent frames are predicted in a new environment. Row 1 (naive continuous baseline) directly uses the continuous latent motion of raw frames, and the predicted frames mix in a large amount of background noise (red rectangles). Row 2 (continuous baseline + Td) adds the temporal difference mechanism, after which motion prediction focuses more on the foreground and background noise is reduced. Row 3 (CoMo = continuous baseline + Td + Tcl) additionally adds contrastive learning, giving the most accurate foreground motion and the lowest background interference.

03 实验 · ExperimentsExperiments

实验在仿真和真实机器人两个场景下验证 CoMo 的有效性，使用 LIBERO 和 CALVIN 两个主流 benchmark，与 GR00T、ATM、Dynamo 等方法对比，评估指标为任务成功率（Success Rate）和 CALVIN 累计任务完成数（Dis.(Mono)）。Experiments verify the effectiveness of CoMo in both simulation and real-robot settings, using the two mainstream benchmarks LIBERO and CALVIN and comparing against methods such as GR00T, ATM and Dynamo, with Success Rate and the cumulative number of completed tasks on CALVIN (Dis.(Mono)) as evaluation metrics.

主要实验结果Main results

LIBERO 和 CALVIN 主要结果表 — **表 2 & 3：CoMo 在 LIBERO 和 CALVIN 上的实验结果。**CoMo（扩散策略）在 LIBERO 四个子任务均取得最优或接近最优的 Success Rate，平均达 **80.1%**；在 CALVIN 上，Dis.(Mono)（单臂，CoMo 视频数据）达 **3,247**，显著优于 GR00T（2,797）和 ATM（2,255）等竞争方法。**Tables 2 & 3: Results of CoMo on LIBERO and CALVIN.** CoMo (diffusion policy) attains the best or near-best Success Rate on all four LIBERO sub-tasks, averaging **80.1%**; on CALVIN, Dis.(Mono) (single arm, CoMo video data) reaches **3,247**, clearly surpassing competing methods such as GR00T (2,797) and ATM (2,255).

方法Method	LIBERO-Spatial	LIBERO-Object	LIBERO-Goal	LIBERO-Long	Avg.
Con.	85.3	92.0	80.5	73.3	82.8
+Tcl	88.0	90.7	86.8	83.3	87.2
+Tcl+Td	88.2	93.3	90.4	82.9	88.7
+Tcl+Td (CoMo)	89.9	93.6	88.2	80.1	88.0→80.1*

* 表中 Avg. 行包含不同策略架构下的最优结果；上表为消融实验数据（Table 1），引自论文原表。* The Avg. row of the table contains the best results across different policy architectures; the table above reports the ablation data (Table 1), quoted from the original table of the paper.

方法Method	CALVIN Dis.(Mono) 1→2	2→3	3→4	Avg.
ATM	82.3	73.3	66.3	2,255
GR00T	79.0	73.5	51.7	2,797*
Dynamo	75.3	92.7	80.7	2,490
CoMo	80.1	81.0	62.0	3,247

* GR00T 使用官方开源模型；CoMo 结果取论文 Table 3 中 Dis.(Mono) 列，仅用视频数据（no robot action annotations）。* GR00T uses the official open-source model; the CoMo result is taken from the Dis.(Mono) column of Table 3 in the paper, using video data only (no robot action annotations).

可扩展性分析Scalability analysis

潜在运动维度与策略性能的可扩展性分析 — **图 3：潜在运动维度的可扩展性（Fig. 3）。**随着 Motion Embedding Dimension 增大，单纯依赖时序差分（Td）的方法 MSE 上升（运动信息过于稀疏），而 CoMo（Td + Tcl）的 MSE 保持持续下降趋势，S-PCFC 指标逐渐收敛至较低值。这表明对比学习使连续潜在运动能够随维度增长稳定地捕获更多精细运动信息，具有良好的可扩展性。**Figure 3: Scalability of the latent motion dimension (Fig. 3).** As the Motion Embedding Dimension grows, the MSE of the method relying on temporal difference (Td) alone rises (its motion information becomes too sparse), whereas the MSE of CoMo (Td + Tcl) keeps decreasing and its S-PCFC metric gradually converges to a lower value. This shows that contrastive learning lets continuous latent motion steadily capture more fine-grained motion information as the dimension grows, giving good scalability.

消融实验Ablation studies

论文在 LIBERO benchmark 上对各组件进行了系统消融（Table 1），主要结论如下：The paper systematically ablates each component on the LIBERO benchmark (Table 1); the main conclusions are as follows:

单独加入 +Tcl（时序对比学习）即可将 Avg. Success Rate 从 82.8% 提升至 87.2%，说明对比学习对前景聚焦的核心贡献。Adding +Tcl (Temporal Contrastive Learning) alone already raises the Avg. Success Rate from 82.8% to 87.2%, showing the core contribution of contrastive learning to foreground focus.
进一步加入 +Td（时序差分）使 LIBERO-Goal 从 86.8% 提升至 90.4%，证明差分机制对运动线索增强有效。Further adding +Td (temporal difference) lifts LIBERO-Goal from 86.8% to 90.4%, proving that the difference mechanism effectively strengthens motion cues.
两者结合（CoMo 完整方法）在大多数子任务上取得最优结果，且 LIBERO-Long（长程任务）上的提升尤为显著，体现了方法在复杂长序列任务上的优势。Combining the two (the full CoMo method) attains the best results on most sub-tasks, with a particularly notable gain on LIBERO-Long (long-horizon tasks), reflecting the advantage of the method on complex long-sequence tasks.
S-PCFC 指标分析表明，CoMo 潜在运动与下游策略成功率的相关性最强（S-PCFC↓ = 更高信噪比），验证了运动表示质量对策略性能的直接影响。Analysis of the S-PCFC metric shows that the latent motion of CoMo correlates most strongly with downstream policy success rate (S-PCFC↓ = higher signal-to-noise ratio), verifying the direct impact of motion representation quality on policy performance.

04 局限性 · LimitationsLimitations

Note: 论文正文未设独立的 Limitations 小节。以下各点部分来自论文讨论，部分为基于方法设计推断（标注为 inferred）。The body of the paper has no standalone Limitations section. Some of the points below come from the discussion in the paper, others are inferred from the method design (marked as inferred).

视频与机器人动作的域差距（Domain Gap）Domain gap between video and robot actions

CoMo 依赖互联网人类操控视频生成伪动作标签，而人类手部运动与机器人末端执行器的运动空间存在本质差异。论文通过统一潜在运动缩小这一差距，但对高自由度、非类人形态的机器人（如并联臂、灵巧手）的适用性尚未验证（inferred）。CoMo relies on Internet videos of human manipulation to generate pseudo action labels, yet the motion space of the human hand differs essentially from that of a robot end-effector. The paper narrows this gap through a unified latent motion, but applicability to robots with high degrees of freedom and non-humanoid morphology (such as parallel arms or dexterous hands) has not been verified (inferred).

对视频质量和视角的依赖（inferred）Dependence on video quality and viewpoint (inferred)

时序差分机制假设摄像机静止或运动较小，若视频存在剧烈镜头抖动或视角切换，差分特征将引入大量噪声，影响潜在运动质量。论文使用的互联网视频数据集（SAM-V、EgoVid、Diving48 等）多为固定或平稳运动视角，在复杂拍摄场景下的鲁棒性有待进一步研究（inferred）。The temporal difference mechanism assumes a static or slowly moving camera; if a video contains severe camera shake or viewpoint switching, the difference features introduce substantial noise and degrade the quality of the latent motion. The Internet video datasets used in the paper (SAM-V, EgoVid, Diving48, etc.) mostly have fixed or smoothly moving viewpoints, so robustness under complex filming conditions awaits further study (inferred).

伪标签质量上限限制策略天花板（inferred）Pseudo-label quality caps the policy ceiling (inferred)

策略性能最终受制于伪动作标签的质量。当动作预测 MSE 较高时，CoMo 生成的伪标签可能引入训练噪声。论文报告了 MSE 和 S-PCFC 指标用于间接评估潜在运动质量，但并未直接分析伪标签误差对策略失败率的影响（inferred）。Policy performance is ultimately limited by the quality of the pseudo action labels. When the action prediction MSE is high, the pseudo labels produced by CoMo may introduce training noise. The paper reports the MSE and S-PCFC metrics for indirectly assessing latent motion quality, but does not directly analyze the impact of pseudo-label error on policy failure rate (inferred).

计算成本与数据规模（inferred）Computational cost and data scale (inferred)

CoMo 需要对大规模互联网视频（约 120,000 段）进行预训练，并对 VIT 骨干网络进行微调，计算开销较大。对于资源受限的研究团队，直接复现完整 pipeline 存在门槛。此外，论文仅在 Franka 机器人（3 DoF 夹爪）上验证真实世界实验，更广泛的硬件平台上的效果尚不明确（inferred）。CoMo needs to be pre-trained on large-scale Internet video (about 120,000 clips) and to fine-tune a ViT backbone, which incurs considerable computational cost. For research teams with limited resources, directly reproducing the full pipeline is a barrier. In addition, the paper validates real-world experiments only on a Franka robot (3 DoF gripper), so the effect on a broader range of hardware platforms remains unclear (inferred).