Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation

01 动机Motivation

机器人操作学习面临两条主流路线的天然裂痕：以动作为中心的行为克隆（behavior cloning）能直接产生动作，但难以利用丰富的视觉先验；以视觉为中心的世界模型或表征预训练能捕捉丰富的视觉语义，却与最终动作预测存在鸿沟。如何将两者无缝融合，是本文要回答的核心问题。Robotic manipulation learning suffers from an inherent rift between two mainstream routes: action-centric behavior cloning directly produces actions but struggles to exploit rich visual priors, whereas vision-centric world models or representation pre-training capture rich visual semantics yet remain disconnected from the final action prediction. How to fuse the two seamlessly is the core question this paper sets out to answer.

"We propose an end-to-end framework called Predictive Inverse Dynamics Models (PIDM), which integrates conditional visual foresight and inverse dynamics prediction to close the vision-action loop."

方法对比示意图 — **Figure 1**：三种范式对比。（左）朴素端到端行为克隆直接从观测预测动作；（中）解耦式方法先单独预测视觉状态再接逆动力学，两个模块无法互相优化；（右）本文提出的 PIDM 将视觉预测与动作预测端到端联合训练，视觉-动作回路完整闭合。**Figure 1**: comparison of three paradigms. (Left) naive end-to-end behavior cloning predicts actions directly from observations; (Middle) decoupled approaches first predict the visual state separately and then apply an inverse dynamics model, so the two modules cannot optimize each other; (Right) the proposed PIDM trains visual prediction and action prediction jointly end to end, fully closing the vision-action loop.

87.7%LIBERO-LONG 成功率
（+9% vs. 最强基线 MPI 77.3%）LIBERO-LONG success rate
(+9% vs. strongest baseline MPI 77.3%)

4.28CALVIN ABC-D 平均任务完成数
Seer-Large，SOTACALVIN ABC-D average tasks completed
Seer-Large, SOTA

78.4%真实机器人平均成功率
（vs. 从零训练 60.0%）Real-robot average success rate
(vs. 60.0% training from scratch)

187%仅用 10% 数据时
相对从零训练的 LIBERO 提升With only 10% of the data
relative LIBERO improvement over training from scratch

02 方法Method

Seer 以一个统一的 Transformer 骨干处理多模态输入（RGB 图像、语言指令、机器人状态），通过两个特殊 token——[FRS] 和 [INV]——分别负责视觉预测和动作预测，并以单向注意力掩码让动作生成能同时看到历史帧和未来预测帧，从而实现视觉与动作的协同优化。Seer processes multi-modal inputs (RGB images, language instructions, robot states) with a unified Transformer backbone. Two special tokens — [FRS] and [INV] — are responsible for visual prediction and action prediction respectively, and a unidirectional attention mask lets action generation attend to both historical frames and predicted future frames, achieving joint optimization of vision and action.

Seer 架构图 — **Figure 2**：Seer 完整流水线。三大模块组成：（1）**Multi-Modal Encoder**——MAE 预训练的 ViT-B 编码 RGB 图像，CLIP 编码语言目标，Perceiver Resampler 压缩视觉 token；（2）**Conditional Visual Foresight**——[FRS] token 在 24 层 GPT-2 式 Transformer 中预测未来 RGB 图像，损失为像素级 MSE：*ℒ_fore = ‖f_fore(g, h_t) − o_t+n‖²₂*；（3）**Inverse Dynamics Prediction**——[INV] token 通过单向注意力掩码同时访问历史与预测的未来帧，输出手臂动作（Smooth-L1 损失）和夹爪状态（BCE 损失）。总损失 *ℒ = αℒ_fore + ℒ_inv*，α=0.5。**Figure 2**: the complete Seer pipeline, composed of three modules. (1) **Multi-Modal Encoder** — an MAE pre-trained ViT-B encodes RGB images, CLIP encodes the language goal, and a Perceiver Resampler compresses the visual tokens; (2) **Conditional Visual Foresight** — the [FRS] token predicts future RGB images inside a 24-layer GPT-2-style Transformer, with a pixel-level MSE loss: *ℒ_fore = ‖f_fore(g, h_t) − o_t+n‖²₂*; (3) **Inverse Dynamics Prediction** — the [INV] token accesses both historical and predicted future frames through a unidirectional attention mask, outputting arm actions (Smooth-L1 loss) and gripper state (BCE loss). The total loss is *ℒ = αℒ_fore + ℒ_inv*, with α=0.5.

Conditional Visual Foresight

[FRS] token 在语言目标 g 和历史观测序列 h_t 的条件下，预测 n 步后的未来 RGB 图像 o_t+n。这一模块迫使模型学习场景的物理动态，而非仅记忆动作-观测对应关系。预训练时使用 DROID（76,000 条轨迹）进行大规模视觉动力学学习，历史帧长度为 7–10 帧。Conditioned on the language goal g and the sequence of historical observations h_t, the [FRS] token predicts the future RGB image o_t+n n steps ahead. This module forces the model to learn the physical dynamics of the scene rather than merely memorizing action-observation correspondences. Pre-training uses DROID (76,000 trajectories) for large-scale visual dynamics learning, with a history length of 7–10 frames.

Inverse Dynamics Prediction

[INV] token 通过单向注意力掩码同时"看到"过去帧和 [FRS] 预测的未来帧，从而以视觉语境为桥梁推断中间动作序列。动作预测步长（action horizon）为 3 步。标准 Seer 骨干含 24 层 GPT-2 block，hidden size 384，12 heads，可训练参数 65M；Seer-Large 可训练参数增至 315M。Through a unidirectional attention mask, the [INV] token simultaneously "sees" past frames and the future frames predicted by [FRS], using the visual context as a bridge to infer the intermediate action sequence. The action horizon is 3 steps. The standard Seer backbone contains 24 GPT-2 blocks with hidden size 384 and 12 heads, and 65M trainable parameters; Seer-Large increases the trainable parameters to 315M.

预训练与微调策略Pre-training and Fine-tuning Strategy

两阶段训练：首先在 DROID 上以 batch size 640–2048、学习率 1e-4 进行 20–30 epoch 预训练，让模型掌握通用的视觉动力学知识；随后在下游任务（每任务仅 100 条演示）上以学习率 1e-3 进行 20–40 epoch 微调。预训练阶段视觉编码器（ViT-B，251M 参数）保持冻结，仅更新 Transformer 骨干和解码器。Training proceeds in two stages: first, 20–30 epochs of pre-training on DROID with batch size 640–2048 and learning rate 1e-4, giving the model general visual dynamics knowledge; then 20–40 epochs of fine-tuning on downstream tasks (only 100 demonstrations per task) with learning rate 1e-3. During pre-training the visual encoder (ViT-B, 251M parameters) is kept frozen and only the Transformer backbone and decoders are updated.

03 实验Experiments

在三个评测场景上验证 Seer：（1）模拟仿真——LIBERO-LONG（长时序操作）和 CALVIN ABC-D（跨场景泛化）；（2）真实机器人——Franka Research 3，6 个任务（4 个泛化任务 + 2 个精度任务），共 900+ 次试验；（3）数据效率与规模扩展实验。Seer is validated in three evaluation settings: (1) simulation — LIBERO-LONG (long-horizon manipulation) and CALVIN ABC-D (cross-scene generalization); (2) real robots — Franka Research 3, 6 tasks (4 generalization tasks + 2 precision tasks), 900+ trials in total; (3) data-efficiency and scaling experiments.

LIBERO-LONG 基准LIBERO-LONG Benchmark

方法Method	平均成功率Average Success Rate
MTACT	41.0%
OpenVLA	54.0%
MVP	68.2%
MPI	77.3%
Seer (scratch)	78.7%
Seer（本文）Seer (ours)	87.7%

CALVIN ABC-D 基准（平均完成任务数）CALVIN ABC-D Benchmark (Average Number of Tasks Completed)

方法Method	平均任务数 (↑)Avg. Tasks (↑)
Roboflamingo	2.47
Susie	2.69
GR-1	3.06
3D Diffusor Actor	3.27
CLOVER	3.53
Seer (scratch)	3.64
Seer（本文）Seer (ours)	3.98
Seer-Large（本文）Seer-Large (ours)	4.28

真实机器人任务Real-Robot Tasks

方法Method	平均成功率Average Success Rate	平均得分Average Score
OpenVLA	16.7%	11.0
MPI	48.4%	29.3
MVP	55.0%	29.8
Seer (scratch)	60.0%	32.8
Seer（本文）Seer (ours)	78.4%	39.5

真实机器人实验 — **Figure 4**：真实世界实验设置（Franka Research 3）与四类泛化测试任务：Flip White Bowl（翻转白碗）、Stack Cups（叠杯）、Wipe Board（擦黑板）、Pick Place Close（拾放并关闭）。Seer 在所有任务上均超越基线。**Figure 4**: the real-world experimental setup (Franka Research 3) and the four categories of generalization test tasks: Flip White Bowl, Stack Cups, Wipe Board, and Pick Place Close. Seer outperforms the baselines on all tasks.

数据效率与鲁棒性Data Efficiency and Robustness

数据效率与规模扩展 — **Figure 3**：（左）数据效率：仅用 10% 下游数据时，Seer 在 LIBERO-LONG 上相对从零训练取得 "187% relative improvement"，在 CALVIN 上取得 "150% relative improvement"。（右）规模扩展：随模型参数增大，Seer 性能单调提升，展现良好的 scalability。**Figure 3**: (Left) data efficiency: using only 10% of the downstream data, Seer achieves a "187% relative improvement" over training from scratch on LIBERO-LONG and a "150% relative improvement" on CALVIN. (Right) scaling: as the model size grows, Seer's performance improves monotonically, demonstrating good scalability.

鲁棒性测试 — **Figure 5**：四类干扰下的鲁棒性对比。多物体干扰：60% vs. 33.3%；新背景：33.3% vs. 6.67%；新颖物体：60% vs. 46.7%；光照变化：66.7% vs. 46.7%。预训练带来显著鲁棒性提升。**Figure 5**: robustness comparison under four categories of disturbance. Distractor objects: 60% vs. 33.3%; unseen background: 33.3% vs. 6.67%; novel objects: 60% vs. 46.7%; lighting changes: 66.7% vs. 46.7%. Pre-training brings a marked gain in robustness.

消融实验Ablation Studies

在 CALVIN ABC-D 上进行消融，结论如下：单独引入视觉预测微调目标从 3.31 提升至 3.41；同时引入视觉预测与逆动力学两个目标达到 3.64。进一步加入预训练：仅预训练视觉预测目标达 3.73，两个目标均预训练达到最优 3.98。说明视觉预测与逆动力学两个模块在预训练和微调两个阶段均能协同互补，缺一不可。Ablations on CALVIN ABC-D yield the following conclusions: adding the visual-prediction fine-tuning objective alone raises the score from 3.31 to 3.41; adding both the visual-prediction and inverse-dynamics objectives reaches 3.64. Adding pre-training on top: pre-training the visual-prediction objective alone reaches 3.73, while pre-training both objectives gives the best result, 3.98. This shows that the visual-prediction and inverse-dynamics modules complement each other in both the pre-training and fine-tuning stages, and neither can be omitted.

04 局限性Limitations

说明：Note:以下局限性由作者在论文中明确陈述（stated），并非推断（inferred）。the limitations below are explicitly stated by the authors in the paper, not inferred.

真实世界评测范围有限Limited scope of real-world evaluation

论文作者明确指出："We only evaluate six downstream tasks, lacking a broader assessment of high-precision and contact-rich manipulation scenarios."（仅评测了六个下游任务，缺乏对高精度和接触密集型操作场景的更全面评估。）精细装配、力控操作等任务尚未涵盖。The authors state explicitly: "We only evaluate six downstream tasks, lacking a broader assessment of high-precision and contact-rich manipulation scenarios." Tasks such as fine-grained assembly and force-controlled manipulation are not yet covered.

跨机器人（cross-embodiment）泛化能力未验证Cross-embodiment generalization not verified

作者明确表示："Evaluating across different robots is also necessary to test Seer's cross-embodiment capability"，但当前所有实验均在 Franka Research 3 上进行，跨机体迁移能力有待探索。The authors state explicitly: "Evaluating across different robots is also necessary to test Seer's cross-embodiment capability", yet all current experiments are conducted on the Franka Research 3, leaving cross-embodiment transfer to be explored.

视觉预测的像素级 MSE 损失过于朴素（inferred）The pixel-level MSE loss for visual prediction is overly naive (inferred)

视觉预测目标使用像素均方误差 ℒ_fore = ‖f_fore(g, h_t) − o_t+n‖²₂，这在预测多步模糊未来时容易产生模糊（blurry）图像，可能限制对细粒度操作状态的预测质量。此条为作者未明确指出、从设计推断的局限性。The visual-prediction objective uses a pixel-wise mean squared error ℒ_fore = ‖f_fore(g, h_t) − o_t+n‖²₂, which tends to produce blurry images when predicting an uncertain multi-step future, potentially limiting the prediction quality for fine-grained manipulation states. This limitation is not stated by the authors but inferred from the design.