Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

01 动机 MotivationMotivation

当前机器人视觉表征方法主要依赖单帧图像或两帧图像学习，忽视了具身任务中至关重要的动态信息。视频扩散模型（VDM）在大规模互联网视频上预训练，隐含地理解了物理世界的演化规律，但如何将这种"未来预见"能力转化为 robot policy 的视觉表征，尚无有效方案。Current robot visual representation methods learn mainly from single-frame or two-frame images, overlooking the dynamic information that is critical for embodied tasks. Video diffusion models (VDM), pretrained on large-scale internet video, implicitly capture how the physical world evolves, yet there is still no effective way to turn this "foresight of the future" into visual representations for a robot policy.

"We hypothesize that VDMs contain both current static information and predicted future dynamics, which can provide more comprehensive guidance for robot policy learning."

teaser：VDM 表征 vs 先前视觉编码器 — 图 1：先前视觉编码器（左）只利用当前静态信息；VPP（右）通过视频扩散模型同时获得当前状态与预测的未来动态，形成更丰富的 predictive visual representation，为策略决策提供前瞻性信息。Figure 1: Prior visual encoders (left) exploit only current static information; VPP (right) obtains both the current state and the predicted future dynamics through a video diffusion model, forming a richer predictive visual representation that supplies forward-looking cues for policy decisions.

+18.6%CALVIN ABC→D 相对先前 SOTA 提升Gain on CALVIN ABC→D over the prior SOTA

+31.6%真实灵巧操作任务成功率提升Success-rate gain on real-world dexterous manipulation tasks

4.33CALVIN 平均完成任务数（先前 SOTA: 3.35）Average tasks completed on CALVIN (prior SOTA: 3.35)

1.5BVDM 基础模型参数量（Stable Video Diffusion）Parameters of the VDM backbone (Stable Video Diffusion)

02 方法 MethodMethod

VPP 分两阶段：首先在机器人与人类操作数据上微调视频扩散模型，使其具备文本引导的视频预测能力（TVP）；然后以 TVP 的内部 latent 特征作为视觉编码器，通过 Video Former 聚合时空信息，最终由 Diffusion Policy 输出动作序列。VPP proceeds in two stages: a video diffusion model is first fine-tuned on robot and human manipulation data so that it acquires text-guided video prediction (TVP) ability; the internal latent features of the TVP model then serve as the visual encoder, Video Former aggregates the spatio-temporal information, and a Diffusion Policy finally outputs the action sequence.

VPP 两阶段训练流程 — 图 2：VPP 整体流程。Stage 1 — 在多样化数据集上微调 Stable Video Diffusion，加入 language conditioning via cross-attention，得到 Text-guided Video Prediction（TVP）模型。Stage 2 — 冻结 TVP，以其 up-sampling layers 的 latent 特征拼接为聚合表征 F_p，经 Video Former 压缩后输入 Diffusion Policy Head 生成动作。Figure 2: Overall VPP pipeline. Stage 1 — fine-tune Stable Video Diffusion on diverse datasets and add language conditioning via cross-attention, yielding a Text-guided Video Prediction (TVP) model. Stage 2 — freeze the TVP model, concatenate the latent features of its up-sampling layers into an aggregated representation F_p, compress it with Video Former, and feed it to the Diffusion Policy Head to generate actions.

Stage 1：Text-guided Video Prediction (TVP)Stage 1: Text-guided Video Prediction (TVP)

以 Stable Video Diffusion（1.5B 参数）为基础，通过 cross-attention 引入语言条件，在三类数据集上联合训练：Built on Stable Video Diffusion (1.5B parameters), language conditioning is introduced through cross-attention and the model is trained jointly on three kinds of datasets:

互联网人类操作数据：193,690 条轨迹，提供丰富的物理先验Internet human manipulation data: 193,690 trajectories, providing rich physical priors
机器人数据：179,074 条轨迹（BridgeV2、Open X-Embodiment 等）Robot data: 179,074 trajectories (BridgeV2, Open X-Embodiment, etc.)
下游任务视频：用于域适应Downstream task videos: used for domain adaptation

训练目标是从带噪样本重建完整视频序列，使模型在 single forward pass 中生成当前帧与未来帧的联合 latent 表征。The training objective is to reconstruct the complete video sequence from noisy samples, so that the model produces a joint latent representation of current and future frames in a single forward pass.

Stage 2：Video Former + Diffusion Policy HeadStage 2: Video Former + Diffusion Policy Head

TVP 充当"视觉编码器"，约 140ms 完成一次前向推理，提取多个 up-sampling layers 的 latent 特征并拼接为 F_p。TVP acts as the "visual encoder", completing one forward pass in about 140ms and extracting the latent features of several up-sampling layers, which are concatenated into F_p. Video Former 利用可学习 token 通过 spatial attention 与 temporal attention 对 F_p 进行时空聚合，压缩多视角信息，无需逐帧生成完整视频（相比 SuSIE 快 3.2×）。Video Former uses learnable tokens to aggregate F_p over space and time via spatial attention and temporal attention, compressing multi-view information without generating the complete video frame by frame (3.2× faster than SuSIE). Diffusion Policy Head 通过 cross-attention 将聚合表征与语言指令结合，生成连续的动作序列，以去噪扩散过程输出 6-DoF 末端执行器轨迹。Diffusion Policy Head combines the aggregated representation with the language instruction via cross-attention to produce a continuous action sequence, outputting 6-DoF end-effector trajectories through a denoising diffusion process.

图 4：TVP 单步前向预测的可视化。虽然单步预测的视频画面并不清晰，但其 latent 特征已编码了物体的物理演化信息（如滑块位置、物体朝向），对策略学习具有有效的引导作用。Figure 4: Visualization of single-step forward prediction by TVP. Although the predicted video frames are not sharp, their latent features already encode the physical evolution of objects (e.g., slider position and object orientation), which effectively guides policy learning.

03 实验 ExperimentsExperiments

在四类平台上系统评估 VPP：仿真基准 CALVIN（跨环境泛化）与 MetaWorld（50 任务多任务操作），以及真实硬件 Franka Panda 机械臂（30+ 任务）和灵巧手（100+ 任务）。与 RT-1、Diffusion Policy、GR-1、RoboUniview、SuSIE、Vidman 等基线对比。VPP is evaluated systematically on four kinds of platforms: the simulation benchmarks CALVIN (cross-environment generalization) and MetaWorld (multi-task manipulation over 50 tasks), plus real hardware — a Franka Panda arm (30+ tasks) and a dexterous hand (100+ tasks). It is compared against baselines such as RT-1, Diffusion Policy, GR-1, RoboUniview, SuSIE and Vidman.

Benchmark	Prior SOTA	VPP（本文）VPP (ours)	相对提升Relative gain
CALVIN ABC→D 平均完成任务数CALVIN ABC→D avg. tasks completed	3.35 (RoboUniview)	4.33	+29.3%
CALVIN 10% 数据CALVIN with 10% of the data	1.41 (GR-1)	3.25	+130.5%
MetaWorld 平均成功率MetaWorld average success rate	57.4% (GR-1)	68.2%	+10.8%
Franka 已见任务成功率Franka seen-task success rate	52% (GR-1)	85.6%	+64.6%
灵巧手已见任务成功率Dexterous hand, seen-task success rate	32% (GR-1)	74.9%	+134.1%
灵巧手工具使用任务Dexterous hand, tool-use tasks	15% (GR-1)	68%	+353.3%

CALVIN 与 MetaWorld 任务场景 — 图 3：CALVIN（左）与 MetaWorld（右）任务环境。CALVIN 要求跨 4 个视觉不同场景（A/B/C/D）的泛化能力；MetaWorld 涵盖 50 种不同的机械臂操作任务。Figure 3: Task environments of CALVIN (left) and MetaWorld (right). CALVIN requires generalization across 4 visually distinct scenes (A/B/C/D); MetaWorld covers 50 different robot-arm manipulation tasks.

真实硬件平台 — 图 5：真实硬件评估平台。左：Franka Panda 机械臂执行日常操作任务（如折叠布料、打开抽屉）。右：灵巧手执行精细操作（如使用剪刀、旋紧瓶盖），VPP 在灵巧手工具使用任务上比 GR-1 高出 53 个百分点（68% vs 15%）。Figure 5: Real-hardware evaluation platforms. Left: a Franka Panda arm performing everyday manipulation tasks (e.g., folding cloth, opening a drawer). Right: a dexterous hand performing fine manipulation (e.g., using scissors, tightening a bottle cap); on dexterous-hand tool-use tasks VPP exceeds GR-1 by 53 percentage points (68% vs 15%).

预测轨迹与实际执行轨迹对比 — 图 6：预测轨迹（红色）与实际执行轨迹（绿色）对比，两者高度吻合，说明 TVP 所学习的预测性表征确实捕捉了物体运动的物理规律。Figure 6: Comparison of the predicted trajectory (red) with the actually executed trajectory (green); the two agree closely, showing that the predictive representations learned by TVP do capture the physics of object motion.

消融实验 AblationsAblations

在 CALVIN 基准上的关键消融（以平均完成任务数衡量）：Key ablations on the CALVIN benchmark (measured by the average number of tasks completed):

配置Setting	CALVIN 平均任务数CALVIN avg. tasks	相对完整版本Relative to the full model
完整 VPPFull VPP	4.33	—
去掉互联网数据w/o internet data	3.97	−8.3%
去掉 SVD 预训练（随机初始化）w/o SVD pretraining (random initialization)	1.63	−62.4%
去掉 Video Former（改用所有帧特征）w/o Video Former (use features of all frames)	3.86	−10.9%，推理速度慢 3.2×−10.9%, inference 3.2× slower
以 VAE 替换 VDMReplace VDM with a VAE	2.58	−40.4%
以 VC-1 编码器替换Replace the encoder with VC-1	1.23	−71.6%
仅用最后一层特征Use only last-layer features	3.60	−16.9%

消融结果揭示：SVD 大规模预训练是最关键的性能来源（去掉后性能下降 62.4%）；Video Former 对效率与性能均有重要贡献；多层特征融合优于只用最后一层。The ablations reveal that large-scale SVD pretraining is the most critical source of performance (removing it drops performance by 62.4%); Video Former contributes substantially to both efficiency and performance; and fusing multi-layer features outperforms using only the last layer.

04 局限性 LimitationsLimitations

注意：论文未设专门的 Limitations 章节。以下第 1 条为作者在正文中明确提及的局限，第 2–3 条为从系统设计中推断的潜在约束，已标注来源。Note: The paper has no dedicated Limitations section. Item 1 below is a limitation explicitly mentioned by the authors in the main text; items 2–3 are potential constraints inferred from the system design, with their source labelled.

单步预测视觉质量有限（作者明确提及）Limited visual quality of single-step prediction (explicitly mentioned by the authors)

作者指出，单步前向预测"do not yield clear video"，生成的预测帧较为模糊。尽管如此，实验表明其 latent 特征已足够编码物理演化信息，对策略学习仍有指导价值。但若任务需要高精度视觉预测，此局限可能影响表现。The authors point out that single-step forward prediction "do not yield clear video": the generated frames are rather blurry. Even so, experiments show that the latent features already encode enough information about physical evolution to guide policy learning. Yet for tasks that require high-precision visual prediction, this limitation may affect performance.

推理延迟较高（设计推断）Relatively high inference latency (inferred from the design)

TVP 单次前向推理约需 140ms，对控制频率有影响。虽然相比 SuSIE 等需要多步去噪的方法快 3.2×，但对于需要高频控制（>10Hz）的任务（如高速避障、接触丰富操作）仍是瓶颈。A single TVP forward pass takes about 140ms, which constrains the control frequency. Although it is 3.2× faster than multi-step denoising methods such as SuSIE, it remains a bottleneck for tasks that need high-frequency control (>10Hz), such as high-speed obstacle avoidance and contact-rich manipulation.

依赖大规模预训练数据与算力（设计推断）Dependence on large-scale pretraining data and compute (inferred from the design)

VPP 的性能高度依赖 Stable Video Diffusion 在海量互联网视频上的预训练（消融去掉 SVD 预训练后性能下降 62.4%）。这意味着方法对计算资源要求较高，难以在资源受限场景下从头训练或快速迁移到全新领域。VPP's performance relies heavily on the pretraining of Stable Video Diffusion over massive internet video (ablating SVD pretraining drops performance by 62.4%). This means the method is demanding in computational resources, and is hard to train from scratch in resource-constrained settings or to transfer quickly to entirely new domains.