将视频生成用作机器人策略学习的代理任务Video generation as a proxy task for robot policy learning
Junbang Liang, Pavel Tokmakov, Ruoshi Liu, Sruthi Sudhakar, Paarth Shah, Rares Ambrus, Carl Vondrick · Columbia University & Toyota Research Institute
Video Policy 提出一个模块化框架，将视频生成与动作生成联合训练——通过学习"生成机器人执行任务的视频"作为代理目标，用极少的演示数据即可习得高度泛化的操作策略，在仿真与真实环境中均显著超越传统 behavior cloning 方法。Video Policy proposes a modular framework that jointly trains video generation and action generation: by learning to "generate videos of a robot performing the task" as a proxy objective, it acquires highly generalizable manipulation policies from very few demonstrations, significantly outperforming conventional behavior cloning in both simulation and the real world.
2025 · arXivStable Video Diffusion (SVD)RoboCasa · Libero10📄 arXiv:2508.00795
视频生成video generation机器人策略robot policyvisuomotor policyvideo diffusionimitation learningsample efficiencygeneralizationStable Video Diffusion
01动机 MotivationMotivation
当前 visuomotor policy 面临两大核心挑战：（1）在感知或行为分布偏移下泛化能力差；（2）性能高度依赖大规模人类演示数据集。这两点制约了机器人策略在真实场景中的可扩展性与鲁棒性。Current visuomotor policies face two core challenges: (1) poor generalization under perceptual or behavioral distribution shifts; (2) heavy reliance on large-scale human demonstration datasets. Together these limit the scalability and robustness of robot policies in real-world settings.
"Despite tremendous progress in dexterous manipulation, current visuomotor policies remain fundamentally limited by two challenges: they struggle to generalize under perceptual or behavioral distribution shifts, and their performance is constrained by the size of human demonstration data."
作者的核心观察是：视频生成是比动作生成更通用的目标。学会预测机器人执行任务的视频，可以从无动作标注的视频数据中学习环境动力学，进而以极少有动作标注的演示数据完成策略提取。这一思路将互联网规模的视频生成预训练引入机器人学习，提供近乎无限的动作无关数据来源。The authors' key observation is that video generation is a more general objective than action generation. Learning to predict videos of a robot performing tasks makes it possible to learn environment dynamics from action-free video data, and then to extract a policy from very few action-labeled demonstrations. This brings internet-scale video generation pre-training into robot learning, offering a nearly unlimited source of action-agnostic data.
图 1：视频生成作为机器人策略学习的代理任务。给定初始观测图像与自然语言任务描述，模型同时生成机器人执行任务的视频（上）与机器人动作序列（中），通过独立的扩散网络实现。模块化设计支持从无动作视频数据中学习，提升对未见场景的泛化能力，是传统 behavior cloning 的可扩展替代方案。Figure 1: Video generation as a proxy task for robot policy learning. Given an initial observation image and a natural-language task description, the model simultaneously generates a video of the robot performing the task (top) and the robot action sequence (middle), through separate diffusion networks. The modular design supports learning from action-free video data and improves generalization to unseen scenes, making it a scalable alternative to conventional behavior cloning.
0.66RoboCasa avg. success (300 demos)
0.94Libero10 avg. success rate
50demos 即可超越多数 baselinedemos suffice to surpass most baselines
3×优于 GR00T (0.50 → 0.66)better than GR00T (0.50 → 0.66)
02方法 MethodMethod
Video Policy 由两个模块化扩散网络组成：Video U-Net（μθ）负责生成未来帧序列，Action U-Net（αθ）以视频特征为条件解码机器人动作。两者联合训练，视频生成网络的中间特征直接为动作预测提供丰富的时空表征。Video Policy consists of two modular diffusion networks: a Video U-Net (μθ) that generates the future frame sequence, and an Action U-Net (αθ) that decodes robot actions conditioned on video features. The two are trained jointly, and the intermediate features of the video generation network directly provide rich spatio-temporal representations for action prediction.
图 2：Video Policy 整体架构。模型以初始环境状态图像与未来帧/动作的噪声向量作为输入，联合扩散视频帧与动作序列。帧的表征通过五个等间距的 decoder 层（layers 9, 14, 17, 20, 23）提取，经 CNN adapter 处理后作为条件信号输入 Action U-Net，引导动作去噪。模块化设计使两个网络可以分阶段训练，支持无动作视频数据的利用。Figure 2: Overall architecture of Video Policy. The model takes an image of the initial environment state together with noise vectors for future frames and actions as input, and jointly diffuses video frames and the action sequence. Frame representations are extracted from five evenly spaced decoder layers (layers 9, 14, 17, 20, 23), processed by a CNN adapter and then fed into the Action U-Net as a conditioning signal that guides action denoising. The modular design lets the two networks be trained in separate stages, enabling the use of action-free video data.
基于 Stable Video Diffusion (SVD) 构建，通过 cross-attention 接受两类条件输入：（1）自然语言任务描述的 CLIP embedding；（2）连结并经 VAE 编码的初始观测图像。在机器人演示数据上 fine-tune 后，模型学会生成符合任务语义的执行视频序列，并在此过程中隐式编码环境动力学。Built on Stable Video Diffusion (SVD), it takes two kinds of conditioning input through cross-attention: (1) the CLIP embedding of the natural-language task description; (2) the initial observation image, concatenated and encoded by a VAE. After fine-tuning on robot demonstration data, the model learns to generate execution video sequences consistent with the task semantics, implicitly encoding environment dynamics in the process.
1D CNN U-Net，从视频解码器五个等间距层抽取时空特征，经 CNN adapter 后输入动作 U-Net，对动作噪声进行去噪以预测连续控制量。训练时对视频网络采用梯度截断（gradient stopping），防止动作损失反向传播影响视频模型，保持视频生成质量。A 1D CNN U-Net extracts spatio-temporal features from five evenly spaced layers of the video decoder; after a CNN adapter they are fed into the action U-Net, which denoises the action noise to predict continuous control values. During training, gradient stopping is applied to the video network so that the action loss cannot back-propagate into the video model, preserving video generation quality.
两阶段训练策略Two-stage training strategy
Stage 1：在全量视频数据（含无动作标注视频）上 fine-tune 视频扩散模型，学习丰富的环境动力学表征。Stage 2：冻结视频网络权重，仅训练动作解码头，从少量有标注演示中提取策略。消融实验（Table 3）证明此两阶段策略相比联合训练（joint）提升 success rate 从 0.57 → 0.63，而完全跳过视频 fine-tuning 则性能崩溃至 0.09。Stage 1: fine-tune the video diffusion model on the full video data (including action-free videos) to learn rich representations of environment dynamics. Stage 2: freeze the video network weights and train only the action decoding head, extracting the policy from a small number of labeled demonstrations. The ablation (Table 3) shows that this two-stage strategy raises the success rate from 0.57 to 0.63 compared with joint training, whereas skipping video fine-tuning entirely collapses performance to 0.09.
03实验 ExperimentsExperiments
在 RoboCasa（24 个任务，涵盖 Pick&Place、Doors、Drawers、Knobs、Buttons 等多类别）与 Libero10 两个 benchmark 上评测；真实环境测试 5 项操作任务，评估对物体位置、外观与背景的泛化能力。主要 baseline 包括 DP-ResNet、DP-CLIP、GR00T、DP-VLA、UVA 等。Evaluated on two benchmarks, RoboCasa (24 tasks spanning Pick&Place, Doors, Drawers, Knobs, Buttons and other categories) and Libero10; real-world testing covers 5 manipulation tasks and assesses generalization to object position, appearance and background. The main baselines include DP-ResNet, DP-CLIP, GR00T, DP-VLA and UVA.
使用 50 个演示的 Video Policy 即超越了所有使用更多数据的 baseline；300 个演示版本进一步达到 0.66，高于最强 baseline DP-VLA（0.57）约 16%。With 50 demonstrations, Video Policy already surpasses every baseline that uses more data; the 300-demonstration version further reaches 0.66, about 16% above the strongest baseline DP-VLA (0.57).
消融实验（Table 3，RoboCasa）Ablation study (Table 3, RoboCasa)
变体Variant
Avg. Success Rate
No Video Tuning
0.09
Joint（联合训练）Joint (joint training)
0.57
2-Stage（本文方法）2-Stage (ours)
0.63
作者指出："learning to generate policy-execution videos is both necessary and sufficient"——无视频 fine-tuning 时成功率仅 0.09，说明视频生成目标是提取有效策略的核心。The authors note: "learning to generate policy-execution videos is both necessary and sufficient" — without video fine-tuning the success rate is only 0.09, showing that the video generation objective is central to extracting an effective policy.
视频预测 horizon 的影响Effect of the video prediction horizon
图 4：基于无动作视频数据的任务泛化。Video Policy 与 baseline DP-ResNet 均仅在 24 个 RoboCasa 任务中的 12 个上训练动作监督，但 Video Policy 的视频生成模型额外使用全部 24 个任务的无动作视频。结果表明 Video Policy 在未见任务（图右侧）上实现了显著的泛化，而无法利用无动作数据的 DP-ResNet 几乎无法泛化到未见任务。Figure 4: Task generalization from action-free video data. Both Video Policy and the DP-ResNet baseline receive action supervision on only 12 of the 24 RoboCasa tasks, but the video generation model of Video Policy additionally uses action-free videos of all 24 tasks. The results show that Video Policy achieves substantial generalization to unseen tasks (right side of the figure), while DP-ResNet, which cannot exploit action-free data, barely generalizes to unseen tasks.
图 5：Pick and Place 真实环境定性结果。Video Policy 在物体位置、外观与背景颜色三类分布偏移下均展现出强鲁棒性。Figure 5: Qualitative real-world results for Pick and Place. Video Policy shows strong robustness under all three kinds of distribution shift: object position, appearance and background color.
04局限性 LimitationsLimitations
Note:以下局限性均为作者在论文中明确陈述（stated by the authors）。All the limitations below are explicitly stated by the authors in the paper.
规模受限：仅单一真实机器人平台与有限仿真 benchmarkLimited scale: a single real-world robot platform and few simulation benchmarks
"Our study has several limitations. First, it is restricted in the scale of simulation benchmarks and a single real-world embodiment." 当前验证仅覆盖 RoboCasa 和 Libero10 两个仿真 benchmark 以及一种真实机器人平台，结论能否推广到更广泛的任务场景仍有待验证。"Our study has several limitations. First, it is restricted in the scale of simulation benchmarks and a single real-world embodiment." The current validation covers only the two simulation benchmarks RoboCasa and Libero10 plus one real robot platform, so whether the conclusions generalize to broader task settings remains to be verified.
仅探索了 Stable Video Diffusion 一种视频生成模型Only one video generation model, Stable Video Diffusion, was explored
"Additionally, we explore only one instantiation of video generation models — Stable Video Diffusion (SVD). While our analysis is more extensive than prior works, broader validation across tasks, environments, and model families would further strengthen the findings." 不同视频生成模型架构（如 DiT-based 模型）是否同样有效尚未验证。"Additionally, we explore only one instantiation of video generation models — Stable Video Diffusion (SVD). While our analysis is more extensive than prior works, broader validation across tasks, environments, and model families would further strengthen the findings." Whether other video generation architectures (such as DiT-based models) are equally effective has not been verified.
推理速度慢，当前不适合实时部署Slow inference; not currently suitable for real-time deployment
"The computational cost of video diffusion models remains a major practical bottleneck, particularly for real-world deployment." 当前实现在 256×256 分辨率下生成 25 帧约需 9 秒，远无法满足实时控制需求。作者指出扩散推理加速方向（如 consistency distillation）有望缓解此问题，但尚未集成。"The computational cost of video diffusion models remains a major practical bottleneck, particularly for real-world deployment." The current implementation takes about 9 seconds to generate 25 frames at 256×256 resolution, far short of real-time control requirements. The authors point to diffusion inference acceleration (such as consistency distillation) as a promising remedy, but it is not yet integrated.
