Solving New Tasks by Adapting Internet Video Knowledge

01 动机 · MotivationMotivation

视频生成模型在机器人领域展现出两种应用路径：作为 visual planner（视觉规划器）或 policy supervisor（策略监督器）。 然而，互联网预训练视频模型与机器人领域数据之间存在根本矛盾：前者拥有强大的语言对齐能力，却缺乏对特定机器人环境的感知；后者则受限于数据规模，难以支持跨任务的自然语言泛化。Video generative models show two application paths in robotics: as a visual planner or as a policy supervisor. However, a fundamental tension exists between internet-pretrained video models and in-domain robot data: the former possess strong language alignment yet lack awareness of the specific robotic environment, while the latter is limited in scale and can hardly support natural-language generalization across tasks.

"Video generative models demonstrate great promise in robotics by serving as visual planners or as policy supervisors. When pretrained on internet-scale data, such video models intimately understand alignment with natural language, and can thus facilitate generalization to novel downstream behavior through text-conditioning. However, they may not be sensitive to the specificities of the particular environment the agent inhabits."

02 方法 · MethodMethod

本文系统比较了四种将领域数据融入预训练视频模型的 adaptation 策略，并在此基础上提出 Inverse Probabilistic Adaptation。 所有方法均使用预训练于互联网数据的视频模型（AnimateDiff）为基础，融入少量机器人操作演示数据。This paper systematically compares four adaptation strategies for incorporating domain data into pretrained video models, and on that basis proposes Inverse Probabilistic Adaptation. All methods build on a video model pretrained on internet data (AnimateDiff) and incorporate a small amount of robot manipulation demonstration data.

Direct Finetuning

直接对预训练视频模型的 motion module 进行微调，使用配对的领域内视频数据。这是最直接的迁移方式，能够编码环境特有的动态信息，但需要高质量的配对视频数据，且可能导致灾难性遗忘（catastrophic forgetting），在次优数据下容易崩溃。The motion module of the pretrained video model is finetuned directly with paired in-domain video data. This is the most straightforward transfer route and can encode environment-specific dynamics, but it requires high-quality paired video data, may lead to catastrophic forgetting, and collapses easily under suboptimal data.

Subject Customization

仅修改 image encoder 和 text encoder，无需视频数据，只需少量静态图像-文字对。使用特殊标识符（special identifier token）标记特定机器人环境的外观，对数据要求最低，但在运动规划方面的泛化能力有限。实现上采用 DreamBooth，以 20 张静态图像训练，通过 AnimateDiff 完成定制化。Only the image encoder and the text encoder are modified; no video data is needed, just a small number of static image-text pairs. A special identifier token marks the appearance of the particular robotic environment, which imposes the lowest data requirement but offers limited generalization for motion planning. The implementation uses DreamBooth, trained on 20 static images, with customization completed through AnimateDiff.

Probabilistic Adaptation

冻结大规模预训练模型，同时训练一个小型领域内视频模型，通过 score composition（分数合成）将两者结合：The large-scale pretrained model is frozen while a small in-domain video model is trained, and the two are combined through score composition:
ε_adapted(x, c, t) = ε_large(x, c, t) + γ · (ε_small(x, c, t) − ε_large(x, c, t))
其中 γ 是 guidance scale。此方法无需更新大模型参数，融合了大模型的语言对齐能力和小模型的环境特异性知识。where γ is the guidance scale. This method requires no update to the parameters of the large model, and fuses the language alignment of the large model with the environment-specific knowledge of the small one.

Inverse Probabilistic Adaptation（本文核心贡献）Inverse Probabilistic Adaptation (core contribution of this paper)

翻转 score composition 中大模型与小模型的方向，以领域内小模型为"基础"，利用大模型的分数引导其向更具语言对齐性的方向演变：The direction between the large and the small model in score composition is flipped: the in-domain small model serves as the "base", and the score of the large model guides it to evolve toward stronger language alignment:
ε_inverse(x, c, t) = ε_small(x, c, t) + γ · (ε_large(x, c, t) − ε_small(x, c, t))
这一简单的方向翻转使得方法对次优演示数据更具鲁棒性——当领域数据质量低下时，大模型提供的先验知识可以纠偏，从而避免小模型过拟合到次优行为上。This simple flip of direction makes the method more robust to suboptimal demonstration data — when the quality of the domain data is poor, the prior knowledge provided by the large model can correct the bias, preventing the small model from overfitting to suboptimal behavior.

03 实验 · ExperimentsExperiments

实验在 MetaWorld 基准的 9 个机器人操作任务上进行，分别评估两种应用路径：Policy Supervision 和 Visual Planning。 所有方法均以统一配置进行比较（context window size = 8，stride = 4，noise level = 100），基线为 Vanilla AnimateDiff（无 adaptation）。Experiments are run on 9 robot manipulation tasks of the MetaWorld benchmark, evaluating the two application paths separately: Policy Supervision and Visual Planning. All methods are compared under a unified configuration (context window size = 8, stride = 4, noise level = 100), with Vanilla AnimateDiff (no adaptation) as the baseline.

Policy Supervision 结果Policy Supervision results

在 Policy Supervision 设置下，适配后的视频模型对机器人执行帧进行 text-conditioned 评分，作为奖励信号训练策略。 结果如下表（成功率，越高越好，所有 9 个任务的平均值及选取任务报告）：Under the Policy Supervision setting, the adapted video model gives text-conditioned scores to the frames of robot execution, which serve as reward signals for training the policy. The results are given in the table below (success rate, higher is better; the average over all 9 tasks and a selection of tasks are reported):

Visual Planning 结果Visual Planning results

在 Visual Planning 设置下，适配后的模型合成执行计划的视频，通过逆动力学模型转化为动作。Inverse Probabilistic Adaptation 同样表现最优，在 9 个任务上取得最高平均成功率，并且在次优演示数据场景下成功率仍保持稳健（优于 Direct Finetuning 大幅下降的表现）。Under the Visual Planning setting, the adapted model synthesizes a video of the execution plan, which is turned into actions by an inverse dynamics model. Inverse Probabilistic Adaptation again performs best, attaining the highest average success rate across the 9 tasks, and its success rate stays robust in the suboptimal demonstration setting (better than the sharp drop of Direct Finetuning).

次优数据下的鲁棒性（Suboptimal Demonstrations）Robustness under suboptimal data (Suboptimal Demonstrations)

额外实验评估了当只有次优演示数据（suboptimal demonstrations，即随机策略采集的）可用时各方法的表现。 Direct Finetuning 的整体成功率大幅下降，而 Inverse Probabilistic Adaptation 仍保持稳健，成功率接近使用最优演示数据的情况。 这说明大模型先验可以有效纠正次优领域数据带来的偏差。Additional experiments evaluate how each method performs when only suboptimal demonstrations (i.e. collected by a random policy) are available. The overall success rate of Direct Finetuning drops sharply, whereas Inverse Probabilistic Adaptation stays robust, with a success rate close to that obtained with optimal demonstrations. This shows that the prior of the large model can effectively correct the bias introduced by suboptimal domain data.

Ablation 研究Ablation study

在策略监督（Policy Supervision）的 continuous control 设置（Dog 和 Humanoid 环境，来自 DeepMind Control Suite）上进行了消融实验，使用 Episode Return 作为评估指标（3 个 seed 取平均）：Ablation experiments are carried out in the continuous control setting of policy supervision (Policy Supervision) — the Dog and Humanoid environments from the DeepMind Control Suite — using Episode Return as the evaluation metric (averaged over 3 seeds):

• Subject Customization 仅修改静态 encoder，在 walking 等运动任务中提升有限。Subject Customization only modifies the static encoders, and brings limited gains on locomotion tasks such as walking.
• Probabilistic Adaptation 在大多数设置下表现一般，但在部分任务中具有竞争力。Probabilistic Adaptation performs unremarkably under most settings, but is competitive on some tasks.
• Inverse Probabilistic Adaptation 在 Dog Walking 和 Humanoid Walking 任务上均取得最高平均回报，证明了方法的跨环境通用性。Inverse Probabilistic Adaptation attains the highest average return on both the Dog Walking and the Humanoid Walking task, which demonstrates the cross-environment generality of the method.
• 当 text prompt 指定新的姿势或动作（如 "dog pumping"）时，direct finetuning 反而能最好地监督关联策略的学习，说明不同适应场景下最优方法存在差异。When the text prompt specifies a novel pose or motion (such as "dog pumping"), direct finetuning instead supervises the learning of the associated policy best, indicating that the optimal method differs across adaptation scenarios.

04 局限性 · LimitationsLimitations