Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

01 动机（Motivation）Motivation

大型预训练视频生成模型捕获了丰富的时序动态与隐式物理先验，理论上是理想的机器人策略骨干网络—— 然而如何将其高效转化为可执行的控制策略，此前仍缺乏简洁的单阶段解决方案。Large pretrained video generation models capture rich temporal dynamics and implicit physical priors, which in principle makes them ideal backbones for robot policies — yet a concise single-stage solution for turning them efficiently into executable control policies was still missing.

"large pretrained video generation models have shown impressive ability to generate physically plausible and temporally coherent videos"—— 这一能力与机器人任务高度契合，但既有方法往往需要多阶段训练或大量架构改动。"large pretrained video generation models have shown impressive ability to generate physically plausible and temporally coherent videos" — this ability aligns well with robotic tasks, but prior methods usually require multi-stage training or extensive architectural changes.

Cosmos Policy 总览 — **图 1 · 系统总览：**Cosmos Policy 将 Cosmos-Predict2-2B 视频生成模型直接 fine-tune 为双臂机器人（ALOHA）的控制策略。左侧展示模型通过预测未来帧同时输出动作 chunk；右侧为真实机器人上四项复杂任务的执行效果。 Cosmos Policy 在 ALOHA 平均得分 93.6，显著超过 π₀.₅（88.6）与 π₀（77.9）。**Figure 1 · System overview:** Cosmos Policy directly fine-tunes the Cosmos-Predict2-2B video generation model into a control policy for a bimanual ALOHA robot. The left shows the model predicting future frames while emitting action chunks; the right shows execution of four complex tasks on the real robot. Cosmos Policy averages 93.6 on ALOHA, clearly surpassing π₀.₅ (88.6) and π₀ (77.9).

98.5%LIBERO 平均成功率（4 个任务套件）LIBERO average success rate (4 task suites)

67.1%RoboCasa 成功率（仅 50 条演示）RoboCasa success rate (only 50 demonstrations)

93.6ALOHA 真实机器人平均得分ALOHA real-robot average score

+12.5planning 在最难任务上的提升（分）Gain from planning on the hardest task (points)

02 方法（Method）Method

Cosmos Policy 的核心思路是将机器人动作、本体感知与价值估计编码为视频模型 diffusion 过程中的 latent frames，从而在不修改任何模型权重结构的前提下，通过单阶段 fine-tuning 同时学习策略、世界模型和价值函数。The core idea of Cosmos Policy is to encode robot actions, proprioception and value estimates as latent frames within the diffusion process of the video model, so that policy, world model and value function are learned jointly through single-stage fine-tuning without altering any model weight structure.

Latent 序列构成与 joint training — **图 2 · Latent 帧序列与联合训练方案：**输入 latent 序列为 (1 + T′) × H′ × W′ × 16，其中新模态（动作 chunk、未来本体感知、未来图像、状态价值 V(s′)）替换占位帧。非图像模态被"normalized and duplicated copies"填充到 latent 体积中。右图展示三种训练目标的 batch 组成：50% 演示数据（策略）、25% rollout（世界模型）、25% rollout（价值函数）。**Figure 2 · Latent frame sequence and joint training scheme:** the input latent sequence is (1 + T′) × H′ × W′ × 16, in which the new modalities (action chunk, future proprioception, future image, state value V(s′)) replace placeholder frames. Non-image modalities are padded into the latent volume as "normalized and duplicated copies". The right panel shows the batch composition of the three training objectives: 50% demonstration data (policy), 25% rollout (world model), 25% rollout (value function).

Latent Frame Injection（模态注入）Latent Frame Injection

区别于添加新模块的传统做法，Cosmos Policy 将所有额外模态——机器人动作、本体感知、价值估计—— 编码为 latent 空间中的"帧"，直接嵌入原始 diffusion 序列。以多相机 ALOHA 为例，latent 序列依次包含：空白占位帧、机器人本体感知、腕部相机、第三视角相机、动作 chunk、未来本体感知、未来图像与未来状态价值。 输出可并行生成（更快，适合直接策略执行）或自回归生成（质量更高，适合 planning）。Unlike the conventional practice of adding new modules, Cosmos Policy encodes every additional modality — robot actions, proprioception, value estimates — as "frames" in latent space and embeds them directly into the original diffusion sequence. Taking the multi-camera ALOHA setup as an example, the latent sequence contains, in order: a blank placeholder frame, robot proprioception, the wrist camera, the third-person camera, the action chunk, future proprioception, the future image and the future state value. Outputs can be generated in parallel (faster, suited to direct policy execution) or autoregressively (higher quality, suited to planning).

Joint Training Scheme（联合训练）Joint Training Scheme

为避免策略过度依赖单一监督信号，训练 batch 按比例混合三类目标：To keep the policy from over-relying on a single supervision signal, the training batch mixes three kinds of objectives in fixed proportions:

50% 演示数据：训练策略 p(a, s′, V(s′) | s)50% demonstration data: trains the policy p(a, s′, V(s′) | s)
25% rollout 数据：训练世界模型 p(s′, V(s′) | s, a)25% rollout data: trains the world model p(s′, V(s′) | s, a)
25% rollout 数据：训练价值函数 p(V(s′) | s, a, s′)25% rollout data: trains the value function p(V(s′) | s, a, s′)

辅助监督（未来状态预测 + 价值估计）显著提升了策略质量。消融实验显示，去掉辅助 loss 后 LIBERO 平均成功率从 98.5% 降至 97.0%（-1.5%）。Auxiliary supervision (future state prediction + value estimation) markedly improves policy quality. Ablations show that after removing the auxiliary loss the LIBERO average success rate drops from 98.5% to 97.0% (-1.5%).

Model-Based Planning（基于模型的规划）Model-Based Planning

推理阶段，Cosmos Policy 实现 best-of-N 采样：并行采样多个动作候选，由 planning 模型预测每个动作导致的未来状态与价值，选择价值最高的动作执行。聚合策略采用 "majority mean"—— 先根据阈值判断多数预测是成功还是失败，再对多数群体内的预测取均值，以提高鲁棒性。实验表明，model-based V(s′) planning 优于 model-free Q(s, a) planning，归因于在有限 rollout 数据下更高效的学习。At inference time, Cosmos Policy performs best-of-N sampling: multiple action candidates are sampled in parallel, the planning model predicts the future state and value induced by each of them, and the action with the highest value is executed. Aggregation uses a "majority mean" — a threshold first decides whether the majority of predictions are successes or failures, then the predictions inside the majority group are averaged, which improves robustness. Experiments show that model-based V(s′) planning outperforms model-free Q(s, a) planning, attributed to more efficient learning under limited rollout data.

03 实验（Experiments）Experiments

在三个基准上评估：LIBERO 仿真（4 个任务套件）、RoboCasa 仿真（24 个厨房任务）以及真实 ALOHA 双臂机器人（4 个复杂操作任务）。与 CogVLA、OpenVLA-OFT、π₀.₅、π₀、Video Policy、FLARE、GR00T-N1.5 等最先进方法对比。Evaluation covers three benchmarks: LIBERO simulation (4 task suites), RoboCasa simulation (24 kitchen tasks) and a real ALOHA bimanual robot (4 complex manipulation tasks). Comparisons are made against state-of-the-art methods such as CogVLA, OpenVLA-OFT, π₀.₅, π₀, Video Policy, FLARE and GR00T-N1.5.

LIBERO 仿真基准LIBERO simulation benchmark

方法Method	Spatial	Object	Goal	Long	平均Average
OpenVLA-OFT	—	—	—	—	97.1%
CogVLA	—	—	—	—	97.4%
π₀.₅	—	—	—	—	96.9%
Cosmos Policy	98.1%	100.0%	98.2%	97.6%	98.5%

RoboCasa 仿真基准（24 项厨房任务）RoboCasa simulation benchmark (24 kitchen tasks)

方法Method	演示数量Demonstrations	平均成功率Average success rate
Video Policy	300 条300 demos	66.0%
FLARE	300 条300 demos	66.4%
GR00T-N1.5	300 条300 demos	64.1%
Cosmos Policy	50 条50 demos	67.1%

Cosmos Policy 仅使用 50 条演示（为对比方法的 1/6），便超越了使用 300 条演示的所有基线，显示出强大的 data efficiency。Cosmos Policy uses only 50 demonstrations (1/6 of what the compared methods use) yet surpasses every baseline trained on 300 demonstrations, showing strong data efficiency.

ALOHA 真实双臂机器人ALOHA real bimanual robot

ALOHA 任务得分对比 — **图 3 · ALOHA 真实机器人任务得分：**四项双臂操作任务（放盘子、折叠衬衫、放糖果入碗、放糖果入袋）。 Cosmos Policy 平均得分 93.6，明显优于 π₀.₅（88.6）、π₀（77.9）和 OpenVLA-OFT+（62.0）。**Figure 3 · Task scores on the real ALOHA robot:** four bimanual manipulation tasks (put X on plate, fold shirt, put candies in bowl, put candy in ziploc bag). Cosmos Policy averages 93.6, clearly ahead of π₀.₅ (88.6), π₀ (77.9) and OpenVLA-OFT+ (62.0).

任务Task	OpenVLA-OFT+	π₀	π₀.₅	Cosmos Policy
Put X on plate	—	—	—	100.0
Fold shirt	—	—	—	99.5
Put candies in bowl	—	—	—	89.6
Put candy in ziploc bag	—	—	—	85.4
平均Average	62.0	77.9	88.6	93.6

Model-Based Planning 效果Model-Based Planning results

Planning 提升效果 — **图 4 · Planning 对最难任务的提升：**在"Put candies in bowl"上 planning 带来 +12.5 分提升； "Put candy in ziploc bag"同样获得显著增益。Model-based V(s′) planning 优于 model-free Q(s, a) planning。**Figure 4 · Planning gains on the hardest tasks:** on "Put candies in bowl" planning adds +12.5 points; "Put candy in ziploc bag" also gains substantially. Model-based V(s′) planning outperforms model-free Q(s, a) planning.

Ablation 分析Ablation analysis

辅助 loss 的作用：去掉未来状态与价值辅助监督 → LIBERO 平均成功率从 98.5% 降至 97.0%（-1.5%）。Effect of the auxiliary loss: removing the future-state and value auxiliary supervision → the LIBERO average success rate drops from 98.5% to 97.0% (-1.5%).
预训练权重的作用：从头训练 → LIBERO 成功率降至 94.6%（-3.9%）；ALOHA 折叠衬衫任务从 99.2 降至 80.8，并出现"jerky motions"（急促不稳定动作）。Effect of the pretrained weights: training from scratch → the LIBERO success rate falls to 94.6% (-3.9%); the ALOHA fold-shirt task drops from 99.2 to 80.8, and "jerky motions" appear.

ALOHA rollout 可视化 — **图 5 · ALOHA 真实机器人执行轨迹可视化：**Cosmos Policy 在四项双臂操作任务（放盘子、折叠衬衫、放糖果入碗、放糖果入袋）上的逐步执行帧。策略生成流畅、精准的双臂协调动作，折叠衬衫任务尤为突出（得分 99.5）。**Figure 5 · Rollout visualization on the real ALOHA robot:** step-by-step execution frames of Cosmos Policy on the four bimanual manipulation tasks (put X on plate, fold shirt, put candies in bowl, put candy in ziploc bag). The policy produces smooth and precise bimanual coordination, most notably on the fold-shirt task (score 99.5).

04 局限性（Limitations）Limitations

注：以下三点均为论文作者明确陈述（stated）的局限性。Note: all three points below are limitations explicitly stated by the authors of the paper.

推理速度较慢（Inference speed）Slow inference speed

Model-based planning 模式下，每次产生一个 action chunk 约需 5 秒，限制了该方案在动态、时间敏感任务上的适用性。对于需要快速反应的场景，当前 planning 延迟是主要瓶颈。In model-based planning mode, producing one action chunk takes about 5 seconds, which limits the applicability of the approach to dynamic, time-sensitive tasks. For scenarios that require fast reactions, the current planning latency is the main bottleneck.

依赖大量 rollout 数据（Rollout data requirements）Reliance on large amounts of rollout data

有效的 planning 需要超出演示分布的大量 rollout 数据，以实现对分布外状态的准确预测。当 rollout 数据不足时，世界模型的泛化能力受限，planning 的收益也随之下降。Effective planning requires a large amount of rollout data beyond the demonstration distribution in order to predict out-of-distribution states accurately. When rollout data is insufficient, the generalization of the world model is limited and the benefit of planning decreases accordingly.

规划深度受限（Planning depth）Limited planning depth

当前方案采用"best-of-N planning with one layer in the search tree"的单步规划策略，无法进行多步前瞻。作者指出，扩展预测 horizon 以及多层规划树有望进一步提升性能，但尚未实现。The current design adopts the single-step planning strategy "best-of-N planning with one layer in the search tree" and cannot look several steps ahead. The authors note that extending the prediction horizon and using a multi-layer planning tree is expected to improve performance further, but this has not yet been implemented.