Cosmos World Foundation Model Platform for Physical AI

01 动机Motivation

当前 Physical AI 发展缓慢，核心瓶颈在于：现实世界中带标注的观测—动作交互数据极度稀缺，且采集成本极高。世界模型（World Model）可以作为物理世界的"数字孪生"，让智能体在仿真中安全、廉价地生成海量训练数据，从而打破数据壁垒、加速 policy 的迭代。Physical AI is advancing slowly, and the core bottleneck is that annotated observation-action interaction data from the real world is extremely scarce and very expensive to collect. A World Model can act as a "digital twin" of the physical world, letting agents generate massive amounts of training data safely and cheaply in simulation, thereby breaking the data barrier and accelerating policy iteration.

"Physical AI needs to be trained digitally first. It needs a digital twin of itself, the policy model, and a digital twin of the world, the world model."

Cosmos platform overview — **图 2：Cosmos 平台全景。** 预训练 WFM 是"世界模型通才（generalist）"，可通过后训练（post-training）特化至具体 Physical AI 场景。平台分五大组件：视频数据处理（Video Curator）、视频分词（Tokenizer）、 WFM 预训练、WFM 后训练（下游适配），以及安全护栏（Guardrail）。**Figure 2: overview of the Cosmos platform.** A pre-trained WFM is a "generalist" world model that can be specialized to a concrete Physical AI setting through post-training. The platform has five components: video data curation (Video Curator), video tokenization (Tokenizer), WFM pre-training, WFM post-training (downstream adaptation), and the safety Guardrail.

20M小时原始训练视频hours of raw training video

~100M处理后视频片段curated video clips

7B / 14B扩散模型参数量diffusion model parameters

4B – 13B自回归模型参数量autoregressive model parameters

02 方法Method

Cosmos 平台由四个相互配合的模块组成：① 视频数据飞轮（Video Curator）； ② Cosmos Tokenizer（连续/离散两种，支持图像与视频联合训练）； ③ 两大预训练 WFM 家族（扩散式 Diffusion WFM 与自回归 Autoregressive WFM）； ④ 面向机器人、自动驾驶的后训练 recipe 与安全护栏。The Cosmos platform consists of four interacting modules: ① the video data flywheel (Video Curator); ② the Cosmos Tokenizer (continuous and discrete variants, jointly trained on images and video); ③ two families of pre-trained WFMs (Diffusion WFM and Autoregressive WFM); ④ post-training recipes and safety guardrails for robotics and autonomous driving.

视频数据飞轮（Video Curator）Video Data Flywheel (Video Curator)

Video curation pipeline — **图 5：Cosmos Video Curator 五步流水线。** 依次为：1) *split*（镜头检测，使用 TransNetV2，BBC 数据集 F1 = 0.967）； 2) *filtering*（运动过滤、视觉质量评估，移除质量最低的 15%，以及文字覆盖/视频类型分类）； 3) *annotation*（VLM 生成字幕，平均 559 字符/片段）； 4) *dedup*（语义聚类去重，移除约 30% 数据）； 5) *sharding*（按分辨率/宽高比/时长分桶）。系统采用 PyNvideoCodec + NVIDIA 硬件加速，转码吞吐量提升约 6.5×。**Figure 5: the five-stage Cosmos Video Curator pipeline.** In order: 1) *split* (shot detection with TransNetV2, F1 = 0.967 on the BBC dataset); 2) *filtering* (motion filtering and visual quality assessment, removing the lowest-quality 15%, plus text-overlay and video-type classification); 3) *annotation* (VLM-generated captions, 559 characters per clip on average); 4) *dedup* (semantic-clustering deduplication, removing about 30% of the data); 5) *sharding* (bucketing by resolution / aspect ratio / duration). The system uses PyNvideoCodec with NVIDIA hardware acceleration, improving transcoding throughput by about 6.5×.

Cosmos Tokenizer

预训练 WFM：扩散式与自回归式Pre-trained WFMs: diffusion and autoregressive

WFM model architecture — **图 6（模型侧）：两大 WFM 家族。** *Diffusion WFM*：基于 EDM 去噪得分匹配，包含 7B / 14B Text2World 模型；通过两阶段流水线（Text2World → Video2World）加入当前观测帧作为条件； Prompt Upsampler（12B Mistral-NeMo 基）弥合 VLM 描述分布差距。 *Autoregressive WFM*：Llama3-style GPT 架构从头训练，包含 4B / 12B 纯视频预测模型和 5B / 13B Video2World 变体（T5 + cross-attention 注入文本）；推理阶段配备 7B 扩散解码器，将离散 token 映射回连续表示以提升画质。**Figure 6 (model side): the two WFM families.** *Diffusion WFM*: built on EDM denoising score matching, including 7B / 14B Text2World models; a two-stage pipeline (Text2World → Video2World) adds the current observed frame as a condition; a Prompt Upsampler (based on 12B Mistral-NeMo) bridges the distribution gap of VLM captions. *Autoregressive WFM*: a Llama3-style GPT architecture trained from scratch, including 4B / 12B video-only prediction models and 5B / 13B Video2World variants (text injected via T5 + cross-attention); at inference a 7B diffusion decoder maps discrete tokens back to continuous representations to improve visual quality.

后训练：相机控制、机器人、自动驾驶Post-training: camera control, robotics, autonomous driving

预训练 WFM 可通过后训练适配三类典型 Physical AI 场景：Pre-trained WFMs can be adapted through post-training to three representative Physical AI settings:

相机控制（Camera Control）：在扩散 WFM 中注入相机位姿条件，生成可导航虚拟环境，用于视觉 SLAM / 导航策略训练。Camera Control: camera-pose conditioning is injected into the diffusion WFM to generate navigable virtual environments, used for visual SLAM / navigation policy training.
机器人操作（Robotic Manipulation）：在视频—动作序列数据集上微调，使模型能预测机器人执行动作后的世界状态，辅助策略学习与数据增强。Robotic Manipulation: fine-tuned on video-action sequence datasets so that the model can predict the state of the world after the robot executes an action, supporting policy learning and data augmentation.
自动驾驶（Autonomous Driving）：在驾驶专属数据集上微调，生成多视角、多条件驾驶场景视频，支持传感器仿真与安全测试。Autonomous Driving: fine-tuned on driving-specific datasets to generate multi-view, multi-condition driving-scene videos, supporting sensor simulation and safety testing.

03 实验Experiments

论文在 TokenBench（作者新建的 benchmark，涵盖 500 段视频：机器人操作 BridgeData V2、自动驾驶 BDD100K、第一视角 EgoExo-4D 与通用网络视频 Panda-70M）及 MS-COCO 上评测分词器，并展示扩散与自回归 WFM 的定性生成结果。The tokenizer is evaluated on TokenBench (a new benchmark built by the authors, covering 500 videos: robotic manipulation from BridgeData V2, autonomous driving from BDD100K, egocentric video from EgoExo-4D and general web video from Panda-70M) and on MS-COCO, and qualitative generation results of the diffusion and autoregressive WFMs are shown.

Cosmos Tokenizer 定量结果Cosmos Tokenizer quantitative results

数据集 / 压缩比Dataset / compression	类型Type	PSNR (dB) ↑	SSIM ↑	rFVD ↓
DAVIS · 4×8×8	Continuous (CV)	35.85	0.920	10.05
DAVIS · 4×8×8	Discrete (DV)	32.97	0.840	53.44
MS-COCO · 8×8	Continuous (CV)	32.79	0.824	1.874 rFID
MS-COCO · 8×8	Discrete (DV)	31.36	0.714	4.133 rFID

论文还报告，Cosmos Tokenizer 推理速度比此前方法快 2×～12×，并以此作为分词器效率的核心亮点。The paper further reports that Cosmos Tokenizer is 2×-12× faster at inference than prior methods, presenting this as the key efficiency highlight of the tokenizer.

TokenBench reconstruction results — **图 10：TokenBench 重建示例。** 在机器人操作、自动驾驶、第一视角、通用网络视频四类场景下， Cosmos Tokenizer（连续/离散）的视觉重建质量对比，验证了在多样化 Physical AI 场景中的通用性。**Figure 10: TokenBench reconstruction examples.** Visual reconstruction quality of the Cosmos Tokenizer (continuous / discrete) is compared across four scenario types -- robotic manipulation, autonomous driving, egocentric video and general web video -- verifying its generality across diverse Physical AI settings.

WFM generation evaluation — **图 11：预训练 WFM 定性生成结果。** 展示了 Cosmos Diffusion WFM（7B / 14B）及 Autoregressive WFM 在 text-to-world 与 video-to-world（Video2World）两种模式下的生成样本，涵盖室内场景、机器人场景和驾驶场景。**Figure 11: qualitative generation results of the pre-trained WFMs.** Samples generated by the Cosmos Diffusion WFM (7B / 14B) and the Autoregressive WFM under the two modes text-to-world and video-to-world (Video2World), covering indoor scenes, robot scenes and driving scenes.

安全护栏（Guardrail）Safety Guardrail

Cosmos 在输入端部署 Pre-Guard（关键词拦截 + Aegis 护栏模型过滤有害 prompt）；在输出端部署 Post-Guard（视频内容安全过滤 + 人脸模糊）。两套护栏均经过形式化红队（Red Team）对抗测试。At the input side Cosmos deploys Pre-Guard (keyword blocking plus the Aegis guardrail model to filter harmful prompts); at the output side it deploys Post-Guard (video content safety filtering plus face blurring). Both guardrails have gone through formal Red Team adversarial testing.

04 局限性Limitations

说明：论文仅在 §5.2.7 中明确列出自回归模型的局限；其余条目为本文根据设计推断（inferred），已标注。Note: the paper explicitly lists limitations of the autoregressive model only in §5.2.7; the remaining entries are inferred here from the design (inferred) and are marked as such.

自回归生成的误差累积（stated）Error accumulation in autoregressive generation (stated)

论文原文："Autoregressive models generate videos sequentially, which can lead to error accumulation over longer generation horizons and may result in degraded quality for extended video sequences." 即自回归式 WFM 在长序列预测时，逐 token 生成导致误差逐帧积累，长视频质量下降。From the paper: "Autoregressive models generate videos sequentially, which can lead to error accumulation over longer generation horizons and may result in degraded quality for extended video sequences." That is, when an autoregressive WFM predicts long sequences, token-by-token generation makes errors accumulate frame by frame and the quality of long videos degrades.

极高训练计算成本（inferred）Extremely high training compute cost (inferred)

预训练使用 10,000 块 NVIDIA H100 GPU，历时约三个月。如此规模的算力门槛使大多数研究者和机构无法从头复现预训练过程，只能在开放权重基础上进行后训练。Pre-training used 10,000 NVIDIA H100 GPUs over roughly three months. A compute barrier of this scale makes it impossible for most researchers and institutions to reproduce pre-training from scratch; they can only post-train on top of the open weights.

物理一致性仍未完全解决（inferred）Physical consistency is still not fully solved (inferred)

论文自述"the world foundation model problem is still far from being solved"，当前生成视频可能出现物理规律违反（穿模、重力异常等），距离真正可靠的物理世界孪生仍需大量后续研究。The paper itself states that "the world foundation model problem is still far from being solved"; generated videos may still violate physical laws (object interpenetration, anomalous gravity and so on), and a truly reliable twin of the physical world requires a great deal of further research.

护栏系统尚不完备（inferred）The guardrail system is not yet complete (inferred)

开源发布附带 Pre-Guard 与 Post-Guard，但作者明确表示安全措施仍在持续演进，护栏并非一次性的完整解决方案，对抗性 prompt 的覆盖率存在盲区。The open release ships with Pre-Guard and Post-Guard, but the authors state explicitly that the safety measures are still evolving; the guardrails are not a one-off complete solution, and coverage of adversarial prompts has blind spots.