Nano World Models

01 动机Motivation

视频预测世界模型已成为学习预测模拟器的核心范式，支撑着生成、规划与决策制定。然而，尽管工业界的交互式视频生成取得了快速进展，更广泛的研究社区仍然缺乏紧凑、可复现、易于扩展的实现来研究现代世界模型背后的设计选择。Video-prediction world models have become a core paradigm for learning predictive simulators, underpinning generation, planning and decision making. Yet despite the rapid progress of interactive video generation in industry, the broader research community still lacks compact, reproducible and easily extensible implementations for studying the design choices behind modern world models.

"The broader research community still lacks compact, reproducible, and easily extensible implementations for studying the design choices underlying modern world models."

现有工业规模的世界模型对研究社区而言难以访问，研究者面临的核心挑战是：视频扩散、diffusion forcing、一致性蒸馏等技术已相当成熟，研究重心应从发明新技术转向理解设计决策。然而当前领域仍高度碎片化——不同论文使用不同数据集、训练方案和评估协议，缺乏"通用语言"。Existing industrial-scale world models are hard for the research community to access. The core challenge researchers face is that techniques such as video diffusion, diffusion forcing and consistency distillation are already fairly mature, so the focus of research should shift from inventing new techniques to understanding design decisions. Yet the field remains highly fragmented — different papers use different datasets, training recipes and evaluation protocols, and lack a common language.

NanoWM 的目标是打造一座"世界模型研究的巴别塔"（"a Babel tower for world model research"），让数据集、目标函数、架构和任务能够说同一种语言，从而系统比较各种设计选项。The goal of NanoWM is to build “a Babel tower for world model research”, so that datasets, objectives, architectures and tasks can speak the same language and design options can be compared systematically.

02 方法Method

NanoWM 以 diffusion forcing 框架为核心，通过向轨迹中不同帧分配不同噪声指标，在同一模型接口下统一表达 teacher-forced 预测、masked future prediction 和自回归 rollout。NanoWM is built around the diffusion forcing framework: by assigning different noise indices to different frames of a trajectory, it expresses teacher-forced prediction, masked future prediction and autoregressive rollout under a single model interface.

Diffusion Forcing 核心框架The Diffusion Forcing Core Framework

系统对轨迹内每帧分配噪声指标（noise index）：上下文帧保持干净，未来帧获得较高噪声指标。仅通过改变噪声调度，同一模型接口便可表达：The system assigns a noise index to every frame of a trajectory: context frames stay clean, while future frames receive higher noise indices. By changing only the noise schedule, the same model interface can express:

• Teacher-forced 预测：所有帧独立去噪Teacher-forced prediction: every frame is denoised independently
• Masked future prediction：遮蔽部分未来帧Masked future prediction: part of the future frames is masked out
• 自回归 rollout：将生成帧作为上下文继续预测Autoregressive rollout: generated frames are fed back as context to keep predicting

生成目标（Prediction Parameterization）Prediction Parameterization

框架支持三种扩散/流匹配预测目标：The framework supports three diffusion / flow-matching prediction targets:

• v-prediction（cosine schedule + ZTSNR）：FID 最优v-prediction (cosine schedule + ZTSNR): best FID
• x-prediction（cosine schedule + ZTSNR）：PSNR/SSIM 最优x-prediction (cosine schedule + ZTSNR): best PSNR/SSIM
• ε-prediction（linear schedule）：整体表现最弱ε-prediction (linear schedule): weakest overall

Transformer 骨干与模型规模Transformer Backbone and Model Scales

使用基于 Transformer 的骨干网络，对潜在视频 token 进行空间 patch 投影，再经过交错的空间-时间注意力块处理。命名规范为 "NanoWM-[规模]/[PatchSize]"，例如 NanoWM-B/2 表示 Base 规模、2×2 的 patch 大小。四种规模：A Transformer-based backbone is used: latent video tokens go through a spatial patch projection and are then processed by interleaved spatial-temporal attention blocks. The naming convention is “NanoWM-[scale]/[PatchSize]”; for example NanoWM-B/2 denotes the Base scale with a 2×2 patch size. Four scales:

• NanoWM-S：39.8M 参数NanoWM-S: 39.8M parameters
• NanoWM-B：158.6M 参数NanoWM-B: 158.6M parameters
• NanoWM-L：约 460M 参数NanoWM-L: about 460M parameters
• NanoWM-XL：830M 参数NanoWM-XL: 830M parameters

动作注入方式（Action Injection）Action Injection

框架实现了五种将动作信号注入模型的方式：The framework implements five ways of injecting the action signal into the model:

• Additive：元素逐位相加，参数最少Additive: element-wise addition, fewest parameters
• AdaLN：自适应层归一化AdaLN: adaptive layer normalization
• AdaLN-fuse：AdaLN 与时间步注入融合AdaLN-fuse: fuses AdaLN with timestep injection
• FiLM：特征线性调制（172.8M 参数）FiLM: feature-wise linear modulation (172.8M parameters)
• Cross-attention：交叉注意力（187.0M 参数）Cross-attention: cross attention (187.0M parameters)

潜在观测空间（Latent Observation Spaces）Latent Observation Spaces

支持三种潜在空间编码：Three latent-space encodings are supported:

• SD-VAE（Stable Diffusion VAE）：面向重建，latent shape [4, 32, 32]SD-VAE (Stable Diffusion VAE): reconstruction-oriented, latent shape [4, 32, 32]
• Web-DINO：自监督语义/几何特征，latent shape [1024, 16, 16]Web-DINO: self-supervised semantic / geometric features, latent shape [1024, 16, 16]
• V-JEPA 2.1：视频预训练预测型特征，latent shape [1024, 16, 16]V-JEPA 2.1: video-pretrained predictive features, latent shape [1024, 16, 16]

长程 RolloutLong-Horizon Rollout

通过在时间轴上应用滑动窗口注意力，模型可生成超越训练长度 4 倍的视频序列。长程生成时将已生成帧作为上下文帧，滑动窗口保证计算效率。By applying sliding-window attention along the temporal axis, the model can generate video sequences four times longer than the training length. During long-horizon generation the already generated frames serve as context frames, and the sliding window keeps computation efficient.

03 实验Experiments

实验在 6 个任务域上展开：Point Maze、Wall、Rope、Granular（来自 D4RL、DeepMind Control Suite）、PushT 和 RT-1（机器人操作数据）。评估指标：PSNR（像素保真度）、SSIM（结构相似性）、LPIPS（感知距离）、FID（分布相似性），以及决策任务的 Success Rate。验证集使用 256 个固定 seed=42 的片段。Experiments span 6 task domains: Point Maze, Wall, Rope, Granular (from D4RL and the DeepMind Control Suite), PushT and RT-1 (robot manipulation data). Evaluation metrics: PSNR (pixel fidelity), SSIM (structural similarity), LPIPS (perceptual distance), FID (distributional similarity), plus Success Rate for decision-making tasks. The validation set uses 256 fixed clips with seed=42.

发现一：预测目标对比（RT-1 Fractal 数据集，NanoWM-B/2）Finding 1: Prediction Target Comparison (RT-1 Fractal dataset, NanoWM-B/2)

v-prediction 在 FID 上最优；x-prediction 在重建指标（PSNR/SSIM/LPIPS）上最优。两者均大幅优于 ε-prediction。v-prediction is best on FID; x-prediction is best on the reconstruction metrics (PSNR/SSIM/LPIPS). Both are substantially better than ε-prediction.

发现二：模型规模消融（RT-1 Fractal）Finding 2: Model Scale Ablation (RT-1 Fractal)

规模扩大在所有指标上均带来一致提升。Scaling up brings consistent gains on every metric.

发现三：动作注入方式对比Finding 3: Action Injection Comparison

动作注入方式的优劣具有任务依赖性：FiLM 在 RT-1 上 FID 最优（40.62）；additive 在 PushT 上以最少参数量取得最佳 PSNR 和 FID。Cross-attention 在 RT-1 上表现最差（FID 51.12），尽管参数量最多。Which action injection scheme wins is task dependent: FiLM gives the best FID on RT-1 (40.62); additive achieves the best PSNR and FID on PushT with the fewest parameters. Cross-attention performs worst on RT-1 (FID 51.12), despite having the most parameters.

发现四：潜在空间对比（PushT 目标条件规划）Finding 4: Latent Space Comparison (PushT goal-conditioned planning)

诊断实验（真实动作 rollout 的 Latent MSE）揭示了失败根源：Web-DINO 和 V-JEPA 2.1 的动作嵌入 RMS 量级（分别为 0.00214 和 0.00129）远低于 SD-VAE（0.1119），表明语义潜在空间下模型几乎完全忽略动作信号，成为"动作无关"模型。这暗示扩散目标函数不足以强制语义表征中的动作利用。A diagnostic experiment (latent MSE of rollouts with ground-truth actions) reveals the root cause: the action-embedding RMS magnitudes of Web-DINO and V-JEPA 2.1 (0.00214 and 0.00129 respectively) are far below that of SD-VAE (0.1119), showing that under semantic latent spaces the model almost entirely ignores the action signal and becomes an action-agnostic model. This suggests that the diffusion objective is not enough to enforce action utilization in semantic representations.

发现五：跨域性能（统一训练方案）Finding 5: Cross-Domain Performance (unified training recipe)

统一训练方案在所有域上均有效。视觉/动态复杂度越高（如 Granular、RT-1），性能越低；简单仿真环境（Wall、Point Maze）表现最优。The unified training recipe is effective on every domain. The higher the visual / dynamic complexity (e.g. Granular, RT-1), the lower the performance; simple simulated environments (Wall, Point Maze) perform best.

发现六：长程 Rollout 误差累积Finding 6: Error Accumulation in Long-Horizon Rollout

消融实验总结Ablation Summary

• 预测目标：v-prediction 在感知质量（FID）最优；x-prediction 在像素保真度最优；ε-prediction 整体最差，差距显著（FID 差约 6–7 点）。Prediction target: v-prediction is best on perceptual quality (FID); x-prediction is best on pixel fidelity; ε-prediction is worst overall, by a clear margin (about 6–7 FID points).
• 模型规模：从 S(39.8M) 到 L(~460M) 所有指标单调提升，规模效应清晰。Model scale: from S(39.8M) to L(~460M) every metric improves monotonically, a clear scaling effect.
• 动作注入：无单一最优方法，任务依赖性强；additive 以最低参数量提供竞争性能。Action injection: there is no single best method and the choice is strongly task dependent; additive delivers competitive performance with the lowest parameter count.
• 采样预算：增加 DDIM 步数可持续改善 rollout 质量，对长程生成尤为重要。Sampling budget: increasing the number of DDIM steps keeps improving rollout quality, which matters especially for long-horizon generation.

04 局限性Limitations