用视频生成模型驱动视觉导航规划，支持动态约束与陌生环境想象Driving visual navigation planning with a video generation model, supporting dynamic constraints and imagination in unfamiliar environments
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, Yann LeCun · Meta AI / UC Berkeley
本文提出 Navigation World Model（NWM）——一个可控视频生成系统，输入过去帧与导航动作后预测未来视觉观测，
进而在不依赖固定策略的情况下进行轨迹规划与约束融合。核心架构为 Conditional Diffusion Transformer（CDiT），
以线性复杂度处理多帧上下文，规模扩展至 10 亿参数，并通过机器人与人类两类第一人称视频联合训练，
实现跨环境、跨实体的泛化能力。This paper proposes the Navigation World Model (NWM), a controllable video generation system that takes past frames together with navigation actions and predicts future visual observations, thereby performing trajectory planning and constraint incorporation without relying on a fixed policy. Its core architecture is the Conditional Diffusion Transformer (CDiT), which processes multi-frame context with linear complexity, scales up to one billion parameters, and is jointly trained on egocentric videos of both robots and humans, achieving generalization across environments and embodiments.
导航世界模型navigation world modelworld model视觉导航visual navigationvideo generationdiffusion transformerrobot navigation轨迹规划trajectory planningegocentric video
01动机Motivation
现有的视觉导航策略通常是"硬编码"的：一旦训练完成，就无法在推理阶段动态引入新的约束或反事实推理。
如果导航过程中遇到障碍、需要绕道，或需要在完全陌生的场景中规划路径，这类策略几乎无从应对。
更根本的问题是：如何让机器人在脑中"想象"轨迹，并以此筛选出最优方案？Existing visual navigation policies are usually "hard-coded": once training is finished, no new constraint or counterfactual reasoning can be introduced dynamically at inference time. If an obstacle is met during navigation, if a detour is needed, or if a path has to be planned in a completely unfamiliar scene, such policies are almost helpless. The more fundamental question is: how can a robot "imagine" trajectories in its mind and thereby select the best plan?
"Unlike fixed policies, NWM can imagine trajectories considering constraints and counterfactuals,
enabling dynamic constraint incorporation during the planning phase."
图 1：NWM 总览。
（a）训练阶段：从机器人与人类的第一人称视频及关联导航动作中学习。
（b）已知环境规划：通过 Cross-Entropy Method 模拟多条轨迹，选出代价最低的路径。
（c）陌生环境想象：仅凭单张图像即可想象可行的导航路径。Figure 1: overview of NWM. (a) Training stage: learning from egocentric videos of robots and humans together with the associated navigation actions. (b) Planning in known environments: simulating many trajectories with the Cross-Entropy Method and picking the lowest-cost path. (c) Imagination in unfamiliar environments: imagining feasible navigation paths from a single image alone.
1BCDiT 最大参数规模Largest CDiT parameter scale
4×CDiT vs. DiT FLOPs 减少（相同 LPIPS 下）FLOPs reduction of CDiT vs. DiT (at the same LPIPS)
908hEgo4D 无标注视频辅助训练Ego4D unlabeled video used as auxiliary training data
02方法Method
NWM 将导航建模为条件视频生成问题：给定历史帧序列和导航动作
ai = (u, φ)（其中 u ∈ ℝ² 控制前后/左右平移，φ ∈ ℝ 控制偏航角），
预测未来某时刻的视觉观测。
训练时额外引入时间偏移量 k（timeshift），允许模型"跳跃"到任意未来时刻，
并采用"每状态多目标"策略防止动作与时间的混淆（action-time entanglement）。NWM casts navigation as a conditional video generation problem: given a sequence of past frames and a navigation action ai = (u, φ) (where u ∈ ℝ² controls forward/backward and left/right translation and φ ∈ ℝ controls the yaw angle), it predicts the visual observation at some future moment. During training an additional time offset k (timeshift) is introduced, letting the model "jump" to an arbitrary future moment, and a "multiple targets per state" strategy is adopted to prevent action-time entanglement.
图 2：Conditional Diffusion Transformer（CDiT）Block。
目标帧 token 之间使用标准 self-attention；历史帧通过 cross-attention 作为条件输入，
而非与目标帧混合计算 full self-attention。
这使复杂度从标准 DiT 的 O(m²n²d) 降至 O(mn²d)，即关于历史帧数呈线性。
动作嵌入、时间偏移嵌入和扩散时间步嵌入通过 AdaLN 联合融入。Figure 2: the Conditional Diffusion Transformer (CDiT) block. Standard self-attention is used among target-frame tokens; context frames are fed in as conditions through cross-attention rather than being mixed with the target frame in a full self-attention computation. This lowers the complexity from O(m²n²d) in a standard DiT to O(mn²d), i.e. linear in the number of context frames. Action embeddings, time-offset embeddings and diffusion timestep embeddings are jointly injected via AdaLN.
Standalone 轨迹规划Standalone trajectory planning
使用 Cross-Entropy Method（CEM）在 NWM 内直接搜索动作序列。
能量函数平衡两个目标：The Cross-Entropy Method (CEM) is used to search action sequences directly inside NWM. The energy function balances two objectives:
对目标图像的感知相似度（perceptual similarity）perceptual similarity to the goal image
约束违反惩罚——约束以"无效动作集 𝒜valid"和"不安全状态集 𝒮safe"编码constraint-violation penalty — constraints are encoded as an "invalid action set 𝒜valid" and an "unsafe state set 𝒮safe"
不依赖任何外部策略，完全由 NWM 的内部想象驱动。It depends on no external policy at all and is driven entirely by NWM's internal imagination.
Trajectory Ranking（与外部策略结合）Trajectory Ranking (combined with an external policy)
从已有导航策略（如 NoMaD）采样多条候选轨迹（如 ×32 条），
再用 NWM 对每条轨迹进行"视频模拟"，
选出最接近目标、约束违反最少的轨迹执行。
无需对外部策略重新训练，即可赋予其动态约束能力。Multiple candidate trajectories (e.g. ×32) are sampled from an existing navigation policy (e.g. NoMaD), NWM then performs a "video simulation" of every trajectory, and the one closest to the goal with the fewest constraint violations is selected for execution. The external policy gains dynamic constraint handling without any retraining.
图 7：NWM 对外部策略轨迹进行排序。
给定起点观测和目标图像，NWM 从 NoMaD 采样的多条轨迹中，
模拟每条路径的视觉演变，最终选出视觉上最接近目标的轨迹（黄色高亮）执行。Figure 7: NWM ranks the trajectories of an external policy. Given the starting observation and the goal image, NWM simulates the visual evolution of every path among the trajectories sampled from NoMaD, and finally selects the trajectory visually closest to the goal (highlighted in yellow) for execution.
03实验Experiments
实验使用多个机器人与人类第一人称视频数据集：SCAND（8.7h）、TartanDrive（5h）、RECON（40h）、
HuRoN（75h）构成已知环境训练集；GO Stanford（25h）作为未知环境评测集；
Ego4D（908h 导航相关片段）提供无标注辅助数据。
导航指标为 Absolute Trajectory Error（ATE）和 Relative Pose Error（RPE）；
视觉质量指标为 LPIPS、DreamSim、PSNR、FVD、FID。The experiments use several egocentric video datasets of robots and humans: SCAND (8.7h), TartanDrive (5h), RECON (40h) and HuRoN (75h) form the known-environment training set; GO Stanford (25h) serves as the unknown-environment evaluation set; Ego4D (908h of navigation-related clips) provides unlabeled auxiliary data. Navigation is measured by Absolute Trajectory Error (ATE) and Relative Pose Error (RPE); visual quality by LPIPS, DreamSim, PSNR, FVD and FID.
图 4 & 5：预测精度随时间的变化（RECON 数据集）。
左：NWM 与 DIAMOND 在 1 FPS 和 4 FPS 下的 LPIPS 对比，时间跨度达 16 秒。
右：CDiT vs. 标准 DiT 的 LPIPS–FLOPs 权衡曲线（越左下越好）。Figures 4 & 5: prediction accuracy over time (RECON dataset). Left: LPIPS of NWM vs. DIAMOND at 1 FPS and 4 FPS, over a horizon of up to 16 seconds. Right: the LPIPS–FLOPs trade-off curve of CDiT vs. standard DiT (the further to the lower left, the better).
"4× fewer FLOPs compared to standard DiT while achieving better future prediction results."
约束规划（Table 3）Constrained planning (Table 3)
在"先向前再转弯"、"先向左/右再向前"等动作约束下规划，
与无约束基线相比，终点位移偏差极小：
前向优先约束 δu +0.36±0.01m、δφ +0.61±0.02；
左右优先约束 δu -0.03±0.01m、δφ +0.20±0.01。Planning under action constraints such as "go forward first, then turn" and "go left/right first, then forward" gives, compared with the unconstrained baseline, a very small deviation of the endpoint displacement: for the forward-first constraint δu +0.36±0.01m, δφ +0.61±0.02; for the left/right-first constraint δu -0.03±0.01m, δφ +0.20±0.01.
图 3：已知环境轨迹跟随的定性对比。
NWM 与竞品在 RECON 数据集上沿真实轨迹的视频生成效果对比，NWM 生成细节更清晰、时序更一致。Figure 3: qualitative comparison of trajectory following in known environments. Video generation along ground-truth trajectories on the RECON dataset, comparing NWM with competing methods; NWM produces sharper details and more temporally consistent results.
泛化到未知环境（Table 4）Generalization to unknown environments (Table 4)
引入 Ego4D 无标注数据后，GO Stanford（未知环境）评测结果改善：
LPIPS 从 0.658 → 0.652，DreamSim 从 0.478 → 0.464；
同时 RECON（已知环境）性能保持在 ~0.30 不退化。After bringing in unlabeled Ego4D data, the evaluation on GO Stanford (unknown environment) improves: LPIPS from 0.658 → 0.652 and DreamSim from 0.478 → 0.464, while performance on RECON (known environment) stays around ~0.30 without degradation.
消融实验（Table 1）Ablation study (Table 1)
在 RECON 上预测未来 4 秒（LPIPS 为主指标，越低越好）：Predicting 4 seconds into the future on RECON (LPIPS is the main metric, lower is better):
变体Variant
LPIPS ↓
1 个目标/样本1 target per sample
0.312
4 个目标/样本（最终设置）4 targets per sample (final setting)
0.296
1 帧上下文1 context frame
0.304
4 帧上下文4 context frames
0.296
仅时间条件（无动作）Time conditioning only (no action)
0.760
动作 + 时间联合条件Joint action + time conditioning
0.296
04局限性Limitations
说明：以下局限性均为论文第 5 节（Section 5 Limitations）明确陈述的内容。Note: every limitation below is explicitly stated in Section 5 (Section 5 Limitations) of the paper.
Mode Collapse（模式坍缩）Mode Collapse
在分布外（out-of-distribution）场景中，模型的输出会逐渐向训练数据靠拢，
出现"mode collapse"现象——生成图像的多样性和忠实度随时间下降。
论文图 10 以陌生环境为例，展示了该失败案例。In out-of-distribution scenes the outputs of the model gradually drift toward the training data, showing the "mode collapse" phenomenon — the diversity and fidelity of the generated images decay over time. Figure 10 of the paper takes an unfamiliar environment as an example and shows this failure case.
动态目标建模困难（Temporal Dynamics）Difficulty in modeling dynamic targets (Temporal Dynamics)
NWM 难以准确模拟行人等动态物体的运动，当前版本对动态场景的预测质量有限。NWM struggles to accurately simulate the motion of dynamic objects such as pedestrians, and the current version has limited prediction quality on dynamic scenes.
动作空间局限于 3-DoFAction space limited to 3-DoF
当前动作表示为 a = (u, φ)，仅覆盖 3 个自由度（前后、左右平移 + 偏航角）。
扩展到完整 6-DoF 以支持更复杂机器人平台被列为未来工作。The current action representation is a = (u, φ), covering only 3 degrees of freedom (forward/backward and left/right translation + yaw angle). Extending it to full 6-DoF so as to support more complex robot platforms is listed as future work.
