WorldMem 提出一种轻量级记忆库机制，将历史帧的 latent token 连同 5D 姿态与时间戳一起存储，通过基于置信度的空间检索与状态感知 cross-attention，让视频扩散模型在生成长达数百帧的第一人称视角时仍能忠实重建先前观测的场景，从根本上解决上下文窗口外的时空不一致问题。WorldMem proposes a lightweight memory bank mechanism that stores the latent tokens of past frames together with their 5D poses and timestamps. Through confidence-based spatial retrieval and state-aware cross-attention, it lets a video diffusion model faithfully reconstruct previously observed scenes even when generating first-person views spanning hundreds of frames, fundamentally resolving the spatio-temporal inconsistency that arises beyond the context window.
world simulationmemory banklong-term consistencyvideo diffusion状态感知注意力state-aware attentionDiffusion TransformerPlücker embedding视频生成video generation3D spatial consistency动态场景建模dynamic scene modeling
01动机Motivation
现有视频扩散模型受固定上下文窗口限制，在生成长序列时不得不丢弃先前帧。当智能体重返已探索区域时，模型无法重现之前的场景，导致严重的视觉不一致。如何在不依赖显式三维重建的前提下实现长程世界一致性，是世界模拟研究的核心挑战。Existing video diffusion models are constrained by a fixed context window and must discard earlier frames when generating long sequences. When the agent returns to an already explored area, the model cannot reproduce the earlier scene, causing severe visual inconsistency. Achieving long-term world consistency without relying on explicit 3D reconstruction is a core challenge for world simulation research.
"Existing approaches either struggle to preserve 3D spatial consistency over time or lack flexibility in modeling dynamic environments."
图1 · Teaser：WorldMem 在生成超过上下文窗口的长序列时，能够忠实重建先前观测的场景（含动态变化，如植被生长），而基线方法（Diffusion Forcing）则产生明显漂移与不一致。Figure 1 · Teaser: when generating sequences longer than the context window, WorldMem faithfully reconstructs previously observed scenes (including dynamic changes such as vegetation growth), whereas the baseline (Diffusion Forcing) exhibits obvious drift and inconsistency.
+6.66PSNR 提升（超窗口对比 Diffusion Forcing）PSNR gain (beyond the context window, vs. Diffusion Forcing)
~21 MB600 帧记忆库存储占用memory bank storage for 600 frames
02方法Method
WorldMem 以 Conditional Diffusion Transformer（CDiT）与 Diffusion Forcing 为基础，在其上引入三个核心组件：记忆库（Memory Bank）、置信度检索（Confidence-based Retrieval）和状态感知记忆注意力（State-aware Memory Attention），共同实现对历史场景的高效感知与忠实重建。WorldMem builds on a Conditional Diffusion Transformer (CDiT) and Diffusion Forcing, adding three core components on top: a Memory Bank, Confidence-based Retrieval and State-aware Memory Attention, which together enable efficient perception and faithful reconstruction of past scenes.
图2 · 架构总览：生成时，模型维护一个记忆库，存储历史帧的 latent token 及对应 5D 姿态与时间戳三元组 {(x_i, p_i, t_i)}。每步推理通过置信度检索选取 L_M 帧记忆，经状态感知 cross-attention 注入当前生成过程。Figure 2 · Architecture overview: during generation the model maintains a memory bank holding the latent tokens of past frames together with the corresponding 5D pose and timestamp triplets. At each inference step, confidence-based retrieval selects L_M memory frames, which are injected into the current generation process through state-aware cross-attention.
记忆库设计（Memory Bank）Memory Bank
记忆库以元组序列 {(𝐱ᵢᵐ, 𝐩ᵢ, tᵢ)} 的形式保存所有已生成帧的 latent token、5D 相机姿态（x, y, z, pitch, yaw）和时间戳。这种几何无关（geometry-free）表示无需维护显式三维结构，天然支持动态场景。存储 600 帧仅需约 21 MB，检索延迟在 1000 个候选帧时也仅为 0.16 秒。The memory bank keeps, as a sequence of tuples, the latent tokens of all generated frames along with their 5D camera poses (x, y, z, pitch, yaw) and timestamps. This geometry-free representation needs no explicit 3D structure to be maintained and naturally supports dynamic scenes. Storing 600 frames takes only about 21 MB, and retrieval latency is merely 0.16 seconds even with 1000 candidate frames.
检索算法综合三项打分维度为每个记忆单元计算置信度：The retrieval algorithm combines three scoring dimensions into a confidence value for every memory unit:
FOV 重叠率：通过 Monte Carlo 采样估算当前视角与历史帧视野的空间交叠，优先选取"能看到当前场景"的历史帧；FOV overlap: Monte Carlo sampling estimates the spatial overlap between the current viewpoint and the field of view of past frames, prioritizing historical frames that "can see the current scene";
时间临近度权重：近期帧与当前生成更相关；Temporal proximity weighting: recent frames are more relevant to the current generation;
相似度过滤（Similarity Filtering）：剔除冗余帧，确保 L_M 帧记忆覆盖多样视角。消融实验中加入相似度过滤后 rFID 从 24.33 降至 15.37。Similarity Filtering: removes redundant frames so that the L_M memory frames cover diverse viewpoints. In the ablation study, adding similarity filtering lowers rFID from 24.33 to 15.37.
与单纯视觉 token 注意力不同，WorldMem 将显式时空状态嵌入融入 cross-attention 的 Q/K 计算：Unlike attention over purely visual tokens, WorldMem folds explicit spatio-temporal state embeddings into the Q/K computation of cross-attention:
Plücker embedding：对相机姿态进行空间编码，提供每帧的射线级别位置感知；Plücker embedding: spatially encodes camera poses and provides ray-level positional awareness for each frame;
时间戳 sinusoidal embedding：允许模型区分同一位置在不同时刻的状态（如白天/夜晚、植被生长）；Sinusoidal timestamp embedding: lets the model distinguish the state of the same location at different moments (day/night, vegetation growth, etc.);
相对状态表达：将当前帧设为零参考，所有历史帧姿态转换为相对坐标，提升泛化性（PSNR: 23.63 → 23.98, LPIPS: 0.1830 → 0.1429）。Relative state representation: the current frame is taken as the zero reference and all historical frame poses are converted to relative coordinates, improving generalization (PSNR: 23.63 → 23.98, LPIPS: 0.1830 → 0.1429).
Cross-attention 公式为：CrossAttn(Q=p_q(X̃_q), K=p_k(X̃_k), V=p_v(X_k))，其中 Q/K 融合状态嵌入，V 保留纯视觉特征。The cross-attention is formulated as CrossAttn(Q=p_q(X_q), K=p_k(X_k), V=p_v(X_k)), where Q/K fuse the state embeddings while V retains the purely visual features.
训练策略Training Strategy
记忆帧使用最低噪声等级 k_min，上下文帧则在 [k_min, k_max] 范围内采样；时序注意力掩码防止记忆单元互相影响，保持因果性。训练时采用渐进式距离采样（Progressive Sampling）：从小范围（2m）逐渐扩展至大范围（8m），使模型学会处理大视角跨度的重访场景，相比固定 8m 采样 PSNR 提升 2.87（21.11 → 23.98）。Memory frames use the lowest noise level k_min, while context frames are sampled within [k_min, k_max]; a temporal attention mask prevents memory units from influencing one another and preserves causality. Training adopts progressive distance sampling: starting from a small range (2m) and gradually expanding to a large range (8m), so that the model learns to handle revisits with large viewpoint gaps, improving PSNR by 2.87 (21.11 → 23.98) over fixed 8m sampling.
03实验Experiments
在 MineDojo（Minecraft 虚拟环境，含平原/沙漠/热带草原等多样地形）和 RealEstate10K（真实室内场景）两个基准上评估。指标：PSNR（像素保真度）、LPIPS（感知相似度）、rFID（重建真实性）。对比基线：Full Sequence、Diffusion Forcing（DFoT）、CameraCtrl、ViewCrafter。Evaluation is carried out on two benchmarks: MineDojo (a Minecraft virtual environment with diverse terrains such as plains, desert and savanna) and RealEstate10K (real indoor scenes). Metrics: PSNR (pixel fidelity), LPIPS (perceptual similarity) and rFID (reconstruction realism). Baselines: Full Sequence, Diffusion Forcing (DFoT), CameraCtrl and ViewCrafter.
注：论文中 RealEstate10K 基线完整数值请见原文 Table 2；此处引用 WorldMem 最终结果。Note: for the full RealEstate10K baseline numbers, see Table 2 of the original paper; only WorldMem’s final results are quoted here.
图5 · 超窗口一致性对比：经过 600 帧记忆后再生成 100 帧。Diffusion Forcing（上行）在重访场景时产生严重漂移；WorldMem（下行）能准确重建先前观测的区域，与 ground truth 高度吻合。Figure 5 · Beyond-context-window comparison: 100 frames are generated after 600 frames of memory. Diffusion Forcing (top row) drifts severely when revisiting a scene, while WorldMem (bottom row) accurately reconstructs the previously observed region, matching the ground truth closely.图3 · 定性结果：上方展示与 ground truth 的对比，涵盖动态场景（植被生长、天气变化）；下方展示交互式世界场景中对象的持久性与环境状态的动态演化。Figure 3 · Qualitative results: the top part compares against ground truth and covers dynamic scenes (vegetation growth, weather changes); the bottom part shows object persistence and the dynamic evolution of environment states in an interactive world.
消融实验Ablation Study
图7 · 长程 PSNR 消融：在 300 帧生成过程中，随着帧数增加，各消融变体的 PSNR 出现明显下降，而完整 WorldMem 保持最高且最稳定的一致性。Figure 7 · Long-horizon PSNR ablation: over 300 generated frames, the PSNR of every ablated variant drops noticeably as the frame count grows, whereas the complete WorldMem keeps the highest and most stable consistency.
关键结论：相似度过滤对 rFID 改善最显著（24.33 → 15.37）；相对状态编码优于绝对编码（LPIPS −0.040）；渐进式距离采样带来最大 PSNR 增益；最优记忆长度为 8 帧（过长反而因冗余干扰性能）。Key findings: similarity filtering brings the largest rFID improvement (24.33 → 15.37); relative state encoding beats absolute encoding (LPIPS −0.040); progressive distance sampling yields the biggest PSNR gain; the optimal memory length is 8 frames (longer memory instead hurts performance through redundancy).
04局限性Limitations
说明：以下局限性均为论文作者明确陈述（stated），非推断。Note: every limitation below is explicitly stated by the authors, not inferred.
极端遮挡场景下检索可能不完整Retrieval may be incomplete under extreme occlusion
当智能体被障碍物遮挡或处于视角盲区时，置信度检索算法无法保证召回所有必要的历史记忆，可能导致局部场景重建失败。When the agent is occluded by obstacles or sits in a viewpoint blind spot, the confidence-based retrieval algorithm cannot guarantee that all necessary historical memories are recalled, which may cause local scene reconstruction to fail.
交互多样性与真实感有限Limited interaction diversity and realism
当前实验中的环境交互种类和物理真实感尚有局限，复杂动态场景（如流体、破坏性交互）的建模能力有待提升。The variety of environment interactions and the physical realism in the current experiments are still limited; the ability to model complex dynamic scenes (fluids, destructive interactions) remains to be improved.
记忆库随序列线性增长The memory bank grows linearly with the sequence
随着生成序列变长，记忆库存储量线性增加，对极长序列（远超本文实验的 600 帧）可能带来显著内存压力。作者指出扩展至真实世界场景及更丰富交互为未来方向。As the generated sequence gets longer, memory bank storage grows linearly, which may impose significant memory pressure for extremely long sequences (far beyond the 600 frames used in this paper). The authors point to extending the method to real-world scenes and richer interactions as future work.
