高效持久记忆的世界行动模型 (Efficient World Action Modeling with Persistent Memory)Efficient World Action Modeling with Persistent Memory
Sizhe Yang · Juncheng Mu · Tianming Wei · Chenhao Lu · Xiaofan Li · Linning Xu · Zhengrong Xue · Zhecheng Yuan · Dahua Lin · Jiangmiao Pang · Huazhe Xu | 香港中文大学 · 清华大学 · 浙江大学The Chinese University of Hong Kong · Tsinghua University · Zhejiang University
机器人操作的长时序任务中，现有世界行动模型（WAM）面临记忆与效率的根本矛盾：滑动窗口方法推理高效但遗忘长程上下文，全历史 KV 缓存方法具备记忆能力却随序列增长导致延迟与显存急剧上升。MemoryWAM 提出混合记忆设计——近期帧、事件边界锚帧、以及压缩长程历史的 gist token——将推理时间与空间复杂度从 O(N) 降至 O(N/d)，同时在仿真与真实机器人任务中超越强 VLA/WAM 基线。In long-horizon robot manipulation tasks, existing world action models (WAMs) face a fundamental conflict between memory and efficiency: sliding-window methods infer efficiently but forget long-range context, whereas full-history KV cache methods retain memory yet suffer sharply growing latency and GPU memory as the sequence lengthens. MemoryWAM proposes a hybrid memory design — recent frames, event-boundary anchor frames, and gist tokens that compress long-range history — reducing the time and space complexity of inference from O(N) to O(N/d), while surpassing strong VLA/WAM baselines on both simulated and real-robot tasks.
真实世界中的机器人操作不仅需要理解当前观测，还需要记忆与动态建模。现有 WAM 在高效推理（短窗口）和长历史保留（全 KV 缓存）之间存在不可调和的权衡，导致非 Markov 任务中两类方法均存在显著缺陷。Real-world robot manipulation requires not only understanding the current observation, but also memory and dynamics modeling. Existing WAMs face an irreconcilable trade-off between efficient inference (short window) and long-history retention (full KV cache), so both families show significant deficiencies on non-Markovian tasks.
"Existing WAMs face a fundamental trade-off: methods with efficient inference typically condition only on a bounded window of recent observations and therefore struggle in non-Markovian environments, whereas methods that preserve long histories incur time and space costs that grow substantially with sequence length."
Figure 1 · 总览。滑动窗口方法（左）推理为 O(1)，但丢失长程上下文；全 KV 缓存方法（中）保留完整历史但推理随轨迹长度 N 线性增长。MemoryWAM（下）引入 hybrid memory：近期帧（短期记忆）+ 锚帧（事件边界）+ gist token（长程摘要），复杂度从 O(N) 降至 O(N/d)。右侧散点图显示 MemoryWAM 在 RMBench 上达到最高成功率且推理延迟最低。Figure 1 · Overview. Sliding-window methods (left) infer in O(1) but lose long-range context; full KV cache methods (middle) preserve the complete history, yet their inference grows linearly with trajectory length N. MemoryWAM (bottom) introduces a hybrid memory: recent frames (short-term memory) + anchor frames (event boundaries) + gist tokens (long-range summary), lowering complexity from O(N) to O(N/d). The scatter plot on the right shows that MemoryWAM reaches the highest success rate on RMBench with the lowest inference latency.
83.0%RMBench 平均成功率（Ours）RMBench average success rate (Ours)
MemoryWAM 以 Mixture-of-Transformers (MoT) 架构为主干，将 video DiT（视觉动态建模）与 action DiT（动作预测）双路并行，并引入三层混合记忆机制，通过定制化注意力掩码实现高效持久上下文检索。MemoryWAM uses a Mixture-of-Transformers (MoT) architecture as its backbone, running a video DiT (visual dynamics modeling) and an action DiT (action prediction) as two parallel pathways, and introduces a three-tier hybrid memory mechanism that achieves efficient persistent context retrieval through a customized attention mask.
Figure 2 · 系统架构。MemoryWAM 由 video DiT (Φ_v) 与 action DiT (Φ_a) 构成 MoT 架构。训练时 video DiT 通过视频预测提供稠密监督；推理时仅前向一次当前帧更新 KV 缓存，无需生成未来帧。持久记忆保留初始锚帧的完整 token 与最近帧，并将长程历史压缩为少量 gist token。Figure 2 · System architecture. MemoryWAM forms an MoT architecture from a video DiT (Φ_v) and an action DiT (Φ_a). During training, the video DiT provides dense supervision via video prediction; at inference, only a single forward pass over the current frame updates the KV cache, with no need to generate future frames. Persistent memory keeps the full tokens of the initial anchor frame together with the most recent frames, and compresses long-range history into a small number of gist tokens.
混合记忆（Hybrid Memory）三层设计Three-Tier Design of Hybrid Memory
保留最近 N 帧的完整视觉 token，为当前动作规划提供高保真短期上下文。计算量固定为 O(N)（N 为滑窗大小，而非轨迹长度）。The full visual tokens of the most recent N frames are kept, providing high-fidelity short-term context for current action planning. The computation is fixed at O(N) (N being the sliding-window size, not the trajectory length).
在任务起始等"事件边界"处保留完整视觉 token 作为锚帧，提供关键初始状态信息。灵感来源于认知心理学中事件边界对记忆的高度显著性。At "event boundaries" such as the start of a task, the full visual tokens are kept as anchor frames, providing critical initial-state information. The idea is inspired by the high salience of event boundaries for memory in cognitive psychology.
长程记忆：Gist TokensLong-Range Memory: Gist Tokens
对超出滑窗的远程帧，用少量（可学习的）gist token 对完整 token 序列进行压缩，将每帧 L 个 token 压缩到 L/d 个（d 为压缩比）。推理时仅保留 gist token 的 KV 缓存，丢弃原始帧缓存，将整体历史复杂度从 O(N) 降至 O(N/d)。For distant frames beyond the sliding window, a small number of (learnable) gist tokens compress the full token sequence, reducing the L tokens per frame to L/d (d being the compression ratio). At inference, only the KV cache of the gist tokens is kept and the original frame cache is discarded, lowering the overall history complexity from O(N) to O(N/d).
Figure 3 · 注意力掩码。以三帧历史、一帧锚帧、一帧近期帧为例：f 为干净视频帧，g 为 gist token，a 为待去噪动作 token。近期帧可全量关注所有历史，远程历史仅通过 gist token 被访问。推理时仅保留 g_i 的 KV 缓存，f_i 的 KV 缓存被驱逐，实现持久记忆与高效推理的统一。Figure 3 · Attention mask. Illustrated with three history frames, one anchor frame and one recent frame: f denotes clean video frames, g denotes gist tokens, and a denotes the action tokens to be denoised. Recent frames may attend to the whole history, while distant history is accessed only through gist tokens. At inference, only the KV cache of g_i is kept and that of f_i is evicted, unifying persistent memory with efficient inference.
训练目标Training Objective
MemoryWAM 继承高效 WAM 的核心优势：训练时通过视频预测任务（预测未来帧 z̃_{t+1:t+k}）提供稠密动态监督，推理时跳过视频生成，仅运行 action DiT 完成动作预测。动作 chunk a_{t:t+h-1} 由 action DiT 对带噪声的动作 token x_τ^a 去噪得到，条件为语言指令 l 和累积视频 KV 缓存 C^v_{≤t}。MemoryWAM inherits the core advantage of efficient WAMs: at training time, a video prediction task (predicting future frames z̃_{t+1:t+k}) provides dense dynamics supervision, while at inference, video generation is skipped and only the action DiT is run to predict actions. The action chunk a_{t:t+h-1} is obtained by the action DiT denoising the noisy action tokens x_τ^a, conditioned on the language instruction l and the accumulated video KV cache C^v_{≤t}.
03 Experiments
在 RMBench（长时序记忆依赖操作基准，9 项任务）和真实双臂机器人（ARX + RealSense D455）上评估，与 π₀.₅、FastWAM、LingBot-VA 对比，每项任务 100 次 rollout 统计成功率。Evaluation is carried out on RMBench (a long-horizon, memory-dependent manipulation benchmark with 9 tasks) and on a real dual-arm robot (ARX + RealSense D455), compared against π₀.₅, FastWAM and LingBot-VA, with success rates measured over 100 rollouts per task.
记忆机制效率对比（Figure 4）Efficiency Comparison of Memory Mechanisms (Figure 4)
Figure 4 · 记忆机制对比。(a) 单次前向延迟随序列长度增长曲线；(b) GPU 显存占用曲线；(c) Press Button 任务成功率。Hybrid memory 在序列长度 1,600 帧时仍比 RNN/TTT-based 方案更高效，且成功率（87%）与 Full Attention 持平。Full Attention 延迟显著更高但成功率不更优。Figure 4 · Comparison of memory mechanisms. (a) Single-forward latency as sequence length grows; (b) GPU memory usage; (c) success rate on the Press Button task. Hybrid memory remains more efficient than RNN/TTT-based schemes even at a sequence length of 1,600 frames, and its success rate (87%) matches Full Attention. Full Attention has markedly higher latency without a better success rate.
短窗口方法（π₀.₅、FastWAM）在记忆依赖任务上大量失败（平均 <11%）。MemoryWAM 平均成功率比全历史 KV 缓存的 LingBot-VA 高 4.8 个百分点，且在每项任务上均达到领先或持平。Short-window methods (π₀.₅, FastWAM) fail extensively on memory-dependent tasks (averaging <11%). The average success rate of MemoryWAM exceeds that of the full-history KV cache model LingBot-VA by 4.8 percentage points, and it leads or ties on every task.
真实机器人实验（Table 2）Real-Robot Experiments (Table 2)
Figure 5 · 真实世界任务示意。左：Shell Game——人手随机交换杯子后，机器人需识别藏有方块的杯子并抓取（需追踪遮挡物体）。右：Look and Press——机器人观察桌上两个数字，依次按对应次数的按钮，最后按下确认键（需计数工作记忆）。Figure 5 · Illustration of the real-world tasks. Left: Shell Game — after a human hand randomly swaps the cups, the robot must identify and grasp the cup hiding the block (requiring tracking of an occluded object). Right: Look and Press — the robot observes two numbers on the table, presses the button the corresponding number of times in turn, and finally presses the confirm key (requiring counting working memory).
任务Task
π₀.₅
LingBot-VA
Ours
Shell Game
5/20
13/20
18/20
Look and Press
0/20
14/20
15/20
真实机器人上 MemoryWAM 两项任务均最优。值得注意的是，LingBot-VA 高推理延迟导致其在 Shell Game 中错过杯子交换时机而失败——印证了效率本身也是操作性能的组成部分。On the real robot, MemoryWAM is the best on both tasks. Notably, the high inference latency of LingBot-VA causes it to miss the moment of the cup swap in Shell Game and fail — confirming that efficiency itself is a component of manipulation performance.
消融实验（Table 3）Ablation Study (Table 3)
去除 gist token 导致最大性能下降，说明长程历史压缩是记忆依赖决策的核心。去除锚帧或滑窗均降低性能，证明三类记忆提供互补收益。Full Attention（保留所有历史）性能反而弱于混合记忆，说明"密集历史并非最优"——冗余信息增加检索难度。消融结论："MemoryWAM's hybrid memory design is not merely an efficiency-oriented compromise, but an effective memory structure."Removing gist tokens causes the largest performance drop, showing that long-range history compression is central to memory-dependent decision making. Removing anchor frames or the sliding window also degrades performance, proving that the three memory types provide complementary gains. Full Attention (keeping all history) is in fact weaker than hybrid memory, indicating that "dense history is not optimal" — redundant information makes retrieval harder. The ablation conclusion: "MemoryWAM's hybrid memory design is not merely an efficiency-oriented compromise, but an effective memory structure."
04 Limitations
Note:本文无显式 Limitations 章节。以下局限性：前两点为论文结论/方法设计中直接推断（inferred from design），第三点基于实验规模。This paper has no explicit Limitations section. Among the limitations below, the first two are inferred directly from the paper's conclusions and method design (inferred from design), while the third is based on the experiment scale.
压缩比 d 是固定超参数（inferred）The compression ratio d is a fixed hyper-parameter (inferred)
gist token 压缩比 d 在训练前固定，无法自适应任务难度或历史重要性动态调整。对于信息密度不均匀的轨迹，固定压缩可能导致部分关键帧被过度压缩或欠压缩。The gist token compression ratio d is fixed before training and cannot be adapted dynamically to task difficulty or to the importance of the history. For trajectories with uneven information density, fixed compression may over-compress or under-compress some key frames.
锚帧选择依赖任务边界先验（inferred）Anchor frame selection relies on a task-boundary prior (inferred)
当前方案以任务初始帧作为锚帧（"initial observations of a task"），需预先知道任务开始时刻。对于无明确起点或多阶段混合的连续操作场景，锚帧定义可能不够通用。The current scheme takes the initial frame of a task as the anchor frame ("initial observations of a task"), which requires knowing the task start time in advance. For continuous manipulation settings without a clear starting point, or mixing multiple stages, the definition of anchor frames may not be general enough.
真实机器人实验规模有限（stated via experiment size）Limited scale of the real-robot experiments (stated via experiment size)
真实机器人实验每任务仅 20 次 rollout，演示数据分别为 50（Shell Game）和 100（Look and Press）条。样本量相对较小，泛化性结论需更大规模验证。The real-robot experiments use only 20 rollouts per task, with 50 (Shell Game) and 100 (Look and Press) demonstrations respectively. The sample size is relatively small, so conclusions about generalization need validation at a larger scale.
