MEM: Multi-Scale Embodied Memory for Vision Language Action Models

01 动机 MotivationMotivation

现有 VLA 模型（如 π₀.₆）在处理单步或短时操作任务时表现出色，但面对需要数分钟乃至十余分钟记忆的长时程任务时，性能急剧下降。它们缺乏一种既能捕捉细粒度局部操作动作、又能跟踪宏观任务状态的记忆机制。Existing VLA models (such as π₀.₆) perform well on single-step or short-horizon manipulation tasks, but their performance degrades sharply on long-horizon tasks that require minutes or more than ten minutes of memory. They lack a memory mechanism that can both capture fine-grained local manipulation actions and track the macroscopic task state.

"Effective robotic memory should operate across multiple levels of abstraction: a short-horizon memory to track recent observations at a fine-grained level, and a long-horizon memory to track the state of a task at a semantic level." — 论文核心论点the paper's core claim

**Figure 0：MEM 系统概览。**MEM 通过两个核心组件为 VLA 赋予长时程记忆：（1）高层策略通过更新语言记忆 mₜ 跟踪长时程语义事件（左侧，语言长时记忆）；（2）低层策略通过高效视频编码器处理短时观测记忆（右侧，视频短时记忆）。**Figure 0: MEM system overview.** MEM endows VLA with long-horizon memory through two core components: (1) the high-level policy tracks long-horizon semantic events by updating the language memory mₜ (left, long-horizon language memory); (2) the low-level policy processes short-horizon observation memory with an efficient video encoder (right, short-horizon video memory).

15 min支持的最长任务记忆时长Longest supported task memory span

54 s短时视频记忆覆盖的观测时长（推理时）Observation span covered by the short-horizon video memory (at inference)

42Recipe Setup 训练菜谱数（5 道菜在未见场景评测）Recipe Setup training recipes (5 dishes evaluated in unseen scenes)

4th每第 4 层插入一次 temporal attentiontemporal attention inserted every 4th layer

核心挑战在于：简单地将多帧图像拼接送入 VLA backbone 会导致推理延迟急剧增加，无法满足实时控制需求；而仅依赖文本记忆则无法捕捉精细的操作细节（如擦拭表面的时间长短、当前抓握状态等）。MEM 通过两级异构记忆的组合解决这一矛盾。The core challenge is that naively concatenating multiple frames into the VLA backbone causes inference latency to rise sharply, failing to meet real-time control requirements; while relying on text memory alone cannot capture fine manipulation details (such as how long a surface has been wiped, or the current grasp state). MEM resolves this tension by combining two heterogeneous levels of memory.

02 方法 MethodMethod

MEM 由两个互补的记忆模块组成：基于高效视频编码器的短时观测记忆（short-horizon video memory）处理近期细粒度观测；基于 LLM 压缩的长时语言记忆（long-horizon language memory）在更长的时间跨度上追踪任务语义状态。两者分别由低层策略和高层策略使用。MEM consists of two complementary memory modules: a short-horizon observation memory (short-horizon video memory) built on an efficient video encoder handles recent fine-grained observations; a long-horizon language memory (long-horizon language memory) based on LLM compression tracks the semantic state of the task over a longer time span. The two are used by the low-level policy and the high-level policy respectively.

短时记忆：高效视频编码器（Short-Horizon Video Memory）Short-Horizon Memory: An Efficient Video Encoder (Short-Horizon Video Memory)

**Figure 2：视频编码器架构。**在标准 ViT 的基础上，每隔第 4 层将空间 attention 替换为同时包含空间与时序上下文的 causal-temporal attention（黑色箭头），其余层保持双向空间 attention（白色箭头）。过去帧的 tokens 在 ViT 上层被丢弃，仅将当前帧的 token 表示传入 VLA backbone，从而压缩 token 数量。**Figure 2: Video encoder architecture.** Building on a standard ViT, every 4th layer replaces spatial attention with causal-temporal attention that carries both spatial and temporal context (black arrows), while the remaining layers keep bidirectional spatial attention (white arrows). Tokens of past frames are discarded in the upper ViT layers, and only the token representation of the current frame is passed to the VLA backbone, thereby compressing the token count.

视频编码器将标准 ViT 扩展为视频输入：在不引入任何新可学习参数的前提下（仅修改 attention 模式 + 固定正弦时序位置编码），每隔第 4 层插入一次 space-time separable attention。该设计将计算复杂度从朴素多帧拼接的 O(n²K²) 降至 O(Kn² + nK²)，其中 K 为帧数，n 为每帧 patch 数。The video encoder extends a standard ViT to video input: without introducing any new learnable parameters (only modifying the attention pattern plus fixed sinusoidal temporal position encodings), space-time separable attention is inserted every 4th layer. This design reduces the computational complexity from O(n²K²) for naive multi-frame concatenation to O(Kn² + nK²), where K is the number of frames and n is the number of patches per frame.

Pre-training：使用 6 帧观测序列（5 帧历史 + 当前帧），帧间隔 1 秒；在机器人遥操作数据、策略 rollout 数据、视频语言任务（视频描述等）的混合数据集上预训练。Pre-training: uses 6-frame observation sequences (5 history frames + the current frame) with a 1-second frame interval; pre-trained on a mixed dataset of robot teleoperation data, policy rollout data, and video-language tasks (video captioning, etc.).
Post-training：灵活扩展至最多 18 帧、覆盖 54 秒的观测记忆，无需重新训练 backbone 结构。Post-training: flexibly scales to at most 18 frames, covering 54 seconds of observation memory, without retraining the backbone architecture.
推理延迟控制：过去帧 tokens 在 ViT 上层被丢弃，只向 VLA backbone 传递当前帧的 patch 表示，保持实时推理吞吐。Inference latency control: tokens of past frames are discarded in the upper ViT layers, and only the patch representation of the current frame is passed to the VLA backbone, preserving real-time inference throughput.

Latency comparison — **Figure 1：推理延迟对比（π₀.₆，4路摄像头，单张 NVIDIA H100 GPU）。**朴素地增加帧数会使推理延迟急剧上升，超出实时控制阈值；MEM 的高效视频编码器在使用多帧历史时仍保持在可接受延迟范围内。**Figure 1: Inference latency comparison (π₀.₆, 4 camera views, a single NVIDIA H100 GPU).** Naively increasing the number of frames makes inference latency rise sharply, exceeding the real-time control threshold; MEM's efficient video encoder stays within an acceptable latency range even when using multi-frame history.

长时记忆：压缩语言记忆（Long-Horizon Language Memory）Long-Horizon Memory: Compressed Language Memory (Long-Horizon Language Memory)

高层策略维护一个自然语言记忆 mₜ，由一个 off-the-shelf 预训练 LLM 在每个子任务结束后负责更新与压缩。LLM 接收子任务指令序列及其成功/失败标记，输出精炼后的语言记忆，去除冗余细节，仅保留对未来决策有用的语义信息。The high-level policy maintains a natural-language memory mₜ, which an off-the-shelf pre-trained LLM updates and compresses at the end of every subtask. The LLM takes the sequence of subtask instructions together with their success/failure flags and outputs a refined language memory, removing redundant details and keeping only the semantic information useful for future decisions.

"Instead of remembering the precise attributes of all objects that were manipulated ('I put a light green bowl, a dark blue bowl and a bright yellow bowl into the top right cabinet'), it is often sufficient to just remember where the bowls were placed ('I placed three bowls in the top right cabinet')." — 论文对语言压缩机制的举例说明the paper's illustration of the language compression mechanism

这种压缩设计还解决了训练-推理分布偏移问题：训练时每个子任务指令通常只出现一次，而推理时策略可能反复失败并重试同一子任务，导致朴素拼接语言指令的方式分布失配。压缩记忆始终维持一个紧凑、语义丰富的历史摘要，避免了这一问题。This compression design also addresses the train-inference distribution shift problem: during training each subtask instruction usually appears only once, whereas at inference the policy may repeatedly fail and retry the same subtask, so naively concatenating language instructions leads to a distribution mismatch. The compressed memory always keeps a compact, semantically rich summary of the history, avoiding this problem.

Long-horizon tasks — **Figure 3：评测任务示例。**MEM 在多个挑战性长时程灵巧操作任务上测试，包括 recipe setup（按菜谱摆放食材）、kitchen cleanup（厨房清理）、grilled cheese sandwich 制作等，任务时长最长可达 15 分钟。**Figure 3: Example evaluation tasks.** MEM is tested on several challenging long-horizon dexterous manipulation tasks, including recipe setup (arranging ingredients according to a recipe), kitchen cleanup, grilled cheese sandwich making, and more, with task durations of up to 15 minutes.

03 实验 ExperimentsExperiments

实验在真实机器人平台上进行，评估 MEM 在长时程任务、in-context 适应、核心记忆能力及非记忆任务（基线对比）四个维度的表现。基线包括：π₀.₆（无记忆）、Pool Memory（历史帧池化）、Proprio Memory（本体感知记忆）以及 Naive Language Memory（未压缩的语言拼接）。Experiments are run on a real robot platform, evaluating MEM along four dimensions: long-horizon tasks, in-context adaptation, core memory capabilities, and non-memory tasks (baseline comparison). Baselines include π₀.₆ (no memory), Pool Memory (pooling over history frames), Proprio Memory (proprioceptive memory), and Naive Language Memory (uncompressed language concatenation).

长时程任务表现（Figure 4）Long-Horizon Task Performance (Figure 4)

Long-horizon task results — **Figure 4：长时程操作任务性能对比。**无记忆的 π₀.₆ 在 recipe setup 和 kitchen cleanup 上成功率极低（约 10-20%）。完整 MEM（短时视频记忆 + 长时语言记忆）显著优于所有消融变体，验证了两级记忆协同的必要性。朴素语言记忆（未压缩）因训练-推理分布偏移而性能受限。**Figure 4: Performance comparison on long-horizon manipulation tasks.** The memoryless π₀.₆ attains very low success rates on recipe setup and kitchen cleanup (about 10-20%). Full MEM (short-horizon video memory + long-horizon language memory) clearly outperforms all ablation variants, confirming that the two levels of memory must work together. Naive language memory (uncompressed) is limited in performance by the train-inference distribution shift.

消融分析揭示了两级记忆各自的贡献：The ablation analysis reveals the respective contributions of the two levels of memory:

去除视频短时记忆：策略难以判断已擦拭表面的时长或盘子是否已拿起，导致性能下降。Removing the short-horizon video memory: the policy struggles to tell how long a surface has been wiped or whether a plate has already been picked up, causing a performance drop.
去除语言长时记忆：策略无法跟踪任务进度（如哪些子任务已完成），成功率进一步下降。Removing the long-horizon language memory: the policy cannot track task progress (such as which subtasks are already done), and the success rate drops further.
朴素语言拼接（无压缩）：面临训练-推理分布偏移，性能介于有无记忆之间。Naive language concatenation (no compression): suffers from the train-inference distribution shift, with performance falling between the memoryless and memory-equipped settings.

In-Context 操作策略适应（Figure 5）In-Context Adaptation of Manipulation Strategies (Figure 5)

In-context adaptation results — **Figure 5：In-context 操作策略适应。**MEM 策略能根据历史失败经验调整操作策略（如调整抓握高度或开门方向）。无记忆的策略则会反复执行相同的失败动作。**Figure 5: In-context adaptation of manipulation strategies.** The MEM policy adapts its manipulation strategy from past failure experience (for example, adjusting the grasp height or the direction in which a door is opened). A memoryless policy instead repeats the same failing action over and over.

核心记忆能力基准（Figure 6）Core Memory Capability Benchmark (Figure 6)

Memory capabilities comparison — **Figure 6：核心记忆能力对比（局部可观测性处理、计数、视觉记忆）。**无记忆的 VLA 在所有记忆任务上均表现不佳（例如从 4 个抽屉中找到物体的成功率接近随机概率 25%）。仅 MEM 在全部核心记忆能力上均表现良好；Pool Memory 和 Proprio Memory 在部分能力上有一定效果，但无法全面覆盖。**Figure 6: Comparison of core memory capabilities (handling partial observability, counting, visual memory).** The memoryless VLA performs poorly on all memory tasks (for example, the success rate of locating an object among 4 drawers is close to the 25% chance level). Only MEM performs well across every core memory capability; Pool Memory and Proprio Memory help on some of them but cannot cover the full range.

非记忆任务上的基线对等（Figure 8）Parity with Baselines on Non-Memory Tasks (Figure 8)

MEM 在不需要记忆的灵巧操作任务（衬衫折叠、搭建积木、收拾餐桌等）上，与 state-of-the-art 无记忆 VLA π₀.₆ 表现持平，表明引入记忆模块不会损害模型的通用操作能力。On dexterous manipulation tasks that need no memory (shirt folding, block building, table bussing, etc.), MEM matches the state-of-the-art memoryless VLA π₀.₆, showing that introducing the memory modules does not harm the model's general manipulation ability.

预训练的重要性（Figure 7）The Importance of Pre-training (Figure 7)

在多样化机器人及非机器人视频数据上进行预训练对 MEM 的记忆利用能力至关重要：仅进行 post-training 而跳过预训练的 MEM，平均任务成功率约为 45%，而完整预训练后约为 75%，差距显著。这表明视频编码器需要从广泛的视频-语言数据中学习通用的时序表征。Pre-training on diverse robot and non-robot video data is crucial for MEM's ability to exploit memory: MEM that only undergoes post-training and skips pre-training reaches an average task success rate of about 45%, versus about 75% after full pre-training, a significant gap. This indicates that the video encoder needs to learn general temporal representations from broad video-language data.

04 局限性 LimitationsLimitations

注：论文正文无独立的 Limitations 章节。以下各点部分来自论文结论段的展望性表述（标注为作者明确提及），其余为从方法设计中合理推断（标注为设计推断）。Note: The paper has no standalone Limitations section. Some points below come from forward-looking statements in the paper's conclusion (marked explicitly mentioned by the authors); the rest are reasonable inferences from the method design (marked design inference).

单 Episode 内的记忆上限（作者明确提及）Memory bounded by a single episode (explicitly mentioned by the authors)

论文结论明确指出：The paper's conclusion states explicitly:"We believe that MEM is only the first step towards building robot policies that can effectively manage very long-horizon memory. Future work can explore how we can scale memory to last beyond the horizon of a single episode, to span weeks, months, or years of deployment." 目前 MEM 的记忆仅覆盖单次任务执行期间，无法跨 episode 积累经验。At present MEM's memory covers only a single task execution and cannot accumulate experience across episodes.

语言记忆依赖高层策略的子任务分割（设计推断）Language memory depends on subtask segmentation by the high-level policy (design inference)

长时语言记忆的更新由高层策略触发，需要子任务级别的语言标注（成功/失败标记）。这对标注质量和高层策略的可靠性有一定依赖，若子任务识别出错，记忆更新可能引入噪声。Updates to the long-horizon language memory are triggered by the high-level policy and require subtask-level language annotations (success/failure flags). This places some dependence on annotation quality and on the reliability of the high-level policy; if subtask recognition goes wrong, memory updates may introduce noise.

视频编码器预训练数据需求（设计推断）Pre-training data requirements of the video encoder (design inference)

实验表明，跳过预训练会导致约 30% 的性能下降。这意味着 MEM 的有效性高度依赖多样化机器人及非机器人视频数据的预训练，在数据受限场景下可能难以复现相同效果。Experiments show that skipping pre-training causes roughly a 30% performance drop. This means MEM's effectiveness depends heavily on pre-training with diverse robot and non-robot video data, and the same results may be hard to reproduce in data-constrained settings.

持续学习能力尚待探索（作者明确提及）Continual learning remains to be explored (explicitly mentioned by the authors)

论文展望未来工作将探索"allow us to build robots that learn continually at deployment time"，表明 MEM 当前版本尚不支持部署时在线学习与记忆更新。The paper's outlook for future work is to explore "allow us to build robots that learn continually at deployment time", indicating that the current version of MEM does not yet support online learning and memory updating at deployment time.