MemER: Scaling Up Memory for Robot Control via Experience Retrieval
通过经验检索为机器人策略赋予长时程视觉记忆Endowing robot policies with long-horizon visual memory via experience retrieval
Ajay Sridhar, Jennifer Pan*, Satvik Sharma, Chelsea Finn · Stanford University · *Equal contribution
当前机器人策略普遍缺乏长时程记忆能力——直接输入长观测序列既计算昂贵，又在分布偏移下容易过拟合。
MemER 提出一个分层策略框架：high-level policy 负责从历史观测中筛选并跟踪任务相关关键帧（keyframes），
再将这些关键帧与最近帧一并传给 low-level VLA 执行具体动作。
在三个需要数分钟记忆的真实长时程操作任务上，MemER 大幅超越无记忆及朴素长上下文基线，并接近人类 high-level oracle 性能。Current robot policies broadly lack long-horizon memory: feeding in long observation sequences directly is computationally expensive and prone to overfitting under distribution shift. MemER proposes a hierarchical policy framework in which a high-level policy selects and tracks task-relevant keyframes from the observation history, and then passes those keyframes together with the most recent frames to a low-level VLA that executes the concrete actions. On three real-world long-horizon manipulation tasks that require minutes of memory, MemER substantially outperforms memoryless and naive long-context baselines and approaches the performance of a human high-level oracle.
现有的机器人操作策略虽然在泛化能力上取得了显著进展，但普遍存在一个关键缺陷——缺乏长时程记忆。
人类执行任务时会自然地依赖记忆（如记住花生酱放在哪个柜子里），机器人却做不到。Existing robot manipulation policies have made remarkable progress in generalization, yet they share one critical flaw: they lack long-horizon memory. Humans naturally rely on memory when carrying out a task (remembering which cabinet the peanut butter is in, for instance), while robots cannot.
"Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift,
while indiscriminate subsampling of history leads to irrelevant or redundant information."
现有的两类解决方向各有缺陷：The two existing families of solutions each have drawbacks:
直接延长观测窗口：计算代价随帧数线性增长，32帧（≈16秒记忆）已使推理延迟逼近1秒上限，远不足以处理数分钟量级的任务；此外，长历史还会引入 covariate shift，导致策略在自身产生的轨迹上泛化失败。Simply extending the observation window: the computational cost grows linearly with the number of frames; 32 frames (≈16 s of memory) already pushes inference latency toward the 1 s ceiling, far short of what minute-scale tasks demand. Long histories additionally introduce covariate shift, so the policy fails to generalize on the trajectories it produces itself.
压缩/下采样历史：均匀采样引入大量无关信息；视觉 trace 等压缩方式难以扩展到需要跨越数百帧的任务。Compressing / subsampling the history: uniform sampling brings in a great deal of irrelevant information, and compression schemes such as visual traces are hard to scale to tasks that span hundreds of frames.
图1 · MemER 总览。
MemER 是一个通过经验检索来扩展机器人控制记忆的框架。利用单个策略（仅需少量专家示范训练），
MemER 能够检索并利用任务相关的历史信息，有效处理三个复杂的长时程任务。Figure 1 · MemER overview. MemER is a framework that scales up memory for robot control via experience retrieval. With a single policy, trained on only a handful of expert demonstrations, MemER retrieves and exploits task-relevant history and effectively handles three complex long-horizon tasks.
>90%三项任务的成功率（MemER）Success rate across the three tasks (MemER)
34%Long History 基线与 MemER 的平均差距Average gap between the Long History baseline and MemER
50每个任务仅需 50 条长时程演示Only 50 long-horizon demonstrations per task
MemER 将策略分解为两层：high-level policy（基于 Qwen2.5-VL-7B-Instruct 微调）负责在每个推理步骤筛选关键帧并预测当前子任务（language subtask）；
low-level policy（基于 π₀.₅ 微调）根据子任务指令执行具体的关节控制动作。
两个策略异步运行：π_h 约 1Hz，π_l 约 2Hz。MemER decomposes the policy into two levels: a high-level policy (fine-tuned from Qwen2.5-VL-7B-Instruct) selects keyframes at every inference step and predicts the current language subtask; a low-level policy (fine-tuned from π₀.₅) executes the concrete joint-control actions according to that subtask instruction. The two policies run asynchronously: π_h at roughly 1Hz and π_l at roughly 2Hz.
图2 · MemER 架构。
High-level policy 接受任务指令、已选关键帧 K_t 和最近 N 帧图像，
输出：① 当前子任务 l'_t，② 候选关键帧 J_t。
候选帧经过 keyframe filter 去冗余后更新选定关键帧集合，供下一步推理使用。
Low-level policy 仅接受当前图像、关节状态和子任务指令，输出动作块（action chunk）。Figure 2 · MemER architecture. The high-level policy takes the task instruction, the selected keyframes K_t and the most recent N frames, and outputs ① the current subtask l'_t and ② the candidate keyframes J_t. After the keyframe filter removes redundancy, the candidates update the set of selected keyframes used by the next inference step. The low-level policy takes only the current image, the joint state and the subtask instruction, and outputs an action chunk.
High-Level Policy：关键帧提名与子任务预测High-Level Policy: Keyframe Nomination and Subtask Prediction
在每个时刻 t，high-level policy 接收三部分输入：At each time step t, the high-level policy receives three kinds of input:
最近 N 帧（每台相机共享同一上下文窗口）：R_t = I_{t−N+1:t}the most recent N frames (all cameras share the same context window): R_t = I_{t−N+1:t}
高层任务指令 l_tthe high-level task instruction l_t
已选关键帧集合 K_t ⊆ I_{0:t−N+1}，实践中 |K_t| ≤ 8the set of already selected keyframes K_t ⊆ I_{0:t−N+1}, with |K_t| ≤ 8 in practice
输出两项：当前子任务 l'_t 和候选关键帧 J_t ⊆ R_t。
这些候选帧将进入 keyframe filter，决定是否纳入长期记忆。It outputs two items: the current subtask l'_t and the candidate keyframes J_t ⊆ R_t. These candidates then enter the keyframe filter, which decides whether they are committed to long-term memory.
Keyframe Filter：1D 单链接聚类去冗余Keyframe Filter: 1D Single-Linkage Clustering for Deduplication
为避免记忆爆炸，MemER 使用一个轻量的在线聚类算法对历次候选帧进行整合。
具体地，将所有候选帧的时间戳汇总为 G_{0:t}，对距离不超过 d 帧的索引进行合并，
形成若干不相交簇（cluster）。每个簇取中位帧作为代表性关键帧。
重复提名的帧会获得更高权重（中位数偏向提名集中处），确保记忆的代表性。
该流程在单台 CPU 上毫秒级完成，不引入额外模型，支持流式实时更新。To keep memory from blowing up, MemER consolidates successive candidate frames with a lightweight online clustering algorithm. Concretely, the timestamps of all candidate frames are pooled into G_{0:t}, and indices no more than d frames apart are merged into disjoint clusters. Each cluster contributes its median frame as the representative keyframe. Repeatedly nominated frames thus carry more weight (the median is pulled toward where nominations concentrate), which keeps the memory representative. The procedure finishes in milliseconds on a single CPU, adds no extra model, and supports streaming real-time updates.
图3 · 关键帧选取的 1D 单链接聚类可视化。
每个时刻，high-level policy 提名候选帧（橙色高亮）。所有候选帧以合并距离 d=5 聚类，
柱高表示某时刻帧被提名的次数，取每簇中位帧加入记忆。Figure 3 · Visualization of the 1D single-linkage clustering used for keyframe selection. At every time step the high-level policy nominates candidate frames (highlighted in orange). All candidates are clustered with a merge distance of d=5; bar height gives how many times the frame at a given time step was nominated, and the median frame of each cluster is added to memory.
关键帧标注与训练Keyframe Annotation and Training
为训练 high-level policy，论文采用半自动标注流程：
首先提取相邻子任务边界帧作为候选，再由人工为每类子任务制定统一规则（取第一帧/最后帧/不取），
最终自动应用于所有演示，每段子任务至多产生一个关键帧标注。
仅需 50 条长时程演示及 10–15 条干预演示即可完成微调。To train the high-level policy, the paper adopts a semi-automatic annotation pipeline: frames at the boundaries between adjacent subtasks are first extracted as candidates, a human then fixes one uniform rule per subtask type (take the first frame / the last frame / none), and that rule is finally applied automatically to all demonstrations, so each subtask segment yields at most one keyframe annotation. Fine-tuning needs only 50 long-horizon demonstrations plus 10–15 intervention demonstrations.
Model Merging：保留预训练泛化性Model Merging: Preserving Pretrained Generalization
纯微调后的 high-level policy 对低层策略出现的重试/冻结行为鲁棒性下降（训练数据全为专家轨迹，缺乏失败恢复样本）。
受 Anonymous (2025) 启发，论文将预训练权重 θ_pre 与微调权重 θ_ft 进行线性插值：
θ = (1−α)·θ_pre + α·θ_ft，其中 α=0.8。
实验表明 model merging 在所有任务上均能维持或提升性能。After plain fine-tuning, the high-level policy becomes less robust to the retry and freeze behaviors of the low-level policy (its training data consists entirely of expert trajectories and lacks failure-recovery examples). Inspired by Anonymous (2025), the paper linearly interpolates the pretrained weights θ_pre with the fine-tuned weights θ_ft: θ = (1−α)·θ_pre + α·θ_ft, with α=0.8. Experiments show that model merging maintains or improves performance on every task.
03 Experiments
实验在真实 Franka 机械臂（DROID 设置，ZED 第三视角相机 + 腕部 miniZED 相机，分辨率 320×180）上开展，
针对三个需要分钟级记忆的长时程任务各进行 20 次试验。Experiments are run on a real Franka arm (DROID setup, a ZED third-person camera plus a wrist-mounted miniZED camera, resolution 320×180), with 20 trials on each of the three long-horizon tasks that demand minute-scale memory.
图4 · 三项评测任务。
每列展示任务的复杂指令、中间子任务及对应关键帧预测。
从左到右：Object Search（搜索并取出指定物体）、Counting Scoops（精准计数舀取）、Dust & Replace（除尘并复位物体）。Figure 4 · The three evaluation tasks. Each column shows a task's complex instruction, its intermediate subtasks and the corresponding keyframe predictions. From left to right: Object Search (find and retrieve the specified object), Counting Scoops (scooping with exact counting), Dust & Replace (dust an object and put it back).
主要结果（Q1：与记忆基线的对比）Main Results (Q1: Comparison with Memory Baselines)
所有方法共享同一 low-level policy，仅 high-level policy 的输入上下文不同：All methods share the same low-level policy; only the input context of the high-level policy differs:
Long History（32帧，≈16秒记忆）仍比 MemER 平均差 34%，且推理延迟接近1秒上限。
MemER 在所有三项任务上成功率均超过 90%，主要失败来源是 low-level policy 执行失误，而非记忆错误。Long History (32 frames, ≈16 s of memory) still trails MemER by 34% on average, and its inference latency approaches the 1 s ceiling. MemER exceeds a 90% success rate on all three tasks; the dominant failure source is execution error in the low-level policy rather than memory error.
Q2：与专有 API VLM 的对比（离线评测）Q2: Comparison with Proprietary API VLMs (Offline Evaluation)
由于 GPT-5 和 Gemini Robotics–ER 1.5 的 API 延迟高达 10–15 秒，无法满足闭环部署的实时要求（<1秒），
论文设计了离线轨迹准确率评测（trajectory accuracy）和子任务切换准确率（boundary accuracy）。Because the API latency of GPT-5 and Gemini Robotics–ER 1.5 reaches 10–15 s, they cannot meet the real-time requirement of closed-loop deployment (<1 s), so the paper designs an offline evaluation of trajectory accuracy and subtask boundary accuracy.
方法Method
Object Search Traj / Boundary
Counting Traj / Boundary
Dust & Replace Traj / Boundary
MemER（本文）MemER (ours)
0.80 / 0.76
0.67 / 0.65
0.87 / 0.86
GPT-5
0.15 / 0.16
0.43 / 0.47
0.67 / 0.63
Gemini Robotics–ER 1.5
0.21 / 0.23
0.13 / 0.14
0.19 / 0.22
零样本 API VLM 的主要失败原因：提名过多无用候选帧，无法识别哪些帧真正重要，
导致子任务预测不准确——说明机器人感知特定的关键帧识别能力需要专门微调。The main failure mode of zero-shot API VLMs: they nominate far too many useless candidate frames and cannot tell which frames really matter, which makes subtask prediction inaccurate — evidence that robot-perception-specific keyframe identification requires dedicated fine-tuning.
Q3：记忆模态对比（图像 vs. 文本）Q3: Comparison of Memory Modalities (Images vs. Text)
论文对比了三种记忆表示方式：① 视觉关键帧（MemER）、② 文本子任务（Short History + Text）、③ 图文混合（MemER + Text）。
结果表明，纯视觉关键帧记忆性能最佳。加入文本后，模型往往过度依赖文本 token，忽视视觉输入；
文本记忆在策略重试/冻结导致轨迹偏离时尤为脆弱，无法捕获物体位置等关键视觉细节（如 Object Search 中已搜索过的 bin）。The paper compares three memory representations: ① visual keyframes (MemER), ② textual subtasks (Short History + Text), ③ a mixture of images and text (MemER + Text). The results show that purely visual keyframe memory performs best. Once text is added, the model tends to over-rely on text tokens and neglect the visual input; textual memory is especially brittle when retries or freezes make the trajectory deviate, and it fails to capture critical visual details such as object locations (for instance which bins have already been searched in Object Search).
消融：Model MergingAblation: Model Merging
在所有三项任务上，将预训练权重（α=0.8）与微调权重融合均能维持或提升性能，
尤其在低层策略出现重试行为时有明显帮助，印证了线性权重插值对抗过拟合的有效性。On all three tasks, merging the pretrained weights (α=0.8) with the fine-tuned weights maintains or improves performance, and helps markedly when the low-level policy exhibits retry behavior, confirming the effectiveness of linear weight interpolation against overfitting.
04 Limitations
Note：以下限制均为论文作者在 Discussion and Future Work 部分明确陈述（stated）。Note: every limitation below is explicitly stated by the authors in the Discussion and Future Work section.
记忆无上限——缺乏关键帧遗忘机制Memory has no upper bound — no keyframe forgetting mechanism
MemER 持续累积信息性关键帧，但目前缺乏在关键帧数量过多时主动丢弃的机制。
对于需要数小时记忆的任务，这一问题可能导致上下文溢出。
论文指出：MemER keeps accumulating informative keyframes, but currently has no mechanism for actively discarding them once there are too many. For tasks that require hours of memory, this could cause context overflow. The paper notes:"enabling the high-level policy to reason about which keyframes to not only add but also delete
for modifiable long-term memory is an exciting direction for future work."
推理吞吐量受 VLM 骨干与调度策略制约Inference throughput is constrained by the VLM backbone and the scheduling policy
π_h 约 1Hz、π_l 约 2Hz 的异步调度已接近可容忍的延迟上限，
无法支持非常高频率的控制（如精细装配任务）。
论文提出 improved model caching 和更好的 tokenization 是进一步降低推理延迟的方向。The asynchronous schedule of π_h at roughly 1Hz and π_l at roughly 2Hz is already close to the tolerable latency ceiling, and cannot support very high-frequency control (such as fine assembly tasks). The paper suggests improved model caching and better tokenization as directions for further cutting inference latency.
记忆仅限于视觉观测Memory is limited to visual observations
当前 MemER 的记忆完全基于图像帧，未纳入触觉、音频等其它感知模态。
论文认为MemER's memory is currently based entirely on image frames and takes in no other sensory modality such as touch or audio. The paper holds that"incorporating other sensory modalities such as tactile or audio is a promising extension."
仅在单一机器人本体 / 单房间场景上验证Validated only on a single robot embodiment / single-room setting
所有实验均在 Franka 手臂 + DROID 台架上进行，未测试移动机器人或跨房间任务。
论文指出扩展到All experiments were carried out on a Franka arm with a DROID rig; mobile robots and cross-room tasks were not tested. The paper notes that extending to"mobile manipulation and multi-room tasks, where memory must interleave spatial mapping
with episodic recall"是使系统更接近人类记忆能力的重要一步。would be an important step toward bringing the system closer to human memory capability.
