A Comprehensive and Challenging Robotic Memory Benchmark
Huashuo Lei, Wenxuan Song, Huarui Zhang, Jieyuan Pei, Jiayi Chen, Haodong Yan, Han Zhao, Pengxiang Ding, Zhipeng Zhang, Lida Huang, Donglin Wang, Yan Wang, Haoang Li
RoboMemArena 是首个系统性评估机器人记忆能力的大规模基准：包含 26 项仿真任务与 5 项真实世界任务，平均轨迹长达 1000+ 步，68.9% 的子任务依赖历史记忆。作者同步提出双系统 VLA 模型 PrediMem，通过分级记忆与 predictive coding 在所有任务类别上显著超越现有方法。RoboMemArena is the first large-scale benchmark that systematically evaluates robotic memory capability: it contains 26 simulation tasks and 5 real-world tasks, with trajectories averaging more than 1000 steps and 68.9% of subtasks depending on historical memory. The authors also propose PrediMem, a dual-system VLA model that significantly surpasses existing methods across all task categories through hierarchical memory and predictive coding.
robotic memory benchmarklong-horizon manipulationVLAPrediMempredictive codingkeyframe bufferdual-system 架构dual-system architecture记忆依赖子任务memory-dependent subtask仿真到现实迁移sim-to-real transfer
01动机 / MotivationMotivation
现有机器人操作基准大多关注单步或短序列动作，对跨时间步记忆能力的评估严重不足。当物体被遮挡、状态发生改变、或任务依赖此前操作结果时，reactive 策略（仅依赖当前观测）将系统性失败——而这恰恰是真实操作场景中的常态。Most existing robotic manipulation benchmarks focus on single-step or short-sequence actions and evaluate cross-timestep memory capability only very inadequately. When objects are occluded, states change, or a task depends on the results of earlier operations, reactive policies (which rely on the current observation alone) fail systematically — yet this is precisely the norm in real manipulation scenarios.
"Existing robotic memory benchmarks suffer from...lack of multimodal annotations...limited task coverage...restricted to simulation."
Figure 1.RoboMemArena 四大任务类别可视化。每行依次展示任务指令、子任务分解与执行轨迹：Multi-Object Counting（计数）、Occlusion（遮挡）、Sequence（序列）、Transferring（转移）。每类任务都要求机器人在长时间跨度内维护并利用记忆状态。Visualization of the four task categories of RoboMemArena. Each row shows, in order, the task instruction, the subtask decomposition and the execution trajectory: Multi-Object Counting, Occlusion, Sequence and Transferring. Every category requires the robot to maintain and exploit memory state over a long time span.
26仿真任务simulation tasks
1000+平均轨迹步数average trajectory steps
68.9%子任务依赖记忆of subtasks depend on memory
8领先于所有对比基准测试的评估维度criteria on which it leads every compared benchmark
Figure 2.RoboMemArena 基准统计。(a) 平均轨迹长度对比先前基准；(b) 任务构成（4 大类分布）；(c) history-dependent subtask ratio——凸显本基准长时域特性与记忆依赖子任务的高占比。RoboMemArena benchmark statistics. (a) Average trajectory length compared with prior benchmarks; (b) task composition (distribution over the 4 major categories); (c) history-dependent subtask ratio — highlighting the long-horizon nature of this benchmark and the high proportion of memory-dependent subtasks.
与已有基准的对比Comparison with Existing Benchmarks
论文在 14 个已有机器人基准上做了系统对比，RoboMemArena 是唯一同时满足以下全部 8 项标准的基准：长时域（>1000 步平均）、自动指令生成、原子子目标、可扩展生成、自主抓取（AnyGrasp）、状态 oracle、原生 keyframe、真实世界评估。The paper systematically compares 14 existing robotic benchmarks, and RoboMemArena is the only one that simultaneously satisfies all 8 of the following criteria: long horizon (>1000 steps on average), automatic instruction generation, atomic subgoals, scalable generation, autonomous grasping (AnyGrasp), state oracle, native keyframes and real-world evaluation.
02方法 / MethodMethod
本文贡献两部分：(1) RoboMemArena 基准的数据生成 pipeline，利用 VLM 驱动任务分解与 AnyGrasp 自主轨迹生成；(2) 用于基准测试的双系统 VLA 模型 PrediMem，结合分级记忆与 predictive coding 机制。This work makes two contributions: (1) the data generation pipeline of the RoboMemArena benchmark, which uses a VLM to drive task decomposition and AnyGrasp for autonomous trajectory generation; (2) PrediMem, a dual-system VLA model used for benchmarking, which combines hierarchical memory with a predictive coding mechanism.
利用 VLM 对每个任务提出有序的可执行子任务序列，自动生成结构化指令标注。A VLM proposes an ordered sequence of executable subtasks for every task and automatically generates structured instruction annotations.
基于两类准则提取 keyframe：物理交互锚点（gripper 状态转变）与运动学拐点（速度极小值、方向变化）。Keyframes are extracted according to two kinds of criteria: physical interaction anchors (gripper state transitions) and kinematic inflection points (velocity minima, direction changes).
评估指标Evaluation Metrics
TSR（Task Success Rate）：所有谓词均满足才算成功。CSR（Cumulative Success Rate）：衡量每项任务通过的 verification stage 比例，量化任务进展。每任务 3–9 个 stage，多数超过 5 个。TSR (Task Success Rate): a task counts as successful only when all predicates are satisfied. CSR (Cumulative Success Rate): measures the fraction of verification stages a task passes, quantifying task progress. Each task has 3–9 stages, most of them more than 5.
PrediMem：双系统 VLAPrediMem: A Dual-System VLA
Figure 3.PrediMem pipeline。系统由两个异步耦合组件构成：S1（低层 action policy，执行当前子任务）与 S2（高层规划器，预测 keyframe 并调度下一子任务）。Memory bank 含 recent buffer（5 帧）与不设上限的 keyframe buffer。PrediMem pipeline. The system consists of two asynchronously coupled components: S1 (a low-level action policy that executes the current subtask) and S2 (a high-level planner that predicts keyframes and schedules the next subtask). The memory bank holds a recent buffer (5 frames) and an unbounded keyframe buffer.
S2 高层规划器S2 High-Level Planner
基于 Qwen3-VL-8B 的高层 VLM 规划器，管理 memory bank（recent buffer + keyframe buffer），负责预测子任务与 keyframe 决策。运行频率约 1.06 Hz，p50 延迟 0.939 秒。A high-level VLM planner based on Qwen3-VL-8B that manages the memory bank (recent buffer + keyframe buffer) and is responsible for subtask prediction and keyframe decisions. It runs at about 1.06 Hz with a p50 latency of 0.939 seconds.
S1 低层执行器S1 Low-Level Executor
以子任务条件生成动作 chunk 的低层 VLA policy，运行频率约 3.40 Hz，每次 S2 更新平均覆盖约 2.92 个 chunk，实现异步流水线。A low-level VLA policy that generates action chunks conditioned on the subtask; it runs at about 3.40 Hz and covers on average about 2.92 chunks per S2 update, realizing an asynchronous pipeline.
Predictive Coding Head
仅在训练阶段使用的辅助损失，提升模型对状态转变的敏感度。损失函数：An auxiliary loss used only during training that increases the model's sensitivity to state transitions. The loss function is:
最优损失权重为 0.1（在 0.0–1.0 消融实验中表现最佳）。Vision tower 冻结，其余模块全量微调，训练 2 个 epoch，使用 4×H100 GPU，学习率 1×10⁻⁵。The optimal loss weight is 0.1 (best performing across the 0.0–1.0 ablation). The vision tower is frozen while the remaining modules are fully fine-tuned, trained for 2 epochs on 4×H100 GPUs with a learning rate of 1×10⁻⁵.
03实验 / ExperimentsExperiments
在 RoboMemArena 的 26 项仿真任务与 5 项真实世界任务上评估 PrediMem，与 π₀.₅、HiF-VLA、MemoryVLA、MemER 以及闭源大模型（Qwen3-VL、GPT）对比。PrediMem is evaluated on the 26 simulation tasks and 5 real-world tasks of RoboMemArena and compared with π₀.₅, HiF-VLA, MemoryVLA, MemER and closed-source large models (Qwen3-VL, GPT).
主要仿真结果（Table 2）Main Simulation Results (Table 2)
方法Method
Transferring TSR / CSR
Occlusion TSR / CSR
Counting TSR / CSR
Sequence TSR / CSR
Avg TSR / CSR
π₀.₅（reactive baseline）π₀.₅ (reactive baseline)
20.0 / 42.8
12.7 / 17.2
14.3 / 50.9
60.0 / 71.6
21.5 / 38.7
HiF-VLA
17.5 / 38.9
12.7 / 27.1
8.6 / 45.9
42.5 / 70.2
16.9 / 39.8
MemoryVLA
15.0 / 37.2
7.3 / 13.1
14.3 / 55.1
37.5 / 65.2
15.0 / 35.3
MemER
20.0 / 36.1
16.4 / 33.2
27.1 / 65.1
65.0 / 79.1
27.3 / 49.1
PrediMem（本文）PrediMem (ours)
22.5 / 45.2
27.3 / 38.4
45.7 / 69.3
72.5 / 89.5
38.5 / 55.2
Ground Truth Oracle
上界参考Upper-bound reference
46.1 / 64.8
闭源基线：Qwen3-VL 冻结 6.0% TSR / 26.2% CSR；GPT-5.4 8.7% TSR / 30.5% CSR——"Closed-source agents transfer poorly to robotic memory-intensive tasks"。Closed-source baselines: frozen Qwen3-VL reaches 6.0% TSR / 26.2% CSR; GPT-5.4 reaches 8.7% TSR / 30.5% CSR — "Closed-source agents transfer poorly to robotic memory-intensive tasks".
真实世界结果（Table 4）Real-World Results (Table 4)
方法Method
Pour×2
Brush
Transfer
Shell Game
IHMB
Avg
π₀.₅
20%
10%
60%
10%
0%
20%
MemER
30%
50%
80%
40%
0%
40%
PrediMem
60%
60%
80%
50%
10%
52%
IHMB（Imitate Human to Make Breakfast）是时长约 3 分钟的最长任务，仅 PrediMem 成功完成。相比 reactive baseline，PrediMem 平均提升真实世界性能 32 个百分点。IHMB (Imitate Human to Make Breakfast) is the longest task, lasting about 3 minutes, and only PrediMem completes it successfully. Compared with the reactive baseline, PrediMem improves real-world performance by 32 percentage points on average.
Figure S2.真实世界任务演示。展示双臂机器人平台上各任务的代表性执行帧，涵盖 Pour Bottle × 2、Brush Plates with Swap、Transfer Objects、Shell Game 与 Imitate Human to Make Breakfast 五类场景。Real-world task demonstrations. Representative execution frames of each task on the bimanual robot platform, covering five kinds of scenario: Pour Bottle × 2, Brush Plates with Swap, Transfer Objects, Shell Game and Imitate Human to Make Breakfast.
消融实验Ablation Studies
Figure 4.记忆行为分析。(a) recent-buffer 大小对平均 CSR 的敏感性（3–5 帧最优）；(b) keyframe-bank 容量影响（不设上限性能最佳）；(c) t-SNE 可视化显示 predictive coding 产生更紧凑、更具判别性的 keyframe cluster。Analysis of memory behavior. (a) Sensitivity of average CSR to recent-buffer size (3–5 frames is optimal); (b) effect of keyframe-bank capacity (unbounded performs best); (c) t-SNE visualization shows that predictive coding produces more compact and more discriminative keyframe clusters.
消融变体Ablation variant
TSR (%)
CSR (%)
w/o Predictive Coding
32.3
49.0
w/o Keyframe Bank
17.7
41.6
PrediMem（完整）PrediMem (full)
38.5
55.2
Predictive coding 单独贡献约 +6.2% TSR；keyframe bank 对记忆依赖任务（尤其是 Occlusion 类）至关重要，移除后 TSR 下降 20.8 个百分点。Predictive coding alone contributes about +6.2% TSR; the keyframe bank is critical for memory-dependent tasks (especially the Occlusion category), and removing it drops TSR by 20.8 percentage points.
规模扩展实验Scaling Experiments
S2 模型规模S2 model scale
TSR (%)
CSR (%)
Qwen3-1.7B
19.9
41.4
Qwen3-4B
31.9
51.7
Qwen3-8B（完整）Qwen3-8B (full)
38.5
55.2
S2 模型规模从 1.7B 扩展至 8B，性能在所有任务类别上一致提升，符合 scaling law 预期。Scaling the S2 model from 1.7B to 8B consistently improves performance across all task categories, in line with scaling-law expectations.
04局限性 / LimitationsLimitations
Note:以下局限性部分来自论文明确陈述（stated），部分由设计本身推断（inferred）。Some of the limitations below are explicitly stated in the paper (stated); others are inferred from the design itself (inferred).
闭源 VLM 在机器人任务上的迁移鸿沟（stated）Transfer gap of closed-source VLMs on robotic tasks (stated)
Qwen3-VL 冻结（6.0% TSR）与 GPT-5.4（8.7% TSR）等闭源大模型尽管具备强大的通用能力，但在记忆密集型机器人操作任务上表现远不如开源微调方案，说明通用视觉-语言能力与具身记忆能力之间存在显著差距。Although closed-source large models such as frozen Qwen3-VL (6.0% TSR) and GPT-5.4 (8.7% TSR) possess strong general capabilities, they perform far worse than open-source fine-tuned solutions on memory-intensive robotic manipulation tasks, which indicates a significant gap between general vision-language capability and embodied memory capability.
真实世界评估规模有限（stated）Limited scale of the real-world evaluation (stated)
真实世界评估仅覆盖 5 项任务，使用双臂平台，尚无法充分代表真实部署场景的多样性。IHMB 任务成功率仅 10%，说明长时域真实操作仍是开放挑战。The real-world evaluation covers only 5 tasks on a bimanual platform and cannot yet adequately represent the diversity of real deployment scenarios. The success rate on the IHMB task is only 10%, showing that long-horizon real-world manipulation remains an open challenge.
专注桌面操作，泛化性待验证（stated + inferred）Focused on tabletop manipulation, with generalization still unverified (stated + inferred)
当前基准与模型均聚焦于 tabletop manipulation，尚未验证能否泛化至移动操作（mobile manipulation）或更复杂的开放世界场景。此外，数据集由仿真自动生成，仿真到现实的 visual gap 可能影响策略在真实世界的泛化。Both the current benchmark and the model focus on tabletop manipulation, and it has not been verified whether they generalize to mobile manipulation or to more complex open-world scenarios. In addition, the dataset is automatically generated in simulation, so the sim-to-real visual gap may affect how well policies generalize in the real world.
