Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation
Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, Gao Huang
主流 VLA 模型仅依赖当前帧，无法处理需要历史信息的长时程操作任务。MemoryVLA 受认知神经科学启发，引入感知-认知双流记忆库（PCMB），通过检索、门控融合与语义合并三步机制，让机器人策略拥有"工作记忆"与"情节记忆"，在多项仿真与真实世界基准上取得显著提升。Mainstream VLA models rely only on the current frame and cannot handle long-horizon manipulation tasks that require historical information. Inspired by cognitive neuroscience, MemoryVLA introduces a perceptual-cognitive dual-stream memory bank (PCMB) whose three-step mechanism of retrieval, gated fusion and semantic consolidation gives the robot policy both "working memory" and "episodic memory", yielding marked gains on a range of simulation and real-world benchmarks.
机器人操作天然具有非马尔可夫性：早期动作决定后续状态，单帧观测往往不足以完成任务。然而，当前主流 VLA 模型均依赖当前帧，忽略了时序依赖，在需要历史记忆的任务（如顺序按按钮、记忆颜色后取物）上表现不佳。Robotic manipulation is inherently non-Markovian: early actions determine later states, and a single-frame observation is often not enough to finish the task. Yet current mainstream VLA models all rely on the current frame and ignore temporal dependency, performing poorly on tasks that require historical memory (such as pushing buttons in order, or memorizing a color before fetching an object).
"Robotic manipulation is inherently non-Markovian, and earlier actions influence later decisions, calling for temporal modeling."
朴素方案——直接拼接多帧——面临两大障碍： （1）自注意力的二次复杂度严重限制了可用的时序上下文长度； （2）多帧输入与模型在单帧机器人数据上预训练的分布不匹配。The naive solution—simply concatenating multiple frames—faces two major obstacles: (1) the quadratic complexity of self-attention severely limits the usable length of temporal context; (2) multi-frame input does not match the distribution of the single-frame robot data the model was pretrained on.
图 1. （a）Push Buttons 任务中，按键前后外观几乎相同，仅凭当前帧无法判断是否已按下；（b）人类双记忆系统：工作记忆处理即时控制，海马体保存情节细节；（c）MemoryVLA 的感知-认知记忆库（PCMB）框架示意；（d）与 baseline 的性能对比。Figure 1. (a) In the Push Buttons task the scene looks almost identical before and after a press, so the current frame alone cannot tell whether the button has been pushed; (b) the human dual-memory system: working memory handles immediate control while the hippocampus stores episodic detail; (c) sketch of the perceptual-cognitive memory bank (PCMB) framework of MemoryVLA; (d) performance comparison against the baseline.
71.9%SimplerEnv-Bridge 成功率 +14.6 pts over CogACTSimplerEnv-Bridge success rate +14.6 pts over CogACT
83%真实长时程任务成功率 +26 pts over CogACTReal-world long-horizon task success rate +26 pts over CogACT
96.5%LIBERO 整体成功率 +3.3 pts over CogACTLIBERO overall success rate +3.3 pts over CogACT
+3.6%推理延迟增量 0.194 s @ RTX 4090Inference latency overhead 0.194 s @ RTX 4090
02方法Method
MemoryVLA 构建了一套"认知-记忆-动作"（Cognition-Memory-Action）流水线：7B 预训练 VLM 提取当前帧的感知与认知表征，PCMB 从历史中检索并融合相关记忆，扩散 Transformer（DiT）基于富含历史信息的表征预测未来 16 步动作序列。MemoryVLA builds a "Cognition-Memory-Action" pipeline: a 7B pretrained VLM extracts the perceptual and cognitive representations of the current frame, PCMB retrieves and fuses the relevant memory from history, and a diffusion Transformer (DiT) predicts the future 16-step action sequence from the history-enriched representation.
图 2. 整体架构。 RGB 观测与语言指令经 7B Prismatic VLM 编码为感知 token（SE-bottleneck 压缩至 256 维）和认知 token（EOS 位置输出）；双流工作记忆作为查询，从 PCMB 检索历史信息；门控融合后更新记忆库；最后送入 DiT 生成动作序列。Figure 2. Overall architecture. The RGB observation and the language instruction are encoded by a 7B Prismatic VLM into perceptual tokens (compressed to 256 dimensions by an SE-bottleneck) and a cognitive token (the output at the EOS position); the dual-stream working memory acts as the query that retrieves historical information from PCMB; the memory bank is updated after gated fusion; finally the result is fed to the DiT to generate the action sequence.
Stage 1 · Vision-Language Cognition Module
系统采用基于 Open-X Embodiment 数据预训练的 7B Prismatic VLM。视觉编码器并联 DINOv2 和 SigLIP backbone，对当前第三视角 RGB 图像编码。感知压缩模块（SE-bottleneck）将视觉 token 压缩为紧凑的感知 token p ∈ ℝNp×dp（Np=256）。原始视觉 token 投影至语言嵌入空间后与分词后的指令拼接，送入 LLaMA-7B，"output at the end-of-sentence (EOS) position is taken as the cognitive token c ∈ ℝ1×dc"，用于捕捉高层语义。The system adopts a 7B Prismatic VLM pretrained on Open-X Embodiment data. The visual encoder runs a DINOv2 and a SigLIP backbone in parallel to encode the current third-person RGB image. The perceptual compression module (SE-bottleneck) compresses the visual tokens into compact perceptual tokens p ∈ ℝNp×dp (Np=256). The raw visual tokens are projected into the language embedding space, concatenated with the tokenized instruction and fed to LLaMA-7B, where "output at the end-of-sentence (EOS) position is taken as the cognitive token c ∈ ℝ1×dc", capturing high-level semantics.
Stage 2 · Perceptual-Cognitive Memory Bank (PCMB)
图 3. PCMB 三大机制。 （a）带时步位置编码的 cross-attention 检索；（b）门控融合：当前 token 与检索结果自适应加权；（c）语义合并：容量满时合并余弦相似度最高的相邻条目，而非 FIFO 丢弃。Figure 3. The three mechanisms of PCMB. (a) cross-attention retrieval with timestep positional encoding; (b) gated fusion: the current token and the retrieved result are weighted adaptively; (c) semantic consolidation: when capacity is full, the adjacent entries with the highest cosine similarity are merged instead of being dropped FIFO.
PCMB 维护两条并行流：感知流存储低级视觉细节，认知流存储高级语义，每条最多保留 L 个条目。PCMB maintains two parallel streams: the perceptual stream stores low-level visual detail and the cognitive stream stores high-level semantics, each keeping at most L entries.
Memory Retrieval：当前工作记忆 token 作为双查询，通过缩放点积注意力检索历史信息。每个记忆条目附加基于 sinusoidal embedding 的时步位置编码 TE(·)，使模型感知时序先后。Memory Retrieval: the current working-memory tokens serve as dual queries and retrieve historical information through scaled dot-product attention. Every memory entry carries a timestep positional encoding TE(·) based on a sinusoidal embedding, so that the model perceives temporal order.
Memory Gate Fusion：检索到的嵌入通过可学习门控与当前表征融合：x̃ = gx ⊙ Hx + (1−gx) ⊙ x，σ 为 sigmoid，⊙ 为逐元素乘法。Memory Gate Fusion: the retrieved embedding is fused with the current representation through a learnable gate: x̃ = gx ⊙ Hx + (1−gx) ⊙ x, where σ is the sigmoid and ⊙ is element-wise multiplication.
Memory Consolidation：容量达到 L 时，在每条流内计算相邻条目的余弦相似度，合并相似度最高的一对（取均值），而非简单的先进先出（FIFO），保留最多样化的历史信息。Memory Consolidation: once capacity reaches L, the cosine similarity of adjacent entries is computed within each stream and the most similar pair is merged (by averaging) rather than simply discarded first-in-first-out (FIFO), which preserves the most diverse historical information.
Stage 3 · Memory-Conditioned Action Expert
动作专家采用基于 DDIM 的扩散 Transformer（DiT），推理时执行 10 步去噪，生成未来 T=16 步的动作序列 {a₁, …, a₁₆}。在每个去噪步骤中，含噪动作 token 注入去噪时步的 sinusoidal 编码，并与富含记忆的认知表征拼接，实现历史条件下的精确动作预测。The action expert is a DDIM-based diffusion Transformer (DiT); at inference it runs 10 denoising steps to generate the future T=16-step action sequence {a₁, …, a₁₆}. At every denoising step the noisy action tokens are injected with the sinusoidal encoding of the denoising timestep and concatenated with the memory-enriched cognitive representation, enabling accurate history-conditioned action prediction.
03实验Experiments
实验涵盖 4 个仿真基准（SimplerEnv-Bridge、SimplerEnv-Fractal、LIBERO、Mikasa-Robo）和 2 套真实机器人任务（Franka + WidowX，共 12 项），对比 CogACT、π₀、OpenVLA、Octo、TraceVLA、RoboVLMs 等多个强 baseline。The experiments span 4 simulation benchmarks (SimplerEnv-Bridge, SimplerEnv-Fractal, LIBERO, Mikasa-Robo) and 2 suites of real-robot tasks (Franka + WidowX, 12 in total), compared against strong baselines including CogACT, π₀, OpenVLA, Octo, TraceVLA and RoboVLMs.
长时程任务中单任务最大提升：Seq. Push Buttons +43 pts over CogACT；Change Food +38 pts；Guess Where +32 pts。Largest single-task gains among the long-horizon tasks: Seq. Push Buttons +43 pts over CogACT; Change Food +38 pts; Guess Where +32 pts.
图 10. 记忆检索可视化。 真实世界"Change Food"任务和仿真"Shell Game Touch"任务中，注意力权重高亮了能消解当前决策歧义的关键历史帧——说明模型确实学到了有意义的时序检索，而非随机查找。Figure 10. Visualizing memory retrieval. In the real-world "Change Food" task and the simulated "Shell Game Touch" task, the attention weights highlight exactly those key historical frames that resolve the ambiguity of the current decision—showing that the model has indeed learned meaningful temporal retrieval rather than random lookup.
消融实验Ablation studies
以 SimplerEnv-Bridge 成功率为指标（括号内为与完整模型 71.9% 的对比）：Measured by SimplerEnv-Bridge success rate (values in parentheses are relative to the full model at 71.9%):
记忆流类型：仅认知流 63.5%（−8.4）；仅感知流 64.6%（−7.3）；双流完整 71.9%——两流互补，缺一不可。Memory stream type: cognitive stream only 63.5% (−8.4); perceptual stream only 64.6% (−7.3); full dual stream 71.9%—the two streams are complementary and neither can be dropped.
检索位置编码：去掉时步 PE 后下降至 69.8%，说明时序位置感知有效。Retrieval positional encoding: removing the timestep PE drops performance to 69.8%, showing that temporal position awareness is effective.
融合方式：加法融合 67.7% vs. 门控融合 71.9%。Fusion scheme: additive fusion 67.7% vs. gated fusion 71.9%.
合并策略：FIFO 丢弃 66.7% vs. 语义合并 71.9%。Consolidation strategy: FIFO discarding 66.7% vs. semantic consolidation 71.9%.
推理效率（RTX 4090）Inference efficiency (RTX 4090)
延迟 0.194 s（较 baseline 增加 +3.6%）；吞吐 82.5 Hz；GPU 显存 16.6 GB（+0.8 GB）。计算开销极小，对实时部署友好。Latency 0.194 s (+3.6% over the baseline); throughput 82.5 Hz; GPU memory 16.6 GB (+0.8 GB). The computational overhead is tiny and friendly to real-time deployment.
鲁棒性与泛化Robustness and generalization
真实世界 OOD 评测（Pick Place Order + Clean Restaurant Table）中，在未见背景（92–100%）、干扰物（86–92%）、光照（94–96%）、未见物体/容器（89–100%）、遮挡（94–96%）等条件下均保持高成功率。In the real-world OOD evaluation (Pick Place Order + Clean Restaurant Table), high success rates are kept under unseen backgrounds (92–100%), distractors (86–92%), lighting (94–96%), unseen objects/containers (89–100%) and occlusion (94–96%).
04局限性Limitations
注：论文未设独立的 Limitations 章节。以下第 1 条为作者在结论部分明确指出的未来方向（stated），第 2–3 条为从设计与实验结果推断（inferred）。Note: the paper has no dedicated Limitations section. Item 1 below is a future direction the authors state explicitly in the conclusion (stated); items 2–3 are inferred from the design and the experimental results (inferred).
未探索记忆反思与长期记忆（stated）Memory reflection and long-term memory left unexplored (stated)
作者指出两个未来方向："(i) developing memory reflection, aligning long-term memory to the LLM input space to enable embedding-space chain-of-thought reasoning; and (ii) building lifelong memory through biologically inspired consolidation that distills frequently reused experiences into permanent representations."——当前 PCMB 仅在单次 episode 内维护记忆，无法跨 episode 积累经验。The authors point out two future directions: "(i) developing memory reflection, aligning long-term memory to the LLM input space to enable embedding-space chain-of-thought reasoning; and (ii) building lifelong memory through biologically inspired consolidation that distills frequently reused experiences into permanent representations."—the current PCMB maintains memory only within a single episode and cannot accumulate experience across episodes.
相机视角泛化能力有限（inferred from ablations）Limited generalization across camera viewpoints (inferred from ablations)
消融实验表明，在"未见相机视角"条件下仿真性能明显下降，说明感知 token 对视角变化仍较敏感，模型的视角不变性有待提升。The ablations show a clear drop in simulation performance under "unseen camera viewpoints", indicating that the perceptual tokens are still sensitive to viewpoint change and that the viewpoint invariance of the model has room to improve.
高度动态场景与实时交互任务（inferred from design）Highly dynamic scenes and real-time interactive tasks (inferred from design)
PCMB 的语义合并策略以余弦相似度为准则，在场景变化极快或存在多个相似状态时可能合并不当；10 步 DDIM 去噪虽已较快（0.194 s），对毫秒级反应要求的任务仍有压力。The semantic consolidation strategy of PCMB uses cosine similarity as its criterion, so it may merge the wrong entries when the scene changes extremely fast or several similar states exist; although 10-step DDIM denoising is already fast (0.194 s), tasks demanding millisecond-level reaction remain a stretch.
