Chameleon：面向视觉运动操作的控制索引前瞻性记忆Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulation
Control-Indexed Prospective Memory for Visuomotor Manipulation
Xinying Guo, Chenxi Jiang, Hyun Bin Kim, Yuhang Han, Ying Sun, Yang Xiao, Jianfei Yang · 南洋理工大学 MARS Lab & A*STAR & 新加坡国立大学Nanyang Technological University MARS Lab & A*STAR & National University of Singapore
机器人在执行操作任务时，往往需要依赖远早于当前决策时刻观察到的历史信息——即"观测-动作延迟"（observation–action delay）问题。Chameleon 提出了一种约 60M 参数的视觉运动策略，通过控制索引前瞻性记忆（control-indexed prospective memory）模块，使机器人能够在正确的时机检索并利用历史轨迹，从而在真实机器人任务中将决策成功率从 22.5% 大幅提升至 80.8%。Robots performing manipulation tasks often have to rely on information observed long before the current decision moment—that is, the "observation–action delay" problem. Chameleon proposes a visuomotor policy of about 60M parameters whose control-indexed prospective memory module lets the robot retrieve and exploit past trajectories at the right moment, raising the decision success rate on real-robot tasks from 22.5% to 80.8%.
现有的记忆增强策略依赖语义相似度或视觉相似度检索历史轨迹，往往召回貌似合理但与当前决策无关的轨迹。这一根本缺陷导致机器人在视觉别名（visual aliasing）严重的任务中频繁犯错——例如"杯子游戏"（shell game）中多个杯子外观相同，历史上哪个杯子藏着球才是关键。Existing memory-augmented policies retrieve past trajectories by semantic or visual similarity, and therefore often recall trajectories that are plausible yet irrelevant to the current decision. This fundamental flaw makes robots err frequently on tasks with severe visual aliasing—for instance, in the shell game several cups look identical, and what matters is which cup hid the ball earlier.
"Memory must be policy-facing—designed to make the right past actionable at the right moment." ——论文核心论点：记忆系统必须面向策略设计，让正确的历史在正确的时刻可用。— The core claim of the paper: a memory system must be designed to face the policy, making the right past available at the right moment.
图1：观测-动作延迟与现有方法的不足。
当前基于语义摘要或视觉相似度的方法可能选取错误的历史轨迹。有效的记忆系统需要三个核心能力：
可分离性（separability）——区分相似历史；
可寻址性（addressability）——检索与控制相关的轨迹；
前瞻性（prospectiveness）——将记忆转换为可直接驱动动作的状态。Figure 1: The observation–action delay and the shortcomings of existing methods. Current methods based on semantic summaries or visual similarity may pick the wrong past trajectory. An effective memory system needs three core capabilities: separability — telling similar histories apart; addressability — retrieving control-relevant trajectories; prospectiveness — turning memory into a state that can directly drive action.
论文将所提三项功能需求类比于人类情景记忆的神经机制：The paper draws an analogy between the three proposed functional requirements and the neural mechanisms of human episodic memory:
可分离性（Separability）Separability
类比海马体齿状回（dentate gyrus）的模式分离机制，保持相似历史的独立表示，避免历史轨迹相互干扰。Analogous to the pattern-separation mechanism of the hippocampal dentate gyrus, it keeps independent representations of similar histories and prevents past trajectories from interfering with one another.
可寻址性（Addressability）Addressability
类比前额叶皮层（PFC）与海马体之间的线索依赖检索机制，以当前控制需求为索引，检索任务相关的历史轨迹。Analogous to the cue-dependent retrieval mechanism between the prefrontal cortex (PFC) and the hippocampus, it uses the current control demand as the index to retrieve task-relevant past trajectories.
前瞻性（Prospectiveness）Prospectiveness
类比前额叶皮层的目标导向预测功能，将历史记忆整合为"动作就绪"的工作状态，而非仅重建过去。Analogous to the goal-directed predictive function of the prefrontal cortex, it integrates past memory into an "action-ready" working state rather than merely reconstructing the past.
非马尔可夫决策Non-Markovian Decision-Making
在"杯子游戏"等任务中，相同的当前观测对应不同的正确动作，取决于历史上未被直接观测的隐变量（如球的位置）。In tasks such as the shell game, the same current observation corresponds to different correct actions, depending on latent variables in the past that were never directly observed (such as the position of the ball).
02方法Method
Chameleon 通过四个核心模块实现控制索引前瞻性记忆：①将多模态观测编码为具身事件 token（embodied event tokens）；②通过选择性状态空间模型（SSM）进行逐 token 轨迹传播；③以学习到的控制索引进行可寻址检索；④将检索到的历史整合为前瞻性策略状态，驱动整流流（rectified-flow）动作生成头。Chameleon realizes control-indexed prospective memory with four core modules: ① encoding multimodal observations into embodied event tokens; ② propagating per-token trajectories with a selective state space model (SSM); ③ performing addressable retrieval with a learned control index; ④ integrating the retrieved history into a prospective policy state that drives the rectified-flow action generation head.
图2：Chameleon 系统总览。
系统按时间步处理多视角图像、本体感知和语言指令，生成具身事件 token，经双层控制索引记忆模块后输出前瞻性策略状态，最终通过整流流策略头生成动作序列。Figure 2: Overview of the Chameleon system. At each time step the system processes multi-view images, proprioception and the language instruction, produces embodied event tokens, passes them through the two-level control-indexed memory module to output a prospective policy state, and finally generates an action sequence with the rectified-flow policy head.
① 具身事件 Token 写入① Embodied Event Token Writing
每个时间步 t，系统将来自多摄像头的视觉补丁（经 DP 风格编码器处理）、本体感知（robot state）以及语言指令（经冻结 DistilBERT 编码）拼接为具身事件 token：At every time step t the system concatenates visual patches from multiple cameras (processed by a DP-style encoder), proprioception (robot state) and the language instruction (encoded by a frozen DistilBERT) into embodied event tokens:Zt⁰ = Concat[Xt¹, ..., Xtᵛ, Prop, Lang]。
每类 token 保持独立，为后续分离性存储奠定基础。. Each token type stays independent, laying the groundwork for the subsequent separability storage.
② 逐 Token 轨迹传播（Separability）② Per-Token Trajectory Propagation (Separability)
使用选择性状态空间模型（SSM）在时间维度上独立传播每条 token 流的轨迹，而非将所有历史压缩为单一递归向量。这确保视觉、本体感知等不同来源的历史各自保持独立表示，实现可分离性。事件绑定步骤（Event Binding）通过自注意力混合器让各类 token 先相互交互，使写入的事件已包含任务与身体的条件化信息。A selective state space model (SSM) propagates the trajectory of every token stream independently along the temporal dimension, instead of compressing all history into a single recurrent vector. This ensures that histories from different sources such as vision and proprioception each keep an independent representation, achieving separability. The Event Binding step first lets the token types interact through a self-attention mixer, so that the events written already carry task- and body-conditioned information.
③ 控制索引与可寻址检索（Addressability）③ Control Index and Addressable Retrieval (Addressability)
模型从本体感知与语言中构建学习到的控制索引（control index），再通过对当前场景证据的注意力加以精化，形成控制上下文（control context）。该上下文以注意力方式查询逐 token 历史轨迹，检索与当前决策相关的历史，实现可寻址性。检索到的轨迹合并为"快"工作状态（fast working state），与"慢"情节级记忆（slow episode-level memory）分离。The model builds a learned control index from proprioception and language, then refines it with attention over evidence from the current scene, forming the control context. This context queries the per-token history trajectories through attention and retrieves the history relevant to the current decision, achieving addressability. The retrieved trajectories are merged into a "fast" working state, kept separate from the "slow" episode-level memory.
④ Control-JEPA 前瞻性训练（Prospectiveness）④ Control-JEPA Prospective Training (Prospectiveness)
区别于重建过去图像或状态的训练目标，Control-JEPA 让当前工作状态预测未来的控制上下文：Unlike training objectives that reconstruct past images or states, Control-JEPA makes the current working state predict the future control context:
ûₜ₊ₖ = gθ([hₜ, ηₖ])
目标为后续策略步真实使用的控制上下文，使记忆具有前瞻性和"动作就绪"特性。训练损失采用 smooth-L1 对齐（带 stop-gradient 目标）与跨多个预测视野 {1, 2, 4, 8, 16, 32} 的方差正则化。动作生成头使用整流流（rectified-flow），以无噪初始化 Aτ = (1−τ)A₀ + τA* 进行训练。The target is the control context actually used at the subsequent policy step, which gives the memory prospectiveness and an "action-ready" character. The training loss uses smooth-L1 alignment (with a stop-gradient target) together with variance regularization across multiple prediction horizons {1, 2, 4, 8, 16, 32}. The action generation head uses rectified flow, trained with the noise-free initialization Aτ = (1−τ)A₀ + τA*.
图5（补充）：真实机器人实验平台。
6自由度 UR5 机械臂，配备自适应夹爪与 22 个标注摄像头视角，用于 Camo-Dataset 数据采集。Figure 5 (supplementary): The real-robot experimental platform. A 6-DoF UR5 arm equipped with an adaptive gripper and 22 annotated camera views, used to collect the Camo-Dataset.
03实验Experiments
实验在三类基准上评估 Chameleon：（1）自建真实机器人数据集 Camo-Dataset，专门测试观测-动作延迟下的非马尔可夫决策；（2）公开仿真基准 LIBERO-10、MemoryBench、MIKASA-Robo；（3）消融研究与机制探针，验证三项功能性质。The experiments evaluate Chameleon on three families of benchmarks: (1) the self-collected real-robot dataset Camo-Dataset, built specifically to test non-Markovian decision-making under observation–action delay; (2) the public simulation benchmarks LIBERO-10, MemoryBench and MIKASA-Robo; (3) ablation studies and mechanistic probes that verify the three functional properties.
真实机器人：Camo-DatasetReal Robot: Camo-Dataset
Camo-Dataset 包含三类任务：清洁指定盘子（Clean a specified plate）、杯子游戏（Play shell game）、添加调料（Add various seasonings）。评估指标包括决策成功率（DSR）和总体成功率（SR）。Camo-Dataset contains three families of tasks: clean a specified plate, play shell game, and add various seasonings. The evaluation metrics include the decision success rate (DSR) and the overall success rate (SR).
方法Method
平均 DSRMean DSR
平均 MSRMean MSR
平均 SRMean SR
Diffusion Policy
22.5%
—
21.3%
Chameleon（完整）Chameleon (full)
80.8%
86.1%
71.3%
w/o memory
20.4%
64.8%
17.6%
w/o Control-JEPA
71.6%
72.2%
52.8%
w/o control index
48.8%
56.5%
26.0%
公开基准测试Public Benchmarks
基准Benchmark
Chameleon
最强基线Strongest baseline
备注Notes
MemoryBench（专项策略）MemoryBench (specialist policy)
97.3% ±4.5
SAM2Act+ 94.3%
3个任务，各100条演示3 tasks, 100 demonstrations each
LIBERO-10（混合策略）LIBERO-10 (mixed policy)
87.1% ±0.8
MemoryVLA 93.4%*
10个任务，各50条演示10 tasks, 50 demonstrations each
MIKASA-Robo（混合策略）MIKASA-Robo (mixed policy)
75.1% ±1.4
GMP 67.8%
5个非马尔可夫任务5 non-Markovian tasks
MIKASA-Robo（专项策略）MIKASA-Robo (specialist policy)
95.6% ±1.0
DP-VPWEM 86.5%
2个任务设置2 task settings
* MemoryVLA 为 7B 参数的大型视觉-语言-动作模型；Chameleon 仅 ~60M 可训练参数。* MemoryVLA is a large 7B-parameter vision-language-action model; Chameleon has only about 60M trainable parameters.
图3：评估基准概览。
左侧为公开长视野与记忆基准；右侧为 Camo-Dataset 真实机器人任务。星号（*）标注非马尔可夫决策阶段，即正确动作取决于当前观测之前的历史变量。Figure 3: Overview of the evaluation benchmarks. On the left are the public long-horizon and memory benchmarks; on the right are the Camo-Dataset real-robot tasks. An asterisk (*) marks non-Markovian decision stages, where the correct action depends on history variables preceding the current observation.
消融分析Ablation Analysis
消融实验清晰揭示各模块贡献：去掉记忆模块后 DSR 从 80.8% 跌至 20.4%，接近随机水平；去掉控制索引后 DSR 降至 48.8%，说明基于任务需求的可寻址检索至关重要；去掉 Control-JEPA 训练目标后 DSR 降至 71.6%，验证了前瞻性训练对于将记忆转化为动作就绪状态的价值。The ablations clearly reveal the contribution of each module: removing the memory module drops DSR from 80.8% to 20.4%, close to chance level; removing the control index drops DSR to 48.8%, showing that addressable retrieval driven by the task demand is essential; removing the Control-JEPA training objective drops DSR to 71.6%, confirming the value of prospective training for turning memory into an action-ready state.
机制探针（Mechanistic Probes）Mechanistic Probes
图4：三项功能性质的机制探针实验。
(a) 可分离性：记忆表示对隐变量的解码准确率（83.3%）显著高于当前观测（46.7%）；
(b) 可寻址性：反事实轨迹编辑实验——替换相关轨迹使选择准确率从 93% 降至 87%，移除相关轨迹则降至 40%；
(c) 前瞻性：有无 Control-JEPA 训练目标时，未来控制信息的解码对比。Figure 4: Mechanistic probe experiments for the three functional properties. (a) Separability: the accuracy of decoding the latent variable from the memory representation (83.3%) is markedly higher than from the current observation (46.7%); (b) Addressability: a counterfactual trajectory-editing experiment—replacing the relevant trajectory lowers selection accuracy from 93% to 87%, while removing the relevant trajectory lowers it to 40%; (c) Prospectiveness: a comparison of how future control information is decoded with and without the Control-JEPA training objective.
04局限性Limitations
说明：论文包含对局限性与未来方向的明确讨论。以下各条均为作者明确陈述（stated），少量细节标注为推断（inferred）。Note: The paper contains an explicit discussion of limitations and future directions. Every item below is stated explicitly by the authors, with a few details marked as inferred.
研究范围局限于片段级模仿策略The scope is limited to episode-level imitation policies
本工作专注于"片段级模仿策略中的控制索引前瞻性记忆"（episode-level imitation policies）。控制索引记忆如何在不同具身形态、传感器布局和任务类型之间扩展，仍是开放性问题。作者指出这是自然的后续研究方向。This work focuses on control-indexed prospective memory in episode-level imitation policies. How control-indexed memory scales across different embodiments, sensor layouts and task types remains an open question. The authors note that this is a natural direction for follow-up research.
LIBERO-10 上的残余误差主要源于执行层面Residual error on LIBERO-10 stems mainly from execution
在 LIBERO-10 基准上，剩余的失败案例主要是执行层面的错误（不稳定的抓取、不精确的放置），而非记忆检索失败。这表明动作生成策略本身的精度仍有提升空间，与记忆模块相对解耦。（stated）On the LIBERO-10 benchmark the remaining failure cases are mainly execution-level errors (unstable grasps, imprecise placement) rather than memory-retrieval failures. This suggests that the precision of the action generation policy itself still has room to improve, relatively decoupled from the memory module. (stated)
尚未实现基础模型规模的跨任务/跨具身泛化Cross-task / cross-embodiment generalization at foundation-model scale is not yet achieved
论文展望将控制索引记忆作为可复用模块嵌入基础级机器人策略，支持跨任务、跨具身的泛化迁移。当前工作尚未在此规模上验证。（stated 作为未来方向）The paper envisions embedding control-indexed memory as a reusable module in foundation-level robot policies, supporting generalization and transfer across tasks and across embodiments. The current work has not verified this at that scale. (stated as a future direction)
主动感知与前瞻记忆的结合尚未探索Combining active perception with prospective memory remains unexplored
作者指出，将前瞻性记忆与主动感知（active perception）相结合——让机器人在信息变得对决策至关重要之前主动获取证据——是一个有前景的扩展方向，但当前工作中未涉及。（stated 作为未来方向）The authors note that combining prospective memory with active perception — letting the robot actively gather evidence before that information becomes critical to a decision — is a promising direction for extension, but it is not addressed in the current work. (stated as a future direction)
推断：对训练数据量与演示质量的依赖Inferred: dependence on training data volume and demonstration quality
推断（inferred）：Chameleon 以监督模仿学习为基础，其性能可能随演示数据质量与数量的变化而波动。Control-JEPA 的多视野预测目标同样依赖足量的历史序列，在低数据量场景下的表现尚不明确。Inferred: Chameleon is built on supervised imitation learning, so its performance may fluctuate with the quality and quantity of demonstration data. The multi-horizon prediction objective of Control-JEPA likewise depends on sufficiently long history sequences, and its behaviour in low-data regimes remains unclear.
