ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver

01 动机Motivation

现有 VLA 模型在视觉感知上存在根本性缺陷——注意力分散，难以精准定位目标操作物体，导致在杂乱场景和精细任务中表现不佳。Existing VLA models have a fundamental flaw in visual perception — attention is dispersed and the target manipulated object is hard to localize precisely, which leads to poor performance in cluttered scenes and fine-grained tasks.

"visual attention is always dispersed" rather than focusing on target objects. Existing visual grounding methods either explicitly input cropped images or output bounding boxes in chain-of-thought fashion, but "do not fundamentally refine the attention allocation."

作者观察到，人眼感知的本质是：眼睛对焦的小区域清晰，周围区域模糊。现有 explicit grounding（输入裁剪图像）和 chain-of-thought grounding（输出 bounding box 坐标）两类方法都无法从根本上改变 LLM 内部的注意力分配。ReconVLA 通过隐式重建凝视区域，让模型在特征层面聚焦目标，而非依赖外部后处理。The authors observe that the essence of human visual perception is: the small region the eye focuses on is sharp, while the surrounding area is blurred. Neither existing explicit grounding (feeding in cropped images) nor chain-of-thought grounding (outputting bounding box coordinates) fundamentally changes the attention allocation inside the LLM. By implicitly reconstructing the gaze region, ReconVLA makes the model focus on the target at the feature level instead of relying on external post-processing.

02 方法Method

ReconVLA 在标准 VLA 骨干（LLaVA-7b，Qwen2-7B LLM + siglip-so400m-patch14-384 视觉编码器）的基础上，新增一个与动作输出并行的重建分支：LLM 同时输出 action tokens 和 reconstructive tokens，后者驱动一个 diffusion transformer denoiser 从噪声中恢复凝视区域的 latent 特征。On top of a standard VLA backbone (LLaVA-7b, a Qwen2-7B LLM plus a siglip-so400m-patch14-384 vision encoder), ReconVLA adds a reconstruction branch that runs in parallel with the action output: the LLM emits action tokens and reconstructive tokens at the same time, and the latter drive a diffusion transformer denoiser that recovers the latent features of the gaze region from noise.

核心损失函数Core Loss Function

总损失同时优化动作预测与视觉重建：The total loss jointly optimizes action prediction and visual reconstruction:

ℒ_ReconVLA = ℒ_VLA^action + ℒ_VLA^visual

其中动作损失为 cross-entropy 监督离散动作 token；视觉重建损失为扩散目标：The action loss is a cross-entropy supervision over discrete action tokens; the visual reconstruction loss is a diffusion objective:

ℒ_VLA^visual(𝒉_R, I′) = 𝔼_{t,ϵ}[||𝒟(𝒛_t; 𝒉_R, t) − ϵ||²]

𝒉_R 为 LLM 输出的 reconstructive tokens，I′ 为凝视区域图像，𝒟 为由 Transformer encoder blocks 构成的 denoiser。通过对重建损失和动作损失同时反向传播，两个分支共享并强化相同的特征空间。𝒉_R denotes the reconstructive tokens output by the LLM, I′ the gaze-region image, and 𝒟 a denoiser built from Transformer encoder blocks. By backpropagating the reconstruction loss and the action loss simultaneously, the two branches share and reinforce the same feature space.

凝视区域（Gaze Region）的含义与作用What the Gaze Region Means and What It Does

凝视区域并非简单的 bounding box 裁剪，而是"the target manipulated region"，其作用有三：(1) 在多物体杂乱场景中聚焦正确目标；(2) 增强对目标物体的细节感知；(3) 在长时序任务中辅助子任务规划。The gaze region is not a simple bounding box crop but "the target manipulated region". It serves three purposes: (1) focusing on the correct target in cluttered multi-object scenes; (2) sharpening perception of the target object's details; (3) assisting subtask planning in long-horizon tasks.

大规模预训练数据集Large-Scale Pre-training Dataset

作者通过在 BridgeData V2、LIBERO 和 CALVIN 数据集上微调 Grounding DINO 获取凝视区域标注，构建了包含超过 100k 条轨迹、200万个样本的配对数据集（原图 + 凝视区域裁剪图），用于预训练重建能力。预训练后再进行任务特定微调。By fine-tuning Grounding DINO on the BridgeData V2, LIBERO and CALVIN datasets to obtain gaze-region annotations, the authors build a paired dataset (original image + cropped gaze region) containing over 100k trajectories and 2 million samples, used to pre-train the reconstruction capability. Pre-training is followed by task-specific fine-tuning.

03 实验Experiments

在 CALVIN 模拟基准（Franka Panda 机器人，34 种任务，4 个场景）和真实机器人平台（AgileX PiPer 6-DoF 机械臂）上进行评测，指标为 500 次 rollout 的逐子任务成功率及平均完成序列长度。Evaluation is carried out on the CALVIN simulation benchmark (a Franka Panda robot, 34 tasks, 4 scenes) and on a real-robot platform (a 6-DoF AgileX PiPer arm), with per-subtask success rate over 500 rollouts and average completed sequence length as metrics.

Paradigm 对比（CALVIN ABC→D，Table 1）Paradigm Comparison (CALVIN ABC→D, Table 1)

CoT Grounding "performance is even worse" 因为仅输出 bounding box 坐标不足以提供精细操作所需的精度。ReconVLA 在所有子任务长度上均取得最高成功率。CoT Grounding "performance is even worse", because emitting only bounding box coordinates does not provide the precision that fine-grained manipulation requires. ReconVLA attains the highest success rate at every subtask length.

与 State-of-the-Art 对比（CALVIN ABC→D，Table 3）Comparison with State-of-the-Art (CALVIN ABC→D, Table 3)

ReconVLA 在 5/5 子任务上超越 GR-1 超过 20%、超越 OpenVLA 20.6%、超越 UniVLA 7.6%。在 CALVIN ABCD→D 上也取得平均序列长度 4.23，与 GR-1 的 4.21 相当，超越 RoboFlamingo（4.08）和 VLAS（3.70）。ReconVLA surpasses GR-1 by more than 20% on the 5/5 subtask, OpenVLA by 20.6% and UniVLA by 7.6%. On CALVIN ABCD→D it also reaches an average sequence length of 4.23, on par with GR-1's 4.21 and ahead of RoboFlamingo (4.08) and VLAS (3.70).

消融实验（Table 2）Ablation Study (Table 2)

消融显示：凝视区域（gaze region）是核心贡献，"proves to be more effective" 于全图重建；预训练"substantially enhances generalization"尤其在长序列任务上。精细操作任务 "stack block" 中，ReconVLA 以 79.5% 胜过 baseline 的 59.3%，提升 20.2%。The ablation shows that the gaze region is the core contribution and "proves to be more effective" than full-image reconstruction; pre-training "substantially enhances generalization", especially on long-horizon tasks. On the fine-grained "stack block" task, ReconVLA reaches 79.5% against the baseline's 59.3%, an improvement of 20.2%.

真实机器人实验Real-Robot Experiments

在四项已知任务上，ReconVLA 的成功率均明显优于 OpenVLA 和 PD-VLA（Put Fruit 和 Stack Bowls 约 90%，Flip Cups 约 75%，Bus Table 约 70%）。在未见物体泛化测试中，OpenVLA 和 PD-VLA 成功率接近 0%，而 ReconVLA 仍维持较高成功率，展现出强大的视觉泛化能力。On the four seen tasks ReconVLA's success rate is clearly better than OpenVLA and PD-VLA (about 90% on Put Fruit and Stack Bowls, about 75% on Flip Cups, about 70% on Bus Table). In the unseen-object generalization test, OpenVLA and PD-VLA drop to nearly 0% success, while ReconVLA still maintains a high success rate, showing strong visual generalization ability.

04 局限性Limitations