主动视觉注意力提升视觉-语言-动作模型Improving Vision-Language-Action models with Active Visual Attention
Lei Xiao, Jifeng Li, Juntao Gao, Feiyang Ye, Yan Jin, Jingjing Qian, Jing Zhang, Yong Wu, Xiaoyuan Yu
AVA-VLA 将 VLA（Vision-Language-Action）策略学习重新表述为 POMDP（Partially Observable Markov Decision Process），
通过引入循环历史状态来近似任务信念，并设计主动视觉注意力（Active Visual Attention）模块，
根据指令与执行历史动态重加权视觉 token——在 LIBERO 和 CALVIN 等机器人操作基准上达到最先进水平，
并成功迁移至真实双臂机械手平台。
AVA-VLA reformulates VLA (Vision-Language-Action) policy learning as a POMDP (Partially Observable Markov Decision Process).
It introduces a recurrent historical state to approximate the task belief and designs an Active Visual Attention module
that dynamically reweights visual tokens according to the instruction and the execution history — reaching state-of-the-art
results on robot manipulation benchmarks such as LIBERO and CALVIN, and transferring successfully to a real dual-arm manipulator platform.
现有 VLA 模型在每个时间步独立处理视觉观测，将机械手操作当作 Markov Decision Process（MDP）来求解。
然而机器人操作本质上是部分可观测的，依赖历史交互才能完成精确判断。
以静态语言指令引导的视觉注意力，被迫在每个决策步从零重新评估视觉信息，
无法抑制时序冗余信息，也无法聚焦因过去动作而变得关键的区域。
Existing VLA models process visual observations independently at every time step, solving robot manipulation as a Markov Decision Process (MDP).
Robot manipulation, however, is inherently partially observable and relies on historical interaction to make precise judgements.
Visual attention guided by a static language instruction is forced to re-evaluate visual information from scratch at every decision step,
and can neither suppress temporally redundant information nor focus on regions that have become critical because of past actions.
"by processing frames in isolation, the visual attention weights, guided by the static language instruction,
are forced to re-evaluate the independent visual information from scratch at each decision step."
图 1：动机对比。
任务为"打开灶台并将摩卡壶放上去"，从两个视角观察。
基线 VLA（左）无法稳定定位关键目标"灶台开关"；
AVA-VLA（右）借助历史上下文，展现出更稳定的注意力聚焦。
红色高亮区域表示视觉注意力权重最高的区域。
Figure 1: Motivation comparison.
The task is "turn on the stove and put the moka pot on it", observed from two viewpoints.
The baseline VLA (left) fails to stably localize the key target, the "stove knob";
AVA-VLA (right) exploits historical context and shows a more stable attention focus.
The red highlighted regions mark where the visual attention weights are highest.
98.0%LIBERO 多任务平均 SR （4 套件统一策略）LIBERO multi-task average SR (unified policy over 4 suites)
4.65CALVIN ABC→D 平均链长（连续任务数）CALVIN ABC→D average chain length (consecutive tasks)
98.2%LIBERO 单任务平均 SR （每套件独立策略）LIBERO single-task average SR (a separate policy per suite)
CVPR 2026Highlight 论文 真实双臂机器人验证Highlight paper validated on a real dual-arm robot
02方法Method
AVA-VLA 将 VLA 策略重新表述为 POMDP：动作生成不仅依赖当前观测，
还依赖对任务历史信念的循环近似状态。
核心模块是 Active Visual Attention（AVA），
它将循环状态与语言条件化视觉特征融合，
生成 soft importance scores 对骨干 LLM 中所有层的视觉 token 注意力矩阵进行调制。
AVA-VLA reformulates the VLA policy as a POMDP: action generation depends not only on the current observation,
but also on a recurrent approximate state of the belief over the task history.
The core module is Active Visual Attention (AVA),
which fuses the recurrent state with language-conditioned visual features and
produces soft importance scores that modulate the visual-token attention matrices in all layers of the backbone LLM.
图 2：AVA-VLA 整体架构。
循环状态 rt-1 由上一时间步动作相关隐藏状态经 MLP 模块 ℬ 压缩而来。
AVA 模块将循环状态与文本条件化视觉特征融合，输出 soft 权重向量 ωt，
对骨干 LLM 所有层的视觉 token 注意力矩阵进行调制。
推理时完全循环（fully recurrent），每次前向传播同时预测动作块并提取下一步循环状态。
Figure 2: Overall architecture of AVA-VLA.
The recurrent state rt-1 is compressed by the MLP module ℬ from the action-related hidden states of the previous time step.
The AVA module fuses the recurrent state with text-conditioned visual features and outputs a soft weight vector ωt,
which modulates the visual-token attention matrices in all layers of the backbone LLM.
Inference is fully recurrent: each forward pass simultaneously predicts an action chunk and extracts the recurrent state for the next step.
POMDP 重表述与循环状态POMDP Reformulation and the Recurrent State
将策略形式化为：Āt ~ Pθ(At | xt, bt-1)，
其中 bt-1 捕获"all relevant historical context, including observations and actions"。
由于理论信念状态难以精确计算，方法学习一个压缩的循环表示 rt-1，
作为"a neural approximation"，从上一时间步动作相关的 LLM 隐藏状态提取，通过 MLP 模块 ℬ 投影。
The policy is formalized as Āt ~ Pθ(At | xt, bt-1),
where bt-1 captures "all relevant historical context, including observations and actions".
Since the theoretical belief state cannot be computed exactly, the method learns a compressed recurrent representation rt-1
as "a neural approximation", extracted from the action-related hidden states of the LLM at the previous time step and projected by the MLP module ℬ.
Active Visual Attention（AVA）模块The Active Visual Attention (AVA) Module
AVA 模块依次执行以下操作：The AVA module performs the following operations in sequence:
特征编码：视觉与指令特征各自通过模态专用 MLP 压缩到低维空间 d' < d。Feature encoding: Visual and instruction features are each compressed by a modality-specific MLP into a low-dimensional space d' < d.
FiLM 条件化：将语言指令通过 Feature-wise Linear Modulation 条件化到视觉特征上。FiLM conditioning: The language instruction is conditioned onto the visual features through Feature-wise Linear Modulation.
Cross-Attention：以语言条件化的视觉 token 为 query、循环状态为 key/value，计算跨模态注意力。Cross-Attention: Cross-modal attention is computed with the language-conditioned visual tokens as queries and the recurrent state as keys/values.
Self-Attention：进一步精炼 cross-attention 输出。Self-Attention: The cross-attention output is further refined.
Importance Scoring：经 FFN 和带 Softmax 的线性层，预测每个视觉 token 的"logits for enhancing or weakening"。Importance Scoring: An FFN followed by a linear layer with Softmax predicts, for every visual token, the "logits for enhancing or weakening".
动态注意力调制：软权重 ωt 作用于 LLM 所有层，修改注意力矩阵：
A'tmij = exp(Ctmij) · Uij / Σl exp(Ctmil) · Uil。
Dynamic attention modulation: The soft weights ωt act on all layers of the LLM and modify the attention matrix:
A'tmij = exp(Ctmij) · Uij / Σl exp(Ctmil) · Uil.
训练策略Training Strategy
采用截断反向传播（truncated BPTT），时间窗口 T=4，平衡计算可行性与时序动态学习。
同时引入 L2 正则化损失 ℒωt,n = ‖μ(ωt,n) − c‖，
约束注意力权重均值接近目标常数 c，使模型"focus on task-relevant regions while suppressing distracting background responses"。
超参数：LIBERO 中 λ=1.0, c=0.6, γ=[1.9, 0.1]；CALVIN 中 c=0.2。
Truncated backpropagation (truncated BPTT) is used with a time window T=4, balancing computational feasibility against the learning of temporal dynamics.
An L2 regularization loss ℒωt,n = ‖μ(ωt,n) − c‖ is additionally introduced,
constraining the mean of the attention weights towards a target constant c so that the model "focus on task-relevant regions while suppressing distracting background responses".
Hyperparameters: λ=1.0, c=0.6, γ=[1.9, 0.1] on LIBERO; c=0.2 on CALVIN.
图 3：LIBERO 注意力动态可视化。
任务"put both moka pots on the stove"，两路视角下 soft 权重随时间步的演化。
可以看到注意力权重逐步集中到机械臂接触区域和目标物体上，体现了 AVA 模块的主动聚焦能力。
Figure 3: Visualization of attention dynamics on LIBERO.
For the task "put both moka pots on the stove", the evolution of the soft weights over time steps is shown for two camera views.
The attention weights progressively concentrate on the contact region of the arm and on the target objects, reflecting the active focusing ability of the AVA module.
03实验Experiments
在 LIBERO（4 个套件：Spatial / Object / Goal / Long）、CALVIN（ABC→D 零样本泛化）
以及真实 Mobile ALOHA 双臂机械手平台上进行全面评估，
主要对比基线为 OpenVLA-OFT。评估指标为任务成功率（SR %）和 CALVIN 平均链长（Avg. len）。
A comprehensive evaluation is carried out on LIBERO (4 suites: Spatial / Object / Goal / Long), CALVIN (ABC→D zero-shot generalization)
and a real Mobile ALOHA dual-arm manipulator platform,
with OpenVLA-OFT as the main comparison baseline. The metrics are task success rate (SR %) and CALVIN average chain length (Avg. len).
图 4：Mobile ALOHA 真实双臂机器人实验结果。
在 Pick and Place、Sequenced Instruction Understanding、Flexible Object Folding、Dexterous Action
四类任务上与 UniVLA、OpenVLA-OFT 对比，AVA-VLA 在跨任务平均成功率上均优于基线。
图中展示了任务中间状态的代表性帧及各方法任务成功率。
Figure 4: Real-robot results on the Mobile ALOHA dual-arm platform.
Across the four task categories Pick and Place, Sequenced Instruction Understanding, Flexible Object Folding and Dexterous Action,
AVA-VLA is compared with UniVLA and OpenVLA-OFT and surpasses the baselines in average success rate across tasks.
The figure shows representative frames of intermediate task states together with the success rate of each method.
消融实验Ablation Studies
组件消融（LIBERO 多任务，Table 4）：
仅使用状态初始化（State-based initialization only）得到 97.5% 平均 SR；
仅使用 AVA 模块（AVA module only）同样得到 97.5%；
两者结合（AVA module + State init）达到最优 98.0%，
说明循环状态初始化与 AVA 模块相互配合缺一不可。
Component ablation (LIBERO multi-task, Table 4):
State-based initialization only reaches 97.5% average SR;
the AVA module only likewise reaches 97.5%;
combining the two (AVA module + State init) achieves the best result, 98.0%,
showing that recurrent state initialization and the AVA module complement each other and are both indispensable.
骨干网络泛化性（Table 3，LIBERO-Long）：
在 OpenVLA-7B（+1.7%）、LLaMA2-7B（+2.6%）、Qwen2.5-0.5B（+1.4%）三种骨干上
AVA-VLA 均优于对应的 OpenVLA-OFT 基线，表明方法的骨干无关性。
Backbone generality (Table 3, LIBERO-Long):
On the three backbones OpenVLA-7B (+1.7%), LLaMA2-7B (+2.6%) and Qwen2.5-0.5B (+1.4%),
AVA-VLA outperforms the corresponding OpenVLA-OFT baseline, showing that the method is backbone-agnostic.
视觉 token 剪枝（Table 5）：
剪枝比例 ≤70% 时性能保持在 97.3% 以上；剪枝 80% 降至 96.0%；剪枝 90% 降至 93.9%，
表明 AVA 学到的权重具有稀疏性，在轻度剪枝下可加速推理而几乎不损性能。
Visual token pruning (Table 5):
Performance stays above 97.3% for pruning ratios ≤70%; pruning 80% drops to 96.0% and pruning 90% drops to 93.9%,
showing that the weights learned by AVA are sparse and that mild pruning can accelerate inference at almost no cost in performance.
04局限性Limitations
Note：以下局限性部分为论文附录 E 中明确陈述（stated），部分为从方法设计推断（inferred）。
Note: Some of the limitations below are explicitly stated in Appendix E of the paper (stated), while others are inferred from the method design (inferred).
截断 BPTT 时间窗口有限（stated）Limited truncated-BPTT time window (stated)
论文采用时间窗口 T=4 的截断反向传播（truncated BPTT），
这是"practical trade-off"而非最优的时序建模。
对需要超长历史推理的任务，有限时间窗口可能无法捕获足够的历史依赖。
The paper uses truncated backpropagation (truncated BPTT) with a time window T=4,
which is a "practical trade-off" rather than optimal temporal modelling.
For tasks that require reasoning over very long histories, the limited time window may fail to capture enough historical dependency.
轨迹展开的内存需求（stated）Memory requirements of trajectory unrolling (stated)
训练时对历史轨迹的展开（trajectory unrolling）带来额外的内存开销，
在显存受限的平台上增加了部署难度，可能限制批大小或序列长度。
Unrolling historical trajectories during training (trajectory unrolling) incurs additional memory overhead,
which increases the difficulty of deployment on platforms with limited GPU memory and may constrain the batch size or sequence length.
超参数对任务分布敏感（stated）Hyperparameters are sensitive to the task distribution (stated)
方法需要针对不同任务分布手动调整超参数（λ, c, γ），
例如 LIBERO 中 c=0.6 而 CALVIN 中 c=0.2。
跨域泛化时可能需要重新调参，降低了开箱即用性。
The method requires manually tuning hyperparameters (λ, c, γ) for different task distributions,
for example c=0.6 on LIBERO but c=0.2 on CALVIN.
Cross-domain generalization may therefore require re-tuning, which reduces out-of-the-box usability.
对循环状态初始化敏感（stated）Sensitivity to recurrent-state initialization (stated)
论文指出方法对循环状态初始化存在一定敏感性，
不良的初始化可能影响早期时间步的动作生成质量，
尤其在任务开始阶段历史信息缺乏时。
The paper notes a certain sensitivity to the initialization of the recurrent state;
a poor initialization may degrade the quality of action generation in the early time steps,
especially at the beginning of a task when historical information is scarce.
仅在双臂操作场景验证（inferred）Validated only in dual-arm manipulation settings (inferred)
真实机器人实验仅在 Mobile ALOHA 双臂机械手平台上展开，
对更广泛的机器人形态（如移动机器人、单臂操作、灵巧手）的泛化能力尚未评估。
The real-robot experiments are conducted only on the Mobile ALOHA dual-arm manipulator platform;
generalization to a wider range of robot morphologies (mobile robots, single-arm manipulation, dexterous hands) has not been evaluated.
