EVOLVE-VLA

01 动机Motivation

当前的 VLA 模型依赖大规模的监督微调（SFT），这在实际部署中存在两大根本缺陷：高昂的数据采集成本以及脆弱的分布内记忆。模型在面对部署环境与训练分布的差异时几乎无法自适应恢复。Current VLA models rely on large-scale supervised fine-tuning (SFT), which suffers from two fundamental drawbacks in real deployment: the high cost of data collection and brittle in-distribution memorization. When the deployment environment differs from the training distribution, such models can hardly adapt or recover.

"How do humans develop manipulation skills? We do not simply watch an expert perform a task once and then flawlessly replicate it. Instead, we learn through practice: attempting the task repeatedly, making mistakes, receiving feedback from the environment, and gradually refining our movements through continued experience."

作者指出，现有 VLA 的两大核心瓶颈为：The authors identify two core bottlenecks of existing VLAs:

• 高昂的数据成本：适应一个新任务往往需要收集数百条人工示范；High data cost: adapting to a new task usually requires collecting hundreds of human demonstrations;
• 脆弱的记忆化泛化（brittle memorization）：通过行为克隆训练的 VLA 仅能模仿示范，一旦执行偏差发生，便缺乏恢复能力，更无法泛化到训练分布之外。Brittle memorized generalization (brittle memorization): a VLA trained by behavior cloning can only imitate demonstrations; once execution deviates, it lacks the ability to recover, let alone generalize beyond the training distribution.

02 方法Method

EVOLVE-VLA 的核心思路是：在测试时让 VLA 与真实或模拟环境交互，用环境反馈信号（而非人工标注）驱动在线强化学习，从而在无需额外示范的情况下持续提升策略。框架由三个相互协作的组件构成。The core idea of EVOLVE-VLA is to let the VLA interact with a real or simulated environment at test time and to drive online reinforcement learning with environment feedback signals rather than human annotations, so that the policy keeps improving without additional demonstrations. The framework consists of three cooperating components.

进度估计器 VLAC（Vision-Language-Action Critic）Progress Estimator VLAC (Vision-Language-Action Critic)

由于现实环境中难以获取稀疏成功信号，EVOLVE-VLA 引入了基础模型级别的 critic 模型 VLAC。它接收两张图像和任务指令作为输入，输出一个连续的 critic 值——正值表示第二张图相对于第一张在任务完成上有所进展。这一设计将稀疏的结果信号转化为稠密的逐步反馈，有效解决了 sparse reward 的学习困难问题。Because sparse success signals are hard to obtain in real environments, EVOLVE-VLA introduces VLAC, a critic model at the foundation-model level. It takes two images and the task instruction as input and outputs a continuous critic value — a positive value indicates that the second image has made progress toward task completion relative to the first. This design turns sparse outcome signals into dense step-wise feedback and effectively addresses the learning difficulty of sparse reward.

Accumulative Progress Estimation（累积进度估计）Accumulative Progress Estimation

直接使用 VLAC 比较相邻帧会引入大量噪声。论文提出了基于"区间里程碑采样"（interval-based milestone sampling）的累积进度估计策略：将当前状态与最近的里程碑状态对比，而非初始状态，并通过比例调整（proportional adjustments）平滑噪声。消融实验表明，在相同的 reward call 数量（32 次）下，该方法的 F-Score 从 0.04 提升至 0.20，成功率从 88.3% 提升至 91.3%。Directly using VLAC to compare adjacent frames introduces considerable noise. The paper proposes an accumulative progress estimation strategy based on "interval-based milestone sampling": the current state is compared with the most recent milestone state rather than with the initial state, and the noise is smoothed by proportional adjustments. Ablations show that under the same number of reward calls (32), this method raises the F-Score from 0.04 to 0.20 and the success rate from 88.3% to 91.3%.

Progressive Horizon Extension（渐进式视野扩展）Progressive Horizon Extension

长时域任务探索困难，策略难以直接从完整轨迹中学习。EVOLVE-VLA 采用渐进式扩展探索视野的训练策略：先让策略掌握较短子目标，再逐步延长任务长度直至完整任务。这一机制带来了额外的 +3.1% 成功率提升（91.3% → 94.4%），如 Table 4 所示。Long-horizon tasks are hard to explore, and a policy can hardly learn directly from complete trajectories. EVOLVE-VLA adopts a training strategy that progressively extends the exploration horizon: the policy first masters shorter sub-goals, and the task length is then gradually extended up to the full task. This mechanism brings an additional +3.1% success rate gain (91.3% → 94.4%), as shown in Table 4.

GRPO 在线优化Online Optimization with GRPO

策略更新采用 Group Relative Policy Optimization（GRPO）。GRPO 在每批轨迹内归一化奖励计算优势，应用 PPO 风格的 clipping 保证稳定更新，且无需独立的 value network，显著降低了计算开销。采样时以 temperature T > 1 采样以增强轨迹多样性。Policy updates use Group Relative Policy Optimization (GRPO). GRPO normalizes rewards within each batch of trajectories to compute advantages, applies PPO-style clipping to guarantee stable updates, and requires no separate value network, which markedly reduces computational overhead. Sampling is performed with temperature T > 1 to enrich trajectory diversity.

03 实验Experiments

实验在 LIBERO 基准上进行，该基准包含 4 个任务套件（Spatial / Object / Goal / Long），每套 10 个任务，共 4 种测试场景。基础模型为 OpenVLA-OFT（autoregressive VLA）。Experiments are conducted on the LIBERO benchmark, which contains 4 task suites (Spatial / Object / Goal / Long) with 10 tasks each, i.e. 4 test settings. The base model is OpenVLA-OFT (autoregressive VLA).

主实验：LIBERO 全套件成功率（Table 1）Main Results: Success Rate on All LIBERO Suites (Table 1)

1-Shot 学习（Table 2）1-Shot Learning (Table 2)

跨任务泛化（Zero-Shot Transfer）Cross-Task Generalization (Zero-Shot Transfer)

将在 LIBERO-Long 上训练的策略直接部署到 LIBERO-Object：SFT 基线成功率为 0%，而 EVOLVE-VLA 通过纯自主探索，无任务示范地达到 20.8% 成功率——证明了 test-time training 赋予 VLA 超出训练分布的泛化能力。Deploying the policy trained on LIBERO-Long directly to LIBERO-Object: the SFT baseline reaches 0% success, whereas EVOLVE-VLA attains 20.8% success through purely autonomous exploration without any task demonstration — showing that test-time training endows the VLA with generalization beyond the training distribution.

消融实验（Ablations）Ablations

Table 3 验证了 Accumulative Progress Estimation 的有效性：在相同 reward call 数（32 次）下，相比直接比较相邻两帧的基线（F-Score 0.04，SR 88.3%），区间采样策略将 F-Score 提升至 0.20，成功率提升至 91.3%，同时避免了均匀采样（Uniform）随 call 数增加导致 SR 下降的问题。Table 3 validates the effectiveness of Accumulative Progress Estimation: under the same number of reward calls (32), compared with the baseline that directly compares two adjacent frames (F-Score 0.04, SR 88.3%), the interval sampling strategy raises the F-Score to 0.20 and the success rate to 91.3%, while avoiding the SR degradation that Uniform sampling suffers as the number of calls grows.

Table 4 展示了逐步叠加各模块的效果：SFT 85.8% → + Binary Outcome Rewards 87.7% → + Dense Reward (Vanilla Critic) 91.3% → + Progressive Horizon Extension 94.4%，每个模块均贡献了显著提升。Table 4 shows the effect of stacking the modules one by one: SFT 85.8% → + Binary Outcome Rewards 87.7% → + Dense Reward (Vanilla Critic) 91.3% → + Progressive Horizon Extension 94.4%, with every module contributing a clear gain.

04 局限性Limitations