Bootstrapping Online RL with Vision-Language-Action Models
Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, Liyiming Ke
将预训练 VLA 的"任务相关知识"压缩成一个紧凑的 RL token，再用这个 token 训练轻量 actor-critic——
无需整体重训 VLA，仅需少量真机交互（15 分钟至 5 小时），即可大幅提升精密操作任务的成功率与速度。Compresses the "task-relevant knowledge" of a pretrained VLA into a compact RL token, then trains a lightweight actor-critic on that token—without retraining the VLA as a whole, and with only a small amount of real-robot interaction (15 minutes to 5 hours), it substantially raises both the success rate and the speed of precise manipulation tasks.
VLA 模型（如 π₀、OpenVLA 等）在多样化操作任务上展现出强大的泛化能力，
但面对需要毫米级精度的关键阶段（如螺钉安装、以太网插头插拔），其成功率往往不尽如人意。
强化学习（RL）理论上可以弥补这一差距，但真机 RL 面临严峻挑战：VLA models (such as π₀ and OpenVLA) show strong generalization across diverse manipulation tasks, yet on critical phases that demand millimeter-level precision (installing a screw, plugging and unplugging an Ethernet connector), their success rates are often unsatisfactory. Reinforcement learning (RL) could in principle close this gap, but real-robot RL faces severe challenges:
"every episode takes time, every failure consumes effort and wear, and meaningful adaptation often has to happen within a few hours of practice."
现有方案的两难困境：The dilemma of existing approaches:
直接微调整个 VLA：参数量巨大，样本效率极低，难以在少量真机数据上收敛。Directly fine-tuning the whole VLA: the parameter count is enormous and sample efficiency is extremely low, so convergence on a small amount of real-robot data is hard.
从头训练小型 actor-critic（如 HIL-SERL）：丢弃了 VLA 预训练的感知与策略知识，探索空间大。Training a small actor-critic from scratch (e.g. HIL-SERL): discards the perceptual and policy knowledge acquired in VLA pretraining, leaving a large exploration space.
残差/蒸馏方案（如 PLD、DSRL）：作用于单步动作或扩散噪声空间，未能充分利用 VLA 的完整表征。Residual / distillation schemes (e.g. PLD, DSRL): they act on single-step actions or on the diffusion noise space, failing to exploit the full VLA representation.
核心问题：如何在保留 VLA 预训练知识的同时，让轻量级 RL 在极少量真机数据上高效工作？The core question: how can lightweight RL work efficiently on very little real-robot data while preserving the pretrained knowledge of the VLA?
图：实验中的 4 个真机操作任务。
每个任务都包含一个需要高精度的关键阶段（critical phase）：
（上）用螺丝刀安装 M3 螺钉；（中）穿扎带；（下）插入以太网线和电源充电器。
VLA 在这些精密阶段成功率有限，RLT 通过在线 RL 显著提升了速度与一致性。Figure: the 4 real-robot manipulation tasks used in the experiments. Each task contains a critical phase that demands high precision: (top) installing an M3 screw with a screwdriver; (middle) threading a zip tie; (bottom) inserting an Ethernet cable and a power charger. The VLA has limited success rates in these precise phases, while RLT markedly improves speed and consistency through online RL.
3×关键阶段最大速度提升倍数Maximum speed-up factor in the critical phase
+45%螺钉任务成功率提升（20%→65%）Success-rate gain on the screw task (20%→65%)
≤5h每任务真机 RL 训练数据量Real-robot RL training data per task
4真实机器人操作任务Real-robot manipulation tasks
02方法Method
RLT（RL Token）分两个阶段工作：首先将 VLA 最终层的 token 嵌入压缩成一个紧凑的"RL token"
（通过 encoder-decoder transformer 的信息瓶颈），然后在这个 RL token 上训练一个轻量 actor-critic，
在 VLA 冻结的情况下用在线 RL 精细化动作。RLT (RL Token) works in two stages: it first compresses the token embeddings of the VLA's final layer into a compact "RL token" (through the information bottleneck of an encoder-decoder transformer), and then trains a lightweight actor-critic on this RL token, refining actions with online RL while the VLA stays frozen.
图 2：RL token 提取的架构细节。
RLT 在预训练 VLA 之上添加一个 encoder-decoder transformer。
Encoder 接收 VLA 最终层的所有 token 嵌入，并引入一个可学习的特殊 token（erl）；
Decoder 从 RL token 自回归地重建原始 VLA 嵌入，形成信息瓶颈，
迫使 RL token 保留足够的任务相关信息。Figure 2: architectural details of RL token extraction. RLT adds an encoder-decoder transformer on top of the pretrained VLA. The encoder takes all token embeddings of the VLA's final layer and introduces a learnable special token (erl); the decoder autoregressively reconstructs the original VLA embeddings from the RL token, forming an information bottleneck that forces the RL token to retain enough task-relevant information.
在特定任务的遥操作演示数据上，用重建损失训练 encoder-decoder（可选同时 SFT 微调 VLA 本身）。
重建目标确保："the representation for the RL token must retain enough information to enable the decoder to reconstruct the inputs。"
此阶段使用任务演示数据（每任务 1–10 小时的遥操作数据）。On task-specific teleoperation demonstration data, the encoder-decoder is trained with a reconstruction loss (optionally SFT-fine-tuning the VLA itself at the same time). The reconstruction objective ensures: "the representation for the RL token must retain enough information to enable the decoder to reconstruct the inputs." This stage uses task demonstration data (1–10 hours of teleoperation data per task).
阶段二：在线 RL（Online RL Stage）Stage 2: Online RL
冻结 VLA 与 RL token encoder，在 RL token 加本体感知状态（proprioceptive state）上训练轻量 actor 和 critic：Freeze the VLA and the RL token encoder, and train a lightweight actor and critic on the RL token plus the proprioceptive state:
Reference action conditioning：Actor 以 VLA 采样的动作块（action chunk）为参考输入，
学习"偏移量"而非从头生成动作，大幅缩小探索空间。Reference action conditioning: the actor takes the action chunk sampled by the VLA as a reference input and learns an "offset" rather than generating actions from scratch, greatly shrinking the exploration space.
Reference action dropout：训练时随机将参考动作块替换为零向量，防止 actor 退化为单纯复制 VLA。Reference action dropout: during training the reference action chunk is randomly replaced by a zero vector, preventing the actor from degenerating into a mere copy of the VLA.
BC regularization：策略目标为最小化 −Qψ(x,a) + β‖a − ã‖²，
其中 ã 为 VLA 参考动作块，β 控制锚定强度，防止在线 RL 走向奇怪的局部最优。BC regularization: the policy objective minimizes −Qψ(x,a) + β‖a − ã‖², where ã is the VLA reference action chunk and β controls the anchoring strength, keeping online RL from drifting into odd local optima.
Chunked actions（C=10）：以 50 Hz 控制频率下 10 步为一个动作块，
有效缩短时序差分（TD）学习的 horizon，使稀疏奖励下的 RL 更容易收敛。Chunked actions (C=10): at a 50 Hz control frequency, 10 steps form one action chunk, effectively shortening the horizon of temporal-difference (TD) learning and making RL easier to converge under sparse rewards.
高 update-to-data ratio（5:1）：每收集 1 步新数据就做 5 次梯度更新，
在低数据在线场景下至关重要。High update-to-data ratio (5:1): 5 gradient updates for every 1 step of new data collected, which is critical in the low-data online regime.
数据来源为 off-policy 混合：VLA rollout、RL 探索轨迹、以及人工干预纠正数据。
这种分工使得"VLA 提供广泛的感知理解与动作建议，轻量 actor-critic 在任务最难的部分做在线适配"。The data source is an off-policy mixture: VLA rollouts, RL exploration trajectories, and human intervention corrections. This division of labor lets "the VLA provide broad perceptual understanding and action proposals, while the lightweight actor-critic adapts online on the hardest part of the task".
03实验Experiments
在 4 个真实机器人精密操作任务上评测，使用关键阶段吞吐量
（throughput：每 10 分钟完成的成功次数）和成功率作为核心指标。
每任务关键阶段评测 50 episodes，RL 训练数据量约 15 分钟至 5 小时。Evaluated on 4 real-robot precision manipulation tasks, using critical-phase throughput (throughput: number of successes completed per 10 minutes) and success rate as the core metrics. The critical phase of each task is evaluated over 50 episodes, with roughly 15 minutes to 5 hours of RL training data.
图 4：RLT 在各任务关键阶段的吞吐量提升。
RLT 在速度与一致性两方面均显著超越基础 VLA 策略，
在最难的关键阶段实现最高约 3× 的速度提升。Figure 4: throughput gains of RLT in the critical phase of each task. RLT significantly surpasses the base VLA policy in both speed and consistency, reaching up to about a 3× speed-up in the hardest critical phases.
关键阶段成功率对比Critical-phase success-rate comparison
任务Task
Base VLA
RLT（本文）RLT (ours)
变化Change
Screw Installation（螺钉安装）Screw Installation
20%
65%
+45%
Ethernet Insertion（以太网插入）Ethernet Insertion
高（维持）High (maintained)
维持 + 速度 ≈3×Maintained + speed ≈3×
速度大幅提升Large speed gain
Charger Insertion（充电器插入）Charger Insertion
高（维持）High (maintained)
维持 + 速度 ≈3×Maintained + speed ≈3×
速度大幅提升Large speed gain
注：论文中成功率以图表形式呈现；Screw 任务明确报告 20%→65%；
其余任务以 Ethernet 任务为代表详细比较基线。Note: the paper presents success rates as figures; the screw task explicitly reports 20%→65%; for the other tasks the Ethernet task serves as the representative case for a detailed baseline comparison.
全任务（full-task）成功率提升Full-task success-rate gains
任务Task
Base VLA
RLT
提升Gain
Screw Installation
基线Baseline
+40%
成功率大幅提升Large success-rate gain
Zip Tie Fastening
基线Baseline
+60%
成功率大幅提升Large success-rate gain
与基线方法对比（Ethernet 任务，图 6）Comparison with baseline methods (Ethernet task, Figure 6)
图 6：Ethernet 任务上 RLT 与各基线的对比。
HIL-SERL 和 PLD 使用单步动作（非 action chunk），效果差；
DSRL 成功率接近 RLT 但速度提升显著更少；
DAgger 受限于人类遥操作速度上限。Figure 6: RLT versus the baselines on the Ethernet task. HIL-SERL and PLD use single-step actions (not action chunks) and perform poorly; DSRL reaches a success rate close to RLT but gains markedly less speed; DAgger is capped by the upper bound of human teleoperation speed.
冻结 VLA 上的残差单步动作Residual single-step actions on a frozen VLA
效果差，单步动作不适合Poor; single-step actions are a bad fit
DSRL
扩散 VLA 潜在噪声空间 RLRL in the latent noise space of a diffusion VLA
成功率接近 RLT，但速度提升明显更少Success rate close to RLT, but clearly less speed gain
DAgger
干预数据微调 VLAFine-tuning the VLA on intervention data
受限于人类演示速度Capped by human demonstration speed
RLT（本文）RLT (ours)
RL token + chunked actor-critic
成功率与速度均最优Best in both success rate and speed
消融实验（Ethernet 任务）Ablations (Ethernet task)
图 7：训练过程中各消融变体的吞吐量曲线。
每个组件的去除都会导致性能下降，证明方法各部分缺一不可。Figure 7: throughput curves of the ablation variants during training. Removing any single component degrades performance, showing that every part of the method is indispensable.
无法可靠超越 VLA 基线Cannot reliably beat the VLA baseline
w/o BC Regularizer（β=0）w/o BC Regularizer (β=0)
单项去除中影响最大Largest impact among the single-component removals
w/o Pass-Through（不输入参考动作）w/o Pass-Through (no reference action input)
学习更慢，训练过程失败更多，最终可部分恢复Slower learning and more failures during training, with partial recovery in the end
涌现行为（Emergent Behavior）Emergent Behavior
RLT 在 Ethernet 任务上学到了演示数据中不存在的策略：
基础 VLA 表现出"探测行为"（反复接近-退出-重调整），
而 RLT 学会了流畅插入并主动施加压力、利用顺从性——
约 50% 的 RLT 关键阶段 episodes 速度快于最快的人类遥操作示范。On the Ethernet task RLT learned a strategy absent from the demonstration data: the base VLA exhibits "probing behavior" (repeatedly approaching, backing off, readjusting), whereas RLT learned to insert smoothly while actively applying pressure and exploiting compliance—about 50% of RLT's critical-phase episodes are faster than the fastest human teleoperation demonstration.
04局限性Limitations
说明：以下限制均为论文作者明确陈述（stated）。Note: all limitations below are explicitly stated by the authors of the paper.
需要持续人工干预（Human-in-the-loop）Requires continuous human intervention (human-in-the-loop)
论文明确指出："RLT...does require additional human intervention during training
to provide reward signals, intervention corrections, and switching between RL
(for the critical phase) and the base policy (for the other phases)。"
奖励信号、干预纠正、阶段切换均需人工参与，无法做到完全自主。The paper states explicitly: "RLT...does require additional human intervention during training to provide reward signals, intervention corrections, and switching between RL (for the critical phase) and the base policy (for the other phases)." Reward signals, intervention corrections and phase switching all require a human, so full autonomy is not achieved.
依赖任务阶段人工分割Depends on manual segmentation of task phases
需要人工识别"关键阶段"的起止点，并为该阶段提供 episode 级别的二值成功/失败标签。
论文将"开发全自主 RL 改进流程"列为未来工作方向。A human must identify the start and end of the "critical phase" and provide episode-level binary success/failure labels for it. The paper lists "developing a fully autonomous RL improvement pipeline" as a direction for future work.
仍需任务特定的遥操作演示数据Still requires task-specific teleoperation demonstration data
适配阶段需要每任务 1–10 小时的遥操作演示数据来训练 RL token 的 encoder-decoder，
限制了其在无演示数据场景下的适用性。The adaptation stage needs 1–10 hours of teleoperation demonstration data per task to train the encoder-decoder of the RL token, which limits applicability in settings without demonstration data.
仅在精密关键阶段上 RL 微调RL fine-tuning only on the precise critical phase
当前实验仅对任务中的"关键阶段"做 RL 优化，而非端到端全任务 RL。
在关键阶段之外（抓取、运输等），仍沿用基础 VLA 策略。
这简化了问题但也限制了适用范围（inferred）。Current experiments optimize with RL only the "critical phase" of a task, not end-to-end full-task RL. Outside the critical phase (grasping, transport and so on), the base VLA policy is still used. This simplifies the problem but also narrows the scope of applicability (inferred).
