VLA-Reasoner 是一个即插即用框架，通过在线蒙特卡洛树搜索（MCTS）与学习型世界模型，赋予已有 VLA 模型在测试时探索长视野动作轨迹的能力，从而避免短视决策导致的累积偏差，在仿真和真实机器人任务上均显著提升操作成功率。VLA-Reasoner is a plug-and-play framework that, through online Monte Carlo Tree Search (MCTS) and a learned world model, endows existing VLA models with the ability to explore long-horizon action trajectories at test time, thereby avoiding the cumulative drift caused by myopic decisions and markedly improving manipulation success rates in both simulation and real-robot tasks.
Vision-Language-ActionMonte Carlo Tree Searchtest-time computeworld modelKDE samplingvalue estimationrobot manipulationVLA机器人操作robot manipulation测试时推理test-time reasoning
01动机Motivation
当前 Vision-Language-Action（VLA）模型在机器人部署时存在"短视"问题：每一步只预测当前最优动作，不考虑后续轨迹后果，导致误差随时间累积，长视野任务失败率高。Current Vision-Language-Action (VLA) models suffer from a "myopia" problem when deployed on robots: each step predicts only the locally optimal action without considering the consequences for the subsequent trajectory, so errors accumulate over time and long-horizon tasks fail at a high rate.
"Can VLAs explore the long-horizon future influence of actions at test time, and decide the optimal action?"
图 1：VLA-Reasoner 与基础 VLA 的对比示意。左侧为直接输出动作的传统 VLA 基线；右侧为加入 MCTS 树搜索后的 VLA-Reasoner——通过多步模拟和回溯更新，在测试时选取具有长视野优势的动作。Figure 1: Comparison between VLA-Reasoner and a base VLA. Left, a conventional VLA baseline that directly outputs actions; right, VLA-Reasoner after adding MCTS tree search — through multi-step simulation and backpropagated updates it selects, at test time, the action with a long-horizon advantage.
+19%OpenVLA 真实任务绝对成功率提升（22% → 41%）Absolute success-rate gain of OpenVLA on real-world tasks (22% → 41%)
+10%π0-FAST 真实任务绝对成功率提升（64% → 74%）Absolute success-rate gain of π0-FAST on real-world tasks (64% → 74%)
+9.8%Octo-Small 在仿真中的绝对提升Absolute gain of Octo-Small in simulation
VLA 模型继承了大规模视觉-语言预训练的泛化能力，在机器人模仿学习上取得了显著成果。然而，这类模型在部署阶段仍然脆弱——它们将机器人控制当作一步步的局部决策，忽略了动作对未来状态的长链式影响。对于需要精确堆叠、抓取或多步协调的任务，即使微小的偏差也会随时间放大，最终导致任务失败。VLA models inherit the generalization ability of large-scale vision-language pretraining and have achieved notable results in robot imitation learning. Such models nevertheless remain fragile at deployment — they treat robot control as a series of local, step-by-step decisions and ignore the long chain of influence an action exerts on future states. For tasks that require precise stacking, grasping or multi-step coordination, even a small deviation is amplified over time and eventually leads to task failure.
VLA-Reasoner 的核心思路是：不修改 VLA 权重，仅在测试时外挂一套基于 MCTS 的规划模块，通过学习型世界模型模拟未来状态，以 value function 引导搜索，从而找到长视野更优的动作。The core idea of VLA-Reasoner is to leave the VLA weights unchanged and attach, only at test time, an MCTS-based planning module that simulates future states with a learned world model and lets a value function guide the search, so as to find actions that are better over a long horizon.
02方法Method
VLA-Reasoner 由三个核心模块组成：基于 MCTS 的在线动作搜索、Kernel Density Estimation（KDE）高效采样机制、以及基于视觉的轻量 value network。最终动作由 VLA 预测与 MCTS 结果加权融合得到。VLA-Reasoner comprises three core modules: online action search based on MCTS, an efficient Kernel Density Estimation (KDE) sampling mechanism, and a lightweight vision-based value network. The final action is obtained by a weighted fusion of the VLA prediction and the MCTS result.
图 2：VLA-Reasoner 完整流程。VLA 模型输出动作作为树搜索的根节点初始化；轻量 MCTS 通过"扩展→模拟→回传→选择"四步循环，借助世界模型预测未来视觉状态，并由 value network 评估节点价值；最终选取最优叶节点动作与原始 VLA 动作加权融合后执行。Figure 2: The complete VLA-Reasoner pipeline. The action output by the VLA model initializes the root node of the tree search; a lightweight MCTS runs the four-step loop of "expansion → simulation → backpropagation → selection", predicting future visual states with the world model and scoring node values with the value network; finally the action of the best leaf node is fused by weighting with the original VLA action and executed.
在线 MCTS 动作搜索Online MCTS Action Search
MCTS 以当前 VLA 的预测动作为根节点，执行四阶段循环：MCTS takes the action currently predicted by the VLA as the root node and runs a four-stage loop:
Expansion（扩展）：从根节点采样 N 个候选动作，按欧氏距离选取 k 个最近邻作为子节点，使得搜索集中在 VLA 分布的高概率区域。Expansion: N candidate actions are sampled from the root node, and the k nearest neighbours by Euclidean distance are taken as child nodes, so that the search concentrates on the high-probability region of the VLA distribution.
Simulation（模拟）：使用学习型 action-aware 世界模型递推预测未来状态：s(i+1) = W(a(i), s(i))，将动作轨迹转化为视觉观测序列。Simulation: A learned action-aware world model recursively predicts future states, s(i+1) = W(a(i), s(i)), turning the action trajectory into a sequence of visual observations.
Backpropagation（回传）：沿路径更新节点 Q 值，综合 value estimate 与子节点 Q 值的访问次数加权平均：Q(o(i)) = [N(o(i))·v(i) + Σ N(o(j))·Q(o(j))] / [N(o(i)) + Σ N(o(j))]Backpropagation: Node Q values are updated along the path as a visit-count weighted average of the value estimate and the Q values of the child nodes: Q(o(i)) = [N(o(i))·v(i) + Σ N(o(j))·Q(o(j))] / [N(o(i)) + Σ N(o(j))]
Selection（选择）：采用 Upper Confidence Bound（UCB）策略平衡探索与利用，选取最优子节点继续扩展。Selection: An Upper Confidence Bound (UCB) strategy balances exploration and exploitation, selecting the best child node for further expansion.
Kernel Density Estimation（KDE）高效采样Efficient Sampling via Kernel Density Estimation (KDE)
反复查询 VLA 成本高昂。VLA-Reasoner 用 KDE 对离线数据中的动作分布建模：π(a) = (1/N) Σ K_h(a − a(i))，以概率密度作为访问次数的代理，在保留 VLA 先验的同时大幅减少对模型的实时调用。Querying the VLA repeatedly is expensive. VLA-Reasoner models the action distribution of the offline data with KDE, π(a) = (1/N) Σ K_h(a − a(i)), and uses the probability density as a proxy for visit counts, which preserves the VLA prior while greatly reducing real-time calls to the model.
视觉 value network 与动作融合Visual Value Network and Action Fusion
轻量 ResNet-34 + 2-layer MLP 在离线轨迹上训练，以线性插值帧位置为 ground-truth value（MSE loss），为中间树节点提供稠密反馈。最终执行动作为：A lightweight ResNet-34 plus a 2-layer MLP is trained on offline trajectories, taking linearly interpolated frame positions as the ground-truth value (MSE loss), which provides dense feedback for intermediate tree nodes. The action finally executed is:
a(t) = α · a(VLA) + (1−α) · a(Reasoner)
其中超参数 α 控制原始 VLA 预测与树搜索结果之间的平衡。消融实验显示 α = 0.6 时效果最佳，适度融合优于完全依赖任一来源。The hyperparameter α controls the balance between the original VLA prediction and the tree-search result. Ablations show that α = 0.6 performs best: moderate fusion is better than relying entirely on either source.
图 3：真实世界实验配置。使用 Galaxea-A1 机械臂，配备侧置摄像头与腕部摄像头，覆盖积木堆叠、水果拣选、杯子堆叠、圆周运动等五类任务，每类任务提供 20 个训练示例。Figure 3: Real-world experimental setup. A Galaxea-A1 arm equipped with a side camera and a wrist camera covers five task families — block stacking, fruit picking, cup stacking, circular motion and others — with 20 training demonstrations provided for each.
03实验Experiments
在仿真环境（LIBERO、SimplerEnv）和真实机器人（Galaxea-A1）上与 OpenVLA、Octo-Small、SpatialVLA、π0-FAST 等多个 VLA 基线对比，评估指标为任务成功率（success rate）。Comparison against several VLA baselines — OpenVLA, Octo-Small, SpatialVLA and π0-FAST — in simulation (LIBERO, SimplerEnv) and on a real robot (Galaxea-A1), with task success rate as the evaluation metric.
值得注意的是，VLA-Reasoner 在不进行大规模后训练的前提下，使部分基线达到了与当前最优变体相当的性能水平。Notably, without any large-scale post-training, VLA-Reasoner brings some baselines to a performance level comparable to that of the current best variants.
图 4：杯子堆叠任务的案例对比。上行为基线 VLA 的失败过程——动作偏移累积，最终无法完成叠放；下行为 VLA-Reasoner 通过测试时模拟主动修正轨迹偏差，实现稳定操作。Figure 4: Case comparison on the cup-stacking task. Top row, the failure process of the baseline VLA — action offsets accumulate and the stack is never completed; bottom row, VLA-Reasoner actively corrects the trajectory deviation through test-time simulation and achieves stable manipulation.
消融实验Ablation Studies
图 5：注入强度 α 的消融。α = 0.6 时性能最优；纯 VLA（α = 1.0）和纯 Reasoner（α = 0.0）均弱于适度融合，说明两者各有互补价值。Figure 5: Ablation on the injection strength α. Performance is best at α = 0.6; both pure VLA (α = 1.0) and pure Reasoner (α = 0.0) are weaker than moderate fusion, indicating that the two are complementary.图 6：关键组件验证。KDE 采样优于高斯噪声基线（有效捕获专家行为先验）；基于图像的 value estimation 优于 token-based reward head（更简洁且可解释）。Figure 6: Validation of the key components. KDE sampling outperforms the Gaussian-noise baseline (it effectively captures the prior of expert behaviour); image-based value estimation outperforms a token-based reward head (simpler and more interpretable).
04局限性Limitations
说明：以下局限性均为论文作者明确陈述（stated），见论文 Limitations 章节及 Future Directions 讨论。Note: All limitations below are explicitly stated by the authors of the paper; see the Limitations section and the Future Directions discussion.
需要部署前的任务专属训练Task-specific training is required before deployment
当前方案在部署前需要针对目标任务训练世界模型、KDE 分布估计器和 value network，增加了新任务的配置成本，不支持零样本直接迁移。Before deployment, the current approach must train a world model, a KDE distribution estimator and a value network for the target task, which raises the setup cost of a new task and precludes direct zero-shot transfer.
依赖世界模型的预测精度Reliance on the prediction accuracy of the world model
MCTS 的搜索质量高度依赖学习型世界模型对未来视觉状态的预测是否准确。论文指出"better visual foresight models would improve feedback quality"——世界模型误差会传导为错误的 value 信号，影响最终动作选择。The search quality of MCTS depends heavily on whether the learned world model predicts future visual states accurately. The paper notes that "better visual foresight models would improve feedback quality" — errors of the world model propagate into wrong value signals and affect the final action selection.
Value estimation 设计较为简单The value estimation design is fairly simple
当前的离线线性插值方式为 value 提供稠密监督，但较为启发式。论文认为"more principled data-driven or learning-based methods"有望带来更稳定的搜索效果。The current offline linear-interpolation scheme provides dense supervision for the value, but it is rather heuristic. The paper argues that "more principled data-driven or learning-based methods" could bring more stable search results.
计算开销限制部署平台Computational cost restricts the deployment platform
实验使用 RTX 4090 进行实时 GPU 推理。在计算资源受限的嵌入式机器人平台上，当前框架的可扩展性尚不明确，面向低算力平台的适配是未来工作方向之一。The experiments use an RTX 4090 for real-time GPU inference. On embedded robot platforms with limited computational resources the scalability of the current framework remains unclear, and adapting it to low-compute platforms is one direction for future work.
