When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering
让校准过的 VLM 验证器决定：直接执行、开口澄清、还是请求人类干预Let a calibrated VLM verifier decide: act directly, ask for clarification, or request human intervention
Jessie Yuan, Yilin Wu, Andrea Bajcsy
论文提出 UPS（Uncertainty-aware Policy Steering），用 conformal prediction 校准 VLM 验证器在“策略引导（policy steering）”中的不确定性，使系统能同时区分任务指令的语义歧义与底层策略的执行能力不足，并据此在执行动作、开口澄清、请求人类干预三种策略间做出有统计保证的选择；干预数据再通过 residual learning 持续改进底层策略。The paper proposes UPS (Uncertainty-aware Policy Steering), which uses conformal prediction to calibrate the uncertainty of a VLM verifier inside "policy steering", so that the system can simultaneously tell semantic ambiguity in the task instruction apart from insufficient execution capability of the low-level policy, and on that basis make a statistically guaranteed choice among three strategies: executing the action, asking for clarification, and requesting human intervention; the intervention data then continually improves the low-level policy through residual learning.
Carnegie Mellon UniversityFranka Emika Panda + Robomimic 仿真Franka Emika Panda + Robomimic simulationconformal prediction 校准conformal prediction calibration📄 arXiv:2602.22474Project page
Policy steering 是一种部署时用学习到的验证器（如 VLM）从预训练策略（如 diffusion policy）采样的动作中筛选对齐任务的候选，从而在线调整机器人行为的新兴范式。VLM 因其推理能力被视为理想的“开放世界”验证器，但既有框架都隐含假设验证器是良好校准（well-calibrated）的——这一假设在实践中常常不成立。Policy steering is an emerging paradigm that adjusts robot behavior online at deployment time by using a learned verifier (such as a VLM) to select task-aligned candidates among the actions sampled from a pretrained policy (such as a diffusion policy). VLMs are seen as ideal "open-world" verifiers thanks to their reasoning ability, yet existing frameworks all implicitly assume that the verifier is well-calibrated — an assumption that often fails in practice.
"In practice, this assumption often fails. VLM verifiers can be overconfident when task instructions are ambiguous or under-specified, leading to confident selection of misaligned behaviors. Moreover, when the low-level policy is fundamentally incapable of accomplishing the task under the current conditions, all candidate samples may be unsatisfactory."
论文指出，未校准的验证器面临两类不确定性：高层的语义任务歧义（task ambiguity）与低层的动作不可行性 / 策略能力不足（action infeasibility / incapability）。作者的实验发现，未校准验证器在这两种情形下都可能"自信地"选出一个实际错误的选项，而不是识别限制并停止执行。UPS 的核心思路是让验证器的不确定性直接映射到三种解决策略：执行高置信度动作、通过自然语言澄清任务歧义、或在策略被判定无能力时请求人类干预来重新训练策略。The paper points out that an uncalibrated verifier faces two kinds of uncertainty: high-level semantic task ambiguity and low-level action infeasibility / policy incapability. The authors’ experiments find that in both situations an uncalibrated verifier may "confidently" pick an option that is in fact wrong, instead of recognizing the limitation and stopping execution. The core idea of UPS is to map the verifier’s uncertainty directly onto three resolution strategies: executing a high-confidence action, clarifying task ambiguity through natural language, or requesting human intervention to retrain the policy when the policy is judged incapable.
Figure 1: Uncertainty-Aware Policy Steering. 框架通过 conformal prediction 校准用于策略引导的 VLM 验证器，使 VLM 能在"用自然语言询问用户"到"请求重新训练底层控制策略"之间选择恰当的不确定性解决方式。Figure 1: Uncertainty-Aware Policy Steering. The framework calibrates, via conformal prediction, the VLM verifier used for policy steering, so that the VLM can pick the appropriate way of resolving uncertainty, anywhere between "asking the user in natural language" and "requesting a retraining of the low-level control policy".
30%相较未校准基线，ambiguous 场景成功率提升Success-rate gain in ambiguous scenarios over the uncalibrated baseline
0.06UPS 在 PnP Cup 任务上的人类干预率（HG-DAgger 0.1 / EnsembleDAgger 0.16）Human intervention rate of UPS on the PnP Cup task (HG-DAgger 0.1 / EnsembleDAgger 0.16)
85%PnP Cup 任务最终整体成功率Final overall success rate on the PnP Cup task
02 Method
UPS 分三步：(1) 用 world model 预测动作样本的长时程结果并生成文本叙述（narration）；(2) 用 conformal prediction 校准 VLM 验证器在这些叙述上的不确定性，得到有覆盖保证的预测集；(3) 根据预测集大小和构成选择解决策略——直接执行、语言澄清，或请求低层干预并通过 residual learning 持续改进策略。UPS proceeds in three steps: (1) use a world model to predict the long-horizon outcomes of action samples and to produce a textual narration; (2) use conformal prediction to calibrate the uncertainty of the VLM verifier over these narrations, obtaining a prediction set with a coverage guarantee; (3) choose a resolution strategy according to the size and composition of the prediction set — direct execution, language clarification, or requesting low-level intervention and continually improving the policy through residual learning.
Figure 2: Outcome Prediction & Narration. 策略与 world model 交替执行，预测低层策略引发的长时程结果；解码出的观测被送入 VLM，用文本叙述这些结果。Figure 2: Outcome Prediction & Narration. The policy and the world model run alternately to predict the long-horizon outcomes induced by the low-level policy; the decoded observations are fed to the VLM, which narrates those outcomes in text.
作者将 VLM-in-the-loop 引导建模为一个开放式多选问答问题：给定当前观测 o_t，从策略采样一个长度为 H 的动作块 a_t:t+H，交给 world model（Dreamer-v3）预测未来观测 ô_t+1:t+H；由于低层策略（diffusion policy）通常只预测较短的动作块，短时程结果对验证器往往难以区分，因此论文将动作生成与"想象"（imagination）交替进行，拼接出对验证器更具信息量的长时程动作-结果序列。K 个并行样本的想象结果经 VLM（Gemini）叙述为文本，并额外附加一个"none of the above"选项以支持识别策略无能力（incapable）的情形。验证阶段，VLM 通过对多选项的 log-likelihood 选出最可能的单个 token 作为选择结果，这一形式与 VLM 的原生训练目标一致，也避免了开放式文本生成概率估计中的长度偏置。The authors cast VLM-in-the-loop steering as an open-ended multiple-choice question answering problem: given the current observation o_t, an action chunk a_t:t+H of length H is sampled from the policy and handed to a world model (Dreamer-v3), which predicts the future observations ô_t+1:t+H; since the low-level policy (a diffusion policy) usually predicts only rather short action chunks, short-horizon outcomes are often hard for the verifier to tell apart, so the paper interleaves action generation with "imagination", concatenating long-horizon action-outcome sequences that are far more informative for the verifier. The imagined outcomes of the K parallel samples are narrated into text by a VLM (Gemini), with an extra "none of the above" option appended so that an incapable policy can be identified. At verification time the VLM picks, by log-likelihood over the multiple choices, the single most likely token as its selection; this form matches the VLM’s native training objective and also avoids the length bias of estimating probabilities over open-ended text generation.
Conformal Prediction 校准与不确定性解决Conformal Prediction Calibration and Uncertainty Resolution
UPS 用 conformal prediction 校准 VLM 似然 ℙ^VLM，构造一个以用户指定概率 1−ϵ 保证包含正确动作的预测集，同时最小化集合大小。由于策略引导是序贯决策过程，输入并非独立同分布（i.i.d.），论文采用序列级校准以维持形式化覆盖保证（Theorem 1，Verification Coverage Guarantee）。预测集大小 > 1 时，机器人通过自然语言向用户澄清歧义并更新指令；当预测集仅包含"none of the above"选项时，机器人判定底层策略无能力，转而请求人类物理干预。收集到的干预数据被用于 residual learning，训练一个 residual policy 学习"干预动作 − 基础动作"的差值，从而在部署过程中持续改进底层策略能力，且只在真正必要时才消耗昂贵的人类干预。UPS uses conformal prediction to calibrate the VLM likelihood ℙ^VLM, constructing a prediction set that contains the correct action with a user-specified probability 1−ε while keeping the set as small as possible. Because policy steering is a sequential decision process, the inputs are not independent and identically distributed (i.i.d.), so the paper adopts sequence-level calibration in order to preserve the formal coverage guarantee (Theorem 1, Verification Coverage Guarantee). When the prediction set size is > 1, the robot clarifies the ambiguity with the user in natural language and updates the instruction; when the prediction set contains only the "none of the above" option, the robot judges the low-level policy to be incapable and instead requests physical human intervention. The collected intervention data is used for residual learning, training a residual policy to learn the difference "intervention action − base action", so that the capability of the low-level policy keeps improving during deployment while expensive human intervention is spent only when truly necessary.
03 Experiments
评测覆盖仿真（Robomimic square nut-on-peg 任务）与真实硬件（Franka Emika Panda，PnP Cup 与 Insert Block 两个任务）。基础策略为 image-conditioned diffusion policy，world model 为 Dreamer-v3；叙述用 Gemini-3-flash-preview，验证用 Gemini-2.0-flash。校准数据集含 80 个（观测，指令）对，40 条为 ambiguous 指令、40 条 straightforward，目标覆盖率 1−ϵ = 0.85。The evaluation covers simulation (the Robomimic square nut-on-peg task) and real hardware (Franka Emika Panda, with the two tasks PnP Cup and Insert Block). The base policy is an image-conditioned diffusion policy and the world model is Dreamer-v3; narration uses Gemini-3-flash-preview and verification uses Gemini-2.0-flash. The calibration dataset contains 80 (observation, instruction) pairs, 40 of them ambiguous instructions and 40 straightforward, with a target coverage of 1−ε = 0.85.
不确定性量化：Bayesian Intent 分数函数最优Uncertainty Quantification: the Bayesian Intent Score Function Wins
"our proposed Bayesian intent score function achieves the best balance between coverage and clarification rate across all three tasks. ... CoT-based methods perform poorly due to conservative reasoning ... Vanilla methods struggle with poor calibration, resulting in either overconfidence or severe under-coverage."
Figure 3: Uncertainty Quantification Results: Simulation. 对比 Vanilla、CoT 与 Bayesian Intent（Ours）三种打分方式下的 coverage、clarification rate 与 set size，虚线为目标覆盖率 1−ϵ=0.85。Figure 3: Uncertainty Quantification Results: Simulation. Coverage, clarification rate and set size are compared across the three scoring schemes Vanilla, CoT and Bayesian Intent (Ours); the dashed line is the target coverage 1−ε=0.85.
策略引导性能Policy Steering Performance
对比 Base Policy、Forewarn（未校准的 VLM-in-the-loop 引导）与 UPS w/ Clarification：Comparing Base Policy, Forewarn (uncalibrated VLM-in-the-loop steering) and UPS w/ Clarification:
"Forewarn improves the success rate in both straightforward scenarios and ambiguous scenarios ... but it achieves much smaller gain in ambiguous cases in both tasks (e.g., in the PnP Cup task improvement in straightforward = 45%, ambiguous = 15%) because the VLM verifier is overconfident. In contrast, UPS + w/ Clarification addresses uncertainty in ambiguous cases, further increasing success rate by 15%."
Figure 6: Success Rates Pre- and Post-Continual Learning: Hardware and Simulation. 在 20 条 straightforward 与 20 条 ambiguous 指令上部署机器人，每种场景取 20 次试验的平均成功率；UPS 收集数据的方式使 residual policy 训练后的最终成功率最大化。Figure 6: Success Rates Pre- and Post-Continual Learning: Hardware and Simulation. The robot is deployed on 20 straightforward and 20 ambiguous instructions, and the success rate is averaged over 20 trials per scenario; the way UPS collects data maximizes the final success rate after residual policy training.
Continual Learning：人类干预率与再部署表现Continual Learning: Intervention Rate and Redeployment Performance
与 HG-DAgger、EnsembleDAgger 相比，UPS 的人类干预率（干预步数 / 轨迹长度）最低：Compared with HG-DAgger and EnsembleDAgger, UPS has the lowest human intervention rate (intervention steps / trajectory length):
"For PnP Cup task, UPS requires interventions at a rate of 0.06, compared to 0.1 for HG-DAgger and 0.16 for EnsembleDAgger ... UPS' intervention rate is 0.05, compared to HG-DAgger's rate of 0.06 and EnsembleDAgger's rate of 0.26 in Insert Block. In simulation, the gap widens: UPS's rate is 0.058, while HG-DAgger has an intervention rate of 0.20 and EnsembleDAgger has 0.27."
任务Task
UPS
HG-DAgger
EnsembleDAgger
PnP Cup（硬件）PnP Cup (hardware)
0.06
0.10
0.16
Insert Block（硬件）Insert Block (hardware)
0.05
0.06
0.26
仿真Simulation
0.058
0.20
0.27
用干预数据训练 residual policy 后，UPS 在两个硬件任务的 ambiguous 场景上均获得 15% 的成功率提升，PnP Cup 任务的最终整体成功率达到 85%，与 Theorem 1 的覆盖保证一致（"the verifier can steer the low-level policy to success in 1−ϵ of scenarios"）。论文也指出，两种 DAgger 基线在从干预数据学习新行为的同时难以保持原有能力，residual policy 的分布可能偏向某些行为模式或引入新的"无能力"模式。After a residual policy is trained on the intervention data, UPS gains 15% in success rate in the ambiguous scenarios of both hardware tasks, and the final overall success rate of the PnP Cup task reaches 85%, in line with the coverage guarantee of Theorem 1 ("the verifier can steer the low-level policy to success in 1−ε of scenarios"). The paper also notes that the two DAgger baselines have trouble retaining their original capability while learning new behaviors from intervention data, and that the distribution of the residual policy may lean towards certain behavior modes or introduce new "incapable" modes.
04 Limitations
Note:以下第一条为论文 "VI Conclusion & Limitations" 中明确陈述（stated）；第二、三条为根据实验设置与方法设计推断（inferred）。The first item below is explicitly stated in the paper’s "VI Conclusion & Limitations"; the second and third are inferred from the experimental setup and the method design.
假设 world model 的想象结果与叙述完全准确（stated）Assumes the world model’s imagined outcomes and its narrations are perfectly accurate (stated)
论文原文："we assumed the imaginations from world model always match future outcomes and behavior narrations are correct and complete." 作者聚焦于 VLM 验证器的不确定性量化，并未对 world model 本身及其文本叙述的不确定性建模，指出未来工作可以纳入 world model（及叙述）的不确定性以获得更鲁棒的系统。The paper states: "we assumed the imaginations from world model always match future outcomes and behavior narrations are correct and complete." The authors focus on uncertainty quantification for the VLM verifier and do not model the uncertainty of the world model itself or of its textual narrations, noting that future work could incorporate the uncertainty of the world model (and of the narration) to obtain a more robust system.
仿真环境下覆盖率略低于目标（inferred）Coverage falls slightly short of the target in simulation (inferred)
论文提到在仿真和 Insert Block 硬件任务中获得的覆盖率略低于目标 1−ϵ，作者将此归因于"the challenges of simulating precise contact-rich manipulation tasks with the current world model"——即接触密集型精细操作任务下 world model 的建模精度限制。The paper mentions that the coverage obtained in simulation and on the Insert Block hardware task is slightly below the target 1−ε, which the authors attribute to "the challenges of simulating precise contact-rich manipulation tasks with the current world model" — that is, the modeling precision of the world model is limited on contact-rich, fine-grained manipulation tasks.
Residual policy 可能引入行为分布偏置（inferred）The residual policy may introduce a behavior-distribution bias (inferred)
论文观察到基线 DAgger 方法在学习新干预行为时难以保持原有能力的问题，并推测 UPS 训练的 residual policy 也可能"bias the final policy distribution towards certain behavior modes of doing the task, or may push the model towards new incapable modes"，这一风险在方法设计上尚未被完全消除。The paper observes that baseline DAgger methods have trouble retaining their original capability while learning new intervention behaviors, and conjectures that the residual policy trained by UPS may likewise "bias the final policy distribution towards certain behavior modes of doing the task, or may push the model towards new incapable modes"; this risk has not yet been fully eliminated by the method design.
