UF-OPS: Update-Free On-Policy Steering via Verifiers
无需更新策略参数——用策略自身的成功/失败 rollout 训练 verifier，在测试时引导动作走向成功No policy-parameter updates required — a verifier is trained on the policy's own successful/failed rollouts and steers actions toward success at test time
Maria Attarian, Ian Vyse, Claas Voelcker, Jasper Gerigk, Evgenii Opryshko, Anas Almasri, Sumeet Singh, Yilun Du, Igor Gilitschenski
University of Toronto · Google DeepMind · University of Alberta · UT Austin · Harvard University
Behavior Cloning (BC) 策略往往在需要精准操作的关键接触点上失败。UF-OPS 提出一种
update-free 框架：收集策略初始评估产生的成功与失败轨迹，训练轻量级
verifier 对 (state, action) 的成功可能性打分，然后在推断时用 verifier 将策略推向
得分更高的动作——全程不修改 base policy 参数。在 5 个真实双臂操作任务上平均提升成功率 49%。Behavior Cloning (BC) policies tend to fail at the critical contact points that demand precise manipulation. UF-OPS proposes an update-free framework: collect the successful and failed trajectories produced by the policy's own initial evaluation, train a lightweight verifier that scores the success likelihood of a (state, action) pair, and then use the verifier at inference time to push the policy toward higher-scoring actions — all without modifying the base policy parameters. On 5 real-world bimanual manipulation tasks it raises the average success rate by 49%.
BC 策略在真实操作中表现出明显的脆弱性（brittleness），失败往往集中在需要精准动作的
fine-grained 交互点上。现有改进方案——如 fine-tuning、额外数据采集——代价高昂，
且可能导致灾难性遗忘（catastrophic forgetting），对 black-box 扩散策略更是难以直接适用。BC policies exhibit pronounced brittleness in real-world manipulation, and their failures concentrate on fine-grained interaction points that require precise motion. Existing remedies — such as fine-tuning or collecting extra data — are expensive, may cause catastrophic forgetting, and are hard to apply directly to black-box diffusion policies.
"failures contain crucial information about bottleneck states that require precise manipulation…
[we] leverage the policy's own evaluation data to improve its performance."
图 1：方法总览。
UF-OPS 依赖策略自身的评估数据来提升性能。训练小型 verifier 并随后在推断时利用它进行 steering，
无需昂贵的数据收集和资源密集型 fine-tuning 即可实现策略性能提升。
（"Our method relies on a policy's own evaluation data to improve its performance."）Figure 1: Method overview. UF-OPS relies on the policy's own evaluation data to improve performance. A small verifier is trained and then used for steering at inference time, delivering policy performance gains without expensive data collection or resource-intensive fine-tuning. ("Our method relies on a policy's own evaluation data to improve its performance.")
49%真实任务平均成功率提升（vs. base policy）average success-rate gain on real-world tasks (vs. base policy)
UF-OPS 分四步运作：① 在专家数据上训练 BC 策略；② 收集成功与失败的 rollout；
③ 训练 verifier 对 (st, at, t) 的成功可能性打分；
④ 在执行时用 verifier 配合 steering 策略引导动作选择。UF-OPS operates in four steps: ① train a BC policy on expert data; ② collect successful and failed rollouts; ③ train a verifier that scores the success likelihood of (st, at, t); ④ at execution time, use the verifier together with a steering strategy to guide action selection.
图 2：方法流程图。
给定基础策略，policy evaluation 提供成功与失败 rollout，用于训练 verifier 函数
（对转移 (s, a) 的成功可能性打分），再将 verifier 与 steering 策略结合提升策略性能。Figure 2: Method pipeline. Given a base policy, policy evaluation supplies successful and failed rollouts, which are used to train a verifier function (scoring the success likelihood of a transition (s, a)); the verifier is then combined with a steering strategy to improve policy performance.
Verifier 训练Verifier training
Verifier 以状态-动作-时间步三元组 (st, at, t) 为输入，输出成功概率。
论文提出两种互补的 verifier 类型：The verifier takes a state-action-timestep triple (st, at, t) as input and outputs a success probability. The paper proposes two complementary verifier types:
Success Classifier C
训练二分类器预测轨迹是否来自成功 rollout。损失函数结合 BCE 与辅助对比损失：Train a binary classifier to predict whether a trajectory comes from a successful rollout. The loss combines BCE with an auxiliary contrastive loss:
L = LBCE + λaux · Laux
（λaux = 0.1，margin m = 1.0）L = LBCE + λaux · Laux (λaux = 0.1, margin m = 1.0)
对比损失 Laux = max(0, ε − ||z(s⁺,a⁺,t⁺) − z(s⁻,a⁻,t⁻)||)²
鼓励成功与失败样本的嵌入在特征空间中分离。The contrastive loss Laux = max(0, ε − ||z(s⁺,a⁺,t⁺) − z(s⁻,a⁻,t⁻)||)² encourages the embeddings of successful and failed samples to separate in feature space.
Time-to-Success Estimator Q
估计折扣后的剩余成功回报：Estimates the discounted remaining success return:
Q(st, at, t) = γ^(T−t) · rT
使用稀疏最终状态奖励与指数折扣，无需 bootstrapping，避免价值估计的误差传播。Uses a sparse final-state reward with exponential discounting; no bootstrapping is required, which avoids error propagation in value estimation.
Steering 策略Steering strategies
Best-of-N (BoN)
从策略采样 N 个候选动作，选取 verifier 得分最高者：Sample N candidate actions from the policy and keep the one with the highest verifier score:
a* = argmaxa ∈ A V(st, at, t)
大多数实验取 N = 10。适用于任意 black-box 策略。Most experiments use N = 10. Applicable to any black-box policy.
Classifier Guidance (CG)
对 diffusion policy 在去噪过程中施加 verifier 梯度（Forward Universal Guidance）：Applies verifier gradients to a diffusion policy during denoising (Forward Universal Guidance):
â⁰t ← â⁰t + λ∇â₀ log C(st, â⁰t, t)
λ 按任务调整（0.05–0.8）。需要可微策略，但能在连续空间中更精细地引导动作分布。λ is tuned per task (0.05–0.8). It requires a differentiable policy, but can steer the action distribution more finely in continuous space.
Verifier 网络结构Verifier network architecture
低维输入：双 2-层 encoder（观测/动作）+ sinusoidal 时间步嵌入（dim=64）+ 2-block MLP
（linear-layernorm-ReLU-dropout 0.5）。图像输入：带 spectral norm 的 encoder +
噪声增强（std=0.02，仅训练时）+ 冻结的 base policy vision encoder，时间嵌入 dim=128–256。Low-dimensional input: two 2-layer encoders (observation/action) + sinusoidal timestep embedding (dim=64) + 2-block MLP (linear-layernorm-ReLU-dropout 0.5). Image input: an encoder with spectral norm + noise augmentation (std=0.02, training only) + the frozen base policy vision encoder, with time embedding dim=128–256.
03实验 ExperimentsExperiments
实验覆盖三个层次：教学性导航任务（验证原理）、单任务仿真（Robomimic）、
多任务 VLA 仿真（Libero），以及 5 个真实 Aloha 双臂操作任务。
基线包括 SAILOR、DSRL、V-GPS 等，均在相同 base diffusion policy 上评测。The experiments span three levels: a pedagogical navigation task (validating the principle), single-task simulation (Robomimic), multi-task VLA simulation (Libero), plus 5 real-world Aloha bimanual manipulation tasks. Baselines include SAILOR, DSRL and V-GPS, all evaluated on the same base diffusion policy.
图 3：Aloha 双臂系统上的 5 个真实任务。
从左到右：(a) 将方块放到纸板上；(b) 将球放入碗中；(c) 右手拿锤传给左手后放入箱子；
(d) 右手拿笔帽左手拿笔套上笔帽；(e) 将绿杯叠到紫杯上。Figure 3: The 5 real-world tasks on the Aloha bimanual system. From left to right: (a) place the block on the cardboard; (b) put the ball into the bowl; (c) the right arm picks up the hammer, hands it to the left arm, and places it in the box; (d) the right arm holds the cap while the left arm holds the pen and caps it; (e) stack the green cup onto the purple cup.
每个任务约 60 条评估轨迹用于 verifier 训练，每种方法评测 20 次（20 steered evaluations）：About 60 evaluation rollouts per task are used for verifier training, and every method is evaluated 20 times (20 steered evaluations):
注意：SAILOR 在图像 Transport 任务上仅得 5.9±1.46%，远低于 base policy 的 58.1%，
说明 off-policy verifier 的迁移失败。UF-OPS 全面优于或持平 base policy。Note: SAILOR reaches only 5.9±1.46% on the image Transport task, far below the base policy's 58.1%, which indicates a transfer failure of the off-policy verifier. UF-OPS is uniformly better than or on par with the base policy.
Q×CG 在单个任务上提升最高达 74.6 pp（task43：9.9% → 84.5%），
最低 14.1 pp（task26：5.9% → 20.0%）。On individual tasks, Q×CG achieves a maximum gain of 74.6 pp (task43: 9.9% → 84.5%) and a minimum of 14.1 pp (task26: 5.9% → 20.0%).
图 4：真实任务结果。
(a) 各任务成功率对比；(b) 各引导变体相对 base policy 的提升量，
附 95% Newcombe 置信区间（二项比例差检验）。Figure 4: Real-world task results. (a) Success-rate comparison across tasks; (b) the gain of each steering variant relative to the base policy, with 95% Newcombe confidence intervals (test for the difference of binomial proportions).
Ablation：对比损失的作用（Table II）Ablation: effect of the contrastive loss (Table II)
在低维 Transport 任务上，加入对比损失（λ=0.1）将 C×BoN 从 56.0±3.08 提升至 62.7±3.00；
在 Square 任务上效果相当（85.5 vs 86.0），说明对比损失对难度更高的任务帮助更大。On the low-dimensional Transport task, adding the contrastive loss (λ=0.1) raises C×BoN from 56.0±3.08 to 62.7±3.00; on Square the effect is comparable (85.5 vs 86.0), suggesting that the contrastive loss helps more on harder tasks.
Ablation：On-Policy vs Off-Policy（Table IV）Ablation: on-policy vs off-policy (Table IV)
用不同数据集训练的策略（PH→MH 或 MH→PH 交叉）的 verifier 无法稳定提升性能，
多数情况与 base policy 持平甚至略低，证实了 on-policy 数据的必要性。Verifiers taken from policies trained on a different dataset (PH→MH or MH→PH cross-transfer) fail to improve performance reliably, in most cases matching or even slightly falling below the base policy, which confirms the necessity of on-policy data.
图 5：教学性导航任务。
(a) 专家轨迹经过两个通道，未引导的 base policy 频繁失败于窄通道。
UF-OPS 将流量重定向至更宽的安全通道。
(b) Classifier 将宽通道区域识别为更安全的选择（紫色区域）。
Base policy 成功率 52%，UF-OPS 提升至 85.1%（N=30 BoN）。Figure 5: Pedagogical navigation task. (a) Expert trajectories pass through two corridors, and the unsteered base policy frequently fails in the narrow one. UF-OPS redirects the traffic to the wider, safer corridor. (b) The classifier identifies the wide-corridor region as the safer choice (purple region). The base policy succeeds 52% of the time; UF-OPS raises this to 85.1% (N=30 BoN).
04局限性 LimitationsLimitations
Note:以下局限性均为论文作者明确陈述（stated by the authors）。All limitations below are explicitly stated by the authors of the paper.
Verifier 泛化性未评估Verifier generalization not evaluated
"verifiers are trained on all downstream tasks and we do not evaluate verifier generalization
beyond training tasks." 即当前 verifier 针对每个任务单独训练，跨任务迁移能力尚未探究。"verifiers are trained on all downstream tasks and we do not evaluate verifier generalization beyond training tasks." That is, each verifier is currently trained separately for a single task, and cross-task transfer ability has not yet been investigated.
真实场景需人工标注成功/失败Real-world use requires manual success/failure labeling
"applying this work to real still maintains a small overhead of manual labeling of successful
and failed rollouts." 在真实机器人上，需要人工判断每条轨迹是成功还是失败后才能训练 verifier，
带来一定的标注成本。"applying this work to real still maintains a small overhead of manual labeling of successful and failed rollouts." On a real robot, a human must judge whether each rollout succeeded or failed before the verifier can be trained, which incurs a certain labeling cost.
Classifier Guidance 对引导强度敏感Classifier Guidance is sensitive to guidance strength
"specifically classifier guidance as a method of steering is proven to be very sensitive to
guidance strength, which is a free hyperparameter tunable on a per-task level. In addition,
tuning this in real potentially poses some safety risks." 引导强度 λ 需要逐任务调整，
且在真实机器人上调参本身存在安全风险。"specifically classifier guidance as a method of steering is proven to be very sensitive to guidance strength, which is a free hyperparameter tunable on a per-task level. In addition, tuning this in real potentially poses some safety risks." The guidance strength λ has to be tuned task by task, and tuning it on a real robot itself carries safety risks.
