不确定性免费获得：基于 Diffusion Model 的人机协作（Human-in-the-Loop）策略Uncertainty Comes for Free: Human-in-the-Loop Policies with Diffusion Models
Uncertainty Comes for Free: Human-in-the-Loop Policies with Diffusion Models
Zhanpeng He · Yifeng Cao · Matei Ciocarlie | 2025 · arXiv:2503.01876
本文提出利用扩散策略（diffusion policy）生成过程中固有的去噪不确定性，免训练地识别机器人何时需要请求人工干预，
从而构建高效的半自主 Human-in-the-Loop（HitL）系统，同时将干预数据用于策略微调以持续改善自主性能。This paper proposes to exploit the denoising uncertainty inherent in the generation process of a diffusion policy to identify, training-free, when a robot needs to request human intervention, thereby building an efficient semi-autonomous Human-in-the-Loop (HitL) system, while the intervention data is used for policy fine-tuning to keep improving autonomous performance.
在实际机器人部署中，人工持续监控代价高昂、难以规模化。现有 HitL 方法要求人工标注干预时机，或训练额外分类器，
成本高且泛化性差。本文发现：扩散模型的去噪过程本身就隐含了丰富的不确定性信号，无需任何额外监督即可驱动智能干预请求。In real-world robot deployment, continuous human monitoring is costly and hard to scale. Existing HitL methods require humans to annotate when to intervene, or to train an extra classifier, which is expensive and generalizes poorly. This paper finds that the denoising process of a diffusion model already encodes rich uncertainty signals, which can drive intelligent intervention requests without any additional supervision.
"We propose using denoising uncertainty as a metric for deciding when to request (human) expert assistance."
——无需在训练时引入人工标注，不确定性信号从扩散策略的生成过程中免费获得。— No human annotation has to be introduced at training time; the uncertainty signal comes for free from the generation process of the diffusion policy.
图 1：方法概览。
HitL 策略部署框架：机器人持续估计去噪不确定性（denoising uncertainty），当不确定性超过阈值时向人类操作员请求远程操控辅助（teleoperation），
所收集的干预数据进一步用于策略微调。不确定性度量无需额外训练，直接来自扩散策略的噪声预测过程。Figure 1: Method overview. The HitL policy deployment framework: the robot continuously estimates the denoising uncertainty, and when the uncertainty exceeds a threshold it requests teleoperation assistance from a human operator; the collected intervention data is further used for policy fine-tuning. The uncertainty measure needs no extra training and comes directly from the noise prediction process of the diffusion policy.
≈8.3%真实机器人实验中需人工干预的时间步比例，即可达到 100% 成功率Fraction of time steps requiring human intervention in the real-robot experiments, enough to reach a 100% success rate
47%HitL 微调后平均成功率提升幅度（相比基线）Average success-rate improvement after HitL fine-tuning (vs. baseline)
8.0开抽屉任务所需最少干预步数（vs. ThriftyDAgger 的 17.2 步）Minimum intervention steps for the open-drawer task (vs. 17.2 steps for ThriftyDAgger)
0 额外标注0 extra labels训练期间无需任何人工干预标注，不确定性"免费"获得No human intervention labels are needed during training; the uncertainty comes "for free"
现有方法的不足Shortcomings of existing methods
现有 HitL 机器人策略方法可分为两类：Existing HitL robot policy methods fall into two categories:
基于人工标注的方法（如 DAgger、ThriftyDAgger）：需要在训练时人工标记"何时应干预"，成本高、难扩展。Annotation-based methods (e.g. DAgger, ThriftyDAgger): humans must label "when to intervene" during training, which is costly and hard to scale.
基于额外模型的方法（如 HULA、Sentinel）：需要训练独立的风险预测模型，引入额外的数据和计算开销。Extra-model-based methods (e.g. HULA, Sentinel): a separate risk prediction model has to be trained, adding data and computation overhead.
本文：直接复用扩散策略已学到的噪声预测机制，零成本获取不确定性，无需改变训练流程。This paper: directly reuses the noise prediction mechanism already learned by the diffusion policy, obtaining uncertainty at zero cost and without changing the training pipeline.
02方法Method
核心思路：在当前末端执行器位姿附近采样多个探针点，通过扩散策略的噪声预测网络（noise prediction network）得到对应的预测动作分布，
用高斯混合模型（GMM）拟合后计算模间散度与模内方差，合并为统一的不确定性度量。部署时若不确定性超过验证集第 95 百分位阈值，则触发人工干预请求。Core idea: sample several probe points around the current end-effector pose, obtain the corresponding predicted action distributions through the noise prediction network of the diffusion policy, fit them with a Gaussian mixture model (GMM), and compute the inter-mode divergence and the intra-mode variance, merging them into a single uncertainty measure. At deployment, an intervention request is triggered whenever the uncertainty exceeds the 95th percentile threshold of the validation set.
图 2：模拟环境中的三类部署挑战。
(a) 分布偏移（Distribution shift）：测试场景与训练分布不同；
(b) 部分可观测性（Partial observability）：遮挡导致关键观测缺失；
(c) 动作多模性（Action multi-modality）：同一状态下存在多个合理动作。
本文方法在三种场景中均能有效触发干预请求。Figure 2: Three types of deployment challenge in simulation. (a) Distribution shift: the test scenario differs from the training distribution; (b) Partial observability: occlusion makes key observations missing; (c) Action multi-modality: several reasonable actions exist in the same state. The proposed method triggers intervention requests effectively in all three scenarios.
步骤一：采样探针点与去噪预测Step 1: Probe sampling and denoising prediction
在当前末端执行器位姿 xt 周围，以采样半径 r（实验中取 0.05 m）均匀采样 n 个探针位姿 {si}。
对每个探针点，从噪声初始化动作轨迹出发，运行扩散策略的去噪过程（仅推理，无梯度）得到预测动作向量：Around the current end-effector pose xt, n probe poses {si} are sampled uniformly within a sampling radius r (0.05 m in the experiments). For each probe point, starting from a noise-initialized action trajectory, the denoising process of the diffusion policy is run (inference only, no gradients) to obtain the predicted action vector:
atk-1 = β(atk − γ · εθ(ot, atk, k) + N(0, σ²I))
其中 εθ 为已训练的噪声预测网络，ot 为当前观测。所有探针点的去噪方向向量构成集合 Vts。Here εθ is the trained noise prediction network and ot is the current observation. The denoising direction vectors of all probe points form the set Vts.
步骤二：GMM 拟合与不确定性计算Step 2: GMM fitting and uncertainty computation
对去噪方向向量集 Vts 拟合高斯混合模型（GMM），每个模代表一种可能的动作意图。不确定性由两部分组成：A Gaussian mixture model (GMM) is fitted to the set of denoising direction vectors Vts, where each mode represents one possible action intent. The uncertainty consists of two parts:
模间散度 D：使用余弦相似度衡量各 GMM 模均值向量之间的方向差异。模式越分散，散度越大。Inter-mode divergence D: cosine similarity is used to measure the directional differences among the mean vectors of the GMM modes. The more dispersed the modes, the larger the divergence.
模内方差 Varg：衡量同一 GMM 模内的预测一致性。Intra-mode variance Varg: measures the prediction consistency within a single GMM mode.
Uncertainty(ot) = D(Vts) + α · Varg(Vts)
其中 α 为缩放系数（实验中取较小值 0.01–0.1，以强调模间散度的贡献）。Here α is a scaling coefficient (set to a small value of 0.01–0.1 in the experiments, so as to emphasize the contribution of the inter-mode divergence).
步骤三：HitL 部署与阈值触发Step 3: HitL deployment and threshold triggering
阈值取验证集（held-out validation set）不确定性分布的 第 95 百分位数，无需人工调参。
当实时不确定性超过阈值时，机器人暂停自主执行并请求人工远程操控（teleoperation）若干步，随后恢复自主。
干预数据实时收集，用于后续微调。The threshold is the 95th percentile of the uncertainty distribution on a held-out validation set, requiring no manual tuning. When the online uncertainty exceeds the threshold, the robot pauses autonomous execution and requests human teleoperation for a few steps, then resumes autonomy. The intervention data is collected online and used for subsequent fine-tuning.
图 3：预测不确定性的定性可视化。
颜色越亮（lighter）代表不确定性越高。可以看出，在机器人遭遇分布偏移或动作歧义的区域，本方法的不确定性估计显著升高，
符合直觉——模型在这些位置"更不确定"应该执行什么动作。Figure 3: Qualitative visualization of the predicted uncertainty. Lighter colors indicate higher uncertainty. In regions where the robot encounters distribution shift or action ambiguity, the uncertainty estimate of this method rises markedly, matching intuition — at these locations the model is "more uncertain" about which action to take.
步骤四：干预数据微调（Fine-tuning）Step 4: Fine-tuning on intervention data
收集到的人工干预轨迹可直接用于对扩散策略进行微调。实验表明，与收集完整轨迹相比，
HitL 干预数据更具针对性，以更少数据量实现更好的性能提升，且不产生灾难性遗忘（catastrophic forgetting）。The collected human intervention trajectories can be used directly to fine-tune the diffusion policy. Experiments show that, compared with collecting full trajectories, HitL intervention data is more targeted, achieving a better performance gain with less data and without catastrophic forgetting.
03实验Experiments
实验在模拟环境（三个操控任务，三类挑战场景）和真实机器人（四个任务）上进行。
评估指标：达到 100% 成功率所需的平均人工干预步数（越少越好）；微调后自主成功率。
基线包括：Diff-DAgger、ThriftyDAgger、HULA-offline、Sentinel、Output Variance。Experiments are run in simulation (three manipulation tasks, three categories of challenge) and on a real robot (four tasks). Evaluation metrics: the average number of human intervention steps needed to reach a 100% success rate (lower is better), and the autonomous success rate after fine-tuning. Baselines include Diff-DAgger, ThriftyDAgger, HULA-offline, Sentinel and Output Variance.
模拟实验：达到 100% 成功率所需干预步数（Table I，越少越好）Simulation: intervention steps needed to reach a 100% success rate (Table I, lower is better)
在四个真实操控任务上，本文方法仅需约 8.3% 的时间步进行人工干预，即可实现 100% 任务成功率；
而全轨迹人工操控基线需要 80.0–111.9 个干预步数。本方法在所有场景下均优于 Sentinel 和 Output Variance 基线。On four real manipulation tasks, this method needs human intervention on only about 8.3% of the time steps to achieve a 100% task success rate, whereas the full-trajectory teleoperation baseline requires 80.0–111.9 intervention steps. The method outperforms the Sentinel and Output Variance baselines in all scenarios.
图 5：真实机器人实验设置与结果。
设计了四个任务以模拟三类挑战（分布偏移、部分可观测、动作多模性）。本文方法在全部真实任务中以极少干预步数实现 100% 成功率，
显著优于所有基线方法。Figure 5: Real-robot experimental setup and results. Four tasks are designed to simulate the three types of challenge (distribution shift, partial observability, action multi-modality). The proposed method achieves a 100% success rate on all real tasks with very few intervention steps, clearly outperforming all baselines.
微调实验（Table IV，Lift-real 任务）Fine-tuning experiments (Table IV, Lift-real task)
使用 HitL 收集的 80 个数据样本对策略微调后，自主成功率达到 0.63；
而使用完整轨迹微调（132 个样本）仅达到 0.31。平均来看，HitL 微调带来 47% 的成功率提升，
且数据量更少、针对性更强。After fine-tuning the policy with the 80 data samples collected by HitL, the autonomous success rate reaches 0.63, whereas fine-tuning with full trajectories (132 samples) reaches only 0.31. On average, HitL fine-tuning brings a 47% gain in success rate, with less and more targeted data.
图 4：微调效果分析。
左图：Lift-sim 任务中，不同干预步数下微调后的平均成功率。HitL 方法以更少步数达到更高成功率。
右图：不确定性阈值选择的敏感性分析，第 95 百分位阈值在多数任务中表现稳定。Figure 4: Analysis of the fine-tuning effect. Left: average success rate after fine-tuning under different numbers of intervention steps on the Lift-sim task. The HitL method reaches a higher success rate with fewer steps. Right: sensitivity analysis of the uncertainty threshold choice; the 95th percentile threshold is stable across most tasks.
消融实验（Ablations）Ablations
采样半径（Table V）：最优半径为 0.05 m，干预步数 20 步，成功率 0.63；
过小（0.01 m）成功率降至 0.46，过大（0.1 m）降至 0.53。 缩放系数 α（Table VI）：α = 0.01–0.1（强调模间散度）时效果最优，
Cup-stacking 微调仅需 5 步干预；α = 0.5 时需 7 步。 阈值选择：95 百分位阈值在多类任务中均能稳定地以较低干预量维持 100% 成功率，无需逐任务调整。Sampling radius (Table V): the optimal radius is 0.05 m, with 20 intervention steps and a success rate of 0.63; too small (0.01 m) lowers the success rate to 0.46, too large (0.1 m) lowers it to 0.53. Scaling coefficient α (Table VI): α = 0.01–0.1 (emphasizing the inter-mode divergence) works best, and Cup-stacking fine-tuning needs only 5 intervention steps; α = 0.5 needs 7 steps. Threshold choice: the 95th percentile threshold stably maintains a 100% success rate with relatively few interventions across task types, with no per-task tuning.
图 7：与 Sentinel 和 Output Variance 基线的定性对比。
(a) Sentinel 方法在错误时机触发干预，干预效率低；(b) Output Variance 方法对不确定性估计不准确；
(c) 本文方法在真正需要帮助时精准触发，干预更少、更有效。Figure 7: Qualitative comparison with the Sentinel and Output Variance baselines. (a) Sentinel triggers interventions at the wrong moments and is inefficient; (b) Output Variance estimates uncertainty inaccurately; (c) the proposed method triggers precisely when help is truly needed, with fewer and more effective interventions.
04局限性Limitations
说明：论文未设立独立的"Limitations"章节。以下各点综合论文正文中的隐含假设整理，
均标注为 推断（inferred），并在可能的情况下注明论文原文依据。Note: The paper has no standalone "Limitations" section. The points below are compiled from implicit assumptions in the main text of the paper and are all marked as inferred, with the supporting original text indicated where possible.
控制空间限制（推断）：仅适用于任务空间绝对末端位姿控制Control-space restriction (inferred): applicable only to task-space absolute end-effector pose control
本方法在探针采样时假设动作空间为末端执行器的绝对位姿（absolute end-effector poses）。
对于关节空间控制、速度控制或其他参数化形式的策略，探针采样策略需重新设计，方法的直接适用性受限。When sampling probes, this method assumes that the action space consists of absolute end-effector poses. For policies using joint-space control, velocity control or other parameterizations, the probe sampling scheme has to be redesigned, so the direct applicability of the method is limited.
阈值依赖验证集（推断）：需要代表性验证数据Threshold depends on the validation set (inferred): representative validation data is required
不确定性阈值取验证集的第 95 百分位。若验证集分布与实际部署环境差异过大，阈值可能失准，导致干预过多或过少。
论文中所有实验的验证集均来自与训练分布相近的场景。The uncertainty threshold is the 95th percentile of the validation set. If the validation distribution differs too much from the actual deployment environment, the threshold may become miscalibrated, leading to too many or too few interventions. In all experiments of the paper, the validation sets come from scenarios close to the training distribution.
部分可观测场景的固有局限（论文明确指出）Inherent limitation under partial observability (explicitly stated in the paper)
论文指出，Case 2（部分可观测性）"is the most difficult, since correct decision making is impossible without changing the available observation"。
即使人工干预也无法解决传感器本身的信息缺失，方法对此类场景的帮助有限。The paper points out that Case 2 (partial observability) "is the most difficult, since correct decision making is impossible without changing the available observation". Even human intervention cannot make up for the information missing from the sensors themselves, so the method offers limited help in such scenarios.
假设人工干预总是有效（推断）Assumes that human intervention is always effective (inferred)
方法假设人工操作员的干预"effective for task completion"，未对操作员技能水平或干预质量进行建模。
若操作员本身表现不佳，收集到的干预数据可能对微调有负面影响。The method assumes that the intervention of the human operator is "effective for task completion", and does not model operator skill level or intervention quality. If the operator himself performs poorly, the collected intervention data may have a negative effect on fine-tuning.
真实机器人实验规模较小（推断）Small scale of the real-robot experiments (inferred)
真实机器人实验仅在四个相对简单的操控任务上进行，任务复杂度和场景多样性有限。
方法在长时程任务、高自由度机器人或非结构化环境中的泛化性尚未验证。The real-robot experiments are conducted on only four relatively simple manipulation tasks, with limited task complexity and scene diversity. The generalization of the method to long-horizon tasks, high-degree-of-freedom robots or unstructured environments has not yet been validated.
