利用不确定性估计加速残差强化学习

01 动机Motivation

Residual RL 是将预训练策略与强化学习结合的流行范式——只训练一个输出"修正量"的小策略，而不是重新训练整个网络。但两个关键瓶颈制约了其实用性：Residual RL is a popular paradigm for combining pretrained policies with reinforcement learning: only a small policy that outputs "corrections" is trained, instead of retraining the entire network. Two key bottlenecks, however, limit its practicality:

问题一：低效探索Problem 1: Inefficient Exploration

残差策略在整个状态空间均匀随机探索，而在基础策略已经表现良好的区域浪费大量样本。在稀疏奖励环境下，这一问题尤为突出。The residual policy explores uniformly at random over the whole state space, wasting a large number of samples in regions where the base policy already performs well. This is especially pronounced in sparse-reward environments.

问题二：仅支持确定性策略Problem 2: Deterministic Policies Only

现有 off-policy Residual RL 方法（如 TD3+BC、SAC）假设基础策略是确定性的。对于 GMM 或 Diffusion Policy 等随机基础策略，动作空间的随机性使 Q 函数训练失效。Existing off-policy Residual RL methods (e.g. TD3+BC, SAC) assume a deterministic base policy. For stochastic base policies such as GMM or Diffusion Policy, the randomness of the action space breaks Q-function training.

"Residual RL is a popular approach for adapting pretrained policies by learning a lightweight residual policy that provides corrective actions. While Residual RL is more sample-efficient than finetuning the entire base policy, existing methods struggle with sparse rewards and are designed for deterministic base policies."

Teaser: 不确定性引导的残差策略探索 — **核心思路概览。** 本文提出两项改进以加速 Residual RL： ①使用不确定性估计将探索约束在基础策略置信度低的区域； ②修改 off-policy critic，使其学习基础动作与残差动作组合后的 Q 函数，从而支持随机基础策略。**Overview of the core idea.** Two improvements are proposed to accelerate Residual RL: ① uncertainty estimation confines exploration to regions where the base policy has low confidence; ② the off-policy critic is modified to learn a Q function over the combination of base and residual actions, thereby supporting stochastic base policies.

6测试任务数
（Robosuite + D4RL + 真实机器人）Evaluation tasks
(Robosuite + D4RL + real robot)

4+对比基线方法
（DPPO, IBRL, Policy Decorator, 标准 Residual RL）Baselines compared
(DPPO, IBRL, Policy Decorator, standard Residual RL)

2不确定性度量
（distance-to-data, ensemble variance）Uncertainty metrics
(distance-to-data, ensemble variance)

sim→real零样本迁移
保留近乎全部仿真性能Zero-shot transfer
Retains nearly all of its performance in simulation

测试任务可视化 — **测试任务。** 实验在 Robosuite 的 Lift、Can、Square 三个机械臂操纵任务以及 D4RL 的 Franka Kitchen 任务上展开。任务难度依次递增，奖励信号稀疏。**Evaluation tasks.** Experiments are run on the three Robosuite manipulation tasks Lift, Can and Square, as well as the Franka Kitchen task from D4RL. Task difficulty increases in that order, and the reward signal is sparse.

02 方法Method

本文的核心思路是让残差策略"知道什么时候该出手"：只在基础策略不确定的状态下叠加修正量，其余情况直接沿用基础策略的输出。同时，通过改造 critic 的输入，使整个框架可以处理随机（非确定性）基础策略。The core idea is to let the residual policy "know when to step in": a correction is added only in states where the base policy is uncertain, and otherwise the base policy output is used directly. At the same time, reshaping the critic input lets the whole framework handle stochastic (non-deterministic) base policies.

改进一：不确定性引导的探索（Uncertainty-Guided Exploration）Improvement 1: Uncertainty-Guided Exploration

在每个时间步，先用不确定性估计量 uncertainty(s) 与阈值 τ 比较。若不确定性低（基础策略置信）则直接执行基础策略动作；否则叠加残差修正量：At every time step the uncertainty estimate uncertainty(s) is first compared with a threshold τ. If the uncertainty is low (the base policy is confident), the base policy action is executed directly; otherwise the residual correction is added:

a_taken = { a_b if uncertainty(s) < τ
{ a_b + a_r otherwise

阈值 τ 随训练步数指数衰减，使策略逐渐从"基础策略主导"过渡到"残差策略主导"：The threshold τ decays exponentially with the number of training steps, so that the policy gradually shifts from "base-policy dominated" to "residual-policy dominated":

τ = U · e^{−step / decay_rate}

论文比较了两种不确定性度量：The paper compares two uncertainty metrics:

距离数据集（distance-to-data）：当前状态到训练数据集中最近样本的 L2 距离。适用于 GMM 等策略，但在高维图像空间中可靠性下降。Distance-to-data: the L2 distance from the current state to the nearest sample in the training dataset. It suits policies such as GMM, but becomes less reliable in high-dimensional image spaces.
uncertainty_d(s) = min_d∈D √( Σ (d_i − s_i)² )
集成方差（ensemble variance）：多个基础策略副本在同一状态下输出动作的方差。当训练数据含随机游走数据时表现更稳健。Ensemble variance: the variance of the actions produced by several copies of the base policy in the same state. It is more robust when the training data contains random-play data.
uncertainty_e(s) = (1/N) Σ (π_b(a|s) − (1/N) Σ π_i(a|s))²

改进二：支持随机基础策略的非对称 Actor-CriticImprovement 2: An Asymmetric Actor-Critic for Stochastic Base Policies

对于 GMM 或 Diffusion Policy 等随机基础策略，每次采样的动作 a_b 不同。若 critic 仅观测残差动作 a_r，则 Q 函数估计偏差严重。本文的解决方案是让 critic 观测完整组合动作（a_b + a_r），而 actor 只预测残差修正量 a_r，形成非对称结构：For stochastic base policies such as GMM or Diffusion Policy, the sampled action a_b differs each time. If the critic observes only the residual action a_r, the Q-function estimate is severely biased. The proposed solution lets the critic observe the full combined action (a_b + a_r) while the actor predicts only the residual correction a_r, forming an asymmetric structure:

Actor 输入：状态 s → 输出：残差动作 a_rActor input: state s → output: residual action a_r
Critic 输入：状态 s + 完整动作 a_b + a_r → 输出：Q 值Critic input: state s + full action a_b + a_r → output: Q value

这样可以保证 Q 函数接收到随机基础动作的信息，同时保持"动作分离不变性"（action-split invariance），确保优化目标合理。This guarantees that the Q function receives information about the stochastic base action while preserving "action-split invariance", keeping the optimization objective sound.

确定性基础策略下完整/分离动作的 Q 学习比较 — **完整动作 vs 分离动作。** 对于确定性基础策略，两种 critic 输入方式效果相当；但对于随机基础策略，只有学习完整组合动作的 Q 函数才能正常收敛。图中结果来自 D4RL Franka Kitchen 任务。**Full action vs. split action.** For a deterministic base policy the two critic inputs perform comparably; for a stochastic base policy, only a Q function learned over the full combined action converges properly. The results shown come from the D4RL Franka Kitchen task.

03 实验Experiments

在 Robosuite（Lift、Can、Square）、D4RL Franka Kitchen（Complete、Mixed、Partial）、图像输入 Can 任务以及真实机器人 Can 任务上评估。所有结果均带 95% 置信区间。基线方法包括：DPPO（Diffusion Policy Policy Optimization）、IBRL（Imitation Bootstrapped RL）、IBRL-RPL、Policy Decorator（均匀探索调度）以及标准 Residual RL。Evaluated on Robosuite (Lift, Can, Square), D4RL Franka Kitchen (Complete, Mixed, Partial), an image-input Can task and a real-robot Can task. All results are reported with 95% confidence intervals. The baselines are DPPO (Diffusion Policy Policy Optimization), IBRL (Imitation Bootstrapped RL), IBRL-RPL, Policy Decorator (uniform exploration schedule) and standard Residual RL.

GMM 基础策略 — Robosuite 任务GMM Base Policy — Robosuite Tasks

GMM 基础策略实验结果 — **GMM 策略结果（图 3）。** "Our method is able to outperform all other baselines in all tasks." 在 Lift、Can、Square 三个任务上，本文方法在样本效率和最终成功率上均优于所有基线方法。误差条表示 95% 置信区间。**GMM policy results (Figure 3).** "Our method is able to outperform all other baselines in all tasks." On the three tasks Lift, Can and Square, the proposed method beats every baseline in both sample efficiency and final success rate. Error bars denote 95% confidence intervals.

Diffusion 基础策略 — Kitchen + Robosuite 任务Diffusion Base Policy — Kitchen + Robosuite Tasks

Diffusion 基础策略实验结果 — **Diffusion 策略结果（图 4）。** "Our method is able to outperform all baselines for Kitchen Complete and Can task, and has comparable performance for Square Task." 在 Kitchen Complete 和 Can 任务上以更高成功率超越所有基线；Square 任务上表现持平。**Diffusion policy results (Figure 4).** "Our method is able to outperform all baselines for Kitchen Complete and Can task, and has comparable performance for Square Task." On Kitchen Complete and Can it surpasses all baselines with a higher success rate; on Square it performs on par.

图像输入的 Can 任务Image-Input Can Task

基于图像观测的 Can 任务结果 — **图像输入结果（图 6）。** 在仅凭 RGB 图像作为观测的 Can 任务上，使用 ensemble variance 度量的方法展现出强劲性能，避免了训练早期频繁发生的意外碰撞。 distance-to-data 度量在高维图像空间下可靠性下降，ensemble 方法更为稳健。**Image-input results (Figure 6).** On the Can task with only RGB images as observations, the method using the ensemble variance metric shows strong performance and avoids the accidental collisions that occur frequently early in training. The distance-to-data metric becomes less reliable in high-dimensional image spaces, whereas the ensemble method is more robust.

真实机器人实验（Zero-Shot Sim-to-Real）Real-Robot Experiments (Zero-Shot Sim-to-Real)

真实机器人实验结果 — **真实机器人部署（图 8）。** 在真实机器人 Can 抓取任务上评估四种策略，每种策略在接触（contact）、抓取（grasp）、放置（place）三个阶段各进行 10 次试验。残差策略在迁移到真实环境后"nearly all of their original performance in simulation"，验证了零样本 sim-to-real 迁移的可行性。**Real-robot deployment (Figure 8).** Four policies are evaluated on a real-robot Can grasping task, each with 10 trials at each of the three stages: contact, grasp and place. After transfer to the real environment the residual policies retain "nearly all of their original performance in simulation", confirming the feasibility of zero-shot sim-to-real transfer.

消融研究Ablation Studies

阈值衰减策略消融 — **衰减策略消融（图 9）。** 比较了指数衰减、线性衰减和固定阈值三种策略。 "Exponential decay has the most stable performance out of the three."**Ablation on the decay schedule (Figure 9).** Exponential decay, linear decay and a fixed threshold are compared. "Exponential decay has the most stable performance out of the three."

**衰减速率消融（图 10）。** 衰减速率在 300k–500k 步范围内表现稳健。 "Lower rates resulted in aggressive exploration, causing performance dips that were hard to recover from, while higher rates slowed the convergence."**Ablation on the decay rate (Figure 10).** The decay rate is robust over the range of 300k–500k steps. "Lower rates resulted in aggressive exploration, causing performance dips that were hard to recover from, while higher rates slowed the convergence."

Kitchen 环境中不确定性度量比较Comparing Uncertainty Metrics in the Kitchen Environment

Kitchen 任务不确定性度量比较 — **不同不确定性度量在 Kitchen 任务中的比较（图 11）。** 在 Kitchen Partial 和 Kitchen Mixed 上，训练数据包含随机游走（random play）数据，导致 distance-to-data 度量失效（基础策略即使在不擅长的状态也"自信"）。 Ensemble variance 在这些场景下能够提供更可靠的不确定性估计，取得了更好的性能。**Comparison of uncertainty metrics on the Kitchen tasks (Figure 11).** On Kitchen Partial and Kitchen Mixed the training data contains random play data, which makes the distance-to-data metric fail (the base policy looks "confident" even in states it handles badly). Ensemble variance gives more reliable uncertainty estimates in these settings and achieves better performance.

任务 / 场景Task / setting	主要基线方法Main baselines	本文方法表现Performance of this method	备注Notes
Robosuite Lift / Can / Square（GMM）Robosuite Lift / Can / Square (GMM)	IBRL, Policy Decorator	超越全部基线Outperforms all baselines	样本效率与最终成功率均更优Better in both sample efficiency and final success rate
Kitchen Complete（Diffusion）Kitchen Complete (Diffusion)	DPPO, Policy Decorator	更高成功率Higher success rate	支持随机策略的关键场景Key setting for supporting stochastic policies
Can 任务（Diffusion）Can task (Diffusion)	DPPO, Policy Decorator	更高成功率Higher success rate	—
Square 任务（Diffusion）Square task (Diffusion)	Policy Decorator, DPPO	性能持平Comparable performance	最难任务，无显著差距Hardest task, no significant gap
Kitchen Partial / Mixed	标准 Residual RLStandard Residual RL	ensemble 有提升；distance 失效ensemble improves; distance fails	含随机游走数据时 distance 不可靠distance is unreliable when random-play data is included
真实机器人 Can（Zero-Shot）Real-robot Can (Zero-Shot)	仿真策略基准Simulation policy reference	保留近乎全部仿真性能Retains nearly all of its performance in simulation	无真实环境额外训练No extra training in the real environment

04 局限性Limitations

说明： 以下第一条为作者在论文中明确陈述的局限性（stated）；其余为根据方法设计推断（inferred）。Note: the first item below is a limitation explicitly stated by the authors (stated); the rest are inferred from the method design (inferred).

不确定性度量假设可能不成立 （作者明确指出）The uncertainty-metric assumption may not hold (explicitly stated by the authors)

本文的核心假设是"基础策略置信的地方，它的动作也是正确的"。但当训练数据中包含随机游走（random play）数据时，该假设失效——基础策略可能在许多状态下都显示出"低不确定性"但实际上动作质量差。作者明确指出："it would also benefit from a more robust epistemic uncertainty metric." 这导致 distance-to-data 在 Kitchen Mixed 和 Kitchen Partial 等任务上失效，ensemble 方法虽然改善了情况，但并非完美解决方案。The core assumption of this paper is that "wherever the base policy is confident, its actions are also correct". But when the training data contains random play data the assumption fails — the base policy may show "low uncertainty" in many states while its action quality is in fact poor. The authors state explicitly: "it would also benefit from a more robust epistemic uncertainty metric." This causes distance-to-data to fail on tasks such as Kitchen Mixed and Kitchen Partial; the ensemble method mitigates the problem but is not a perfect solution.

高维观测下 distance-to-data 可靠性下降 （作者明确指出）distance-to-data becomes less reliable under high-dimensional observations (explicitly stated by the authors)

在图像输入场景中，distance-to-data 度量在高维像素空间中变得不可靠。论文图像实验中已观测到这一问题，并改用 ensemble variance 方法。对于更复杂的视觉观测场景（如多物体、复杂背景），不确定性估计的鲁棒性需要进一步研究。In image-input settings the distance-to-data metric becomes unreliable in high-dimensional pixel space. The paper observes this in its image experiments and switches to ensemble variance. For more complex visual observation settings (multiple objects, cluttered backgrounds), the robustness of uncertainty estimation needs further study.

尚未验证对大型基础模型的适用性 （作者指出，推断）Applicability to large foundation models has not been verified (noted by the authors, inferred)

作者指出："We believe that, with reliable uncertainty metrics, our approach could also be applied to larger models including robot foundation models." 然而目前实验仅限于 GMM 和 Diffusion Policy 等相对小型的基础策略，尚未在大型视觉-语言-动作模型（如 RT-2、OpenVLA 等）上验证。The authors note: "We believe that, with reliable uncertainty metrics, our approach could also be applied to larger models including robot foundation models." Yet the experiments so far are limited to relatively small base policies such as GMM and Diffusion Policy, and have not been verified on large vision-language-action models (such as RT-2 or OpenVLA).

超参数（衰减速率）需要任务相关调整 （从消融结果推断）Hyperparameters (the decay rate) need task-specific tuning (inferred from the ablation results)

尽管实验表明衰减速率在 300k–500k 步范围内较为稳健，但极低的衰减速率会导致过激探索，而极高的衰减速率则会拖慢收敛。在不同任务难度和数据集规模下，最优超参数仍需一定程度的人工调整。Although the experiments show the decay rate to be fairly robust over the range of 300k–500k steps, a very low decay rate causes overly aggressive exploration while a very high one slows convergence. Across different task difficulties and dataset sizes, the optimal hyperparameters still require a certain amount of manual tuning.