DSRL: 用潜在空间强化学习引导扩散策略DSRL: Steering Your Diffusion Policy with Latent Space Reinforcement Learning
Steering Your Diffusion Policy with Latent Space Reinforcement Learning
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, Sergey Levine · UC Berkeley, University of Washington, Amazon
DSRL 的核心思路：不改变扩散策略权重，而是在其潜在噪声空间（latent noise space）中运行强化学习，通过学习最优的采样起点来引导预训练策略。该方法仅需黑盒访问 BC 策略，样本效率高，已在真实机器人上验证了从离线数据到自主在线改进的全流程。The core idea of DSRL: without changing the diffusion policy weights, run reinforcement learning in its latent noise space, steering the pretrained policy by learning the optimal sampling starting point. The method needs only black-box access to the BC policy, is sample efficient, and has been validated on real robots over the full pipeline from offline data to autonomous online improvement.
arXiv 2506.15799提交: 2025年6月18日 (v1) · 修订: 2025年6月25日 (v2)Submitted: 18 Jun 2025 (v1) · Revised: 25 Jun 2025 (v2)机器人控制 · Policy AdaptationRobot Control · Policy Adaptation📄 arXiv:2506.15799PDF
扩散策略（diffusion policy）在机器人操控领域表现出色，但从 behavioral cloning（BC）训练所得的策略往往性能不足，且在真实场景中存在分布偏移。传统的强化学习微调需要对整个策略网络进行梯度更新，计算代价高昂，且在真实机器人上采集数据既危险又耗时。如何以极低成本、少量样本将预训练扩散策略适配到目标任务，是一个核心挑战。Diffusion policies perform remarkably well in robot manipulation, but policies obtained from behavioral cloning (BC) are often insufficiently performant and suffer from distribution shift in real-world settings. Conventional RL fine-tuning requires gradient updates over the entire policy network, which is computationally expensive, while collecting data on real robots is both dangerous and time-consuming. Adapting a pretrained diffusion policy to a target task at very low cost and with few samples is therefore a central challenge.
"We propose to instead perform RL in the latent-noise space of diffusion policies, without modifying the diffusion policy weights themselves. This is simple, sample efficient, requires only black-box access to the BC policy, and enables effective real-world autonomous policy improvement."
图 1：DSRL 方法概览。标准扩散策略部署（Standard Diffusion Policy Deployment）从 Gaussian 噪声 x_N 出发，经去噪过程生成动作。DSRL 不修改策略权重，而是在动作噪声空间中学习一个 latent actor，通过选择更好的初始噪声 x_N 来引导整个去噪过程，从而提升任务成功率。右侧直觉图（Steering via Latent Space RL）展示了在一个简单二维环境中，选择不同的初始点 x_N 可以将扩散轨迹引导至不同的最终动作，DSRL 学习选择能导向目标区域的初始点。Figure 1: overview of DSRL. Standard Diffusion Policy Deployment starts from Gaussian noise x_N and produces actions through a denoising process. DSRL does not modify the policy weights; instead it learns a latent actor in the action-noise space and steers the entire denoising process by choosing a better initial noise x_N, thereby improving the task success rate. The intuition panel on the right (Steering via Latent Space RL) shows that in a simple two-dimensional environment different initial points x_N steer the diffusion trajectory toward different final actions, and DSRL learns to pick the initial points that lead to the target region.
黑盒Black-box仅需黑盒访问 BC 策略，无需修改权重Only black-box access to the BC policy is needed; no weight modification
50 ep真实机器人上典型适配所需样本数量量级Order of magnitude of samples typically needed for adaptation on a real robot
5×+相比 fine-tuning 基线的相对成功率提升（部分任务）Relative success-rate improvement over the fine-tuning baseline (on some tasks)
现有工作主要分为两类局限：Existing work suffers from two kinds of limitation:
直接 RL 微调扩散权重（如 DPPO、SRPO 等）：需要对整个神经网络做反向传播，计算量大，且梯度需穿越整个去噪链（通常 10–100 步），内存和时间成本极高。Fine-tuning the diffusion weights directly with RL (e.g. DPPO, SRPO): requires backpropagation through the entire neural network, which is computationally heavy, and gradients must traverse the whole denoising chain (typically 10–100 steps), making memory and time costs extremely high.
引导（guidance）方法：依赖可微分奖励函数或分类器，在真实机器人场景中往往不可行，且通常只能用于 offline 设置。Guidance methods: rely on differentiable reward functions or classifiers, which is often infeasible on real robots, and are usually restricted to the offline setting.
DSRL 的关键洞察是：扩散策略的去噪过程完全由初始噪声 x_N 决定，只要找到合适的 x_N，就能控制最终生成的动作，而这一优化可以在远低维的潜在空间中高效完成。The key insight of DSRL is that the denoising process of a diffusion policy is fully determined by the initial noise x_N: finding a suitable x_N is enough to control the final generated action, and this optimization can be carried out efficiently in a much lower-dimensional latent space.
02方法Method
DSRL 将扩散策略的初始噪声 x_N 视为 RL 的"行动空间"（action space）：给定当前观测 o，latent actor π^L 输出一个噪声向量 x_N，扩散策略以此为起点完成去噪并执行动作，环境反馈奖励用于训练 π^L。整个过程对扩散策略完全透明，只需黑盒推理调用。DSRL treats the initial noise x_N of the diffusion policy as the "action space" of RL: given the current observation o, the latent actor π^L outputs a noise vector x_N, the diffusion policy denoises from it and executes the resulting action, and the environment reward is used to train π^L. The whole procedure is entirely transparent to the diffusion policy and requires only black-box inference calls.
图 2：扩散策略的潜在噪声空间优化（Section 4）。上图展示了 latent actor 如何选择 x_N 并通过固定的扩散策略得到最终动作；右侧直觉图（Figure 3，来自论文）说明对 x_N 的微小扰动可引导动作轨迹向目标区域收束。DSRL 考虑了：(1) 利用 x_N 直接作为连续动作空间运行任意 RL 算法；(2) 使用 noise aliasing 技巧处理噪声等价类；(3) 高效的无噪声（noise-free）采样变体以降低样本数。Figure 2: latent-noise-space optimization of diffusion policies (Section 4). The upper panel shows how the latent actor selects x_N and obtains the final action through the frozen diffusion policy; the intuition panel on the right (Figure 3, from the paper) illustrates that small perturbations of x_N steer the action trajectory to converge toward the target region. DSRL considers: (1) using x_N directly as a continuous action space on which any RL algorithm can be run; (2) a noise aliasing trick for handling noise equivalence classes; (3) an efficient noise-free sampling variant that reduces the number of samples.
Latent Noise 作为 RL 动作空间Latent Noise as the RL Action Space
给定 MDP 状态 s（包含观测 o 和任务信息），latent actor π^L(x_N | s) 输出初始噪声 x_N ∈ ℝ^Z，Z 为动作 chunk 维度。扩散策略 π^BC 以 x_N 为起点执行 N_T 步 DDIM/DDPM 去噪，生成动作序列 a^{0:H}，与环境交互获得奖励。由于不需要通过去噪链做反向传播，梯度计算代价远低于直接微调扩散权重的方法。论文表明该框架与任意 off-policy RL 算法（如 SAC）兼容。Given an MDP state s (containing the observation o and task information), the latent actor π^L(x_N | s) outputs an initial noise x_N ∈ ℝ^Z, where Z is the action-chunk dimension. Starting from x_N, the diffusion policy π^BC runs N_T steps of DDIM/DDPM denoising to generate the action sequence a^{0:H} and interacts with the environment to obtain a reward. Since no backpropagation through the denoising chain is needed, the gradient computation cost is far lower than for methods that fine-tune the diffusion weights directly. The paper shows that the framework is compatible with any off-policy RL algorithm such as SAC.
Noise Aliasing 与高效采样Noise Aliasing and Efficient Sampling
扩散过程并非单射：不同 x_N 可能产生相同的动作（aliasing）。论文提出利用这一性质，将 x_N 的搜索空间等价压缩，避免冗余探索。此外，DSRL 提出了无噪声采样（noise-free sampling）变体：将 x_N 映射到最终动作的确定性变换（score-distillation 近似），以更少步骤完成从潜在空间到动作的转换，可将所需样本数进一步降低约 50%（论文 Section 4.3）。对于 DDIM，去噪链变为确定性过程，latent space 与 action space 间存在可微双射，理论上可用梯度方法；本文选择无梯度 RL 路径以保证通用性。The diffusion process is not injective: different x_N may produce the same action (aliasing). The paper proposes to exploit this property to compress the search space over x_N up to equivalence and avoid redundant exploration. In addition, DSRL introduces a noise-free sampling variant: a deterministic transformation mapping x_N to the final action (a score-distillation approximation) that completes the conversion from latent space to action in fewer steps and further reduces the required number of samples by about 50% (paper Section 4.3). For DDIM the denoising chain becomes deterministic, so a differentiable bijection exists between latent space and action space and gradient methods would in principle apply; this work takes the gradient-free RL route for generality.
算法：DSRL（Algorithm 1）Algorithm: DSRL (Algorithm 1)
论文给出了完整的 DSRL 算法流程（Algorithm 1，Standard Diffusion Steering via Reinforcement Learning）：The paper gives the complete DSRL procedure (Algorithm 1, Standard Diffusion Steering via Reinforcement Learning):
初始化 latent actor π^L，replay buffer D，critic Q；initialize the latent actor π^L, the replay buffer D and the critic Q;
每次迭代：采样观测 o，由 π^L 输出 x_N，调用（黑盒）扩散策略得到动作 a，执行并收集奖励 r；each iteration: sample an observation o, let π^L output x_N, call the (black-box) diffusion policy to obtain the action a, execute it and collect the reward r;
将 (o, x_N, r, o') 存入 D，用 SAC 或其他 off-policy 算法更新 π^L 和 Q；store (o, x_N, r, o') in D and update π^L and Q with SAC or another off-policy algorithm;
策略权重 θ^{BC} 全程冻结，不参与梯度更新。the policy weights θ^{BC} remain frozen throughout and receive no gradient updates.
论文还讨论了离线版本（Offline DSRL）：利用已有的离线演示数据（不包含噪声标注）通过逆向扩散恢复对应 x_N，从而无需任何在线交互即可完成初步适配，然后再切换至在线 RL 继续改进。The paper also discusses an offline version (Offline DSRL): from existing offline demonstration data (which carries no noise annotations), the corresponding x_N is recovered by reverse diffusion, so that an initial adaptation can be completed without any online interaction, after which one switches to online RL for further improvement.
03实验Experiments
实验在三个维度验证 DSRL：(1) 离线数据适配（Franka Kitchen、D4RL、RoboMimic 等标准 benchmark）；(2) 离线-在线联合改进；(3) 真实 Franka 机器人多任务操控（pick、place、apply 等共 6 个任务）。基线方法包括：直接 BC 部署、fine-tuning 类方法（DPPO、SRPO、Cal-QL、IQL、ReBRAC）、以及 V-GPS 等引导方法。The experiments validate DSRL along three dimensions: (1) offline data adaptation (standard benchmarks such as Franka Kitchen, D4RL and RoboMimic); (2) joint offline-to-online improvement; (3) multi-task manipulation on a real Franka robot (6 tasks in total, including pick, place and apply). Baselines include direct BC deployment, fine-tuning methods (DPPO, SRPO, Cal-QL, IQL, ReBRAC) and guidance methods such as V-GPS.
5.1 离线适配：标准 Benchmark 对比5.1 Offline Adaptation: Comparison on Standard Benchmarks
图 3 & 4（论文）：DSRL 在离线适配任务上的性能对比。图 3 评估了在 Schwefel 基准（[25]）上，使用不同 BC 策略检查点时 DSRL 与其他方法的成功率；图 4 展示了在 Franka Kitchen 数据集上对不同预训练扩散策略进行离线适配的结果，包括 DDPM 和 flow-based 策略的对比，时间轴为 epoch 数。DSRL 在绝大多数场景下优于或持平于所有 fine-tuning 和引导基线，且对不同 BC 策略质量（"suboptimal"，"transfer"，"noisy"）均保持稳健。Figures 3 & 4 (paper): performance of DSRL on offline adaptation tasks. Figure 3 evaluates the success rate of DSRL and competing methods on the Schwefel benchmark ([25]) across different BC policy checkpoints; Figure 4 reports offline adaptation of different pretrained diffusion policies on the Franka Kitchen dataset, including a comparison of DDPM and flow-based policies, with epochs on the time axis. In the vast majority of settings DSRL matches or outperforms every fine-tuning and guidance baseline, and it stays robust across BC policies of different quality ("suboptimal", "transfer", "noisy").
在 Franka Kitchen 离线数据集上（Table 1，论文 p.8），DSRL 相比 BC 基线大幅提升了所有任务的成功率。以 Franka Transport 任务为例，BC 成功率为 57%，而 DSRL 达到 74%；Cal-QL、IQL、ReBRAC 等 offline RL 基线在多数任务上均不及 DSRL。值得注意的是，论文指出"we use DDIM sampling for DSRL's BC policy so that we can use the latent actor MDP"，因此与使用 DDPM 的其他基线存在采样器差异。On the Franka Kitchen offline dataset (Table 1, paper p.8), DSRL substantially raises the success rate on every task relative to the BC baseline. On the Franka Transport task, for instance, BC reaches 57% while DSRL attains 74%; offline RL baselines such as Cal-QL, IQL and ReBRAC fall short of DSRL on most tasks. Notably, the paper states that "we use DDIM sampling for DSRL's BC policy so that we can use the latent actor MDP", so there is a sampler difference with respect to the other baselines, which use DDPM.
方法Method
Transport (pick)
Transport (place)
Apply
说明Notes
BC (基线)BC (baseline)
57%
—
—
纯 BC 部署Pure BC deployment
DSRL (离线)DSRL (offline)
74%
高于 BCHigher than BC
高于 BCHigher than BC
不修改策略权重Policy weights untouched
Cal-QL / IQL / ReBRAC
多数任务低于 DSRLBelow DSRL on most tasks
需要修改权重Weight modification required
注：上表数值来自论文 Table 1，部分结果以相对趋势呈现，精确数值见原文。Note: the figures above come from Table 1 of the paper; some results are given as relative trends, and the exact values can be found in the original text.
5.3 离线→在线改进 与 真实机器人实验5.3 Offline-to-Online Improvement and Real-Robot Experiments
图 5 & 6（论文）：真实 Franka 机器人操控实验。图 5（左）展示了 6 个真实任务的示意图，包括 pick cup, place cup on plate, fold cloth, apply tape, insert plug 等；图 6（右）展示了在多任务策略（multi-task BC policy）上应用 DSRL 的成功率曲线（横轴为 adaptation step），相比 BC 基线和 fine-tuning 基线（fine-Diff），DSRL 在约 50 个 episode 内即显著提升各任务成功率，且在大多数任务上最终超越单任务 BC（task-specific BC）。fine-Diff 基线在部分任务上出现"遗忘"问题（catastrophic forgetting），DSRL 因不修改权重而完全规避。Figures 5 & 6 (paper): real Franka robot manipulation experiments. Figure 5 (left) illustrates the 6 real tasks, including pick cup, place cup on plate, fold cloth, apply tape and insert plug; Figure 6 (right) shows success-rate curves obtained by applying DSRL to a multi-task BC policy (horizontal axis: adaptation step). Compared with the BC baseline and the fine-tuning baseline (fine-Diff), DSRL raises the success rate of each task markedly within about 50 episodes and eventually surpasses task-specific BC on most tasks. The fine-Diff baseline exhibits "forgetting" (catastrophic forgetting) on some tasks, which DSRL avoids entirely because it never modifies the weights.
5.4 消融实验（Ablations）5.4 Ablations
图 8–11（论文）：消融实验。分别探究了：(1) x_N 的采样步数 N_T 对 DSRL 性能的影响——更多去噪步数未必更好，DSRL 在 N_T=10 时即可达到较好平衡；(2) 潜在空间维度 Z 对策略的影响；(3) latent noise space 宽度（varying σ）对探索效率的影响；(4) noise-free 采样变体与标准 DSRL 的对比——noise-free 版本在部分任务上样本效率更高（约提升 50%）。论文还对比了在 action space 直接运行 RL（即 action-space RL，跳过扩散）与 DSRL 的差异，验证了在潜在噪声空间操作的优势。Figures 8–11 (paper): ablations. The following are studied in turn: (1) the effect of the number of sampling steps N_T for x_N on DSRL performance — more denoising steps are not necessarily better, and DSRL already strikes a good balance at N_T=10; (2) the effect of the latent space dimension Z on the policy; (3) the effect of the latent noise space width (varying σ) on exploration efficiency; (4) a comparison between the noise-free sampling variant and standard DSRL — the noise-free version is more sample efficient on some tasks (about 50% better). The paper also contrasts running RL directly in action space (action-space RL, skipping diffusion) with DSRL, confirming the advantage of operating in the latent noise space.
5.5 引导扩散策略 (VLA, π0 等通用策略)5.5 Steering Diffusion Policies (VLA, π0 and other generalist policies)
论文在 Section 5.4 展示了 DSRL 可用于引导大型预训练视觉-语言-动作模型（如 π0-style VLA 策略）：在给定少量任务演示的情况下，DSRL 可从多任务通用策略的潜在空间出发，在特定任务上快速提升成功率，而无需对数十亿参数的模型做任何梯度更新。这表明 DSRL 在引导"生成式"通用机器人策略方面具有很强的实用价值。In Section 5.4 the paper shows that DSRL can steer large pretrained vision-language-action models (such as π0-style VLA policies): given only a handful of task demonstrations, DSRL can start from the latent space of a multi-task generalist policy and rapidly improve the success rate on a specific task, without any gradient update to a model with billions of parameters. This indicates that DSRL is of considerable practical value for steering "generative" generalist robot policies.
04局限性Limitations
说明：论文在 Section 6 Discussion 中明确讨论了若干局限性和未来方向，以下条目均为作者明确陈述（stated），少量推断之处已标注为（inferred）。Note: In Section 6 (Discussion) the paper explicitly addresses a number of limitations and future directions. All items below are explicitly stated by the authors, with the few inferred points marked as (inferred).
潜在空间的局限性（Limitations of the Latent Space）Limitations of the Latent Space
DSRL 的优化受限于预训练 BC 策略的能力边界：若策略本身覆盖的动作空间不包含最优行为，调整 x_N 也无法生成超出策略分布的动作。论文指出："if the base policy is extremely poor, steering over its latent-noise space may be unable to discover any action which yields high reward"。此外，latent noise space 的有效性依赖于策略的 mode coverage——若 BC 策略对某类动作模式的覆盖率极低，DSRL 的优化空间也会相应受限。DSRL's optimization is bounded by the capability of the pretrained BC policy: if the action space covered by the policy itself does not contain the optimal behavior, adjusting x_N cannot produce actions outside the policy distribution. The paper notes: "if the base policy is extremely poor, steering over its latent-noise space may be unable to discover any action which yields high reward". Furthermore, the effectiveness of the latent noise space depends on the mode coverage of the policy — if the BC policy covers a certain class of action modes only very sparsely, the optimization room available to DSRL shrinks accordingly.
对 DDIM 采样的依赖（DDIM Requirement）Dependence on DDIM Sampling (DDIM Requirement)
DSRL 当前实现需要扩散策略支持 DDIM（确定性去噪）以建立 latent noise space 与 action space 之间的双射关系。对于使用 DDPM（随机采样）的策略，latent → action 的映射是随机的，增加了 RL 优化的方差，论文在 offline 实验中使用 DDIM 以规避此问题。这在一定程度上限制了对原始 DDPM 策略的直接应用。（部分为 inferred，基于论文 Section 4 的描述。）The current DSRL implementation requires the diffusion policy to support DDIM (deterministic denoising) so as to establish a bijection between latent noise space and action space. For policies that use DDPM (stochastic sampling), the latent → action mapping is stochastic, which increases the variance of RL optimization; the paper uses DDIM in the offline experiments to sidestep this issue. This limits to some extent the direct application to original DDPM policies. (Partly inferred, based on the description in Section 4 of the paper.)
奖励函数设计（Reward Function Design）Reward Function Design
DSRL 需要一个任务奖励函数。在模拟实验中，奖励通常基于 ground-truth 状态；在真实机器人实验中，论文使用了基于检测的奖励（例如物体到达目标区域）。对于更复杂、语义层面的任务，设计合适的自动奖励函数仍是一个挑战，这限制了 DSRL 在无监督场景中的直接应用。DSRL requires a task reward function. In simulation the reward is usually based on ground-truth state; in the real-robot experiments the paper uses detection-based rewards (for example, an object reaching a target region). For more complex, semantic-level tasks, designing a suitable automatic reward function remains a challenge, which restricts the direct application of DSRL in unsupervised settings.
探索效率与收敛速度Exploration Efficiency and Convergence Speed
尽管 DSRL 比直接微调策略权重更高效，论文在 Discussion 中指出，对于高维操控任务，在 latent noise space 中的探索仍需大量样本，对于完全从头学习（即 BC 策略极差）的场景，效率提升有限。论文建议的未来方向包括结合更好的探索策略和利用离线数据进行初始化。Although DSRL is more efficient than fine-tuning the policy weights directly, the paper notes in the Discussion that for high-dimensional manipulation tasks exploration in the latent noise space still consumes many samples, and that for scenarios learned entirely from scratch (i.e. when the BC policy is very poor) the efficiency gain is limited. Future directions suggested by the paper include combining better exploration strategies and using offline data for initialization.
