LP-DS: 拉格朗日扰动扩散引导 — 生成策略的潜在空间强化学习

01 动机Motivation

基于行为克隆（Behavior Cloning）的高容量生成策略在模仿学习中表现优异，但受限于演示数据覆盖范围不足以及分布偏移问题。直接用强化学习（RL）微调大型生成解码器往往不稳定且样本效率低下。已有的噪声空间引导方法 DSRL 虽能绕开解码器修改，却存在潜在查询漂移和行为模式坍塌两大缺陷。High-capacity generative policies built on Behavior Cloning perform excellently in imitation learning, but are limited by insufficient coverage of the demonstration data and by distribution shift. Fine-tuning large generative decoders directly with reinforcement learning (RL) is often unstable and sample-inefficient. DSRL, an existing noise-space steering method, avoids modifying the decoder, yet suffers from two flaws: latent query drift and behavioral mode collapse.

"LP-DS shifts Gaussian noise inputs via w = ε + Δ_θ(s) and optimizes Δ_θ with a Lagrangian trust-region objective that improves downstream value while limiting deviation from the latent prior."

玩具多目标导航实验 — **图1：玩具多目标导航（toy multi-goal navigation）。** 四个对称高斯奖励峰定义四个等价目标模式（红点）。冻结骨干策略可覆盖所有模式；DSRL 立即收缩到单一模式（模式坍塌）； LP-DS 通过不同信任域大小 δ ∈ {0.01, 0.05, 0.1} 提供**可控的多样性-性能权衡**， δ=0.05 时既保持多模态覆盖又获得高质量轨迹，而 δ=0.1 时则发生模式坍塌。**Figure 1: Toy multi-goal navigation.** Four symmetric Gaussian reward peaks define four equivalent goal modes (red dots). The frozen backbone policy covers all modes; DSRL immediately contracts to a single mode (mode collapse); LP-DS provides a **controllable diversity-performance trade-off** through different trust-region sizes δ ∈ {0.01, 0.05, 0.1}: at δ=0.05 it retains multimodal coverage while obtaining high-quality trajectories, whereas at δ=0.1 mode collapse occurs.

核心挑战：DSRL 的两大失败模式Core challenge: the two failure modes of DSRL

① 潜在查询偏离流形① Latent queries drift off the manifold

DSRL 用无约束潜在策略替代预训练先验（prior），导致生成的噪声向量偏离解码器训练时所用的标准高斯分布支撑集，引发不规则动作输出和性能退化。图2 清晰展示了这一现象：DSRL 预测出高幅值潜在查询，与解码器不稳定行为高度相关。DSRL replaces the pretrained prior with an unconstrained latent policy, so the generated noise vectors leave the support of the standard Gaussian distribution on which the decoder was trained, causing irregular action outputs and performance degradation. Figure 2 shows this clearly: DSRL predicts high-magnitude latent queries that correlate strongly with unstable decoder behavior.

② 行为多样性坍塌② Collapse of behavioral diversity

无约束的潜在空间优化会将概率质量集中到少数高奖励区域，骨干策略的多模态结构（multimodal structure）随之消失。 LP-DS 通过拉格朗日信任域机制显式约束扰动幅度，防止过度激进的模式集中。Unconstrained latent-space optimization concentrates probability mass on a few high-reward regions, and the multimodal structure of the backbone policy disappears with it. LP-DS explicitly constrains the perturbation magnitude through the Lagrangian trust-region mechanism, preventing overly aggressive mode concentration.

LP-DS vs DSRL 对比 — **图2：弱约束噪声空间引导的失败模式（HalfCheetah-v2，3 seeds）。** DSRL 预测出高幅值潜在查询（右图），与离流形解码行为及性能退化相关； LP-DS 通过自适应信任域约束保持更小的扰动幅度，始终贴近骨干支撑集。**Figure 2: Failure modes of weakly constrained noise-space steering (HalfCheetah-v2, 3 seeds).** DSRL predicts high-magnitude latent queries (right), which correlate with off-manifold decoding behavior and performance degradation; LP-DS keeps perturbation magnitudes smaller through an adaptive trust-region constraint and stays close to the backbone support.

25%Walker2D 回报提升（对比最强 baseline）Walker2D return gain (vs. the strongest baseline)

33/40Franka 真实机器人拾放成功率（冻结 baseline 为 18/40）Franka real-robot pick-and-place success rate (frozen baseline: 18/40)

17/20Franka 挂杯成功率（冻结 baseline 为 11/20）Franka mug-hanging success rate (frozen baseline: 11/20)

δ=0.35多数实验中默认信任域目标值，对超参不敏感Default trust-region target in most experiments; insensitive to hyperparameters

02 方法Method

LP-DS 将冻结生成解码器 Φ: S × W → A 视为黑盒，仅学习一个轻量级状态条件残差网络 Δ_θ: S → W，在采样噪声上添加偏移 w = ε + Δ_θ(s)，然后通过拉格朗日松弛（Lagrangian relaxation）进行约束优化，使得扰动在提升下游值函数的同时，保持对原始先验分布的接近。LP-DS treats the frozen generative decoder Φ: S × W → A as a black box and learns only a lightweight state-conditioned residual network Δ_θ: S → W, which adds an offset to the sampled noise, w = ε + Δ_θ(s), and is then optimized under constraints via Lagrangian relaxation, so that the perturbation improves the downstream value function while staying close to the original prior distribution.

4.1 残差扰动引导（Steering via Residual Perturbation）4.1 Steering via Residual Perturbation

设 Φ 为冻结的确定性生成解码器（扩散或流匹配模型），正常情况下从 ε ~ N(0, I) 采样噪声。 LP-DS 引入可学习扰动网络 Δ_θ，并按如下方式生成潜在查询：Let Φ be a frozen deterministic generative decoder (a diffusion or flow-matching model), which normally samples noise from ε ~ N(0, I). LP-DS introduces a learnable perturbation network Δ_θ and produces latent queries as follows:

w = ε + Δ_θ(s)，其中 ε ~ N(0, I)w = ε + Δ_θ(s), where ε ~ N(0, I)

当 Δ_θ(s) = 0 时，策略精确恢复到原始行为克隆分布。扰动网络是整个系统中唯一需要优化的组件，极大降低了计算成本。When Δ_θ(s) = 0, the policy exactly recovers the original behavior-cloning distribution. The perturbation network is the only component that has to be optimized in the entire system, which greatly reduces computational cost.

4.2 拉格朗日信任域约束优化（Constrained Optimization via Lagrangian Relaxation）4.2 Constrained Optimization via Lagrangian Relaxation

无约束的潜在空间优化会导致 Δ_θ(s) 幅值无限增大，使 w 偏离标准高斯流形。 LP-DS 将学习问题表述为一个约束优化：在最大化期望 Q 值的同时，将期望扰动幅值的平方控制在阈值 δ 以内（该约束来自 KL 散度的二阶近似）：Unconstrained latent-space optimization makes the magnitude of Δ_θ(s) grow without bound, driving w off the standard Gaussian manifold. LP-DS formulates learning as a constrained optimization: maximize the expected Q value while keeping the expected squared perturbation magnitude below a threshold δ (a constraint derived from a second-order approximation of the KL divergence):

max_θ E[Q^W(s, ε + Δ_θ(s))] s.t. E[‖Δ_θ(s)‖²] ≤ δ

通过引入拉格朗日乘子 α ≥ 0，形成如下拉格朗日函数，并交替更新 θ（梯度上升）和 α（投影对偶梯度上升）：Introducing a Lagrange multiplier α ≥ 0 yields the following Lagrangian, with θ (gradient ascent) and α (projected dual gradient ascent) updated alternately:

L(θ, α) = E[Q^W(s, w) − α(‖Δ_θ(s)‖² − δ)]

当扰动超出信任域时 α 自动增大，强迫 actor 优先满足先验约束；当扰动较小时 α 下降，允许更激进的引导。这一自适应正则化机制是 LP-DS 区别于 DSRL 的核心所在。When the perturbation exceeds the trust region, α automatically increases and forces the actor to satisfy the prior constraint first; when the perturbation is small, α decreases and allows more aggressive steering. This adaptive regularization mechanism is the core of what distinguishes LP-DS from DSRL.

4.3 双重 Critic 架构4.3 Dual-Critic Architecture

LP-DS 维护两个 critic：动作空间 critic Q^A（通过 TD 误差更新）和潜在噪声空间 critic Q^W（通过蒸馏 Q^A 更新）。 actor 的梯度直接作用在潜在空间 critic 上，避免了对高容量解码器进行反向传播的计算开销。整个在线训练流程见 Algorithm 1。LP-DS maintains two critics: an action-space critic Q^A (updated by the TD error) and a latent-noise-space critic Q^W (updated by distilling Q^A). The gradient of the actor acts directly on the latent-space critic, avoiding the computational cost of backpropagating through the high-capacity decoder. The full online training procedure is given in Algorithm 1.

各基准域综合对比结果 — **图3：各域基准对比结果（6 random seeds，均值 ± 1 标准差）。** 上行：RoboMimic 操控成功率；第二行：OpenAI Gym 运动回报；第三行：Adroit 灵巧手操控成功率；第四行：Adroit 回报。 LP-DS 在大多数环境中优于 DSRL、DPPO、IDQL 和 DQL 等 baseline。**Figure 3: Benchmark comparison across domains (6 random seeds, mean ± 1 std).** Top row: RoboMimic manipulation success rate; second row: OpenAI Gym locomotion return; third row: Adroit dexterous-hand manipulation success rate; fourth row: Adroit return. LP-DS outperforms baselines such as DSRL, DPPO, IDQL and DQL in most environments.

03 实验Experiments

在 RoboMimic 操控、OpenAI Gym 运动控制、Adroit 灵巧手操控三大基准上，以及 LIBERO（使用大型视觉-语言-动作模型 π₀ 骨干）和 Franka 真实机器人部署中， LP-DS 全面评估了方法的有效性、泛化性和行为多样性保持能力。Across three major benchmarks — RoboMimic manipulation, OpenAI Gym locomotion control and Adroit dexterous-hand manipulation — as well as LIBERO (using the large vision-language-action model π₀ as backbone) and a real Franka robot deployment, LP-DS is comprehensively evaluated for effectiveness, generalization and preservation of behavioral diversity.

5.1 主要基准结果5.1 Main Benchmark Results

环境/基准Environment / benchmark	最强 BaselineStrongest baseline	LP-DS	提升Gain
Walker2D-v2（回报）Walker2D-v2 (return)	~4000	~5000	~25%
RoboMimic Square（成功率）RoboMimic Square (success rate)	DSRL/DPPO	最高（快速收敛）Highest (fast convergence)	精度敏感任务优势显著Clear advantage on precision-sensitive tasks
Adroit 灵巧操控Adroit dexterous manipulation	各 baselineAll baselines	整体最强Strongest overall	成功率与回报均领先Leads in both success rate and return
LIBERO-90（VLA backbone π₀）LIBERO-90 (VLA backbone π₀)	冻结 π₀Frozen π₀	显著提升Substantial improvement	大型 Transformer 骨干可扩展Scales to large Transformer backbones
Franka 拾放（物理机器人）Franka pick-and-place (physical robot)	18/40（冻结骨干）18/40 (frozen backbone)	33/40	+83% 成功次数+83% successes
Franka 挂杯（物理机器人）Franka mug hanging (physical robot)	11/20（冻结骨干）11/20 (frozen backbone)	17/20	+55% 成功次数+55% successes

**图9：Franka 真实机器人任务。** 左：空间拾放任务，在 2×4 网格（40 个位置）上评估；右：挂杯任务，机器人需抓住杯子并将杯柄对准木架插入。 LP-DS 在仿真中进行潜在空间 RL 适应，并将所得的引导策略直接迁移至物理机器人执行。**Figure 9: Franka real-robot tasks.** Left: a spatial pick-and-place task, evaluated on a 2×4 grid (40 positions); right: a mug-hanging task, where the robot must grasp the mug and align its handle with the wooden rack for insertion. LP-DS performs latent-space RL adaptation in simulation and transfers the resulting steering policy directly to the physical robot.

5.2 行为多样性分析5.2 Behavioral Diversity Analysis

通过 Kozachenko–Leonenko k-NN 动作熵估计器，LP-DS 在在线适应过程中始终保持比 DSRL 更高的动作空间熵，同时成功率也更高。噪声空间引导方法（LP-DS 和 DSRL）相对冻结骨干均会降低熵，这是将概率质量集中于高价值噪声区域的固有代价；但 LP-DS 的信任域约束有效减缓了这一过程。 DPPO 直接在动作空间微调，因此熵变化幅度最小，但整体性能通常不及 LP-DS。Measured with the Kozachenko–Leonenko k-NN action-entropy estimator, LP-DS consistently maintains higher action-space entropy than DSRL during online adaptation, while also reaching a higher success rate. Noise-space steering methods (LP-DS and DSRL) both reduce entropy relative to the frozen backbone, the inherent price of concentrating probability mass on high-value noise regions; the trust-region constraint of LP-DS, however, effectively slows this process. DPPO fine-tunes directly in action space and therefore changes entropy the least, but its overall performance is usually below that of LP-DS.

动作熵对比 — **图5：在线适应期间的动作熵估计（Adroit Pen，3 seeds）。** LP-DS 在整个训练过程中保持比 DSRL 更高的动作熵，印证了拉格朗日信任域机制能有效缓解过早的模式坍塌。 DPPO 熵最高但成功率通常低于 LP-DS。**Figure 5: Action-entropy estimates during online adaptation (Adroit Pen, 3 seeds).** LP-DS maintains higher action entropy than DSRL throughout training, confirming that the Lagrangian trust-region mechanism effectively mitigates premature mode collapse. DPPO attains the highest entropy but usually a lower success rate than LP-DS.

5.3 消融实验与超参灵敏度5.3 Ablations and Hyperparameter Sensitivity

在 Adroit Pen 的消融实验中：In the ablation study on Adroit Pen:

移除拉格朗日惩罚：训练不稳定，最终成功率下降，动作熵持续衰减（向单一模式收缩）。Removing the Lagrangian penalty: training becomes unstable, the final success rate drops, and action entropy decays continuously (contracting toward a single mode).
移除拉格朗日惩罚 + 噪声边界：行为极度不稳定，频繁探索高斯先验下几乎不可能出现的极端噪声区域，成功率极低。Removing the Lagrangian penalty + the noise bounds: behavior becomes extremely unstable, frequently exploring extreme noise regions that are all but impossible under the Gaussian prior, and the success rate is very low.
完整 LP-DS：最高成功率 + 最高动作熵，证明拉格朗日信任域是防止分布外引导的核心机制。Full LP-DS: the highest success rate together with the highest action entropy, showing that the Lagrangian trust region is the key mechanism preventing out-of-distribution steering.

对信任域目标 δ 的扫描实验（Hopper、Walker2d、RoboMimic Square、Adroit Relocate）表明： δ > 0.1 均能取得强劲性能；0.35、0.5、0.66 等邻近值表现相近。 δ 主要充当粗粒度行为旋钮而非需要精细调节的脆弱超参数。A sweep over the trust-region target δ (Hopper, Walker2d, RoboMimic Square, Adroit Relocate) shows that any δ > 0.1 achieves strong performance, and neighboring values such as 0.35, 0.5 and 0.66 perform similarly. δ acts mainly as a coarse-grained behavioral knob rather than a brittle hyperparameter needing fine tuning.

Avoiding 环境成功率与目标覆盖 — **图：Avoiding 环境中的多路径多样性。** LP-DS 在不同信任域大小 δ 下展现出可控的多样性—专业化权衡。小信任域（δ=0.01）保留更广的障碍物绕行路径集合；较大信任域（δ=0.3）使策略集中于较窄的成功路线。 DSRL 则坍塌到单一主导路径。**Figure: Multi-path diversity in the Avoiding environment.** LP-DS exhibits a controllable diversity-specialization trade-off across trust-region sizes δ. A small trust region (δ=0.01) preserves a broader set of obstacle-avoiding paths; a larger one (δ=0.3) concentrates the policy on a narrower set of successful routes. DSRL, by contrast, collapses to a single dominant path.

5.4 跨架构鲁棒性5.4 Cross-Architecture Robustness

在 Hopper-v2 上进行的扩散 vs. 流匹配骨干对比实验表明，LP-DS 在两种骨干架构下取得相当的最终性能，证明残差扰动和信任域框架并不依赖于特定的去噪扩散链，可泛化到连续时间流模型。在 LIBERO-90 基准上使用大型 π₀ VLA 骨干同样获得显著性能提升，验证了 LP-DS 可扩展至大型 Transformer 基础模型。A diffusion vs. flow-matching backbone comparison on Hopper-v2 shows that LP-DS attains comparable final performance under both backbone architectures, demonstrating that the residual perturbation and trust-region framework does not depend on a particular denoising diffusion chain and generalizes to continuous-time flow models. Using a large π₀ VLA backbone on the LIBERO-90 benchmark likewise yields a substantial performance gain, confirming that LP-DS scales to large Transformer foundation models.

04 局限性Limitations

说明：本文未设置独立的"局限性"章节，但在结论（Section 6）和 Impact Statement 中有明确论述。以下条目标注来源：明确陈述（stated）或 推断（inferred）。Note: This paper has no standalone "Limitations" section, but discusses them explicitly in the conclusion (Section 6) and the Impact Statement. Each item below is labeled by source: stated or inferred.

信任域目标 δ 仍需人工选择（stated + inferred）The trust-region target δ still has to be chosen manually (stated + inferred)

论文指出 δ > 0.1 时方法较为鲁棒，且 0.35、0.5、0.66 等值效果相近，但对于不同任务和骨干架构，最优 δ 仍需在一定范围内选取。结论中将"自适应信任域目标"（adaptive trust-region targets）列为未来工作方向。The paper notes that the method is fairly robust for δ > 0.1, and that values such as 0.35, 0.5 and 0.66 perform comparably, but the optimal δ still has to be picked within some range for different tasks and backbone architectures. The conclusion lists "adaptive trust-region targets" as a direction for future work.

噪声空间引导固有地降低行为熵（stated）Noise-space steering inherently reduces behavioral entropy (stated)

论文明确指出："Noise-space steering methods (LP-DS, DSRL) exhibit an inherent reduction in entropy relative to the backbone." 虽然 LP-DS 比 DSRL 保留了更多多样性，但相比直接在动作空间微调（DPPO），其动作空间熵仍会随训练进行而下降，这是潜在空间模式选择的内在代价。The paper states explicitly: "Noise-space steering methods (LP-DS, DSRL) exhibit an inherent reduction in entropy relative to the backbone." Although LP-DS retains more diversity than DSRL, its action-space entropy still decreases as training proceeds compared with fine-tuning directly in action space (DPPO); this is the intrinsic price of mode selection in latent space.

依赖仿真适应再迁移的 sim-to-real 流程（inferred）Relies on a sim-to-real pipeline of adapting in simulation and then transferring (inferred)

真实机器人实验（Section 5.7）采用"仿真中进行潜在空间 RL 适应，再迁移到物理机器人"的流程。这意味着方法在一定程度上依赖高质量的任务匹配仿真器；在仿真-真实差距（sim-to-real gap）较大的场景中，其性能有多大程度可维持尚不清楚。The real-robot experiments (Section 5.7) adopt a pipeline of "latent-space RL adaptation in simulation, then transfer to the physical robot". This means the method depends to some extent on a high-quality, task-matched simulator; how far its performance can be maintained in settings with a large sim-to-real gap remains unclear.

部分可观测与大规模 VLA 场景的验证有限（stated）Limited validation on partially observable and large-scale VLA settings (stated)

论文结论明确将"部分可观测环境"（partially observable）和"大规模视觉-语言-动作设置"（large-scale VLA settings）列为未来工作方向，暗示当前验证覆盖范围有限，尤其是在需要长时间记忆或复杂感知的场景中。The conclusion of the paper explicitly lists "partially observable" environments and "large-scale VLA settings" as directions for future work, implying that the current validation has limited coverage, especially in scenarios that require long-horizon memory or complex perception.

双 Critic 引入额外计算开销（inferred）The dual critics introduce extra computational overhead (inferred)

LP-DS 维护动作空间 critic Q^A 和潜在空间 critic Q^W 共两个独立 critic，以及独立的 actor 和拉格朗日乘子 α 的更新。相比仅有单个 critic 的 DSRL，每步的计算和存储成本更高，在大规模 VLA 骨干上的实际开销尚未量化报告。LP-DS maintains two separate critics, the action-space critic Q^A and the latent-space critic Q^W, plus separate updates for the actor and the Lagrange multiplier α. Compared with DSRL, which has only a single critic, the per-step compute and memory cost is higher, and the actual overhead on large-scale VLA backbones has not yet been quantitatively reported.