CFGRL: Diffusion Guidance Is a Controllable Policy Improvement Operator

01 动机Motivation

离线强化学习（offline RL）的核心挑战是：如何在不超出数据支撑的情况下，从固定数据集中提取比行为策略更优的策略？现有方法（如 Advantage-Weighted Regression / AWR）通过对数据样本赋予奖励权重来进行策略提升，但这会造成梯度幅度在 batch 内严重不均衡——大量低优势样本几乎不贡献梯度，学习效率低下。The core challenge of offline reinforcement learning (offline RL) is: how can a policy better than the behavior policy be extracted from a fixed dataset without stepping outside the data support? Existing methods (such as Advantage-Weighted Regression / AWR) improve the policy by assigning reward weights to data samples, but this leaves gradient magnitudes severely imbalanced within a batch — a large share of low-advantage samples contribute almost no gradient, so learning is inefficient.

"We derive a direct relation between policy improvement and guidance of diffusion models, and show that CFGRL is trained with the simplicity of supervised learning, yet can further improve on the policies in the data."

本文的洞察：扩散模型中的 classifier-free guidance（CFG）天然对应着策略改进算子。条件分支学习"最优行为"，无条件分支学习"参考行为"，两者的分数函数之差正是优势函数（advantage）对策略的梯度方向。推理时放大引导权重 w，等价于对最优性进行更强的"衰减（attenuation）"，从而单调地提升期望回报。The insight of this paper: classifier-free guidance (CFG) in diffusion models corresponds naturally to a policy improvement operator. The conditional branch learns "optimal behavior" and the unconditional branch learns "reference behavior"; the difference between their score functions is exactly the gradient direction of the advantage with respect to the policy. Amplifying the guidance weight w at inference time is equivalent to imposing stronger "attenuation" on optimality, and thus monotonically raises the expected return.

CFGRL 概念图 — **Figure 1：**CFGRL 框架概念。对最优性条件（optimality conditioning）施加衰减，可进一步提升策略性能；增大引导权重在理论上保证期望回报单调不降。（左）AWR 加权梯度分布不均；（右）CFGRL 采用均匀损失权重，引导权重在推理时调节。**Figure 1:** Concept of the CFGRL framework. Applying attenuation to the optimality conditioning further improves policy performance; increasing the guidance weight theoretically guarantees a monotonically non-decreasing expected return. (Left) AWR yields an uneven distribution of weighted gradients; (Right) CFGRL adopts uniform loss weights, with the guidance weight tuned at inference time.

+30%walker-stand 相对 AWR 提升
(782 vs 603)walker-stand gain over AWR
(782 vs 603)

+29%walker-walk 相对 AWR 提升
(608 vs 444)walker-walk gain over AWR
(608 vs 444)

15×pointmaze-giant CFGRL vs GCBC
(30 vs 2)

37×visual-cube-single CFGRL vs GCBC
(37 vs 1)

02 方法Method

CFGRL 将策略建模为参考策略与最优性函数的乘积，训练一个 flow-matching 双条件策略，并在推理时通过调节引导权重 w 控制"遵从参考行为"与"追求最优性"之间的权衡。CFGRL models the policy as the product of a reference policy and an optimality function, trains a dual-conditioned flow-matching policy, and at inference time controls the trade-off between "following the reference behavior" and "pursuing optimality" by tuning the guidance weight w.

乘积策略（Product Policy）Product Policy

核心框架将目标策略分解为：The core framework decomposes the target policy as:

π(a|s) ∝ π̂(a|s) · f(A(s,a))

其中 π̂ 是参考策略（reference policy），f(A) 是关于优势函数的单调递增非负函数。 Remark 1 证明：只要 f 关于优势单调递增，乘积策略在期望回报上严格优于参考策略。 Remark 2（Attenuation）：对 f 施加更大的指数幂（w₁ < w₂）可进一步单调提升期望回报。在 KL 正则化 RL 目标下，最优策略恰为：Here π̂ is the reference policy and f(A) is a monotonically increasing, non-negative function of the advantage. Remark 1 proves that as long as f increases monotonically in the advantage, the product policy is strictly better than the reference policy in expected return. Remark 2 (Attenuation): raising f to a larger exponent (w₁ < w₂) further raises the expected return monotonically. Under the KL-regularized RL objective, the optimal policy is exactly:

π(a|s) ∝ π̂(a|s) · exp(A(s,a))^(1/β)

Classifier-Free Guidance 与策略改进的等价性Equivalence Between Classifier-Free Guidance and Policy Improvement

CFG 在扩散/flow 模型中的采样速度场为：The sampling velocity field of CFG in diffusion / flow models is:

v = (1−w)·vθ(a,t,s,∅) + w·vθ(a,t,s,o=1)

利用 Bayes 规则可将最优性预测器 p(o|s,a) 转化为两个策略分支的差值：Bayes' rule turns the optimality predictor p(o|s,a) into the difference between the two policy branches:

∇ₐ log π(a|s) = ∇ₐ log π̂(a|s) + (∇ₐ log π̂(a|s,o) − ∇ₐ log π̂(a|s))

无需单独训练分类器——差值天然对应策略改进方向。w 越大，朝最优性方向的"推力"越强。No separate classifier has to be trained — the difference already points along the direction of policy improvement. The larger w is, the stronger the "push" toward optimality.

训练：Flow-Matching 均匀损失Training: A Uniform Flow-Matching Loss

训练目标（flow-matching loss）：The training objective (flow-matching loss):

ℒ(θ) = 𝔼_{s,a∼D}[‖vθ(aₜ,t,s,o) − (a−a₀)‖²]

其中 aₜ = (1−t)a₀ + ta（线性插值），o ∈ {0,1} 为最优性标签（离线 RL 中按 A(s,a)≥0 二值化；goal-conditioned 场景中用目标到达概率 pγ(g|s,a)）。所有样本权重相等，消除了 AWR 的梯度不均衡问题。Here aₜ = (1−t)a₀ + ta (linear interpolation) and o ∈ {0,1} is the optimality label (in offline RL it is binarized by A(s,a)≥0; in goal-conditioned settings the goal-reaching probability pγ(g|s,a) is used instead). All samples carry equal weight, which removes the gradient imbalance of AWR.

梯度分布对比 — **Figure 2：**加权回归（AWR）在 batch 内产生极度不均衡的梯度幅度，大量低优势样本几乎不贡献学习信号。CFGRL 采用均匀 diffusion modeling 损失，所有样本均等地参与训练，引导权重仅在推理时调节。**Figure 2:** Weighted regression (AWR) produces extremely imbalanced gradient magnitudes within a batch, where a large share of low-advantage samples contribute almost no learning signal. CFGRL adopts a uniform diffusion modeling loss so that all samples take part in training equally, with the guidance weight tuned only at inference time.

Goal-Conditioned 场景的无 value function 扩展A Value-Function-Free Extension for the Goal-Conditioned Setting

在目标条件行为克隆（goal-conditioned BC）中，利用：In goal-conditioned behavior cloning (goal-conditioned BC), one uses:

π(a|s,g) ∝ π̂(a|s) · pγ(g|s,a)

将目标到达概率 pγ(g|s,a) 作为最优性信号，直接通过 CFG 的两个分支（有目标 vs 无目标）之差来估计，完全不需要显式训练 value function。The goal-reaching probability pγ(g|s,a) serves as the optimality signal and is estimated directly from the difference between the two CFG branches (with goal vs. without goal), so no value function needs to be trained explicitly at all.

03 实验Experiments

三大评测设置：(1) ExORL 离线 RL benchmark（9 个任务，4 seeds）； (2) OGBench（9 个任务，4 seeds）； (3) Goal-Conditioned BC（17 个状态任务 × 8 seeds，7 个视觉任务 × 4 seeds）。主要基线：AWR（离线 RL）、GCBC（目标条件）。Three evaluation settings: (1) the ExORL offline RL benchmark (9 tasks, 4 seeds); (2) OGBench (9 tasks, 4 seeds); (3) Goal-Conditioned BC (17 state-based tasks × 8 seeds, 7 visual tasks × 4 seeds). Main baselines: AWR (offline RL) and GCBC (goal-conditioned).

ExORL Benchmark（与 AWR 对比，部分任务）ExORL Benchmark (vs. AWR, selected tasks)

任务Task	AWR	CFGRL
walker-stand	603±8	782±8
walker-walk	444±4	608±32
cheetah-run	168±7	216±15
jaco-reach-top-right	33±2	72±6

OGBench（部分任务）OGBench (selected tasks)

任务Task	AWR	CFGRL
pointmaze-large-navigate	70±25	100±0
pointmaze-teleport-navigate	3±7	57±7
antmaze-teleport-navigate	22±19	30±22

Goal-Conditioned BC（与 GCBC 对比）Goal-Conditioned BC (vs. GCBC)

任务Task	GCBC	CFGRL
pointmaze-giant	2±2	30±10
antmaze-medium	25±8	53±12
visual-cube-single	1±1	37±9

引导权重与性能的关系 — **Figure 3：**引导权重 w 控制参考策略遵从度与最优性追求之间的权衡，其效果与 AWR 的温度参数类似，但 CFGRL 的关键优势是：w 可在推理时调节，无需重新训练。随着 w 增大，性能在多任务上持续提升。**Figure 3:** The guidance weight w controls the trade-off between adherence to the reference policy and the pursuit of optimality. Its effect resembles the temperature parameter of AWR, but the key advantage of CFGRL is that w can be tuned at inference time, with no retraining required. As w grows, performance keeps improving across tasks.

Goal-Conditioned 外推结果 — **Figure 4：**在 goal-conditioned 任务上，CFGRL（w > 1）能够超越 GCBC 基线，通过增大引导权重实现稳定的性能增益，跨多个环境均有效外推。**Figure 4:** On goal-conditioned tasks, CFGRL (w > 1) surpasses the GCBC baseline, obtaining stable performance gains by increasing the guidance weight and extrapolating effectively across multiple environments.

完整实验结果 — **Figure 5：**在 17 个状态任务和 7 个视觉任务上的完整 goal-conditioned 结果。 CFGRL 在绝大多数任务上显著超越 GCBC，部分视觉任务（visual-cube-single）提升最为突出（1→37）。**Figure 5:** Complete goal-conditioned results on 17 state-based tasks and 7 visual tasks. CFGRL clearly surpasses GCBC on the vast majority of tasks, and the gain is most pronounced on some visual tasks (visual-cube-single, 1→37).

消融分析（Ablation）Ablation Analysis

引导权重敏感性实验表明：增大 w 在几乎所有评测任务上均单调提升性能，与理论推导（Remark 2：单调衰减保证期望回报不降）相吻合。无条件分支的加入相比纯监督 GCBC 带来了显著改善，验证了 CFG 双分支设计的必要性。A sensitivity study on the guidance weight shows that increasing w monotonically improves performance on almost every evaluated task, matching the theoretical derivation (Remark 2: monotone attenuation guarantees a non-decreasing expected return). Adding the unconditional branch brings a marked improvement over purely supervised GCBC, confirming that the dual-branch CFG design is necessary.

04 局限性Limitations

说明：以下局限性均为论文作者在 Section 7 中明确陈述（stated by authors）。Note: All limitations below are explicitly stated by the authors in Section 7.

不替代完整 RL 流程Does Not Replace a Full RL Procedure

作者明确指出："Our method does not claim to replace full RL procedures — we assume a given value function and do not make any prescriptions about how to train it." CFGRL 是策略改进的附加工具，而非端到端 RL 算法，其性能上界仍受数据质量和 value function 精度约束。The authors state explicitly: "Our method does not claim to replace full RL procedures — we assume a given value function and do not make any prescriptions about how to train it." CFGRL is an add-on tool for policy improvement rather than an end-to-end RL algorithm, and its performance ceiling remains bounded by data quality and by the accuracy of the value function.

外推能力有限Limited Extrapolation Ability

"By itself, CFGRL does not represent a state-of-the-art RL algorithm, but rather an additional tool in the algorithm designer's toolbox." 更强的外推（超越数据分布）需要结合更高级的在线 RL 技术。作者提到： "More advanced policy extraction methods and online RL techniques … could provide for stronger extrapolation.""By itself, CFGRL does not represent a state-of-the-art RL algorithm, but rather an additional tool in the algorithm designer's toolbox." Stronger extrapolation (beyond the data distribution) calls for combining the method with more advanced online RL techniques. The authors note: "More advanced policy extraction methods and online RL techniques … could provide for stronger extrapolation."

依赖优势函数估计Reliance on Advantage Estimation

离线 RL 场景中，CFGRL 需要预先估计优势函数 A(s,a) 以生成二值最优性标签 o。优势估计的质量直接影响训练标签的准确性。Goal-conditioned 场景虽可绕开显式 value function，但仍依赖目标到达概率 pγ(g|s,a) 的合理建模。In the offline RL setting, CFGRL must estimate the advantage function A(s,a) beforehand in order to generate the binary optimality label o. The quality of that advantage estimate directly affects the accuracy of the training labels. Although the goal-conditioned setting can bypass an explicit value function, it still relies on sensible modeling of the goal-reaching probability pγ(g|s,a).