Planning with Diffusion for Flexible Behavior Synthesis
Michael Janner · Yilun Du · Joshua B. Tenenbaum · Sergey Levine | UC Berkeley · MIT
Diffuser 将轨迹优化直接嵌入扩散概率模型的训练过程，使"从模型采样"与"用模型规划"几乎等价。通过对噪声轨迹的迭代去噪，Diffuser 天然支持长视野规划、可变规划窗口，以及基于奖励梯度或目标约束的灵活 test-time 任务适配，无需重新训练。Diffuser folds trajectory optimization directly into the training of a diffusion probabilistic model, so that "sampling from the model" and "planning with it" become nearly identical. By iteratively denoising noisy trajectories, Diffuser naturally supports long-horizon planning, variable-length planning windows, and flexible test-time task adaptation driven by reward gradients or goal constraints, all without retraining.
基于模型的强化学习通常将动力学建模与轨迹优化分离：先学一个近似动力学模型，再将其接入经典规划器。然而，这两个步骤在实践中很难协调——学到的模型对标准轨迹优化器并不友好，容易被对抗样本利用；而当代模型方法往往退而求其次，转向单步策略梯度或 value function，放弃了真正的序列规划优势。Model-based reinforcement learning usually separates dynamics modeling from trajectory optimization: an approximate dynamics model is learned first, then plugged into a classical planner. In practice, however, these two stages are hard to reconcile — the learned model is unfriendly to standard trajectory optimizers and is easily exploited by adversarial samples, while contemporary model-based methods often settle for less, turning to single-step policy gradients or value functions and giving up the real advantages of sequential planning.
"We consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem, such that sampling from the model and planning with it become nearly identical."
Figure 1. Diffuser 是一个扩散概率模型，通过对随机噪声轨迹的迭代去噪来规划。论文封面图展示了扩散过程（从噪声到清晰轨迹）以及 Diffuser 在 Maze2D 环境中的规划可视化。Figure 1. Diffuser is a diffusion probabilistic model that plans by iteratively denoising randomly sampled noise trajectories. The teaser figure of the paper shows the diffusion process (from noise to a clean trajectory) together with a visualization of Diffuser planning in the Maze2D environment.
100Block Stacking 满分 = 100（BCQ/CQL 均为 0.0）Block Stacking max score = 100 (BCQ/CQL both 0.0)
N=256扩散去噪步数（推理时使用）Diffusion denoising steps (used at inference)
02方法Method
Diffuser 以整条轨迹 τ = (s₀, a₀, s₁, a₁, …, sₜ, aₜ) 作为建模单元，用扩散模型学习轨迹的数据分布 p(τ)，然后在推理时通过 classifier-guided sampling 或 inpainting 将规划条件（奖励、目标状态）注入采样过程，从而直接输出高质量的未来行动序列。Diffuser takes an entire trajectory τ = (s₀, a₀, s₁, a₁, …, sₜ, aₜ) as its modeling unit, uses a diffusion model to learn the data distribution p(τ) over trajectories, and at inference injects planning conditions (rewards, goal states) into the sampling process via classifier-guided sampling or inpainting, thereby directly emitting high-quality future action sequences.
Figure 2. Diffuser 架构示意图。模型以整段轨迹为输入，在 planning horizon 维度展开，采用 local receptive field（时域卷积）在相邻时步间传递信息。去噪过程从上往下迭代，逐步精化噪声轨迹；可选的目标函数 J 通过梯度引导采样方向。Figure 2. Schematic of the Diffuser architecture. The model takes a whole trajectory as input, unrolled along the planning horizon dimension, and uses a local receptive field (temporal convolutions) to pass information between adjacent timesteps. Denoising iterates from top to bottom, progressively refining the noisy trajectory; an optional objective J steers the sampling direction through its gradient.
Diffuser 将轨迹表示为二维数组 τ ∈ ℝT×(|s|+|a|)，行为时间步，列为拼接的 state-action 向量。这一表示使 state 与 action 完全对等，模型对二者联合去噪，而非像传统动力学模型那样只预测 state。规划时先采样整条轨迹，再执行第一个 action——类似于 model-predictive control（MPC）的 receding-horizon 执行。Diffuser represents a trajectory as a two-dimensional array τ ∈ ℝT×(|s|+|a|), whose rows are timesteps and whose columns are concatenated state-action vectors. This representation puts state and action on completely equal footing: the model denoises both jointly, instead of predicting only the state as conventional dynamics models do. At planning time an entire trajectory is sampled first and only the first action is executed — analogous to the receding-horizon execution of model-predictive control (MPC).
扩散模型（Diffusion Probabilistic Model）Diffusion Probabilistic Model
前向过程（forward process）逐步向轨迹加高斯噪声，参数化为 q(τi | τi-1) = N(τi; √(1-β_i) τi-1, β_i I)。反向过程（reverse process）由参数 θ 优化，学习去噪转移 p_θ(τi-1 | τi)。网络 ε_θ(τi, i) 预测每步残差噪声，损失为：L = E_{i,τ⁰,ε} [‖ε - ε_θ(τi, i)‖²]。网络骨干借鉴 U-Net 思路，使用全时域卷积（temporal convolutions）处理轨迹序列，local receptive field 在相邻时步间共享信息而不依赖全局 attention，保证推理效率。The forward process gradually adds Gaussian noise to the trajectory and is parameterized as q(τi | τi-1) = N(τi; √(1-β_i) τi-1, β_i I). The reverse process is optimized over parameters θ to learn the denoising transition p_θ(τi-1 | τi). The network ε_θ(τi, i) predicts the residual noise at each step, with the loss L = E_{i,τ⁰,ε} [‖ε - ε_θ(τi, i)‖²]. The backbone follows a U-Net design and processes the trajectory sequence with fully temporal convolutions; the local receptive field shares information between adjacent timesteps without relying on global attention, which keeps inference efficient.
在每步去噪时，用奖励函数 J(τ) 的梯度修正均值：μ̃ = μ + Σ∇_τ J(τ)。奖励函数可以是测试时临时定义的任意可微函数——无需重新训练模型。具体实现中，论文另外训练一个小型奖励回归网络作为 J，其梯度用于引导采样向高奖励轨迹偏移。At every denoising step the mean is corrected by the gradient of a reward function J(τ): μ̃ = μ + Σ∇_τ J(τ). The reward can be an arbitrary differentiable function defined ad hoc at test time — no retraining of the model is required. In the concrete implementation, the paper additionally trains a small reward regression network as J, whose gradient shifts sampling toward high-reward trajectories.
将目标状态视为"已知像素"，在每步去噪后强制覆盖目标时步的状态值：δ(τ) = 1 若 s_t 已知（如起点/终点/约束），0 否则。Inpainting 操作为 τi ← δ ⊙ (s_cond + σ_i ε) + (1-δ) ⊙ τi，将已知约束直接"锚定"在对应位置。论文指出，该方法源自图像修复技术，无需额外训练条件模型即可灵活指定任意时步的约束。Goal states are treated as "known pixels": after each denoising step the state values at the constrained timesteps are overwritten, with δ(τ) = 1 if s_t is known (e.g. start, goal or constraint) and 0 otherwise. The inpainting operation is τi ← δ ⊙ (s_cond + σ_i ε) + (1-δ) ⊙ τi, which "anchors" the known constraints directly at their positions. The paper notes that this technique originates from image inpainting and allows constraints at arbitrary timesteps to be specified flexibly, without training an extra conditional model.
Figure 2（论文）. Diffuser 的采样示意：轨迹以二维噪声数组初始化，经 N 步去噪后得到连贯轨迹。整个过程等价于隐式的轨迹优化，不依赖显式的动力学模型展开。Figure 2 (paper). Sampling in Diffuser: the trajectory is initialized as a two-dimensional array of noise and becomes coherent after N denoising steps. The whole procedure is equivalent to implicit trajectory optimization and does not rely on explicit rollouts of a dynamics model.
四大关键特性Four Key Properties
Long-horizon planning
模型在整条轨迹上去噪，自然支持多步规划，不像单步策略那样在长视野下误差积累。The model denoises over the whole trajectory and thus naturally supports multi-step planning, avoiding the error accumulation that single-step policies suffer over long horizons.
Variable-length plans
在规划时通过调整噪声初始化张量的时间维度，即可动态改变规划窗口长度，无需重新训练。Adjusting the time dimension of the noise-initialization tensor at planning time changes the length of the planning window on the fly, with no retraining needed.
Task compositionality
多个奖励函数的梯度可直接求和，实现多目标的即兴组合，例如同时满足 reaching + avoiding。Gradients of several reward functions can simply be summed, composing multiple objectives on the fly — for example satisfying reaching + avoiding at the same time.
Effective non-greedy planning
扩散训练过程本身就是对整条轨迹建模，学到的规划器能规避 Markovian 模型固有的短视偏差。Because diffusion training itself models the entire trajectory, the learned planner avoids the myopic bias inherent to Markovian models.
03实验Experiments
实验在三类环境上评估：Maze2D（长视野目标导航）、D4RL MuJoCo locomotion（offline RL 基准，9 个 dataset-environment 组合）、Block Stacking（多任务 test-time 适配）。主要基线为：BCQ、CQL、IQL、Decision Transformer（DT）、Trajectory Transformer（TT）、MOPO、MOReL、MBOP。Evaluation covers three families of environments: Maze2D (long-horizon goal navigation), D4RL MuJoCo locomotion (the offline RL benchmark, 9 dataset-environment combinations) and Block Stacking (multi-task test-time adaptation). The main baselines are BCQ, CQL, IQL, Decision Transformer (DT), Trajectory Transformer (TT), MOPO, MOReL and MBOP.
Maze2D：长视野规划Maze2D: Long-Horizon Planning
Table 1 & Figure 4（论文）. Maze2D 基准上 Diffuser 与各基线的 MPPI、CQL、IQL 得分对比，以及 Diffuser 通过 inpainting 生成与目标一致轨迹的可视化。Diffuser multi-task 平均得分 129.4，远超次优方法。Table 1 & Figure 4 (paper). Scores of Diffuser against the MPPI, CQL and IQL baselines on the Maze2D benchmark, together with a visualization of goal-consistent trajectories generated by Diffuser via inpainting. The multi-task average score of Diffuser is 129.4, far ahead of the runner-up.
Environment
MPPI
CQL
IQL
Diffuser
Maze2D U-Maze
33.2
5.7
47.4
113.9 ± 3.1
Maze2D Medium
10.2
5.7
34.9
121.5 ± 2.7
Maze2D Large
3.0
12.5
58.6
123.0 ± 0.4
Single-task Avg
16.2
7.7
47.0
119.5
Maze2D U-Maze (multi)
—
—
—
129.4 ± 3.4
表中数值直接摘自论文 Table 1。Multi-task 变体在每轮 episode 开始时随机指定目标位置，IQL/CQL 无对应多任务结果。The values in the table are taken directly from Table 1 of the paper. In the multi-task variant the goal location is randomized at the start of each episode; IQL and CQL report no corresponding multi-task results.
Offline RL（D4RL MuJoCo locomotion）
Table 2（论文）. D4RL locomotion 基准上 Diffuser 与 BC、CQL、IQL、DT、TT、MOPO、MOReL、MBOP 的综合对比（平均分 ± 标准误差，50 trajectories×5% 截断）。Diffuser 平均得分 88.9（Medium-Expert HalfCheetah）。Table 2 (paper). Overall comparison of Diffuser with BC, CQL, IQL, DT, TT, MOPO, MOReL and MBOP on the D4RL locomotion benchmark (mean score ± standard error, 50 trajectories with 5% truncation). The average score of Diffuser is 88.9 (Medium-Expert HalfCheetah).
Dataset
Environment
BC
CQL
IQL
DT
TT
MOPO
MOReL
MBOP
Diffuser
Medium-Expert
HalfCheetah
55.2
91.6
86.7
86.8
95.0
63.3
53.3
105.4
88.9 ± 0.3
Medium-Expert
Hopper
52.5
105.4
91.5
107.6
110.0
23.7
93.6
55.1
103.3 ± 1.4
Medium-Expert
Walker2d
107.5
108.8
109.6
108.1
101.9
44.6
95.6
70.2
106.9 ± 0.6
Average (all 9)
—
—
—
—
—
—
—
—
88.9
所有数值直接摘自论文 Table 2，Diffuser 对应第一列标注 Diffuser 的列。All values are taken directly from Table 2 of the paper; Diffuser corresponds to the column labelled Diffuser.
论文在 block stacking 任务上测试 test-time task compositionality，评估三种变体：(1) Unconditional Stacking（堆叠尽量高）；(2) Conditional Stacking（指定颜色顺序）；(3) Rearrangement（重排颜色块）。Diffuser 在所有三个变体上均显著超越 BCQ 和 CQL（后两者得分均为 0.0），见论文 Table 3。The paper tests test-time task compositionality on block stacking, evaluating three variants: (1) Unconditional Stacking (stack as high as possible); (2) Conditional Stacking (a specified colour order); (3) Rearrangement (rearranging the coloured blocks). Diffuser clearly outperforms BCQ and CQL on all three variants (both baselines score 0.0); see Table 3 of the paper.
Task
BCQ
CQL
Diffuser
Unconditional Stacking
0.0
21.4
58.7 ± 4.1
Conditional Stacking
0.0
0.0
45.6 ± 1.7
Rearrangement
0.0
0.0
58.9 ± 3.4
Average
0.0
8.1
54.4
数值摘自论文 Table 3。满分为 100（执行一段完美轨迹）。The values are taken from Table 3 of the paper. The maximum score is 100 (executing one perfect trajectory).
消融分析（Ablations）Ablations
论文 Appendix A 给出了各组件的消融：去掉 classifier guidance 时 block stacking 性能大幅下滑；将轨迹表示改为仅预测 state 时，action 质量明显劣化；缩短 planning horizon 会降低 Maze2D 性能，验证了长视野建模的必要性。Appendix A of the paper ablates the individual components: removing classifier guidance sharply degrades block-stacking performance; changing the trajectory representation to predict states only markedly worsens action quality; and shortening the planning horizon lowers Maze2D performance, confirming the necessity of long-horizon modeling.
Figure 3（论文）. 扩散规划器的四大特性可视化：(a) 长视野规划，Diffuser 能从训练数据中拼接出可行长轨迹；(b) 时域组合性，通过拼接分布内子段来泛化新轨迹形状；(c) 可变长规划窗口，改变噪声初始化大小即可；(d) 任务组合性，不同奖励函数可在测试时灵活组合。Figure 3 (paper). Visualization of the four properties of diffusion planners: (a) long-horizon planning — Diffuser stitches feasible long trajectories out of the training data; (b) temporal compositionality — new trajectory shapes are generalized by stitching in-distribution segments; (c) variable-length planning windows — only the size of the noise initialization changes; (d) task compositionality — different reward functions can be combined flexibly at test time.
04局限性Limitations
Note: 论文未设专门的 Limitations 节。以下第 1 条为论文正文明确提及，第 2–4 条为从方法设计推断（标注为 inferred）。Note: The paper has no dedicated Limitations section. Item 1 below is stated explicitly in the body of the paper; items 2–4 are inferred from the design of the method (marked inferred).
推理速度较慢（论文明确指出）Slow inference (explicitly noted in the paper)
扩散去噪需要 N=256 次前向传播，与单步策略或 value-based 方法相比推理延迟显著更高。论文指出这是当前扩散规划的主要实际限制，并认为 DDIM 等加速去噪方案可在未来缓解该问题。Diffusion denoising requires N=256 forward passes, so inference latency is markedly higher than that of single-step policies or value-based methods. The paper calls this the main practical limitation of diffusion planning today and suggests that accelerated denoising schemes such as DDIM could alleviate it in the future.
依赖高质量 offline 数据（inferred）Reliance on high-quality offline data (inferred)
Diffuser 以 offline 数据集学习轨迹分布，覆盖范围受限于数据质量与多样性。若训练数据中缺乏高回报轨迹，模型学到的分布将以低回报行为为主，classifier guidance 的提升空间也相应受限。Diffuser learns its trajectory distribution from an offline dataset, so its coverage is bounded by the quality and diversity of that data. If high-return trajectories are absent from the training data, the learned distribution will be dominated by low-return behavior and the headroom for classifier guidance shrinks accordingly.
奖励模型需要额外训练（inferred）The reward model requires separate training (inferred)
Classifier-guided sampling 需要一个单独训练的奖励回归网络，该网络本身也依赖 offline 数据，与主模型的训练解耦。在奖励信号稀疏或难以参数化的场景下，引导效果可能不稳定。Classifier-guided sampling requires a separately trained reward regression network, which itself depends on offline data and is decoupled from the training of the main model. Where reward signals are sparse or hard to parameterize, the guidance may be unstable.
连续动作空间限制与离散场景适配（inferred）Restriction to continuous action spaces and adaptation to discrete settings (inferred)
当前框架主要面向连续 state-action 空间（MuJoCo、Maze2D 等），对离散动作空间的适配需要额外修改（如离散扩散或量化表示）。此外，对极高维 observation（如原始像素）的扩展尚未在论文中验证。The current framework mainly targets continuous state-action spaces (MuJoCo, Maze2D and the like); adapting it to discrete action spaces requires extra modifications, such as discrete diffusion or quantized representations. Moreover, scaling to very high-dimensional observations, such as raw pixels, has not been validated in the paper.
