Dream to Control: Learning Behaviors by Latent Imagination
Dreamer：纯粹在潜在空间中想象，学习长视野控制行为Dreamer: learning long-horizon control behaviors by imagining purely in latent space
Danijar Hafner (University of Toronto / Google Brain) · Timothy Lillicrap (DeepMind) · Jimmy Ba (University of Toronto) · Mohammad Norouzi (Google Brain)
Dreamer 是一个基于世界模型的强化学习智能体，能够仅凭图像输入，通过在紧凑潜在状态空间中"想象"未来轨迹来学习复杂的长视野行为。其核心创新是将价值估计的解析梯度沿想象轨迹反向传播，从而高效训练策略，无需在真实环境中大量交互。在 20 个视觉控制任务上，Dreamer 在数据效率、计算速度和最终性能三方面均超越现有最佳方法。Dreamer is a world-model-based reinforcement learning agent that learns complex, long-horizon behaviors from image inputs alone, by "imagining" future trajectories in a compact latent state space. Its core innovation is to backpropagate analytic gradients of value estimates through imagined trajectories, which trains the policy efficiently without extensive interaction in the real environment. On 20 visual control tasks, Dreamer surpasses previous best methods in data efficiency, computation time, and final performance.
world modellatent imaginationactor-criticRSSMvisual controlmodel-based RL潜在动力学latent dynamics长视野规划long-horizon planningDeepMind Control Suite数据效率data efficiency
01动机Motivation
从高维图像中学习复杂行为是强化学习的核心挑战。已有工作可以学习潜在动力学模型，但如何从世界模型中高效推导出行为，仍是一个开放问题。Learning complex behaviors from high-dimensional images is a central challenge of reinforcement learning. Prior work can learn latent dynamics models, but how to efficiently derive behaviors from a world model remains an open problem.
"Learned world models summarize an agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them."
现有方法的主要缺陷在于：The main shortcomings of existing methods are:
短视行为（Shortsighted behaviors）：仅在固定想象 horizon 内最大化奖励，无法处理需要长期信用分配的任务。Shortsighted behaviors: They maximize rewards only within a fixed imagination horizon and cannot handle tasks that require long-term credit assignment.
依赖无梯度优化：为了对模型误差具有鲁棒性，前人工作通常采用 derivative-free 优化（如 CEM），而非利用神经网络提供的解析梯度，导致策略学习效率低。Reliance on derivative-free optimization: To be robust to model errors, prior work usually adopts derivative-free optimization (such as CEM) instead of exploiting the analytic gradients provided by neural networks, which makes policy learning inefficient.
数据效率低：model-free 方法（如 D4PG）需要 108 步环境交互才能达到与 Dreamer 使用 5×106 步相当的性能。Low data efficiency: Model-free methods (such as D4PG) need 108 environment steps to reach performance comparable to what Dreamer attains with 5×106 steps.
图 1（论文 Figure 2）：20 个视觉控制任务的图像观测示例，涵盖 cup、acrobot、hopper、walker、quadruped 等。这些任务挑战包括接触动力学、稀疏奖励、多自由度和三维环境。智能体的输入是 64×64×3 的图像。Figure 1 (paper Figure 2): Example image observations of the 20 visual control tasks, covering cup, acrobot, hopper, walker, quadruped and others. These tasks pose challenges including contact dynamics, sparse rewards, many degrees of freedom, and 3D environments. The input to the agent is a 64×64×3 image.
823Dreamer 平均分 (5×106 步)Dreamer mean score (5×106 steps)
786D4PG 平均分 (108 步，模型无关)D4PG mean score (108 steps, model-free)
332PlaNet 平均分 (5×106 步)PlaNet mean score (5×106 steps)
20×数据效率优势 vs D4PGdata efficiency advantage vs D4PG
02方法Method
Dreamer 由三个协同运作的模块构成：潜在动力学学习、在潜在空间中的行为学习，以及在真实环境中执行策略收集经验。整个行为学习过程纯粹在世界模型的潜在状态空间中完成，通过对多步价值估计的解析梯度反向传播来训练 actor 和 critic。Dreamer consists of three modules that operate jointly: learning the latent dynamics, learning behaviors in the latent space, and executing the policy in the real environment to collect experience. The whole behavior learning process takes place purely in the latent state space of the world model, training the actor and critic by backpropagating analytic gradients of multi-step value estimates.
图 2（论文 Figure 3）：Dreamer 的三个核心组件。(a) 从经验数据集中学习潜在动力学，将观测与动作编码为紧凑的潜在状态（●），并预测奖励（●）。(b) 在紧凑的潜在空间中，预测状态价值（●）和动作（●），通过沿想象轨迹反向传播梯度来最大化未来价值。(c) 编码当前历史计算模型状态，并在真实环境中执行动作模型。Figure 2 (paper Figure 3): The three core components of Dreamer. (a) Learn the latent dynamics from the dataset of past experience, encoding observations and actions into compact latent states (●) and predicting rewards (●). (b) In the compact latent space, predict state values (●) and actions (●), maximizing future values by backpropagating gradients along imagined trajectories. (c) Encode the history so far to compute the model state, and execute the action model in the real environment.
世界模型：Recurrent State Space Model (RSSM)World model: Recurrent State Space Model (RSSM)
Dreamer 使用 RSSM 作为潜在动力学模型，包含三个关键组件：Dreamer uses RSSM as its latent dynamics model, comprising three key components:
Representation model（表示模型）：p(st | st-1, at-1, ot) — 将观测和动作编码为紧凑的连续向量状态，具有 Markovian 转移性质。Representation model: p(st | st-1, at-1, ot) — encodes observations and actions into compact continuous vector states with Markovian transitions.
Transition model（转移模型）：q(st | st-1, at-1) — 在不观测对应图像的情况下预测未来状态，支持在潜在空间中快速想象数千条轨迹。Transition model: q(st | st-1, at-1) — predicts future states without observing the corresponding images, which allows thousands of trajectories to be imagined quickly in the latent space.
Reward model（奖励模型）：q(rt | st) — 从潜在状态预测奖励。Reward model: q(rt | st) — predicts rewards from latent states.
世界模型通过图像重建的变分下界（ELBO）联合优化，包含图像重建项、奖励预测项和 KL 正则项。The world model is optimized jointly through the variational lower bound (ELBO) of image reconstruction, comprising an image reconstruction term, a reward prediction term, and a KL regularization term.
行为学习：Actor-Critic in Latent ImaginationBehavior learning: Actor-Critic in Latent Imagination
行为学习的核心创新是在潜在空间中训练 actor 和 value 模型，不依赖真实环境交互：The core innovation of behavior learning is to train the actor and value models in the latent space, without relying on real-environment interaction:
Action model（动作模型）：qφ(aτ | sτ)，输出 tanh 变换的 Gaussian 分布，支持 reparameterized 梯度。Action model: qφ(aτ | sτ), outputs a tanh-transformed Gaussian distribution and supports reparameterized gradients.
Value model（价值模型）：vψ(sτ)，估计从状态 sτ 开始的期望累计奖励。Value model: vψ(sτ), estimates the expected cumulative reward starting from state sτ.
λ-return 价值估计：Vλ(sτ) 是 N 步收益的指数加权平均，平衡偏差与方差。λ-return value estimate: Vλ(sτ) is an exponentially weighted average of N-step returns, balancing bias and variance.
动作模型通过最大化价值估计来更新，梯度通过神经网络动力学解析地反向传播：The action model is updated by maximizing the value estimates, with gradients backpropagated analytically through the neural network dynamics:
"We propagate stochastic gradients of multi-step returns through neural network predictions of actions, states, rewards, and values using reparameterization."
图 3（论文 Figure 5 及 Figure 4 ablation）：上部分展示了 RSSM 的长期预测能力：仅给定前 5 帧图像，模型能准确预测 45 步后的状态（Context 为真实值，右侧为预测）。下部分为想象 horizon 消融实验——引入价值模型后，Dreamer 对 horizon 长度更加鲁棒，在多个任务上均稳定超越无价值模型的 action model 和 PlaNet 在线规划。Figure 3 (paper Figure 5 and the Figure 4 ablation): The upper part shows the long-term prediction ability of RSSM: given only the first 5 image frames, the model accurately predicts the state 45 steps ahead (Context is ground truth, predictions on the right). The lower part is the imagination horizon ablation — once the value model is introduced, Dreamer becomes more robust to the horizon length and consistently outperforms both the action model without a value model and PlaNet online planning across many tasks.
03实验Experiments
在 DeepMind Control Suite 的 20 个视觉控制任务上进行评估，图像输入 64×64×3，每个任务固定动作重复 R=2，每个任务运行 5 个随机种子。单块 Nvidia V100 GPU + 10 CPU 核心，每 106 环境步约 3 小时训练时间。Evaluation is performed on the 20 visual control tasks of the DeepMind Control Suite, with 64×64×3 image inputs, a fixed action repeat of R=2 for every task, and 5 random seeds per task. On a single Nvidia V100 GPU plus 10 CPU cores, every 106 environment steps take about 3 hours of training time.
主要性能对比Main performance comparison
方法Method
类型Type
环境步数Environment steps
平均得分（20 任务）Mean score (20 tasks)
Dreamer
Model-based (latent imagination)
5×106
823
D4PG
Model-free (pixel input)
108
786
A3C
Model-free (state input)
108
—
PlaNet
Model-based (online planning)
5×106
332
Dreamer 在使用 D4PG 20 倍更少的环境步数下，平均得分 (823) 超越 D4PG (786)。计算时间方面，Dreamer 约 3 小时/106 步，而 PlaNet 在线规划约 11 小时，D4PG 达到类似性能需约 24 小时。With 20 times fewer environment steps than D4PG, Dreamer attains a mean score (823) that surpasses D4PG (786). In terms of computation time, Dreamer takes about 3 hours per 106 steps, whereas PlaNet online planning takes about 11 hours and D4PG needs about 24 hours to reach comparable performance.
图 4（论文 Figure 6）：各任务性能对比柱状图。Dreamer 在 5×106 步后的平均性能为 823，而 PlaNet（同样 5×106 步）仅为 332，模型无关的 D4PG（108 步）为 786。Dreamer 继承了 PlaNet 的数据效率，同时超越了最强的 model-free 智能体的渐近性能。Figure 4 (paper Figure 6): Bar chart comparing performance across tasks. Dreamer reaches a mean performance of 823 after 5×106 steps, while PlaNet (with the same 5×106 steps) reaches only 332, and the model-free D4PG (108 steps) reaches 786. Dreamer inherits the data efficiency of PlaNet while surpassing the asymptotic performance of the strongest model-free agent.
长视野任务分析Analysis of long-horizon tasks
在需要长期信用分配的任务（如 Acrobot Swingup、Hopper Hop）上，Dreamer 明显优于：(1) 无价值模型的 action model（仅最大化有限 horizon 内的想象奖励），(2) PlaNet 在线规划。论文报告 Dreamer 在 horizon=20 时，在 20 个任务中的 16 个超越这两种替代方案，另 4 个打平。On tasks that require long-term credit assignment (such as Acrobot Swingup and Hopper Hop), Dreamer clearly outperforms: (1) an action model without a value model (which only maximizes imagined rewards within a finite horizon), and (2) PlaNet online planning. The paper reports that with horizon=20, Dreamer surpasses both alternatives on 16 of the 20 tasks and ties on the remaining 4.
表示学习目标消融Ablation of representation learning objectives
对比三种表示学习目标与 Dreamer 配合的效果（论文 Figure 8）：Comparison of three representation learning objectives paired with Dreamer (paper Figure 8):
图像重建（Reconstruction）：大多数任务上性能最佳，Dreamer 默认使用此目标。Reconstruction: Best performance on most tasks; Dreamer uses this objective by default.
对比估计（Contrastive estimation）：能解决约一半任务，但整体弱于重建。Contrastive estimation: Solves about half of the tasks, but is weaker overall than reconstruction.
纯奖励预测（Reward only）："Reward prediction alone was not sufficient in our experiments."Reward only:"Reward prediction alone was not sufficient in our experiments."
这一消融实验表明，表示学习的未来进步很可能直接转化为 Dreamer 在更高视觉复杂度环境中的性能提升。This ablation indicates that future progress in representation learning is likely to translate directly into better Dreamer performance in environments of higher visual complexity.
Atari 和 DeepMind Lab 的扩展验证Extended validation on Atari and DeepMind Lab
论文附录进一步验证 Dreamer 可扩展到离散动作（Atari）和早期终止任务（DeepMind Lab），证明方法的通用性。直通梯度（straight-through gradients）用于离散动作的梯度传播。The appendix of the paper further verifies that Dreamer scales to discrete actions (Atari) and to tasks with early termination (DeepMind Lab), demonstrating the generality of the approach. Straight-through gradients are used to propagate gradients through discrete actions.
04局限性Limitations
说明：论文没有独立的 Limitations 章节。以下限制部分来自结论和相关工作的隐含表述（标注 推断），部分来自论文明确陈述（标注 明确）。Note:The paper has no standalone Limitations section. Some of the limitations below are implied by the conclusion and the related work (marked inferred), and some are stated explicitly in the paper (marked explicit).
视觉复杂度的局限（推断 + 明确）Limits on visual complexity (inferred + explicit)
论文在结论中指出："Future research on representation learning can likely scale latent imagination to environments of higher visual complexity."这说明当前版本在高视觉复杂度环境（如真实世界图像、复杂光照/遮挡）中的性能尚不确定。所有实验均在 64×64×3 分辨率的仿真环境（DeepMind Control Suite）中进行。The conclusion of the paper states: "Future research on representation learning can likely scale latent imagination to environments of higher visual complexity." This indicates that the performance of the current version in environments of high visual complexity (such as real-world images, complex lighting or occlusion) remains uncertain. All experiments were run in simulated environments (DeepMind Control Suite) at 64×64×3 resolution.
模型误差积累（推断）Accumulation of model error (inferred)
在紧凑潜在空间中想象长达数十步的轨迹，模型误差会积累。论文通过引入价值模型部分缓解了这一问题（使策略对 horizon 长度更鲁棒），但模型精度仍是瓶颈——如实验所示，在模型难以准确建模的任务上，基于想象的学习可能失败。When trajectories of up to tens of steps are imagined in the compact latent space, model errors accumulate. The paper partly alleviates this by introducing the value model (making the policy more robust to the horizon length), but model accuracy remains a bottleneck — as the experiments show, imagination-based learning can fail on tasks that the model cannot capture accurately.
纯奖励预测表示不足（明确）Reward-only representations are insufficient (explicit)
"Reward prediction alone was not sufficient in our experiments."当奖励稀疏时，缺乏图像重建信号的表示模型难以学习足够丰富的状态表示，导致 Dreamer 在此配置下性能大幅下降。"Reward prediction alone was not sufficient in our experiments."When rewards are sparse, a representation model that lacks the image reconstruction signal struggles to learn sufficiently rich state representations, causing a large drop in Dreamer performance under this configuration.
计算资源要求（推断）Computational requirements (inferred)
Dreamer 每 106 步约需 3 小时训练时间（单块 Nvidia V100 GPU + 10 CPU 核心），虽然比 PlaNet（11 小时）快，但仍属于较重的计算开销，限制了在计算受限场景下的应用。世界模型与行为学习需要同时训练多个神经网络组件（RSSM、CNN 编解码器、actor、critic）。Dreamer needs about 3 hours of training time per 106 steps (a single Nvidia V100 GPU plus 10 CPU cores). Although this is faster than PlaNet (11 hours), it is still a fairly heavy computational cost, which limits its use in compute-constrained settings. The world model and behavior learning require training several neural network components at the same time (RSSM, CNN encoder-decoder, actor, critic).
