DreamerV2：首个在独立世界模型中实现 Atari 人类水平的 RL 智能体DreamerV2: the first RL agent to reach human-level Atari performance inside a separately trained world model
Danijar Hafner (Google Research) · Timothy Lillicrap (DeepMind) · Mohammad Norouzi (Google Research) · Jimmy Ba (University of Toronto)
DreamerV2 通过在离散世界模型的紧凑潜空间内纯粹预测来学习行为，首次以单 GPU 在 Atari 55 款游戏基准上实现人类水平性能，同时超越顶级单 GPU model-free 算法 IQN 和 Rainbow。其核心创新在于以 categorical 离散隐变量替代 Gaussian 隐变量，并引入 KL balancing 技术稳定世界模型训练。DreamerV2 learns behaviors purely from predictions inside the compact latent space of a discrete world model, for the first time reaching human-level performance on the Atari benchmark of 55 games with a single GPU, while surpassing the strongest single-GPU model-free algorithms IQN and Rainbow. Its core innovation is replacing the Gaussian latents with categorical discrete latents and introducing KL balancing to stabilize world model training.
世界模型理论上能使智能体从想象中的经历学习行为，从而大幅提升样本效率。然而，多年来在 Atari 这一最具竞争力的基准上，没有任何世界模型方法能与主流 model-free 算法竞争。核心问题是：如何让世界模型足够精确，从而完全在其预测中学习出成功的游戏策略？In principle, world models let an agent learn behaviors from imagined experience and thereby greatly improve sample efficiency. Yet for years, on Atari, the most competitive benchmark, no world model approach was able to compete with mainstream model-free algorithms. The core question is: how can a world model be made accurate enough to learn a successful game-playing policy entirely within its own predictions?
"We introduce DreamerV2, the first reinforcement learning agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model."
Figure 1（论文原图）：DreamerV2 在 Atari 55 款游戏基准（sticky actions，200M steps）上的 gamer normalized median score 达到 2.15，超越 IQN（1.29）、Rainbow（1.47）及所有单 GPU model-free 算法，是首个以 model-based 方式达到人类水平的智能体。Figure 1 (from the paper): on the Atari benchmark of 55 games (sticky actions, 200M steps), DreamerV2 reaches a gamer normalized median score of 2.15, surpassing IQN (1.29), Rainbow (1.47) and all single-GPU model-free algorithms, and is the first agent to reach human-level performance in a model-based way.
2.15×Gamer Median Score（DreamerV2，200M frames）Gamer Median Score (DreamerV2, 200M frames)
0.28Clipped Record Mean（推荐的鲁棒评估指标）Clipped Record Mean (the recommended robust evaluation metric)
10 天10 days单 V100 GPU 完成 200M 环境步200M environment steps on a single V100 GPU
468B世界模型内部想象的 compact states 数量Number of compact states imagined inside the world model
02方法 MethodMethod
DreamerV2 由三部分组成：世界模型（World Model）从过去经验中学习环境的紧凑表征；Actor-Critic 在世界模型的想象轨迹中学习策略；环境交互不断扩充经验数据集。世界模型与策略分开训练，使策略可以充分利用世界模型的表征而不相互干扰。DreamerV2 consists of three components: the World Model learns compact representations of the environment from past experience; the Actor-Critic learns the policy inside the imagined trajectories of the world model; environment interaction keeps growing the experience dataset. The world model and the policy are trained separately, so that the policy can fully exploit the world model's representations without the two interfering.
Figure 2（论文原图）：世界模型结构。图像序列经 CNN 编码，RSSM 维护确定性 recurrent state ht；在每个时间步同时计算后验随机状态 zt（融入当前图像 xt）和先验随机状态 ẑt（不访问当前图像，用于后续想象）。DreamerV2 的关键改进是将随机状态由 Gaussian 改为 32 × 32 classes 的 categorical 变量（32 个 categorical，每个 32 类），并用 straight-through gradients 优化。Figure 2 (from the paper): world model architecture. The image sequence is encoded by a CNN, and the RSSM maintains a deterministic recurrent state ht; at every time step it computes both the posterior stochastic state zt (which incorporates the current image xt) and the prior stochastic state ẑt (which has no access to the current image and is used for later imagination). The key change in DreamerV2 is turning the stochastic state from Gaussian into categorical variables of 32 × 32 classes (32 categoricals with 32 classes each), optimized with straight-through gradients.
DreamerV2 用一组 32 个 categorical 变量（每个含 32 个类别）替代 DreamerV1 中的 Gaussian 潜变量。Flatten 后得到长度 1024、仅 32 位为 1 的稀疏二值向量。作者列出四条为何 categorical 变量优于 Gaussian 变量的假设：DreamerV2 replaces the Gaussian latents of DreamerV1 with a set of 32 categorical variables (each with 32 classes). Flattened, they give a sparse binary vector of length 1024 in which only 32 entries are 1. The authors list four hypotheses for why categorical variables are preferable to Gaussian ones:
先验拟合能力更强：categorical 的混合仍是 categorical，因此先验可以精确拟合聚合后验；而 Gaussian 混合无法由单 Gaussian 先验完全覆盖，对多模态状态转移建模困难。Stronger prior fitting: a mixture of categoricals is still a categorical, so the prior can fit the aggregate posterior exactly; a mixture of Gaussians cannot be fully covered by a single Gaussian prior, which makes multimodal state transitions hard to model.
稀疏性利于泛化：稀疏二值向量结构对 Atari 游戏中离散事件（新房间、消失的敌人）是更好的归纳偏置。Sparsity aids generalization: the sparse binary vector structure is a better inductive bias for the discrete events in Atari games (a new room, a vanishing enemy).
梯度估计更稳定：straight-through gradient estimator 忽略了会导致梯度爆炸/消失的一个项，训练更稳定。More stable gradient estimation: the straight-through gradient estimator ignores a term that would cause exploding or vanishing gradients, which makes training more stable.
非平滑建模能力：categorical 变量更适合建模 Atari 中的非连续状态变化。Modeling non-smooth changes: categorical variables are better suited to modeling the discontinuous state changes in Atari.
KL Balancing
世界模型的损失函数包含 KL 项，同时训练先验（transition predictor）和正则化后验（representation model）。标准 KL 的问题是：先验在早期训练不充分，若过度正则化后验会损害表征质量。DreamerV2 引入 KL balancing：以不同学习率优化先验和后验，具体地用混合系数 α = 0.8：The world model loss contains a KL term that both trains the prior (transition predictor) and regularizes the posterior (representation model). The problem with the standard KL is that the prior is not trained enough early on, so over-regularizing the posterior hurts representation quality. DreamerV2 introduces KL balancing: the prior and the posterior are optimized at different learning rates, concretely with a mixing coefficient α = 0.8:
这使先验被更快地拉向后验（促进精确先验动力学），而不是让后验熵增大以减小 KL（避免表征退化）。KL balancing 与 beta-VAE 方法正交，可配合使用。This pulls the prior toward the posterior faster (encouraging accurate prior dynamics) instead of increasing the posterior entropy to reduce the KL (which would degrade the representation). KL balancing is orthogonal to beta-VAE methods and can be combined with them.
Figure 3（论文原图）：Actor-Critic 行为学习。从世界模型训练中的后验状态出发，利用 transition predictor 向前展开 H = 15 步的想象轨迹（无需生成图像），actor 采样动作，critic 用 λ-return（λ = 0.95）估计价值。Actor 同时使用 Reinforce 梯度和 straight-through 梯度的组合（Atari 上 ρ = 1 倾向 Reinforce），critic 每 100 梯度步更新一次 target network。单 GPU 可并行模拟 2500 条潜空间轨迹。Figure 3 (from the paper): actor-critic behavior learning. Starting from the posterior states of world model training, the transition predictor rolls out imagined trajectories of H = 15 steps (without generating images); the actor samples actions and the critic estimates values with a λ-return (λ = 0.95). The actor uses a combination of Reinforce gradients and straight-through gradients (on Atari ρ = 1 favors Reinforce), and the critic updates its target network every 100 gradient steps. A single GPU can simulate 2500 latent trajectories in parallel.
行为学习（Behavior Learning）Behavior Learning
Actor 和 Critic 都是 MLP（各约 1M 参数），在世界模型固定后的想象 MDP 中训练。Critic 用 λ-return 的均方误差损失，Actor 同时最大化 λ-return（通过 Reinforce 梯度和 straight-through 梯度的加权和）并添加熵正则化（Atari 上 η = 10⁻³）。世界模型总参数约 20M。The actor and the critic are both MLPs (about 1M parameters each), trained in the imagination MDP with the world model held fixed. The critic uses a mean squared error loss on the λ-return, while the actor maximizes the λ-return (through a weighted sum of Reinforce gradients and straight-through gradients) with an added entropy regularizer (η = 10⁻³ on Atari). The world model has about 20M parameters in total.
03实验 ExperimentsExperiments
在 55 款 Atari 游戏上评估（sticky actions，200M steps，action repeat 4，单 V100 GPU，单环境实例），对比 IQN、Rainbow、C51、DQN 四个 model-free 基准（分数来自 Dopamine 框架）。作者同时提出四种聚合方式并推荐 Clipped Record Mean 作为最鲁棒评估指标。Evaluated on 55 Atari games (sticky actions, 200M steps, action repeat 4, a single V100 GPU, a single environment instance) against the four model-free baselines IQN, Rainbow, C51 and DQN (scores taken from the Dopamine framework). The authors also propose four aggregation methods and recommend Clipped Record Mean as the most robust evaluation metric.
Figure 4（论文原图）：200M steps 内四种聚合方式下的性能曲线。DreamerV2 在所有指标上均优于 model-free 基准，在 Record Mean 上领先最为显著（0.44 vs IQN 0.21、Rainbow 0.17）。Figure 4 (from the paper): performance curves under the four aggregation methods within 200M steps. DreamerV2 outperforms the model-free baselines on every metric, with the largest margin on Record Mean (0.44 vs IQN 0.21 and Rainbow 0.17).
算法Algorithm
Gamer Median
Gamer Mean
Record Mean
Clipped Record Mean
DreamerV2
2.15
11.33
0.44
0.28
DreamerV2 (schedules)
2.64
10.45
0.43
0.28
IQN
1.29
8.85
0.21
0.21
Rainbow
1.47
9.12
0.17
0.17
C51
1.09
7.70
0.15
0.15
DQN
0.65
2.84
0.12
0.12
Table 1（论文原表）：200M steps 时各算法在 55 款游戏上的汇总分数。DreamerV2 在全部四项指标上超越所有单 GPU 基准。值得注意的是，Rainbow 在 Gamer Median 上优于 IQN（1.47 > 1.29），但在其余三项指标上 IQN 均优于 Rainbow，说明聚合方式的选择对排名有显著影响。Table 1 (from the paper): aggregate scores of each algorithm over the 55 games at 200M steps. DreamerV2 surpasses all single-GPU baselines on all four metrics. Notably, Rainbow beats IQN on Gamer Median (1.47 > 1.29), but IQN beats Rainbow on the other three metrics, which shows that the choice of aggregation method has a significant effect on the ranking.
Figure 5 & Table 2（论文原图/表）：消融实验结果（使用略早版本的 DreamerV2）。移除 Discrete Latents 导致 Clipped Record Mean 从 0.25 下降至 0.19；移除 KL Balancing 降至 0.16；移除 Image Gradients 导致几乎完全失效（0.01）。停止 Reward Gradients 在部分任务上反而略有提升，说明 DreamerV2 的表征来自图像重建而非奖励预测。Figure 5 & Table 2 (from the paper): ablation results (using a slightly earlier version of DreamerV2). Removing Discrete Latents lowers Clipped Record Mean from 0.25 to 0.19; removing KL Balancing lowers it to 0.16; removing Image Gradients makes the agent fail almost completely (0.01). Stopping Reward Gradients even helps slightly on some tasks, indicating that DreamerV2's representations come from image reconstruction rather than from reward prediction.
消融实验（Ablations）Ablations
论文通过逐一移除组件进行消融（Table 2）：The paper ablates the components one at a time (Table 2):
Categorical vs Gaussian 潜变量：categorical 在 42 款游戏上优于 Gaussian，仅 8 款略差，5 款平局。Categorical vs Gaussian latents: categorical is better than Gaussian on 42 games, slightly worse on only 8, and tied on 5.
KL balancing：启用 KL balancing 在 44 款游戏上优于禁用，仅 6 款略差。KL balancing: enabling KL balancing is better than disabling it on 44 games and slightly worse on only 6.
Image Gradients：停止图像梯度在 51 款游戏上导致性能下降（仅 3 款提升），表明 DreamerV2 高度依赖图像重建信号。Image Gradients: stopping image gradients degrades performance on 51 games (with an improvement on only 3), which shows that DreamerV2 relies heavily on the image reconstruction signal.
Reward Gradients：停止奖励梯度对 15 款游戏有所提升、22 款下降、18 款平局，说明通用图像表征比专门化的奖励表征泛化性更好。Reward Gradients: stopping reward gradients improves 15 games, degrades 22 and ties on 18, indicating that general image representations generalize better than specialized reward representations.
04局限性 LimitationsLimitations
Note:论文未设专门的 Limitations 小节。以下标注为来源：作者在正文中明确指出（stated），或从方法设计推断（inferred from the design）。The paper has no dedicated Limitations section. Each item below is labeled by source: explicitly stated by the authors in the text (stated), or inferred from the design (inferred from the design).
Video Pinball 等游戏的失败案例（stated）Failure cases in games such as Video Pinball (stated)
作者明确指出："We hypothesize that the reconstruction loss of the world model does not encourage learning a meaningful latent representation because the most important object in the game, the ball, occupies only a single pixel." 即对于关键物体极小的游戏，图像重建损失无法提供足够的学习信号，导致表征质量差。The authors state explicitly: "We hypothesize that the reconstruction loss of the world model does not encourage learning a meaningful latent representation because the most important object in the game, the ball, occupies only a single pixel." That is, in games whose most important object is tiny, the image reconstruction loss cannot provide enough learning signal, which leads to poor representation quality.
DreamerV2 在 10 天内完成 200M 步（单 V100），与 Rainbow 相当；但世界模型需要额外维护 20M 参数，并在每步同时更新世界模型和策略，内存与计算开销更高。MuZero 虽更强大，但需 2 个月以上的 GPU 计算（stated：论文明确指出 MuZero "would require over 2 months of computation to train even one agent on a GPU"）。DreamerV2 completes 200M steps within 10 days on a single V100, comparable to Rainbow; but the world model requires an extra 20M parameters and updates both the world model and the policy at every step, so its memory and compute overhead are higher. MuZero is more powerful, yet it needs more than 2 months of GPU computation (stated: the paper explicitly notes that MuZero "would require over 2 months of computation to train even one agent on a GPU").
Categorical 潜变量优势机制仍不明确（stated）The mechanism behind the advantage of categorical latents is still unclear (stated)
作者诚实地承认："While we do not know the reason why the categorical variables are beneficial, we state several hypotheses that can be investigated in future work." 该设计在实验上有效，但理论解释尚不充分。The authors honestly admit: "While we do not know the reason why the categorical variables are beneficial, we state several hypotheses that can be investigated in future work." The design works empirically, but its theoretical explanation is still insufficient.
连续控制场景下表现有限（inferred）Limited demonstration on continuous control (inferred)
论文展示了 DreamerV2 在 humanoid stand-up 和 walking（连续动作）上的初步结果，但主要评估集中在 Atari（离散动作）。连续控制下使用 dynamics backpropagation（ρ = 0）而非 Reinforce，超参数需分别调整，跨任务泛化能力未经系统验证。The paper shows preliminary results of DreamerV2 on humanoid stand-up and walking (continuous actions), but the main evaluation is concentrated on Atari (discrete actions). For continuous control it uses dynamics backpropagation (ρ = 0) instead of Reinforce, the hyperparameters have to be tuned separately, and cross-task generalization has not been systematically verified.
