DreamerV3：用世界模型掌控多样领域

01 动机Motivation

强化学习已在围棋、电子游戏等单一领域取得突破，但每换一个新任务就需要重新调参——这严重限制了 RL 的实用价值。作者提出的核心问题是：Reinforcement learning has produced breakthroughs in individual domains such as Go and video games, but every new task requires retuning the hyperparameters — which severely limits the practical value of RL. The core question the authors raise is:

"Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence."

现有专用算法（MuZero、PPG、IMPALA 等）在各自领域表现优异，却无法直接迁移：不同任务的奖励量纲、观测模态（图像/本体感知/向量）、回报范围差异极大，导致相同的损失函数和归一化假设在跨域时崩溃。DreamerV3 的目标是用一套固定超参数覆盖连续控制、离散游戏、3D 导航、开放世界等所有场景。Existing specialized algorithms (MuZero, PPG, IMPALA and others) excel in their own domains yet cannot transfer directly: tasks differ enormously in reward scale, observation modality (images / proprioception / vectors) and return range, so identical loss functions and normalization assumptions collapse across domains. The goal of DreamerV3 is to cover continuous control, discrete games, 3D navigation and open worlds with one fixed set of hyperparameters.

benchmark summary — **Figure 1：基准汇总。** (a) 固定超参数下，Dreamer 在所有基准和数据预算上超越专用调参算法，同时大幅超过广泛适用的 PPO 高质量实现。(b) 开箱即用，Dreamer 学会从零在 Minecraft 中获取钻石——此前方法均需人类数据或领域特定启发式。**Figure 1: Benchmark summary.** (a) Under fixed hyperparameters, Dreamer outperforms specialized tuned algorithms across all benchmarks and data budgets, while also substantially exceeding a high-quality implementation of the widely applicable PPO. (b) Out of the box, Dreamer learns to obtain diamonds in Minecraft from scratch — all previous methods required human data or domain-specific heuristics.

150+评测任务总数evaluation tasks in total

1×超参数配置（无调参）hyperparameter configuration (no tuning)

1 GPUMinecraft 训练仅需 1 张 A100 × 9 天Minecraft training needs only 1 A100 × 9 days

100%Dreamer 代理均发现 Minecraft 钻石of the Dreamer agents discover Minecraft diamonds

02 方法Method

DreamerV3 由三个模块组成：世界模型（World Model）将真实轨迹压缩为紧凑表征；Actor 在想象轨迹中学习策略；Critic 估计状态价值。三者联合训练，但 actor 与 critic 仅在想象空间更新，大幅提升样本效率。DreamerV3 consists of three modules: the World Model compresses real trajectories into compact representations; the Actor learns a policy inside imagined trajectories; the Critic estimates state values. The three are trained jointly, but actor and critic are updated purely in imagination, which greatly improves sample efficiency.

**Figure 3：DreamerV3 训练流程。** (a) World Model Learning：编码器将感知输入 x_t 映射到离散表征 z_t，序列模型（GRU）维护循环状态 h_t，并在给定动作 a_t 的条件下预测下一表征；解码器重建输入以塑造表征质量。(b) Actor-Critic Learning：actor 和 critic 在世界模型想象的抽象表征轨迹上更新，预测动作 a_t 和价值 v_t。**Figure 3: The DreamerV3 training process.** (a) World Model Learning: the encoder maps the sensory input x_t to a discrete representation z_t; the sequence model (GRU) maintains the recurrent state h_t and predicts the next representation conditioned on the action a_t; the decoder reconstructs the input to shape representation quality. (b) Actor-Critic Learning: actor and critic are updated on trajectories of abstract representations imagined by the world model, predicting the action a_t and the value v_t.

世界模型：RSSM 架构World Model: the RSSM Architecture

世界模型基于 Recurrent State Space Model (RSSM)，包含以下组件：The world model builds on the Recurrent State Space Model (RSSM) and comprises the following components:

Encoder：图像输入用 CNN，向量输入用 MLP，输出随机表征 z_t（32 类别 × 32 维 one-hot）。Encoder: a CNN for image inputs and an MLP for vector inputs, emitting the stochastic representation z_t (32 categories × 32 dimensional one-hot).
Sequence Model（GRU）：维护确定性循环状态 h_t，接收上一动作 a_{t-1} 预测下一表征。Sequence Model (GRU): maintains the deterministic recurrent state h_t and predicts the next representation from the previous action a_{t-1}.
Dynamics Predictor：从 h_t 预测 z_t（用于 imagination 阶段）。Dynamics Predictor: predicts z_t from h_t (used during imagination).
Reward / Continue Predictors：预测奖励 r_t 和 episode 终止标志 c_t。Reward / Continue Predictors: predict the reward r_t and the episode termination flag c_t.
Decoder：重建输入 x_t，监督表征学习。Decoder: reconstructs the input x_t and supervises representation learning.

世界模型损失：L(φ) = β_pred · L_pred + β_dyn · L_dyn + β_rep · L_rep，权重分别为 1、1、0.1。动态损失和表征损失均通过 KL 散度（free bits 剪裁至 1 nat）相互约束，让序列模型与编码器共同进步。World model loss: L(φ) = β_pred · L_pred + β_dyn · L_dyn + β_rep · L_rep, with weights 1, 1 and 0.1 respectively. The dynamics loss and the representation loss constrain each other through the KL divergence (clipped by free bits at 1 nat), letting the sequence model and the encoder improve together.

鲁棒性关键技术：SymLog 压缩与收益归一化Key Robustness Techniques: SymLog Compression and Return Normalization

跨域适用性的核心在于对不同量纲目标的统一处理：Cross-domain applicability rests on handling objectives of very different scales uniformly:

SymLog Transformation：symlog(x) = sign(x) · ln(|x|+1)，对大正负值对称压缩，用于重建损失和 critic 目标；避免了截断大目标、引入非平稳性等替代方案的缺陷。SymLog Transformation: symlog(x) = sign(x) · ln(|x|+1) compresses large positive and negative values symmetrically and is used for the reconstruction loss and the critic targets; it avoids the flaws of alternatives such as clipping large targets or introducing non-stationarity.
SymExp TwoHot 分类损失：Critic 输出对数量化 bin 上的 softmax 分布（指数间距），用软标签分类交叉熵替代回归，梯度尺度与目标量级解耦。SymExp TwoHot Classification Loss: the critic outputs a softmax distribution over exponentially spaced bins, replacing regression with a soft-label classification cross-entropy, so the gradient scale is decoupled from the target magnitude.
收益归一化（Return Normalization）：Actor 目标用 5th-95th 百分位范围 S = EMA(Per(R, 95) − Per(R, 5), 0.99) 归一化，固定熵系数 η = 3×10⁻⁴，消除不同任务绝对奖励量纲的影响。Return Normalization: actor targets are normalized by the 5th-95th percentile range S = EMA(Per(R, 95) − Per(R, 5), 0.99), with a fixed entropy coefficient η = 3×10⁻⁴, removing the influence of absolute reward scale across tasks.
Unimix 分类分布：所有分类分布混合 1% 均匀分布，防止确定性塌陷。Unimix Categorical Distributions: every categorical distribution is mixed with 1% of a uniform distribution to prevent deterministic collapse.
零初始化输出层：奖励预测器和 critic 的输出层权重初始化为零，稳定早期训练。Zero-Initialized Output Layers: the output layer weights of the reward predictor and the critic are initialized to zero, stabilizing early training.

03 实验Experiments

DreamerV3 在 7 大基准、150+ 任务上用同一套固定超参数进行评测，对比方法包括各基准的专用 SOTA 算法及高质量 PPO 实现。所有实验均在单张 A100 GPU 上完成。DreamerV3 is evaluated with one and the same fixed set of hyperparameters across 7 benchmark suites and 150+ tasks, against the specialized SOTA algorithm of each benchmark as well as a high-quality PPO implementation. All experiments run on a single A100 GPU.

基准Benchmark	数据量Data	主要对比方法Main baselines	结果Result
Atari 57（200M frames）	200M 帧200M frames	MuZero, Rainbow, IQN	超越 MuZero（使用更少算力）Outperforms MuZero while using less compute
Atari 100k（26 games, 400K frames）	400K 帧400K frames	IRIS, TWM, SPR, SimPLe	超越所有方法（EfficientZero 用重置不公平）Outperforms every method (EfficientZero's resets make it an unfair comparison)
ProcGen（16 games, 50M frames）	50M 帧50M frames	PPG（调参专用）, RainbowPPG (tuned per task), Rainbow	匹敌调参 PPG，超越 RainbowMatches tuned PPG, outperforms Rainbow
DMLab（30 tasks, 100M frames）	100M 帧100M frames	IMPALA, R2D2+（1B 步）IMPALA, R2D2+ (1B steps)	数据效率提升超过 1000%Data efficiency improved by over 1000%
本体感知控制（18 tasks, 500K steps）Proprioceptive control (18 tasks, 500K steps)	500K 步500K steps	D4PG, DMPO, MPO	新 SOTANew SOTA
视觉控制（20 tasks, 1M steps）Visual control (20 tasks, 1M steps)	1M 步1M steps	DrQ-v2, CURL	新 SOTANew SOTA
BSuite（468 configurations）	—	Boot DQN	新 SOTANew SOTA

minecraft diamond progress — **Figure 5：Minecraft 钻石任务中各代理发现关键物品的比例。** 对比算法（IMPALA、Rainbow）最多到达铁镐阶段，从未发现钻石。*"All the Dreamer agents we trained on Minecraft discover diamonds in 100M environment steps."* Dreamer 是唯一可靠获得钻石的算法，且无需人类数据或课程学习。**Figure 5: Fraction of agents that discover each key item in the Minecraft diamond task.** The baselines (IMPALA, Rainbow) reach the iron pickaxe stage at best and never find a diamond. *"All the Dreamer agents we trained on Minecraft discover diamonds in 100M environment steps."* Dreamer is the only algorithm that reliably obtains diamonds, and it needs neither human data nor a curriculum.

消融实验与规模扩展Ablations and Scaling

消融实验表明："The performance of Dreamer predominantly rests on the unsupervised reconstruction loss of its world model, unlike most prior algorithms that rely predominantly on reward and value prediction gradients." 其中 KL 目标（世界模型动态损失与表征损失）是最关键的学习信号；收益归一化和 symexp twohot 损失次之。The ablations show that "The performance of Dreamer predominantly rests on the unsupervised reconstruction loss of its world model, unlike most prior algorithms that rely predominantly on reward and value prediction gradients." Among the components, the KL objective (the dynamics loss and the representation loss of the world model) is the most critical learning signal, followed by return normalization and the symexp twohot loss.

04 局限性Limitations

说明：论文未设专门的 Limitations 章节。以下条目来源：前两项为作者在正文中明确提及的约束条件，后两项为从方法设计推断（标注为inferred）。Note: The paper has no dedicated Limitations section. Sources of the items below: the first two are constraints the authors state explicitly in the text, the last two are inferred from the method design (marked inferred).

计算资源需求较高（stated）Substantial compute requirements (stated)

Minecraft 实验需要单张 A100 GPU 训练 9 天。尽管与 VPT 的 720 GPU × 9 天相比已大幅降低，但对于普通研究者仍是不小的成本。模型规模从 12M 到 400M 参数的实验也需要相应算力支撑。The Minecraft experiment needs 9 days of training on a single A100 GPU. Although far cheaper than the 720 GPUs × 9 days of VPT, it is still a considerable cost for an ordinary researcher. The experiments scaling model size from 12M to 400M parameters also demand corresponding compute.

超参数仍需人工选定（stated）Hyperparameters still have to be chosen by hand (stated)

论文声称"固定超参数"跨域使用，但这套超参数本身是在广泛实验后选定的。作者在附录中指出，某些超参数（如折扣因子 γ = 0.997）在一些任务上并非最优，只是在所有任务上"足够好"。The paper claims that "fixed hyperparameters" work across domains, but that very set was chosen after extensive experimentation. In the appendix the authors note that some hyperparameters (such as the discount factor γ = 0.997) are not optimal on certain tasks, merely "good enough" on all of them.

离散动作空间与连续动作空间的统一处理（inferred）Unified treatment of discrete and continuous action spaces (inferred)

DreamerV3 在两种动作空间均有评测，但 actor 的梯度估计方式不同（连续用重参数化，离散用 straight-through estimator），可能在某些任务上引入额外方差。论文未针对这一差异做专项分析。DreamerV3 is evaluated on both kinds of action space, but the actor gradient is estimated differently in each (reparameterization for continuous actions, a straight-through estimator for discrete ones), which may introduce extra variance on some tasks. The paper offers no dedicated analysis of this difference.

世界模型的表征能力瓶颈（inferred）Representation capacity of the world model as a bottleneck (inferred)

RSSM 的离散表征（32×32 one-hot）和固定预测 horizon（T=16 步）在极长时序依赖或高度随机环境中可能成为瓶颈。论文在 DMLab（需要空间+时序推理）上已体现出世界模型的优势，但更长期规划的场景未被深入分析。The discrete representation of RSSM (32×32 one-hot) and the fixed prediction horizon (T=16 steps) may become bottlenecks in environments with extremely long temporal dependencies or high stochasticity. The paper already shows the advantage of the world model on DMLab (which requires spatial and temporal reasoning), but scenarios demanding longer-horizon planning are not analyzed in depth.