Learning Latent Dynamics for Planning from Pixels（PlaNet）

01 动机（Motivation）Motivation

强化学习在控制任务上取得了令人瞩目的成果，但绝大多数成功案例都依赖 model-free 方法——需要与环境进行数以百万计的交互。当传感器只能提供高维图像时，样本效率问题更为严峻。利用学习到的动态模型（dynamics model）进行规划，理论上可以大幅减少所需交互，但长期以来难以在复杂的图像域中可靠地实现。Reinforcement learning has produced striking results on control tasks, but the overwhelming majority of successes rely on model-free methods—which require millions of interactions with the environment. When the sensors provide only high-dimensional images, the sample-efficiency problem becomes even more severe. Planning with a learned dynamics model could in principle cut the required interactions drastically, but it has long been hard to realize reliably in complex image domains.

"Planning using learned models offers several benefits over model-free reinforcement learning. First, model-based planning can be more data efficient because it leverages a richer training signal and does not rely on propagating rewards through Bellman backups."

六种连续控制任务示例 — **Figure 1（论文原图）**：PlaNet 所测试的六种 DeepMind Control Suite 任务——Cartpole Swingup、Reacher Easy、Cheetah Run、Finger Spin、Cup Catch、Walker Walk。图像分辨率降采样至 64×64 像素后作为唯一观测，不提供物理状态。Cartpole/Finger Spin 仅有稀疏奖励；Cheetah/Walker 存在接触动力学。**Figure 1 (from the paper)**: The six DeepMind Control Suite tasks on which PlaNet is evaluated—Cartpole Swingup, Reacher Easy, Cheetah Run, Finger Spin, Cup Catch, Walker Walk. Images are downsampled to 64×64 pixels and serve as the only observation; no physical state is provided. Cartpole/Finger Spin give sparse rewards only; Cheetah/Walker involve contact dynamics.

6连续控制任务（DeepMind Control Suite）continuous control tasks (DeepMind Control Suite)

50×比 D4PG 少的样本（Finger Spin 任务，100K episodes）fewer samples than D4PG (Finger Spin task, 100K episodes)

64×64像素输入分辨率（唯一观测）pixel input resolution (the only observation)

1 GPUNvidia V100，10–20 小时训练Nvidia V100, 10–20 hours of training

02 方法（Method）Method

PlaNet 的核心是 Recurrent State Space Model（RSSM）——一个同时包含确定性（deterministic）和随机性（stochastic）转移分量的隐状态空间模型。通过变分自编码器（VAE）框架从像素中学习该模型，再利用 Cross-Entropy Method（CEM）在隐空间中规划动作序列，无需任何策略网络。At the core of PlaNet is the Recurrent State Space Model (RSSM)—a latent state space model whose transition contains both a deterministic and a stochastic component. The model is learned from pixels within a variational autoencoder (VAE) framework, and action sequences are then planned in latent space with the Cross-Entropy Method (CEM), without any policy network.

RSSM 模型架构对比 — **Figure 2（论文原图）**：三种隐状态空间模型对比。(a) 纯确定性 RNN：转移完全确定，难以捕捉多模态未来；(b) 纯随机状态空间模型（SSM）：每步独立采样随机状态，信息传播受限；(c) RSSM（本文）：将隐状态分为确定性部分 *h_t*（由 GRU 维护）和随机部分 *s_t*，二者共同决定观测和奖励预测。这种设计使模型既能记忆长程历史，又能表示不确定性。**Figure 2 (from the paper)**: Comparison of three latent state space models. (a) Purely deterministic RNN: the transition is fully determined, making multimodal futures hard to capture; (b) purely stochastic state space model (SSM): the stochastic state is sampled independently at each step, which limits information propagation; (c) RSSM (this paper): the latent state is split into a deterministic part *h_t* (maintained by a GRU) and a stochastic part *s_t*, which together determine the observation and reward predictions. This design lets the model both remember long-range history and represent uncertainty.

Recurrent State Space Model（RSSM）Recurrent State Space Model (RSSM)

RSSM 将隐状态 z_t 拆分为两部分：RSSM splits the latent state z_t into two parts:

确定性状态 h_t：由 GRU（Gated Recurrent Unit）根据上一步确定性状态、随机状态和动作递推得到，负责跨步骤传递长期信息。Deterministic state h_t: obtained recurrently by a GRU (Gated Recurrent Unit) from the previous deterministic state, stochastic state and action; it carries long-term information across steps.
随机状态 s_t：以 Gaussian 分布为先验，以当前确定性状态为条件独立采样，使模型能够表达对未来的不确定性。Stochastic state s_t: sampled independently with a Gaussian prior conditioned on the current deterministic state, so that the model can express uncertainty about the future.

观测模型（decoder）和奖励模型（reward model）均以 (h_t, s_t) 为输入进行预测。训练使用变分下界（ELBO），同时最大化图像重建对数似然和奖励预测准确性，并对 KL 散度施加正则化。Both the observation model (decoder) and the reward model take (h_t, s_t) as input for prediction. Training uses the variational lower bound (ELBO), jointly maximizing the image reconstruction log-likelihood and the reward prediction accuracy while regularizing the KL divergence.

Latent Overshooting（隐空间多步预测目标）Latent Overshooting (a multi-step prediction objective in latent space)

标准 VAE 目标（ELBO）只对单步转移进行变分推断，导致模型仅在一步预测上做优化，而规划需要准确的多步预测。论文提出 latent overshooting：在所有预测距离 d = 1, 2, …, D 上都附加 KL 散度惩罚项，迫使先验和后验在多个时间步上保持一致：The standard VAE objective (ELBO) performs variational inference over one-step transitions only, so the model is optimized for one-step prediction, whereas planning needs accurate multi-step prediction. The paper proposes latent overshooting: a KL divergence penalty is attached at every prediction distance d = 1, 2, …, D, forcing prior and posterior to stay consistent over multiple time steps:

Latent Overshooting 示意图 — **Figure 3（论文原图）**：三种变分目标对比。(a) 标准变分界（只有单步 KL）；(b) observation overshooting（对每个距离的预测都加重建损失，计算量极大）；(c) latent overshooting（本文）：只在隐空间计算多步 KL，避免了高开销的像素解码，但提供了更强的多步一致性监督信号。**Figure 3 (from the paper)**: Comparison of three variational objectives. (a) The standard variational bound (one-step KL only); (b) observation overshooting (a reconstruction loss is added for the prediction at every distance, which is extremely expensive); (c) latent overshooting (this paper): the multi-step KL is computed in latent space only, avoiding costly pixel decoding while still providing a stronger supervision signal for multi-step consistency.

Latent overshooting 的核心思路："If we could train our model to make perfect one-step predictions, it would also make perfect multi-step predictions, so this would not be a problem. However, when using a model with limited capacity and restricted distributional family, training the model only on one-step predictions until convergence does in general not coincide with the model that is best at multi-step predictions."The core idea of latent overshooting: "If we could train our model to make perfect one-step predictions, it would also make perfect multi-step predictions, so this would not be a problem. However, when using a model with limited capacity and restricted distributional family, training the model only on one-step predictions until convergence does in general not coincide with the model that is best at multi-step predictions."

在线规划：CEM（Cross-Entropy Method）Online planning: CEM (Cross-Entropy Method)

测试时，agent 利用学习到的 RSSM 进行 MPC（Model Predictive Control）：在隐空间中以 CEM 采样并评估 H 步候选动作序列（默认 H=12，迭代 10 次，每次 1000 个候选），选取期望累积奖励最高的序列执行第一步动作，再重新规划。整个过程不需要任何策略网络或价值函数——模型是唯一"知识"来源。At test time the agent uses the learned RSSM for MPC (Model Predictive Control): candidate action sequences of H steps are sampled and evaluated with CEM in latent space (by default H=12, 10 iterations, 1000 candidates each); the sequence with the highest expected cumulative reward is chosen, its first action is executed, and planning is redone. The whole procedure needs no policy network and no value function—the model is the only source of "knowledge".

03 实验（Experiments）Experiments

在 DeepMind Control Suite 的 6 个连续控制任务上，对比 A3C（模型无关、像素输入）、D4PG（模型无关、像素输入，proprioceptive 版本作为上界）、带真实动态的 CEM（oracle），以及纯随机随机策略。所有 pixel-based 方法使用相同 64×64 三阶段下采样图像。指标为最终性能的 median ± IQR（5 seeds × 10 trajectories）。On the 6 continuous control tasks of the DeepMind Control Suite, PlaNet is compared with A3C (model-free, pixel input), D4PG (model-free, pixel input, with the proprioceptive version as an upper bound), CEM with the true dynamics (oracle), and a purely random policy. All pixel-based methods use the same 64×64 three-stage downsampled images. The metric is the median ± IQR of final performance (5 seeds × 10 trajectories).

训练曲线对比 — **Figure 4（论文原图）**：六个任务上 PlaNet（RSSM）vs. 纯随机策略（Random collection）vs. A3C（100K episodes）vs. D4PG（1M episodes，proprioceptive，作为 oracle 上界）的训练曲线。横轴为收集的 episode 数，纵轴为 episode 奖励。PlaNet 在 Finger Spin、Cup Catch、Walker Walk 等任务上用远少于 D4PG 的样本超越或接近其性能；Cheetah Run 上 D4PG 占优。**Figure 4 (from the paper)**: Training curves on the six tasks for PlaNet (RSSM) vs. a purely random policy (Random collection) vs. A3C (100K episodes) vs. D4PG (1M episodes, proprioceptive, serving as an oracle upper bound). The horizontal axis is the number of episodes collected, the vertical axis is the episode reward. On tasks such as Finger Spin, Cup Catch and Walker Walk, PlaNet surpasses or approaches D4PG with far fewer samples; on Cheetah Run, D4PG is ahead.

Table 1 与模型设计消融 — **Table 1（论文原图）**：方法比较——模态（proprioceptive vs. pixels）、所需 episodes 数与最终性能（各任务 median reward，100 episodes 评估）。PlaNet（RSSM）在 pixels 模态下以 1,000 个 episodes 实现最优总分 862；D4PG 使用 proprioceptive 观测和 100,000 个 episodes 得分 961（作为 oracle 上界）；A3C 用 100,000 episodes 只达到 214。**Table 1 (from the paper)**: Method comparison—modality (proprioceptive vs. pixels), the number of episodes required, and final performance (median reward per task, evaluated over 100 episodes). In the pixels modality, PlaNet (RSSM) attains the best total score of 862 with 1,000 episodes; D4PG scores 961 with proprioceptive observations and 100,000 episodes (as an oracle upper bound); A3C reaches only 214 with 100,000 episodes.

方法Method	观测模态Observation modality	Episodes	总 median 奖励Total median reward
A3C (pixels)	pixels	100,000	214
D4PG (pixels)	pixels	100,000	462
D4PG (proprioceptive)	proprioceptive	100,000	961
PlaNet / RSSM（本文）PlaNet / RSSM (this paper)	pixels	1,000	862
CEM + 真实动态（oracle）CEM + true dynamics (oracle)	state	—	941

**Figure 5（论文原图）**：PlaNet 与 model-free 算法（A3C、D4PG）的样本效率对比，图中展示了在相同 episode 预算下各方法的性能。PlaNet 使用仅约 2,000 个 episodes 便超越使用 100,000 个 episodes 的 A3C pixel 版本。在 Finger Spin 任务上，PlaNet 以 100K episodes 的表现超越 D4PG 约 20%（relative improvement）。**Figure 5 (from the paper)**: Sample-efficiency comparison between PlaNet and model-free algorithms (A3C, D4PG); the figure shows the performance of each method under the same episode budget. With only about 2,000 episodes PlaNet surpasses the A3C pixel version that uses 100,000 episodes. On the Finger Spin task, PlaNet exceeds the 100K-episode performance of D4PG by about 20% (relative improvement).

消融实验（Ablations）Ablations

论文通过控制变量验证了各设计选择的必要性：The paper verifies the necessity of each design choice by controlled ablations:

RSSM vs. 纯确定性（GRU）vs. 纯随机（SSM）：在所有任务上 RSSM 均优于另外两种，说明确定性和随机分量缺一不可。纯随机 SSM 在 Cheetah Run 等接触动力学任务上表现最差。RSSM vs. purely deterministic (GRU) vs. purely stochastic (SSM): RSSM beats the other two on all tasks, showing that neither the deterministic nor the stochastic component can be dropped. The purely stochastic SSM performs worst on contact-dynamics tasks such as Cheetah Run.
Latent overshooting 的效果：加入多步 KL 目标后，多步预测准确性显著提升，规划性能随之改善，尤其在需要长视野的稀疏奖励任务上。Effect of latent overshooting: after adding the multi-step KL objective, multi-step prediction accuracy improves markedly and planning performance improves with it, especially on sparse-reward tasks that need a long horizon.
规划视野 H 与迭代次数：H=12、迭代 10 次在性能与计算开销之间取得较好平衡；过短的视野导致规划短视，过长则计算开销上升且预测误差累积。Planning horizon H and number of iterations: H=12 with 10 iterations strikes a good balance between performance and computational cost; too short a horizon makes planning myopic, while too long a one raises the cost and lets prediction error accumulate.

04 局限性（Limitations）Limitations

Note：论文在第 7 节（Limitations and Future Work）中明确列出了以下局限，属于作者stated陈述；部分条目为从设计出发的inferred推断，已标注。Note:The paper explicitly lists the following limitations in Section 7 (Limitations and Future Work); they are stated by the authors, while some entries are inferred from the design and are marked as such.

规划计算开销随视野增加（stated）Planning cost grows with the horizon (stated)

CEM 规划在每个环境步骤都需要在隐空间中评估大量候选序列（默认 1000 × 10 次迭代 × 12 步），随着规划视野 H 增加，计算量线性增长。论文指出"对于实际实时控制问题这仍是一个挑战"。At every environment step, CEM planning has to evaluate a large number of candidate sequences in latent space (by default 1000 candidates × 10 iterations × 12 steps), and the amount of computation grows linearly as the planning horizon H increases. The paper notes that this "remains a challenge for practical real-time control problems".

视觉复杂度与部分可观测性的局限（stated）Limits under visual complexity and partial observability (stated)

论文明确表示该方法"在视觉复杂度很高或者部分可观测性严重时可能面临困难"。实验中的 64×64 像素任务视觉相对简单；在更真实的图像域中模型容量可能成为瓶颈。The paper states explicitly that the method "may run into difficulty when visual complexity is very high or partial observability is severe". The 64×64 pixel tasks used in the experiments are visually rather simple; in more realistic image domains model capacity may become the bottleneck.

仅限连续动作空间与低维任务（inferred）Restricted to continuous action spaces and low-dimensional tasks (inferred)

PlaNet 的 CEM 规划天然适配连续动作空间，对高维离散动作空间或复杂操控任务（如机械臂抓取）的扩展性尚未验证。实验中的六个任务动作维度均较低（1–6维）。PlaNet's CEM planning fits continuous action spaces naturally; its extensibility to high-dimensional discrete action spaces or complex manipulation tasks (such as robot-arm grasping) has not been verified. The action dimensionality of all six tasks in the experiments is low (1–6 dimensions).

模型复合误差（compounding errors，inferred）Compounding model errors (inferred)

基于模型规划的固有问题：在分布外（out-of-distribution）的状态区域，learned dynamics 的预测误差会随规划步数累积，导致规划效果退化。论文通过缩短 MPC 重规划间隔来缓解，但未根本解决。An inherent problem of model-based planning: in out-of-distribution state regions, the prediction error of the learned dynamics accumulates with the number of planning steps and degrades planning quality. The paper mitigates this by shortening the MPC replanning interval, but does not solve it fundamentally.