DayDreamer: World Models for Physical Robot Learning

01 动机Motivation

深度强化学习 (deep RL) 在机器人学习中广受欢迎，但现有算法需要大量与环境的试错交互， 在真实机器人上部署成本极高。为此，大多数工作依赖仿真器进行预训练， 再通过 sim-to-real 迁移到实体机器人。然而，仿真器难以精确复现真实世界的动态特性， 且训练出的策略无法自动适应环境变化。 Deep reinforcement learning (deep RL) is widely used in robot learning, but existing algorithms need a large amount of trial-and-error interaction with the environment, which makes deployment on real robots extremely costly. Most work therefore relies on simulators for pre-training, and then transfers to physical robots via sim-to-real. Simulators, however, can hardly reproduce real-world dynamics faithfully, and the resulting policies cannot adapt to changes of the environment on their own.

"Learning inside of simulators fails to capture the complexity of the real world, is prone to simulator inaccuracies, and the resulting behaviors do not adapt to changes in the world."

世界模型 (world model) 通过从少量真实交互中学习环境动态，使智能体能够在"想象"中规划， 从而大幅减少现实中的试错次数。Dreamer 算法已在视频游戏中展现了卓越的数据效率， 但能否在真实机器人上同样奏效仍是开放问题——本文正面回答了这一问题。 A world model learns environment dynamics from a small amount of real interaction, letting the agent plan in imagination and thereby cutting down the number of trials needed in reality. The Dreamer algorithm has already shown outstanding data efficiency in video games, but whether it works just as well on real robots remained an open question — this paper answers it head-on.

02 方法Method

DayDreamer 直接复用 DreamerV2 算法，核心是将 世界模型学习与行为学习 (actor-critic) 解耦并异步运行：learner 线程持续更新神经网络，actor 线程并行与真实机器人交互， 满足高控制频率下的低延迟需求。 DayDreamer directly reuses the DreamerV2 algorithm; the key idea is to decouple world model learning from behavior learning (actor-critic) and run them asynchronously: a learner thread keeps updating the neural networks while an actor thread interacts with the real robot in parallel, which meets the low-latency requirement of high control frequencies.

世界模型：Recurrent State-Space Model (RSSM)World model: Recurrent State-Space Model (RSSM)

世界模型基于 RSSM，包含四个组件： The world model is built on the RSSM and comprises four components:

• Encoder：将所有传感器模态（图像 + 本体感知）融合为离散潜在码 z_t。Encoder: fuses all sensor modalities (images + proprioception) into a discrete latent code z_t.
• Dynamics network：利用循环隐状态 h_t 预测未来潜在码序列，无需观测中间输入。Dynamics network: predicts the sequence of future latent codes from the recurrent hidden state h_t, with no need for intermediate observations as input.
• Decoder：从潜在码重建传感器输入，提供丰富的学习信号，并支持人工检视模型预测质量。Decoder: reconstructs the sensor inputs from the latent codes, providing a rich learning signal and letting humans inspect the quality of the model's predictions.
• Reward network：预测奖励信号，支持从真实世界自动发现任务奖励。Reward network: predicts the reward signal, enabling task rewards to be discovered automatically from the real world.

所有组件联合优化（stochastic backpropagation）。由于预测的是紧凑潜在表示而非高维原始观测， 累积误差大幅降低，可在单块 GPU 上以 batch size 16K 进行大规模并行训练。 All components are optimized jointly (stochastic backpropagation). Because what is predicted is a compact latent representation rather than high-dimensional raw observations, compounding error is greatly reduced, and large-scale parallel training with batch size 16K fits on a single GPU.

Actor-Critic 行为学习Actor-Critic Behavior Learning

Actor 网络 π(a_t|s_t) 与 Critic 网络 v(s_t) 完全在世界模型的潜在空间中通过想象轨迹优化， 无需解码观测。Critic 通过 temporal difference learning 预测 λ-returns（平均 N∈[1,H-1]， imagination horizon H=15），Actor 通过最大化 λ-returns 学习策略。 连续动作任务使用 reparameterization gradients，离散动作使用 Reinforce gradients。 Actor 还通过熵正则化防止策略过早收敛。 The Actor network π(a_t|s_t) and the Critic network v(s_t) are optimized entirely inside the latent space of the world model through imagined trajectories, without decoding observations. The Critic predicts λ-returns by temporal difference learning (averaged over N∈[1,H-1], imagination horizon H=15), and the Actor learns its policy by maximizing the λ-returns. Continuous-action tasks use reparameterization gradients, discrete actions use Reinforce gradients. The Actor is further regularized by entropy to keep the policy from converging prematurely.

超参数设置：replay buffer 容量 10⁶，batch size 32，batch length 32， RSSM 隐状态维度 512，latent codes 32×32，discount γ=0.95，λ=0.95，学习率 10^-4， 所有机器人实验使用完全相同的超参数。 Hyperparameters: replay buffer capacity 10⁶, batch size 32, batch length 32, RSSM hidden state size 512, latent codes 32×32, discount γ=0.95, λ=0.95, learning rate 10^-4, and every robot experiment uses exactly the same hyperparameters.

03 实验Experiments

在 4 款机器人上评估 Dreamer，对比各自领域最强的 model-free baseline： SAC（连续控制）、Rainbow DQN + PPO（离散视觉控制）、DrQv2（连续视觉控制）， 以及人类操作员作为近似上界。 Dreamer is evaluated on 4 robots against the strongest model-free baseline of each domain: SAC (continuous control), Rainbow DQN + PPO (discrete visual control) and DrQv2 (continuous visual control), with a human operator as an approximate upper bound.

A1 四足机器人行走（A1 Quadruped Walking）A1 Quadruped Walking

任务：从背部朝上躺着出发，无任何 reset，学会翻身、站起、以目标速度行走。 动作为 12 个关节角度（20 Hz），输入为关节角、姿态、角速度。 奖励函数由 5 个分量组成（upright、髋/肩/膝关节角度、前向速度），最大奖励为 14。 Task: starting from lying on its back, with no resets at all, learn to flip over, stand up and walk at a target velocity. The action consists of 12 joint angles (20 Hz), and the input comprises joint angles, orientation and angular velocity. The reward function is composed of 5 terms (upright, hip/shoulder/knee joint angles, forward velocity), with a maximum reward of 14.

UR5 多目标视觉抓放（UR5 Multi-Object Visual Pick and Place）UR5 Multi-Object Visual Pick and Place

任务：从第三方相机 RGB 图像中定位 3 个球，将其从一个料仓移到另一料仓。 稀疏奖励：抓住 +1，放回同仓 −1，放入对仓 +10。 动作离散（X/Y/Z 增量 + 夹爪开关，2 Hz）。对比 Rainbow DQN、PPO 以及人类操作员（20 分钟演示）。 Task: locate 3 balls in the RGB image from a third-person camera and move them from one bin to another. Sparse reward: +1 for a grasp, −1 for returning a ball to the same bin, +10 for placing it in the other bin. Actions are discrete (X/Y/Z increments + gripper open/close, 2 Hz). Compared against Rainbow DQN, PPO and a human operator (20 minutes of demonstration).

XArm 视觉抓放（XArm Visual Pick and Place）XArm Visual Pick and Place

XArm 为低成本 7-DOF 机械臂（约 0.5 Hz），使用 RGB-D 相机（深度+彩色）， 需学习将软物体从一仓移到另一仓（物体用绳连接夹爪，避免卡角）。 在改变光照条件（日出时强烈阴影）下，Dreamer 性能短暂下降后约 5 小时自适应恢复并超越原有性能。 XArm is a low-cost 7-DOF arm (about 0.5 Hz) using an RGB-D camera (depth + color), which has to learn to move a soft object from one bin to another (the object is tied to the gripper with a string to keep it from getting stuck in corners). Under changed lighting conditions (harsh shadows at sunrise), Dreamer's performance drops briefly and then recovers within about 5 hours, adapting beyond its previous level.

Sphero 视觉导航（Sphero Navigation）Sphero Visual Navigation

Sphero Ollie 轮式机器人，仅凭俯视 RGB 图像（无本体感知），连续力矩控制（2 Hz）， 导航至固定目标点。奖励为负 L2 距离。 Dreamer 2 小时内达到平均目标距离 0.15（场地归一化），与专门为像素连续控制设计的 DrQv2 持平。 The Sphero Ollie wheeled robot navigates to a fixed goal from top-down RGB images alone (no proprioception), under continuous torque control (2 Hz). The reward is the negative L2 distance. Within 2 hours Dreamer reaches an average goal distance of 0.15 (normalized by arena size), on par with DrQv2, which was designed specifically for continuous control from pixels.

适应性与泛化（Appendix A）Adaptivity and generalization (Appendix A)

XArm 实验在日落后进行，日出时光照剧变导致性能下滑，但 Dreamer 无需算法改动， 约 5 小时自动适应并超越原有性能——展现了世界模型在 continual learning 场景下的潜力。 A1 机器人在行走策略学成后，10 分钟内自适应抵抗外力推扰或迅速翻身恢复。 The XArm experiment ran after sunset, and the drastic change of lighting at sunrise caused a performance drop; yet Dreamer needed no algorithmic change, adapting automatically within about 5 hours and surpassing its previous performance — showing the potential of world models in continual learning settings. Once the A1 robot had learned its walking policy, it adapted within 10 minutes to resist external pushes or to flip back and recover quickly.

04 局限性Limitations