Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation

01 Motivation

机器人在新环境中往往只有有限的、质量参差不齐的交互数据可用——既有示教，也有失败尝试、探索性动作和旧策略的 rollout。论文指出："Robot learning requires adaptation methods that improve reliably from limited, mixed-quality interaction data." 这一挑战在长时程、接触密集（contact-rich）的任务中尤为突出，此时端到端策略微调（end-to-end policy finetuning）"remains inefficient and brittle"。In a new environment a robot usually has only limited, mixed-quality interaction data available — demonstrations, but also failed attempts, exploratory actions and rollouts of older policies. The paper notes: "Robot learning requires adaptation methods that improve reliably from limited, mixed-quality interaction data." This challenge is especially acute in long-horizon, contact-rich tasks, where end-to-end policy finetuning "remains inefficient and brittle".

世界模型（world model）被认为是更合适的抽象："World models offer a compelling alternative: by predicting the outcomes of candidate action sequences, they enable online planning through counterfactual reasoning." 但直接在真实世界里训练带动作条件的机器人世界模型，需要规模上不现实的多样化数据——"training action-conditioned robotic world models directly in the real world requires diverse data at impractical scale."The world model is held to be the more suitable abstraction: "World models offer a compelling alternative: by predicting the outcomes of candidate action sequences, they enable online planning through counterfactual reasoning." But training an action-conditioned robotic world model directly in the real world requires diverse data at an unrealistic scale — "training action-conditioned robotic world models directly in the real world requires diverse data at impractical scale."

"world models, rather than monolithic end-to-end policies, provide the right abstraction for leveraging prior experience to improve efficiently in new environments."

02 Method

SimDist 分两个阶段：仿真预训练阶段用特权状态训练的专家策略 + 次优策略 + 随机动作扰动，生成覆盖失败与恢复行为的多样化轨迹，联合训练 latent dynamics、reward、value 与 behavior-cloning 四个损失项；真实世界自适应阶段冻结编码器、奖励模型与价值函数，只用真实世界的预测误差更新 latent dynamics 模型，将适应问题化简为监督式系统辨识，同时保留稠密、长时程的规划信号供在线改进使用。SimDist has two stages. In the simulation pretraining stage, an expert policy trained on privileged state + sub-optimal policies + random action perturbations generate diverse trajectories covering failure and recovery behaviour, and the four loss terms latent dynamics, reward, value and behavior cloning are trained jointly; in the real-world adaptation stage the encoder, reward model and value function are frozen and only the latent dynamics model is updated from real-world prediction error, reducing the adaptation problem to supervised system identification while preserving dense, long-horizon planning signals for online improvement.

仿真中的多样化数据生成（Diverse Data Generation）Diverse Data Generation in Simulation

论文强调：规划器会主动搜索高价值动作序列，并在覆盖薄弱处利用模型误差，因此模型必须在专家分布与真实数据分布之外仍然可靠。为此，SimDist 通过"alternating between an expert policy and a set of sub-optimal policies, and by periodically injecting random action perturbations over short temporal windows"来生成多样化 rollout，从而覆盖专家流形之外的失败与恢复行为。预训练损失同时监督 latent dynamics、reward、value 和 behavior cloning 四项：The paper stresses that the planner actively searches for high-value action sequences and exploits model error wherever coverage is thin, so the model must remain reliable outside the expert distribution and the real data distribution. SimDist therefore generates diverse rollouts by "alternating between an expert policy and a set of sub-optimal policies, and by periodically injecting random action perturbations over short temporal windows", covering failure and recovery behaviour off the expert manifold. The pretraining loss supervises latent dynamics, reward, value and behavior cloning at the same time:

ℒ_t^sim(θ) 由 latent-dynamics 项 ‖ẑ_t+i+1 − sg(E_θ(o_t+i+1))‖²、reward 项 c₁(r̂_t+i−r_t+i)²、value 项 c₂(v̂_t+i+1−v_t+i+1)² 以及仅对专家动作生效的 behavior-cloning 项 c₃·𝟙_e(a_t+i)‖â_t+i−a_t+i‖² 组成（原文 Eq. 2）。ℒ_t^sim(θ) consists of a latent-dynamics term ‖ẑ_t+i+1 − sg(E_θ(o_t+i+1))‖², a reward term c₁(r̂_t+i−r_t+i)², a value term c₂(v̂_t+i+1−v_t+i+1)², and a behavior-cloning term that applies only to expert actions, c₃·𝟙_e(a_t+i)‖â_t+i−a_t+i‖² (Eq. 2 in the paper).

真实世界自适应 = 系统辨识（System Identification）Real-World Adaptation = System Identification

部署阶段，SimDist "freezes the encoder, reward model, and value function learned in simulation, and updates only the latent dynamics model using real-world prediction losses"。这一设计"reduces adaptation to supervised system identification while preserving dense, long-horizon planning signals for online improvement"——即真实世界只需纠正仿真与现实之间的动力学差距，而不必重新学习奖励和价值判断。At deployment SimDist "freezes the encoder, reward model, and value function learned in simulation, and updates only the latent dynamics model using real-world prediction losses". This design "reduces adaptation to supervised system identification while preserving dense, long-horizon planning signals for online improvement" — that is, the real world only has to correct the dynamics gap between simulation and reality, without relearning reward and value judgements.

基于 MPPI 的在线规划MPPI-Based Online Planning

使用 Model Predictive Path Integral (MPPI)——一种基于采样的 MPC 方法——评估候选动作序列：reward/value 模型对整条预测 latent 轨迹 ẑ_t:t+T 做 attention，从而得到更准确的回报估计，再据此挑选并执行动作。Model Predictive Path Integral (MPPI) — a sampling-based MPC method — is used to evaluate candidate action sequences: the reward/value model attends over the entire predicted latent trajectory ẑ_t:t+T to obtain a more accurate return estimate, and the action is then selected and executed accordingly.

03 Experiments

操作任务在 UR5e 机械臂上进行：Peg Insertion（16 mm 方形插销任务）与 Table Leg（桌腿插孔装配），初始条件分别取自 Narrow / Wide 网格；仿真预训练用 100k 条轨迹。四足任务在 Unitree Go2 上进行：Slippery Slope（3.0°/5.7° 覆盖 PTFE 的低摩擦斜坡，需走完 1.82 m）与 Foam（两块 5 cm 厚记忆棉，仿真中未建模其柔顺动力学，需走完 3.00 m）；仿真预训练用 100M 条轨迹。对照基线为使用相同编码器、但策略头为 MLP 的 state-of-the-art 在线 RL 方法（IQL、RLPD 等），操作任务上另加行为克隆基线；真实世界微调数据量为操作任务 15–30 分钟，四足 Slippery Slope 35.7 分钟、Foam 32.1 分钟。The manipulation tasks run on a UR5e arm: Peg Insertion (a 16 mm square peg) and Table Leg (assembling a table leg into its hole), with initial conditions drawn from a Narrow / Wide grid respectively; simulation pretraining uses 100k trajectories. The quadruped tasks run on a Unitree Go2: Slippery Slope (a 3.0°/5.7° low-friction slope covered with PTFE, 1.82 m to be traversed) and Foam (two 5 cm thick memory-foam blocks whose compliant dynamics are not modelled in simulation, 3.00 m to be traversed); simulation pretraining uses 100M trajectories. The baselines are state-of-the-art online RL methods (IQL, RLPD, etc.) that use the same encoder but an MLP policy head, plus a behavior cloning baseline on the manipulation tasks; real-world finetuning data amounts to 15–30 minutes for manipulation, 35.7 minutes for quadruped Slippery Slope and 32.1 minutes for Foam.

整体结果（Fig. 4，定性）："Across all tasks, SimDist consistently outperforms prior approaches, achieving substantially higher success rates with far greater sample efficiency than online RL baselines, while autonomously improving well beyond the performance of behavior cloning methods... Across the board SimDist typically reaches scores around 2× higher than any baseline." 标准 RL 微调方法常出现灾难性遗忘（catastrophic forgetting），性能在适应过程中崩溃；SGFT（迁移仿真价值函数的基线）虽避免了崩溃，但采样效率仍显著低于 SimDist。此外，为 SimDist 额外提供示教数据只会进一步提升表现，说明它能自然吸收异质、混合质量的数据来源。Overall results (Fig. 4, qualitative): "Across all tasks, SimDist consistently outperforms prior approaches, achieving substantially higher success rates with far greater sample efficiency than online RL baselines, while autonomously improving well beyond the performance of behavior cloning methods... Across the board SimDist typically reaches scores around 2× higher than any baseline." Standard RL finetuning methods frequently show catastrophic forgetting, with performance collapsing during adaptation; SGFT (a baseline that transfers the simulation value function) avoids the collapse but is still markedly less sample-efficient than SimDist. Moreover, supplying SimDist with additional demonstration data only improves it further, showing that it naturally absorbs heterogeneous, mixed-quality data sources.

Table IX：四足真实世界结果（success / 5 次试验）Table IX: quadruped real-world results (success / 5 trials)

数据取自论文 Table IX；SimDist/IQL/RLPD 结果均为使用 35.7 分钟（Slippery Slope）/ 32.1 分钟（Foam）真实数据微调后的表现。"RLPD results on the Foam task are not reported, as the method destabilized the robot prior to evaluation."Data taken from Table IX of the paper; the SimDist/IQL/RLPD results are all after finetuning on 35.7 minutes (Slippery Slope) / 32.1 minutes (Foam) of real data. "RLPD results on the Foam task are not reported, as the method destabilized the robot prior to evaluation."

Ablations

Unfreezing world-model components（Fig. 10）："Unfreezing the encoder causes complete performance loss, as frozen reward and value heads receive latents outside their training distribution. Unfreezing the value function... reintroduces long-horizon credit assignment from limited real-world data and causes catastrophic forgetting." 换言之，只冻结 dynamics 之外的一切、且只微调 dynamics 是关键设计，而非可有可无的选择。Unfreezing world-model components (Fig. 10): "Unfreezing the encoder causes complete performance loss, as frozen reward and value heads receive latents outside their training distribution. Unfreezing the value function... reintroduces long-horizon credit assignment from limited real-world data and causes catastrophic forgetting." In other words, freezing everything except the dynamics and finetuning the dynamics alone is a critical design choice, not an optional one.

此外论文还报告：解冻策略在数据规模消融（Table I，10%/50% 仿真数据）中验证了大规模、多样化仿真数据对性能的贡献；Fig. 6 显示 SimDist 训练吞吐量（success/min）"monotonically improves throughput by ∼1.5×–2× over zero-shot performance"；Fig. 7 的散点图显示，相比 Diffusion Policy，SimDist 在 Peg Wide 任务上学到的策略对初始条件的鲁棒性明显更强，体现了大规模仿真预训练带来的状态空间覆盖优势。The paper further reports: the data-scale ablation (Table I, 10%/50% of the simulation data) confirms the contribution of large-scale, diverse simulation data to performance; Fig. 6 shows that SimDist training throughput (success/min) "monotonically improves throughput by ∼1.5×–2× over zero-shot performance"; and the scatter plot in Fig. 7 shows that, compared with Diffusion Policy, the policy SimDist learns on the Peg Wide task is markedly more robust to initial conditions, reflecting the state-space coverage that large-scale simulation pretraining brings.

04 Limitations