AdaWorld: Learning Adaptable World Models with Latent Actions
无需大量动作标注，通过自监督 latent action 实现跨环境世界模型高效适应Efficient cross-environment world model adaptation via self-supervised latent actions, with no need for large-scale action annotations
Shenyuan Gao · Siyuan Zhou · Yilun Du · Jun Zhang · Chuang Gan | ICML 2025
现有世界模型严重依赖带有动作标注的大量数据，面对新环境时适应成本极高。AdaWorld 提出以自监督方式从无标注视频中提取 latent action，并以此为条件预训练一个自回归世界模型，使模型能够在极少交互数据下快速迁移至新环境，同时支持无需额外训练的跨场景 action transfer 与语义连续的 action composition。Existing world models rely heavily on large amounts of action-labeled data, which makes adapting to a new environment extremely costly. AdaWorld proposes to extract latent actions from unlabeled videos in a self-supervised manner, and to pretrain an autoregressive world model conditioned on them, so that the model can transfer quickly to new environments with very little interaction data, while also supporting training-free cross-scene action transfer and semantically continuous action composition.
ICML 2025Video Prediction · PlanningDiffusion-based World Model📄 arXiv:2503.18938PDF
world modellatent actionself-supervised learningvideo predictionvisual planning自适应世界模型adaptive world modeldiffusion model迁移学习transfer learning
01动机Motivation
世界模型（world model）旨在学习由动作控制的未来帧预测，是构建智能 agent 的核心组件。然而，绝大多数现有方法严重依赖大量动作标注数据和高昂的训练成本，难以通过有限交互适应异构动作空间的新环境，极大限制了其跨域可用性。A world model aims to learn action-controlled prediction of future frames, and is a core component for building intelligent agents. However, the vast majority of existing methods rely heavily on substantial action-labeled data and costly training, making it hard to adapt to new environments with heterogeneous action spaces through limited interactions, which severely limits their cross-domain usability.
"most existing world models rely heavily on substantial action-labeled data and costly training, making it challenging to adapt to novel environments with heterogeneous actions through limited interactions."
Figure 1：不同世界模型学习范式对比。传统方法需要昂贵的动作标注才能获得可控性；AdaWorld 引入 latent action 作为统一条件进行动作感知预训练，无需训练即可跨场景迁移动作，也可以高效 fine-tune 为专用世界模型，并取得显著优于 action-agnostic baseline 的规划结果。Figure 1: Comparison of world model learning paradigms. Conventional methods require expensive action annotations to obtain controllability; AdaWorld introduces latent actions as a unified condition for action-aware pretraining, transfers actions across scenes without any training, and can also be fine-tuned efficiently into a specialized world model, achieving planning results clearly better than the action-agnostic baseline.
AdaWorld 的核心思路是：在预训练阶段就将动作信息注入世界模型——通过 latent action autoencoder 从无标注视频中自监督提取 latent action，再以这些 latent action 为条件预训练一个基于扩散的自回归世界模型。推理时可无训练迁移、少样本 fine-tune 或 MLP 映射等多种方式适应新环境。The core idea of AdaWorld is to inject action information into the world model already at the pretraining stage—a latent action autoencoder extracts latent actions from unlabeled videos in a self-supervised way, and a diffusion-based autoregressive world model is then pretrained conditioned on these latent actions. At inference time, a new environment can be adapted to in several ways: training-free transfer, few-shot fine-tuning, or an MLP mapping.
Latent Action Autoencoder
Figure 2：Latent action autoencoder。编码器基于时空 Transformer，以信息瓶颈（information bottleneck）设计从连续帧中提取最关键的帧间变化，压缩为连续 latent action；解码器则依据 latent action 和当前帧重建下一帧。该压缩机制迫使 latent action 只保留"相对前帧最关键的变化"，从而形成与场景上下文无关的表示。训练目标采用 β-VAE 形式：𝓛 = 𝓛_pred + β·KL，在表达力与解耦之间取得平衡。Figure 2: Latent action autoencoder. The encoder is built on a spatio-temporal Transformer and, with an information bottleneck design, extracts the most essential inter-frame changes from consecutive frames and compresses them into a continuous latent action; the decoder reconstructs the next frame from the latent action and the current frame. This compression forces the latent action to keep only "the most essential change relative to the previous frame", yielding a representation independent of scene context. The training objective takes a β-VAE form: 𝓛 = 𝓛_pred + β·KL, balancing expressiveness against disentanglement.
Figure 3：动作感知预训练流程。以 Stable Video Diffusion 为基础，用 latent action encoder 从无标注视频中提取 latent action，再以其为条件驱动自回归视频预测模型的预训练。推理时支持多步自回归展开（autoregressive rollout）。Figure 3: The action-aware pretraining pipeline. Built on Stable Video Diffusion, a latent action encoder extracts latent actions from unlabeled videos, which in turn condition the pretraining of the autoregressive video prediction model. At inference, multi-step autoregressive rollout is supported.
跨环境适应机制Cross-Environment Adaptation Mechanisms
Action Transfer（零样本迁移）Action Transfer (Zero-Shot)
直接将源视频中提取的 latent action 用于控制目标场景的世界模型，无需任何训练。由于 latent action 与上下文解耦，同一动作可无缝迁移到不同视觉场景。The latent actions extracted from a source video are directly used to control the world model of a target scene, without any training. Because latent actions are disentangled from context, the same action transfers seamlessly to different visual scenes.
对于离散动作空间：以每个动作对应的多条轨迹的 latent action 平均值初始化控制接口，再少量 fine-tune；对于连续动作空间：用 MLP 将真实动作映射为 latent action 空间后 fine-tune，均仅需极少标注样本与步数。For a discrete action space, the control interface is initialized with the average of the latent actions of multiple trajectories corresponding to each action, followed by light fine-tuning; for a continuous action space, an MLP maps ground-truth actions into the latent action space before fine-tuning. Both need only very few labeled samples and steps.
Action Composition（动作组合）Action Composition
Figure 4：动作迁移与组合。左：AdaWorld 可准确识别演示动作并将其迁移到不同场景，而 baseline 方法失败。右：通过在 latent space 中对两个 latent action 做平均，可组合出语义上融合二者功能的新动作，表明 latent action 空间在动作语义上是连续的（"our latent action space is semantically continuous in the meanings of actions"）。Figure 4: Action transfer and composition. Left: AdaWorld accurately recognizes the demonstrated action and transfers it to different scenes, whereas the baseline method fails. Right: averaging two latent actions in the latent space composes a new action that semantically merges the functions of both, indicating that the latent action space is continuous in the meanings of actions ("our latent action space is semantically continuous in the meanings of actions").
03实验Experiments
实验跨多个领域展开：Action Transfer 评估（LIBERO、Something-Something v2），世界模型适应质量（Habitat、Minecraft、DMLab、nuScenes，每类环境 800 步 fine-tune），以及 visual planning（4 个 Procgen 游戏 + VP² 机器人任务）。Baseline 包括：action-agnostic 预训练、flow conditioning、discrete conditioning，以及 Q-learning、ground truth simulator。The experiments span several domains: action transfer evaluation (LIBERO, Something-Something v2), world model adaptation quality (Habitat, Minecraft, DMLab, nuScenes, with 800 fine-tuning steps per environment), and visual planning (4 Procgen games + VP² robot tasks). Baselines include action-agnostic pretraining, flow conditioning, discrete conditioning, as well as Q-learning and a ground truth simulator.
Action Transfer（动作迁移）
方法Method
LIBERO FVD ↓
LIBERO Human ↑
SSv2 FVD ↓
SSv2 Human ↑
Act-agnostic
1545.2
0%
847.2
1%
Flow cond.
1409.5
2%
702.8
10.5%
Discrete cond.
1504.5
3.5%
726.8
21.5%
AdaWorld
767.0
70.5%
473.4
61.5%
AdaWorld 在 FVD 和人类评估成功率上均大幅领先所有 baseline，展现出极强的跨场景动作迁移能力。AdaWorld leads all baselines by a wide margin on both FVD and human-evaluated success rate, demonstrating very strong cross-scene action transfer ability.
世界模型适应质量（800 步 fine-tune）World Model Adaptation Quality (800 Fine-tuning Steps)
方法Method
Habitat PSNR ↑
Habitat LPIPS ↓
Minecraft PSNR ↑
nuScenes PSNR ↑
Act-agnostic
20.34
0.450
19.44
20.86
Flow cond.
22.49
0.373
20.71
20.94
Discrete cond.
23.31
0.342
21.33
21.28
AdaWorld
23.58
0.327
21.59
21.60
适应速度（PSNR 曲线）Adaptation Speed (PSNR Curves)
Figure 6：世界模型适应 PSNR 曲线。在样本数和训练步数均极少的情况下，AdaWorld 比传统预训练方法更快收敛至更高 PSNR，体现出"highly adaptable"的设计目标。Figure 6: PSNR curves of world model adaptation. With extremely few samples and training steps, AdaWorld converges faster and to a higher PSNR than conventional pretraining methods, reflecting the "highly adaptable" design goal.
Visual Planning（视觉规划）Visual Planning
方法Method
Heist
Jumper
Maze
CaveFlyer
平均Avg.
Random
19.33%
22.00%
41.33%
22.00%
26.17%
Act-agnostic
20.67%
20.67%
39.33%
23.33%
26.00%
AdaWorld w/o FT
38.67%
68.00%
41.33%
31.33%
44.83%
AdaWorld w/ FT
66.67%
58.67%
68.00%
33.33%
56.67%
即使不经 fine-tune，AdaWorld 平均成功率（44.83%）也已显著超过 Q-learning（27.17%）和 action-agnostic 方法（26.00%）。机器人任务聚合归一化成功率：AdaWorld 21.54，action-agnostic baseline 仅 5.03。Even without fine-tuning, the average success rate of AdaWorld (44.83%) already clearly exceeds Q-learning (27.17%) and the action-agnostic method (26.00%). On the aggregated normalized success rate of robot tasks, AdaWorld reaches 21.54, while the action-agnostic baseline attains only 5.03.
消融（Ablations）Ablations
数据多样性消融（Table 5）表明，增加训练数据多样性能显著提升 latent action 对新领域的泛化能力。架构泛化性消融（Table 6）表明，将动作感知预训练应用于 iVideoGPT 同样大幅提升其适应性，验证了该方法的通用性。UMAP 可视化（Figure 7）显示，较小的 β 值增强表达力但牺牲了与上下文的解耦程度。The data diversity ablation (Table 5) shows that increasing the diversity of training data significantly improves the generalization of latent actions to new domains. The architecture generality ablation (Table 6) shows that applying action-aware pretraining to iVideoGPT likewise greatly improves its adaptability, confirming the generality of the method. The UMAP visualization (Figure 7) shows that a smaller β enhances expressiveness but sacrifices disentanglement from context.
04局限性Limitations
Note：以下局限性均为作者在论文中明确陈述（stated by the authors）。Note: All limitations below are explicitly stated by the authors in the paper.
推理速度不支持实时Inference speed is not real-time
"it does not operate at real-time frequency." 未来可通过蒸馏（distillation）与加速采样技术改善推理速度。"it does not operate at real-time frequency." Inference speed may be improved in the future through distillation and accelerated sampling techniques.
长序列展开时难以生成新内容Difficulty generating novel content in long rollouts
"AdaWorld struggles to create novel content when the rollout exceeds the initial scene." 作者认为该问题可通过扩大模型规模与训练数据解决。"AdaWorld struggles to create novel content when the rollout exceeds the initial scene." The authors argue this can be resolved by scaling up the model and the training data.
超长期展开（extremely long-term rollouts）能力不足Insufficient capability for extremely long-term rollouts
"our model falls short in achieving extremely long-term rollouts, and we will explore potential solutions in future work."
存在若干典型失败案例A number of typical failure cases remain
作者在附录中列举了部分主要失败案例（"We also append some primary failure cases"），但论文主体未作详细讨论。The authors list some primary failure cases in the appendix ("We also append some primary failure cases"), but the main body of the paper does not discuss them in detail.
