无动作标注，直接从真实世界视频中学习潜在动作空间No action labels: learning a latent action space directly from in-the-wild videos
Quentin Garrido · Tushar Nagarajan · Basile Terver · Nicolas Ballas · Yann LeCun · Michael Rabbat | FAIR at Meta, Inria, NYU
世界模型需要动作标注才能预测未来，但大规模视频数据几乎没有标注。本文研究如何在多样化的真实世界（in-the-wild）视频上训练潜在动作世界模型（LAM），无需任何动作标签。研究发现，连续且受约束的潜在动作能够有效捕捉复杂动作，而常用的向量量化方法在此场景下表现不佳；同时，通过训练轻量级控制器将已知动作映射到潜在动作，LAM 可用于机器人操作和导航的视觉规划任务。World models need action annotations in order to predict the future, yet large-scale video data is almost entirely unlabeled. This paper studies how to train latent action world models (LAM) on diverse in-the-wild videos without any action labels. It finds that continuous and constrained latent actions capture complex actions effectively, whereas the commonly used vector quantization approach performs poorly in this setting; in addition, by training a lightweight controller that maps known actions to latent actions, the LAM can be used for visual planning in robot manipulation and navigation.
智能体要在现实世界中推理与规划，必须能够预测动作的后果。现有世界模型大多依赖动作标注——而大规模视频数据几乎没有标签。这催生了潜在动作模型（Latent Action Model, LAM）的研究方向：从视频本身学习动作空间，无需任何显式动作标注。To reason and plan in the real world, an agent must be able to predict the consequences of its actions. Most existing world models rely on action annotations, yet large-scale video data is almost entirely unlabeled. This has given rise to the research direction of Latent Action Models (LAM): learning an action space from video itself, without any explicit action annotation.
"To learn a truly general and transferable latent action world model, we argue that we must go beyond these targeted data sources. Sources of natural in-the-wild videos such as HowTo100M or YoutubeTemporal-1B provide a much richer and general learning environment than usually studied."
以往 LAM 工作大多局限于视频游戏、桌面机械臂操作等窄领域，泛化性受限。真实世界视频（in-the-wild）涵盖更丰富的动作分布——摄像机运动、人员进入画面、物体变化等——提供了学习通用动作空间的机会，同时也带来了新挑战：视频多样性导致的环境噪声、不同视频间缺乏统一具身形式（embodiment）。Prior LAM work has largely been confined to narrow domains such as video games and tabletop robot-arm manipulation, which limits generalization. In-the-wild videos cover a much richer action distribution — camera motion, people entering the frame, objects changing — offering an opportunity to learn a general action space, but they also raise new challenges: environmental noise caused by the diversity of videos, and the lack of a common embodiment across different videos.
Figure 1 — Action diversity. 传统导航或操作数据仅涵盖摄像机/手部等通用动作；真实世界视频（in-the-wild）则包含更广泛的动作分布，例如物体进入画面、人物跳舞等。LAM 需要能够建模这种丰富性。Figure 1 — Action diversity. Conventional navigation or manipulation data covers only generic actions such as camera or hand motion; in-the-wild videos instead contain a far broader action distribution, for example objects entering the frame or people dancing. A LAM must be able to model such richness.
×2+scene cut 后预测误差倍增，证明 latent action 未编码完整下一帧prediction error more than doubles after a scene cut, showing that the latent action does not encode the complete next frame
30kYoutubeTemporal-1B 上的训练迭代次数（默认设置）training iterations on YoutubeTemporal-1B (default setting)
≈V-JEPA 2-AC规划性能与有标注基线相当planning performance on par with the action-labeled baseline
02方法Method
本文采用标准的逆动力学模型（IDM）+ 世界模型（前向模型）联合训练框架，并系统研究如何在真实世界视频上约束潜在动作的信息量，以避免编码冗余噪声。The paper adopts the standard framework of jointly training an inverse dynamics model (IDM) and a world model (forward model), and systematically studies how to constrain the information content of latent actions on in-the-wild videos so as to avoid encoding redundant noise.
整体架构Overall architecture
给定视频帧序列 s0:T，首先使用冻结的 V-JEPA 2-L 编码器将每帧映射为抽象表示。IDM gφ 以过去帧与未来帧为输入，输出潜在动作 zt = gφ(st, st+1)；世界模型 pψ（实现为 ViT-L + RoPE 位置编码，以 AdaLN-zero 接受 zt 条件）预测下一帧 st+1。两者联合以 L1 预测损失优化：Lt = ‖st+1 − pψ(s0:t, zt)‖1。训练使用 teacher forcing。Given a sequence of video frames s0:T, a frozen V-JEPA 2-L encoder first maps every frame into an abstract representation. The IDM gφ takes past and future frames as input and outputs a latent action zt = gφ(st, st+1); the world model pψ (implemented as a ViT-L with RoPE positional encoding, taking zt as a condition through AdaLN-zero) predicts the next frame st+1. The two are optimized jointly with an L1 prediction loss: Lt = ‖st+1 − pψ(s0:t, zt)‖1. Training uses teacher forcing.
Figure 2 — Latent action world model. 世界模型被赋予以潜在变量表示的动作。IDM 从过去帧和未来帧推断潜在动作，再经由正则化约束其信息量（稀疏化、噪声添加或量化）。最终潜在动作通过 AdaLN-zero 条件化至 ViT-L 世界模型。Figure 2 — Latent action world model. The world model is given actions represented as latent variables. The IDM infers the latent action from past and future frames, and regularization then constrains its information content (sparsity, added noise, or quantization). The resulting latent action conditions the ViT-L world model through AdaLN-zero.
信息量约束：三种正则化策略Constraining information content: three regularization strategies
潜在动作的核心挑战是防止 IDM 将未来帧完整编码进 zt（"cheating"）。本文研究三种策略，各有优劣：The core challenge for latent actions is to prevent the IDM from encoding the entire future frame into zt ("cheating"). The paper studies three strategies, each with its own strengths and weaknesses:
稀疏化（Sparsity）Sparsity
对潜在动作施加 L1 正则项，辅以 VCM（Variance-Covariance-Mean）正则化防止模式坍缩。参数 λl1 控制约束强度。优点：可灵活调节容量；缺点：实现相对复杂。An L1 regularization term is applied to the latent action, complemented by VCM (Variance-Covariance-Mean) regularization to prevent mode collapse. The parameter λl1 controls the strength of the constraint. Advantage: capacity can be tuned flexibly; drawback: the implementation is relatively involved.
类似 VAE 的 KL 散度正则：L(zt) = −β DKL(q(zt|st, st+1) ‖ N(0,1))。通过目标标准差添加噪声、目标均值降低 zt 范数。β 参数调节信息容量。A VAE-like KL-divergence regularizer: L(zt) = −β DKL(q(zt|st, st+1) ‖ N(0,1)). Noise is added through a target standard deviation, while a target mean reduces the norm of zt. The β parameter tunes the information capacity.
使用向量量化（VQ）将连续向量离散化为 codebook 编码，与 UniVLA 相同的量化方案。这是 Genie、LAPA、UniVLA 等主流方法的选择，本文将其作为对照基线。Vector quantization (VQ) discretizes the continuous vector into codebook entries, using the same quantization scheme as UniVLA. This is the choice made by mainstream methods such as Genie, LAPA and UniVLA, and the paper adopts it as the comparison baseline.
控制器（Controller）Controller
为将 LAM 用于规划，训练一个轻量级控制器，将已知动作映射到潜在动作。仅用动作时采用 MLP；结合过去表示时采用基于 cross-attention 的 adapter，以处理潜在动作与相机位置相关的问题。In order to use the LAM for planning, a lightweight controller is trained to map known actions into latent actions. When only actions are used it is an MLP; when past representations are also used it is a cross-attention based adapter, which addresses the fact that latent actions are tied to camera position.
03实验Experiments
所有模型在 YoutubeTemporal-1B 上训练（16帧片段，4fps，batch size 1024，30k 迭代），评测涵盖：IDM 预测误差（LPIPS）、未来帧泄漏（scene cut）、动作迁移（cycle consistency）、具身特性分析，以及下游规划性能（DROID 机械臂操作 / RECON 导航）。All models are trained on YoutubeTemporal-1B (16-frame clips, 4fps, batch size 1024, 30k iterations). The evaluation covers: IDM prediction error (LPIPS), future-frame leakage (scene cut), action transfer (cycle consistency), analysis of embodiment properties, and downstream planning performance (DROID robot-arm manipulation / RECON navigation).
正则化方式比较Comparison of regularization schemes
Figure 3 — Sample predictions using the IDM. 稀疏（Sparse）和噪声（Noisy）潜在动作均能捕捉"一个人走进场景"这一复杂动作；而离散（Discrete/VQ）方式仅能呈现模糊的移动团块，无法精确还原细节。这直接说明向量量化在真实世界视频的复杂性面前能力受限。Figure 3 — Sample predictions using the IDM. Sparse and Noisy latent actions both capture the complex action of a person walking into the scene; the discrete (Discrete/VQ) variant produces only a blurry moving blob and cannot recover the details precisely. This directly shows that vector quantization is limited when confronted with the complexity of in-the-wild video.
预测误差与信息容量Prediction error and information capacity
稀疏和噪声潜在动作通过调节超参数（λl1 或 β），可在无约束连续动作（最低误差）与纯确定性世界模型（最高误差）之间灵活权衡，形成连续的容量谱系。而向量量化方法难以提升容量，始终靠近确定性基线，"the vector quantization based approach struggles to scale its capacity and remains very close to the deterministic baseline."By tuning their hyperparameters (λl1 or β), sparse and noisy latent actions can trade off flexibly between unconstrained continuous actions (lowest error) and a purely deterministic world model (highest error), forming a continuous spectrum of capacity. Vector quantization, by contrast, has difficulty increasing its capacity and always stays close to the deterministic baseline, "the vector quantization based approach struggles to scale its capacity and remains very close to the deterministic baseline."
未来帧泄漏（Future Leakage）Future leakage
正则化方式Regularization
容量Capacity
无场景切换（LPIPS）No scene cut (LPIPS)
有场景切换（LPIPS）With scene cut (LPIPS)
倍率Ratio
Sparse
Low
0.28
0.66
×2.3
Sparse
High
0.20
0.50
×2.4
Noisy
Low
0.33
0.69
×2.1
Noisy
High
0.21
0.54
×2.5
Discrete
Low
0.34
0.69
×2.0
Discrete
High
0.29
0.68
×2.3
所有模型在场景切换时预测误差均翻倍以上，表明没有模型通过编码整个未来帧来"作弊"。数据集本身的复杂性使此类捷径难以习得。For every model the prediction error more than doubles at a scene cut, indicating that no model "cheats" by encoding the entire future frame. The complexity of the dataset itself makes such a shortcut hard to learn.
动作迁移（Cycle Consistency）Action transfer (Cycle Consistency)
在 Kinetics（人类活动视频）和 RECON（导航）上，将视频 A 的潜在动作迁移至视频 B 再反推，cycle consistency 误差增幅极小（约 ×1.03 ~ ×1.34），证明潜在动作具有跨视频迁移性，且此迁移并非来自未来帧泄漏。On Kinetics (human activity videos) and RECON (navigation), transferring the latent action of video A to video B and then inverting it yields only a marginal increase in cycle consistency error (about ×1.03 ~ ×1.34), showing that latent actions transfer across videos and that this transfer does not come from future-frame leakage.
定性分析（Figure 7/8）：可将"一人向左行走"的动作迁移至"飞行球"，球随即向左运动；还能仅对画面中空间位置最近的人施加动作，体现出空间局部性——"due to a lack of common embodiment in natural videos, the model learns generic actions that are applied relative to the camera."Qualitative analysis (Figure 7/8): the action of a person walking to the left can be transferred to a flying ball, which then moves to the left; the action can also be applied only to the person spatially closest in the frame, revealing spatial locality — "due to a lack of common embodiment in natural videos, the model learns generic actions that are applied relative to the camera."
下游规划性能Downstream planning performance
通过训练控制器，LAM 可用于 DROID（Franka Emika Panda 机械臂）和 RECON（户外导航）上的视觉规划任务，采用 CEM 在 H=3 步规划。关键结论：With a trained controller, the LAM can be used for visual planning on DROID (Franka Emika Panda arm) and RECON (outdoor navigation), planning with CEM over a horizon of H=3 steps. Key conclusions:
DROID 操作：规划性能与 V-JEPA 2-AC（使用已知动作标注训练的基线）相当。DROID manipulation: planning performance is on par with V-JEPA 2-AC, a baseline trained with known action annotations.
RECON 导航：在 Relative Pose Error (RPE) 指标上超越策略基线 NoMaD，但仍低于专为导航设计的 NWM。RECON navigation: on the Relative Pose Error (RPE) metric it surpasses the policy baseline NoMaD, but remains below NWM, which is designed specifically for navigation.
整体最优：噪声潜在动作（Noisy）在容量适中时取得最好规划性能，过约束或欠约束均次优。Best overall: noisy latent actions (Noisy) obtain the best planning performance at a moderate capacity, while both over-constrained and under-constrained settings are worse.
规模化实验Scaling experiments
Figure 12 — Scaling trends. 在模型规模（Large→Giant）、训练时长（30k→120k 步）和数据量三个维度上扩展均能提升 IDM 自然视频预测质量（上行）；规划性能方面，训练时长的提升最为显著，模型规模对噪声潜在动作有帮助，而数据量的影响不显著——"these results would suggest that while scaling can improve the quality of a latent action world model…this may not always be visible in downstream tasks that mainly evaluate simple actions."Figure 12 — Scaling trends. Scaling along the three axes of model size (Large→Giant), training length (30k→120k steps) and data volume all improve the IDM's prediction quality on natural video (top row); for planning performance, longer training brings the most pronounced gain, model size helps noisy latent actions, and the amount of data has no significant effect — "these results would suggest that while scaling can improve the quality of a latent action world model…this may not always be visible in downstream tasks that mainly evaluate simple actions."
04局限性Limitations
Note: 以下局限性均由作者在论文"Limitations and future work"节明确阐述。Note: All limitations below are stated explicitly by the authors in the paper's "Limitations and future work" section.
静态信息约束系数（Variable latent information content）Static information-constraint coefficients (Variable latent information content)
本文中潜在动作的信息约束系数（如 λl1、β）在训练期间固定不变。然而不同视频中动作复杂程度差异很大，有些片段甚至是确定性的。未来工作应根据视频复杂度动态调整约束，实现更好校准的潜在动作，尽管这可能增加潜在动作空间的复杂性。In this work the information-constraint coefficients of the latent action (such as λl1 and β) are held fixed during training. Yet the complexity of the action varies greatly across videos, and some clips are even deterministic. Future work should adapt the constraint dynamically to the complexity of the video, so as to obtain better calibrated latent actions, although this may increase the complexity of the latent action space.
潜在动作空间中的采样与规划（Sampling and planning in latent action space）Sampling and planning in latent action space
本文主要研究将已知动作映射到潜在动作（控制器方案），而非直接在潜在动作空间内采样规划。作者承认"most of the work is ahead for high dimensional structured latent actions"，直接利用潜在动作进行规划（如 Rybkin et al., 2019 的方法）仍是开放问题。The paper mainly studies mapping known actions into latent actions (the controller scheme) rather than sampling and planning directly within the latent action space. The authors concede that "most of the work is ahead for high dimensional structured latent actions"; planning directly with latent actions (as in Rybkin et al., 2019) remains an open problem.
冻结表示限制单阶段训练（Shaping representations with single stage training）Frozen representations preclude single-stage training (Shaping representations with single stage training)
世界模型训练在冻结的 V-JEPA 2 表示之上进行，而该表示空间并非针对预测设计。这可能制约逆动力学模型训练质量及预测精度。若能将潜在动作引入 V-JEPA 2 预训练（单阶段编码器/世界模型联合训练），可能带来显著提升——作者视之为"an exciting direction for future work."The world model is trained on top of frozen V-JEPA 2 representations, and that representation space was not designed for prediction. This may constrain both the training quality of the inverse dynamics model and the accuracy of prediction. Introducing latent actions into V-JEPA 2 pretraining (single-stage joint training of encoder and world model) could bring a substantial gain, which the authors regard as "an exciting direction for future work."
规划性能与视频生成质量不相关（inferred）Planning performance is uncorrelated with video generation quality (inferred)
实验发现，unrolling 视觉质量（LPIPS）与规划性能（Δxyz / RPE）之间的相关性较弱——规划最好的模型往往并非视频生成最清晰的模型。这一"common challenge in the world model literature"意味着评估指标的选择对 LAM 研究结论至关重要。Experiments show that the correlation between the visual quality of unrolling (LPIPS) and planning performance (Δxyz / RPE) is weak: the model that plans best is often not the one that generates the clearest video. This "common challenge in the world model literature" means that the choice of evaluation metric is crucial for conclusions drawn about LAMs.
