从纯视频中恢复隐变量动作，无需任何动作标签即可预训练策略Recovering latent actions from pure video: pre-training policies without any action labels
Dominik Schmidt & Minqi Jiang · UCL, Meta AI
LAPO（Latent Action Policies）提出了一种仅靠视频观测序列学习行为策略的框架。通过联合训练逆向动态模型（IDM）与前向动态模型（FDM），并以 Vector Quantization（VQ）作为信息瓶颈，LAPO 可以从无标注视频中自动发现结构化的隐动作空间。学到的 latent policy 经过少量标注数据的解码器微调，便可在 Procgen 基准的 16 个游戏上快速达到甚至超越专家水平。LAPO (Latent Action Policies) proposes a framework that learns behaviour policies from sequences of video observations alone. By jointly training an inverse dynamics model (IDM) and a forward dynamics model (FDM), with Vector Quantization (VQ) serving as an information bottleneck, LAPO automatically discovers a structured latent action space from unlabelled video. Once the learned latent policy is decoded by a decoder fine-tuned on a small amount of labelled data, it rapidly matches or even surpasses expert performance on the 16 games of the Procgen benchmark.
大语言模型通过在海量网络文本上预训练，获得了强大的通用能力。然而在强化学习领域，互联网上存在大量无标注的行为视频（游戏回放、操作演示等），却因为缺少动作标签而无法直接用于策略学习。如何从纯视频中提取行为信息，是通往具身 AI 大规模预训练的关键障碍。Large language models acquire strong general-purpose capabilities by pre-training on massive amounts of web text. In reinforcement learning, however, the vast quantities of unlabelled behaviour video available on the internet (game replays, task demonstrations and the like) cannot be used directly for policy learning because action labels are missing. Extracting behavioural information from pure video is therefore the key obstacle on the road to large-scale pre-training for embodied AI.
"a first step towards pre-training powerful, generalist policies and world models on the vast amounts of videos readily available on the web"
图 1：在 Miner 环境中，LAPO 学到的隐动作空间的 UMAP 投影，以及 FDM 对每簇 latent action 生成的下一帧预测。每个聚类对应玩家真实按下的不同按键方向，表明模型在无任何动作标签的情况下自动发现了与真实动作语义对齐的结构化表征。（图源：原论文 Figure 1）Figure 1: UMAP projection of the latent action space learned by LAPO in the Miner environment, together with the next-frame predictions the FDM generates for each cluster of latent actions. Each cluster corresponds to a different key direction actually pressed by the player, showing that the model automatically discovers structured representations aligned with true action semantics without any action label. (Source: Figure 1 of the paper)
16Procgen 游戏数Procgen games
4M帧内达到专家水平frames to reach expert level
<256离线解码所需标注转换数labelled transitions needed for offline decoding
44%同等帧数下 PPO from scratch 仅达专家的of expert level is all PPO from scratch reaches at the same frame count
02方法Method
LAPO 的核心是两个协同训练的动态模型：逆向动态模型（IDM）根据相邻两帧预测隐动作，前向动态模型（FDM）则仅凭当前帧和隐动作预测下一帧。两者共同以最小化下一状态预测误差为目标，而 Vector Quantization（VQ）瓶颈迫使 IDM 只编码状态差异而非完整的未来信息，从而学到真正具有动作语义的压缩表征。At the core of LAPO are two jointly trained dynamics models: the inverse dynamics model (IDM) predicts a latent action from two adjacent frames, while the forward dynamics model (FDM) predicts the next frame from the current frame and that latent action alone. Both are trained to minimise the next-state prediction error, and the Vector Quantization (VQ) bottleneck forces the IDM to encode only the state difference rather than complete information about the future, so that it learns a compressed representation that genuinely carries action semantics.
图 2：LAPO 架构。IDM 同时观测 ot 和 ot+1，输出经 VQ 量化的 latent action；FDM 仅观测 ot 和该 latent action，预测 ôt+1。训练信号为 ||ôt+1 − ot+1||²。（图源：原论文 Figure 2）Figure 2: the LAPO architecture. The IDM observes both ot and ot+1 and outputs a VQ-quantised latent action; the FDM sees only ot and that latent action and predicts ôt+1. The training signal is ||ôt+1 − ot+1||². (Source: Figure 2 of the paper)
逆向动态模型（IDM）与信息瓶颈The inverse dynamics model (IDM) and the information bottleneck
IDM 以 (ot, ot+1) 为输入，输出连续的 latent action 向量，再经过 VQ 离散化。VQ 作为信息瓶颈的关键作用在于：若无约束，IDM 可以直接"抄写"未来帧信息绕过真正的动作推断；VQ 压缩迫使模型只保留状态转变中最关键的那部分信息，即The IDM takes (ot, ot+1) as input and outputs a continuous latent action vector, which is then discretised by VQ. The crucial role of VQ as an information bottleneck is the following: without any constraint the IDM could simply copy information about the future frame and bypass genuine action inference; VQ compression forces the model to retain only the most essential part of the state transition, that is,"the IDM learns to encode only the difference between ot+1 and ot…rather than full information about ot+1."实验表明，去掉 VQ 后（no-VQ ablation）虽然 FDM loss 更低，但下游策略性能显著下降，印证了 VQ 的不可或缺性。Experiments show that removing VQ (the no-VQ ablation) yields a lower FDM loss but markedly degrades downstream policy performance, confirming that VQ is indispensable.
前向动态模型（FDM）与 Latent PolicyThe forward dynamics model (FDM) and the latent policy
FDM 仅接收当前观测 ot 与 latent action，预测下一帧。训练完成后，IDM 的编码器部分即构成 latent policy——给定观测，输出 latent action。该 latent policy 可通过两种途径解码为真实可执行策略：The FDM receives only the current observation ot and the latent action, and predicts the next frame. After training, the encoder part of the IDM constitutes the latent policy—given an observation, it outputs a latent action. This latent policy can be decoded into a genuinely executable policy in either of two ways:
离线解码（Offline Decoding）：训练一个小型解码器（decoder），在少量带标注的 (latent action → 真实动作) 对上做监督学习，实现标注效率极高的策略迁移。Offline decoding: train a small decoder by supervised learning on a small number of labelled (latent action → true action) pairs, achieving extremely label-efficient policy transfer.
在线解码（Online Decoding / RL Fine-tuning）：以 latent policy 初始化 RL 智能体，在环境中直接做 RL 微调（以 latent action 为动作空间），完全不需要任何标注数据。Online decoding (RL fine-tuning): initialise an RL agent with the latent policy and fine-tune it directly in the environment with RL (using the latent actions as the action space), requiring no labelled data at all.
图 5：16 个 Procgen 游戏的 latent action 空间 UMAP 投影。每个点代表 IDM 对一次状态转变输出的连续（VQ 前）latent action。可见大多数游戏中已自然形成与真实离散动作语义对齐的聚类结构。（图源：原论文 Figure 5）Figure 5: UMAP projections of the latent action spaces of the 16 Procgen games. Each point is the continuous (pre-VQ) latent action that the IDM outputs for one state transition. In most games, clusters aligned with the true discrete action semantics have already emerged naturally. (Source: Figure 5 of the paper)
03实验Experiments
实验在 Procgen 基准的全部 16 个游戏上进行。无标注视频数据集包含约 8M 帧，由用 PPO 训练 50M 步后的专家策略采集。核心对比基线为 PPO from scratch 和 ILPO（另一种无动作标签方法）。Experiments are run on all 16 games of the Procgen benchmark. The unlabelled video dataset contains about 8M frames, collected by an expert policy trained with PPO for 50M steps. The main baselines for comparison are PPO from scratch and ILPO (another action-free method).
图 3（左）：16 个 Procgen 游戏训练过程中的平均回报（3 seeds 平均）。（右）：相对专家策略的归一化平均测试回报。LAPO 在 4M 帧内达到专家水平，而 PPO from scratch 在同等帧数下仅达专家水平的 44%。（图源：原论文 Figure 3）Figure 3 (left): mean return over the course of training on the 16 Procgen games (averaged over 3 seeds). (right): normalised mean test return relative to the expert policy. LAPO reaches expert level within 4M frames, whereas PPO from scratch attains only 44% of expert level at the same frame count. (Source: Figure 3 of the paper)
设置Setting
基线Baseline
LAPO
备注Notes
在线解码 @4M 帧online decoding @4M frames
PPO scratch: 44% expert
≈100% expert
16 games 均值mean over 16 games
超越专家的游戏数games surpassing the expert
—
9 / 16
在 4M 帧内within 4M frames
ILPO 对比ILPO comparison
ILPO 在多关卡任务崩溃ILPO collapses on multi-level tasks
LAPO 保持稳定LAPO stays stable
多关卡环境multi-level environments
离线解码（少量标注）Offline decoding (few labels)
图 4：离线解码性能随标注转换数量的变化曲线（3 seeds 均值 ± 标准差）。用少于 256 条标注转换训练的解码器，其性能已超过 PPO from scratch 训练 4M 步的水平。（图源：原论文 Figure 4）Figure 4: offline decoding performance as a function of the number of labelled transitions (mean ± standard deviation over 3 seeds). A decoder trained on fewer than 256 labelled transitions already exceeds the level reached by PPO from scratch after 4M steps of training. (Source: Figure 4 of the paper)
消融实验（Ablations）Ablations
两个关键消融证实了设计选择的重要性：Two key ablations confirm the importance of the design choices:
RL, no-SL：移除监督学习组件（仅 RL 微调），性能下降，说明离线解码阶段的监督信号有帮助。RL, no-SL: removing the supervised learning component (RL fine-tuning only) degrades performance, showing that the supervision signal of the offline decoding stage helps.
RL, no-SL, no-VQ：同时移除 VQ，尽管 FDM loss 更低，下游策略性能显著劣化——直接验证 VQ 信息瓶颈是学习有意义 latent action 的关键组件。RL, no-SL, no-VQ: removing VQ as well yields a lower FDM loss yet markedly worse downstream policy performance—direct evidence that the VQ information bottleneck is the key component for learning meaningful latent actions.
此外，UMAP 可视化显示，有 VQ 时 latent action 聚类更清晰，与真实动作对齐更好；无 VQ 时表征碎片化，难以解码为有效策略。In addition, the UMAP visualisations show that with VQ the latent actions form clearer clusters that align better with the true actions, whereas without VQ the representation fragments and is hard to decode into an effective policy.
04局限性Limitations
Note:以下局限性均为作者在论文中明确陈述（stated by the authors）。all limitations below are explicitly stated by the authors in the paper.
"Actions that have a delayed effect in observations will be predicted to take place with the same delay."——若某个动作在视觉上的效果延后若干帧才出现，IDM 学到的 latent action 会对应延迟后的可见变化，而非动作本身发生的时刻。作者指出可通过多时间步架构（multi-timestep architectures）加以缓解。—if the visual effect of an action appears only some frames later, the latent action learned by the IDM corresponds to the delayed visible change rather than to the moment the action itself takes place. The authors note that this can be mitigated with multi-timestep architectures.
环境随机性（Stochasticity）Environment stochasticity
"Significant stochasticity can make it difficult for the IDM to compress the useful bits of information among the noise."——高随机性环境中，状态转变中的噪声与真实动作信号混杂，IDM 难以准确分离，导致 latent action 质量下降。更大的数据集可部分缓解此问题。—in highly stochastic environments the noise in state transitions is entangled with the true action signal, the IDM cannot separate them accurately, and the quality of the latent actions degrades. Larger datasets can partly alleviate this problem.
大规模训练的扩展挑战（Scaling Challenges）Scaling challenges for large-scale training
"Training on much larger datasets…would require scaling up the model architecture, which introduces new challenges in balancing the strength of the FDM and the capacity of latent actions representations."——在互联网规模视频数据上训练需要更大的模型，但 FDM 能力与 latent action 容量之间的平衡尚未得到充分研究，是迈向通用视频预训练策略的开放问题。—training on internet-scale video data calls for larger models, but the balance between the strength of the FDM and the capacity of the latent action representation has not yet been studied enough; it remains an open problem on the way to generalist video-pre-trained policies.
依赖部分可观测性结构（Inferred）Dependence on partial-observability structure (inferred)
（从设计推断）实验表明，部分可观测性较低的环境中 latent action 聚类更清晰；对于完全可观测或高维观测的场景，IDM 可能更难学到紧凑的动作表征，泛化能力有待验证。(Inferred from the design) Experiments suggest that latent actions cluster more clearly in environments with less partial observability; in fully observable or high-dimensional-observation settings, the IDM may find it harder to learn compact action representations, and its generalisation remains to be verified.
