Imitating Latent Policies from Observation

01 动机Motivation

传统模仿学习需要同时获取专家的状态观测与动作标注，这在现实场景中往往难以满足—— 视频、人类示范或跨体型代理的行为记录通常只包含状态序列而没有对应动作。现有"仅从观测模仿"方法（如 BCO）虽不依赖专家动作，但需要先在环境中进行大量采样来学习逆向动力学模型，在危险或高成本的真实场景中代价极高。Conventional imitation learning requires expert state observations and action labels at the same time, a requirement that is often impossible to meet in practice: videos, human demonstrations or behavior recordings of agents with a different embodiment usually contain only state sequences and no corresponding actions. Existing “imitation from observation only” methods (such as BCO) do not depend on expert actions, but they must first sample extensively in the environment to learn an inverse dynamics model, which is extremely costly in dangerous or expensive real-world settings.

"We describe a novel approach to imitation learning that infers latent policies directly from state observations … we only need a mechanism for learning policies from observation alone without requiring access to expert actions and with only a few interactions within the environment."

作者的核心洞察是：即便不知道真实动作标签，状态转移之间仍存在可预测的潜在因（latent causes）—— 这些因可以用离散的 latent actions 描述。先离线从专家数据中学习这些 latent actions 和对应的动力学模型，然后只用极少量真实环境交互，将 latent actions 映射到真实动作空间。这就像先观察朋友打游戏，再亲自上手几步确认手柄按键对应关系。The core insight of the authors is that even without ground-truth action labels there are still predictable latent causes (latent causes) behind state transitions, and these causes can be described by discrete latent actions. One first learns these latent actions and the corresponding dynamics model offline from expert data, and then uses only a very small amount of real environment interaction to map the latent actions onto the real action space. It is like watching a friend play a game first and then taking the controller for a few steps to confirm which button does what.

<100Cartpole / Acrobot 达到专家水平所需环境交互步数Environment interaction steps needed to reach expert level on Cartpole / Acrobot

4测试环境（Cartpole, Acrobot, Mountain car, CoinRun）Test environments (Cartpole, Acrobot, Mountain car, CoinRun)

0专家动作标注需求（仅需状态观测序列）Expert action labels required (only state observation sequences are needed)

50k经典控制任务使用的专家状态观测数量Expert state observations used for the classic control tasks

02 方法Method

ILPO 分为两个阶段：Step 1 离线学习 latent policy 和 forward dynamics model； Step 2 用少量真实环境交互学习 action remapping network，将 latent actions 对齐到真实动作。推理时，先选最可能的 latent action，再映射到对应的真实动作。ILPO has two stages: Step 1 learns a latent policy and a forward dynamics model offline; Step 2 learns an action remapping network from a small amount of real environment interaction, aligning the latent actions to the real actions. At inference time the most likely latent action is selected first and then mapped to the corresponding real action.

ILPO 网络架构图 — **图 1：ILPO 双网络架构。** 左侧 Latent Policy Network：包含 embedding E_p、生成器 G（前向动力学模型）和 latent policy π(z|s)。右侧 Action Remapping Network：包含 embedding E_a 和映射网络 π(a|s_t, z)，将学到的 latent action z 对齐到真实环境动作 a。两个 embedding 与各自网络同步训练。**Figure 1: the two-network architecture of ILPO.** Left, the Latent Policy Network: an embedding E_p, a generator G (the forward dynamics model) and the latent policy π(z|s). Right, the Action Remapping Network: an embedding E_a and a mapping network π(a|s_t, z) that aligns the learned latent action z to the real environment action a. Each embedding is trained jointly with its own network.

Step 1：学习 Latent PolicyStep 1: Learning the Latent Policy

给定专家状态序列 {s_t, s_t+1}，ILPO 联合训练两个目标：Given expert state sequences {s_t, s_t+1}, ILPO jointly trains two objectives:

Latent forward dynamics（L_min）： 生成器 G 对每个 latent action z 各自预测下一状态变化量 Δ_t = s_t+1 − s_t，损失只惩罚预测最近的那个模式： L_min = min_z ‖Δ_t − G_θ(E_p(s_t), z)‖₂。这使每个生成器收敛到一类转移簇（transition cluster），而非预测所有转移的均值。Latent forward dynamics (L_min): for every latent action z the generator G separately predicts the next-state change Δ_t = s_t+1 − s_t, and the loss penalizes only the closest prediction: L_min = min_z ‖Δ_t − G_θ(E_p(s_t), z)‖₂. This makes each generator converge to one transition cluster instead of predicting the mean of all transitions.
Latent policy learning（L_exp）： πω(z|s_t) 表示在状态 s_t 下观察到 z 类转移的概率。通过最小化期望下一状态与真实下一状态的差异来训练： ŝ_t+1 = Σ_z πω(z|s_t) G_θ(E_p(s_t), z)，损失 L_exp = ‖s_t+1 − ŝ_t+1‖₂。总损失 L_policy = L_min + L_exp。Latent policy learning (L_exp): πω(z|s_t) denotes the probability of observing a transition of type z in state s_t. It is trained by minimizing the difference between the expected next state and the true next state: ŝ_t+1 = Σ_z πω(z|s_t) G_θ(E_p(s_t), z), with loss L_exp = ‖s_t+1 − ŝ_t+1‖₂. The total loss is L_policy = L_min + L_exp.

Step 2：Action Remapping（动作对齐）Step 2: Action Remapping

通过 ε-greedy 策略在真实环境中采集少量 {s_t, a_t, s_t+1} 三元组。利用已训练好的 G，找到与观测转移最接近的 latent action：A small number of {s_t, a_t, s_t+1} triples is collected in the real environment with an ε-greedy policy. Using the already trained G, the latent action closest to the observed transition is found:

低维状态：z_t = argmin_z ‖s_t+1 − G_θ(E_p(s_t), z)‖₂Low-dimensional states: z_t = argmin_z ‖s_t+1 − G_θ(E_p(s_t), z)‖₂
高维视觉输入：z_t = argmin_z ‖E_p(s_t+1) − E_p(G_θ(E_p(s_t), z))‖₂（在 embedding 空间度量距离）High-dimensional visual inputs: z_t = argmin_z ‖E_p(s_t+1) − E_p(G_θ(E_p(s_t), z))‖₂ (the distance is measured in embedding space)

得到 z_t 后，以真实动作 a_t 为标签，通过 cross-entropy loss 监督训练 action remapping network πξ(a_t|z_t, E_a(s_t))。由于神经网络的泛化能力，相同动作在不同状态下通常产生相似的状态转移，因此只需少量交互就能学会可泛化的 latent-to-real 映射。Once z_t is obtained, the real action a_t is used as the label to supervise the action remapping network πξ(a_t|z_t, E_a(s_t)) with a cross-entropy loss. Thanks to the generalization ability of neural networks, the same action usually produces similar state transitions in different states, so only a few interactions suffice to learn a latent-to-real mapping that generalizes.

推理：组合两步策略Inference: Composing the Two Steps

给定状态 s_t：先选最可能的 latent action z* = argmax_z πω(z|s_t)，再选最可能对应该 latent action 的真实动作 a* = argmax_a πξ(a|z*, s_t)。整个推理过程无需额外环境交互。Given a state s_t: first select the most likely latent action z* = argmax_z πω(z|s_t), then select the real action most likely to correspond to that latent action a* = argmax_a πξ(a|z*, s_t). The whole inference procedure needs no additional environment interaction.

03 实验Experiments

在 4 个环境中评估 ILPO：经典控制任务（Cartpole、Acrobot、Mountain car）和视觉平台游戏 CoinRun（OpenAI）。基线方法：专家策略（Expert）、随机策略（Random）、 Behavioral Cloning（BC，使用真实动作标签）、 BCO（Behavioral Cloning from Observation，不使用专家动作但需要大量环境采样）。实验均使用 OpenAI Baselines 生成专家策略。ILPO is evaluated in 4 environments: the classic control tasks (Cartpole, Acrobot, Mountain car) and the visual platform game CoinRun (OpenAI). The baselines are the expert policy (Expert), a random policy (Random), Behavioral Cloning (BC, which uses ground-truth action labels) and BCO (Behavioral Cloning from Observation, which does not use expert actions but needs extensive environment sampling). All experiments use OpenAI Baselines to generate the expert policies.

经典控制任务结果Results on the Classic Control Tasks

**图 2：经典控制任务模仿学习结果（50 次运行平均）。** 横轴为环境交互步数，纵轴为策略奖励。 Cartpole（左）和 Acrobot（中）中，ILPO 在不足 100 步环境交互后即达到与专家和 BC 相当的水平，且持续优于 BCO。Mountain car（右）中 ILPO 和 BCO 均未能完全追上专家，但 ILPO 明显优于 BCO。奖励设定：Cartpole 每步竿保持直立 +1，Acrobot 和 Mountain car 每步 −1（时间惩罚）。**Figure 2: imitation learning results on the classic control tasks (averaged over 50 runs).** The horizontal axis is the number of environment interaction steps and the vertical axis is the policy reward. On Cartpole (left) and Acrobot (middle) ILPO reaches a level comparable to the expert and to BC after fewer than 100 environment interaction steps, and consistently outperforms BCO. On Mountain car (right) neither ILPO nor BCO fully catches up with the expert, but ILPO is clearly better than BCO. Reward setting: Cartpole gives +1 for every step the pole stays upright, while Acrobot and Mountain car give −1 per step (a time penalty).

CoinRun 视觉任务结果Results on the CoinRun Visual Task

**图 4+5：CoinRun 环境与模仿学习结果。** 上方：CoinRun easy（左）和 hard（中、右）关卡示意，状态输入为 128×128×3 RGB 图像，含 7 个离散动作。 Hard 关卡中间有无法恢复的缺口，右侧还有陷阱。下方曲线：Easy 和 Hard 任务上，ILPO 均显著优于 BCO。尽管高维视觉输入使对齐需要更多步数，好的初始化有时让 ILPO 近乎零样本解决任务。结果为包括失败 seed 在内的所有 seed 平均。**Figures 4+5: the CoinRun environment and imitation learning results.** Top: illustrations of the CoinRun easy (left) and hard (middle, right) levels; the state input is a 128×128×3 RGB image with 7 discrete actions. The hard level has an unrecoverable gap in the middle and a trap on the right. Bottom curves: on both the Easy and Hard tasks ILPO is markedly better than BCO. Although the high-dimensional visual input makes the alignment take more steps, a good initialization sometimes lets ILPO solve the task almost zero-shot. The results are averaged over all seeds, including the failing ones.

下一状态预测示例Next-State Prediction Examples

ILPO 在 CoinRun 中的下一状态预测 — **图 6：ILPO 在 CoinRun easy 任务中的下一状态预测。** 给定当前状态（左），ILPO 生成多个 latent action 对应的预测下一状态（右），高亮框表示由 embedding 距离度量选出的最接近真实下一状态的预测。模型成功学会了"向右移动"和"跳跃"两类主要转移模式。**Figure 6: next-state predictions of ILPO on the CoinRun easy task.** Given the current state (left), ILPO generates the predicted next states corresponding to several latent actions (right); the highlighted box marks the prediction selected by the embedding-distance measure as closest to the true next state. The model successfully learned the two dominant transition modes, “move right” and “jump”.

Latent Action 数量的消融实验Ablation on the Number of Latent Actions

论文测试了 |Z| ≠ |A| 的情形（图 3）：以 |Z| = |A| 为初始猜测效果最好，但智能体在其他大小下仍能学习。 |Z| = 1 在 Cartpole 和 Acrobot 中表现很差，因为所有动作会坍缩到同一个 latent，状态预测无法解耦。这验证了"latent action 数量 = 真实动作数量"是合理的先验假设。The paper tests the case |Z| ≠ |A| (Figure 3): taking |Z| = |A| as the initial guess works best, but the agent can still learn with other sizes. |Z| = 1 performs very poorly on Cartpole and Acrobot, because all actions collapse into the same latent and the state predictions cannot be disentangled. This confirms that “the number of latent actions = the number of real actions” is a reasonable prior assumption.

环境Environment	BCO（基线）BCO (baseline)	ILPO（本文）ILPO (this paper)	BC（有动作标签）BC (with action labels)	Expert
Cartpole（步数 ≤100）Cartpole (≤100 steps)	低于 ExpertBelow Expert	≈ Expert	≈ Expert	—
Acrobot（步数 ≤100）Acrobot (≤100 steps)	低于 ExpertBelow Expert	≈ Expert	≈ Expert	—
Mountain car	低于 ExpertBelow Expert	优于 BCOBetter than BCO	≈ Expert	—
CoinRun Easy / Hard	低于 ExpertBelow Expert	显著优于 BCOMarkedly better than BCO	—	—

注：上表为对论文图表的定性归纳，具体数值请参见原文图 2、图 5。 BC 使用真实动作标签，ILPO 和 BCO 均不使用专家动作。Note: the table above is a qualitative summary of the figures in the paper; for the exact values see Figures 2 and 5 in the original. BC uses ground-truth action labels, while neither ILPO nor BCO uses expert actions.

04 局限性Limitations

注：以下局限性部分来自论文 Discussion & Conclusion 中的明确表述（标注为"stated"），部分由方法设计推断（标注为"inferred"）。Note: some of the limitations below are stated explicitly in the Discussion & Conclusion of the paper (marked “stated”), while others are inferred from the design of the method (marked “inferred”).

仅支持离散动作空间Discrete action spaces only [stated]

"future work could address … that it requires that actions are discrete"。当前 ILPO 假设动作空间为离散集合，连续控制（如机械臂扭矩控制）无法直接应用。此外，方法还假设状态转移是确定性的（deterministic transitions）。“future work could address … that it requires that actions are discrete”. ILPO currently assumes the action space is a discrete set, so continuous control (such as torque control of a robot arm) cannot be applied directly. The method additionally assumes that state transitions are deterministic (deterministic transitions).

仍需要少量真实环境交互Still requires a small amount of real environment interaction [stated]

虽然比 BCO 需要少得多的交互，但 ILPO 第二步仍需在真实环境中采集 {s, a, s'} 样本进行 action remapping。在完全无法与环境交互的场景（如医疗机器人冷启动）依然面临挑战。论文也指出可通过"enforcing stronger local consistencies between latent actions and generated predictions"进一步减少所需样本。Although far fewer interactions are needed than for BCO, the second step of ILPO still has to collect {s, a, s'} samples in the real environment for action remapping. Settings in which the environment cannot be interacted with at all (such as the cold start of a medical robot) remain a challenge. The paper also notes that the number of required samples can be reduced further by “enforcing stronger local consistencies between latent actions and generated predictions”.

高维视觉输入下对初始化敏感Sensitive to initialization with high-dimensional visual inputs [stated]

CoinRun 实验中，"some random seeds led to bad initialization that resulted in the agent not learning at all"。高维情形（128×128×3）下动力学学习难度更大，整体结果包含了表现很差的 seed。In the CoinRun experiments, “some random seeds led to bad initialization that resulted in the agent not learning at all”. In the high-dimensional setting (128×128×3) learning the dynamics is considerably harder, and the overall results include seeds that performed very poorly.

假设 latent action 数量已知Assumes the number of latent actions is known [inferred]

方法需要预先指定 |Z|（通常设为 |A|）。消融实验表明 |Z| ≠ |A| 仍可学习，但在实际应用中真实动作数量未必已知，自动确定合适的 latent action 数量是未来工作方向之一。The method requires |Z| to be specified in advance (usually set to |A|). The ablation shows that learning is still possible when |Z| ≠ |A|, but in practice the number of real actions is not necessarily known, and automatically determining a suitable number of latent actions is one direction for future work.

需要随机（stochastic）专家演示Requires stochastic expert demonstrations [stated]

"ILPO requires stochastic demonstrations … although the agent was capable of performing well with deterministic demonstrations, the performance decreased in this setting"。如果专家演示过于确定性，某些 latent action 对应的转移可能从未在数据中出现，影响动力学模型的覆盖度。“ILPO requires stochastic demonstrations … although the agent was capable of performing well with deterministic demonstrations, the performance decreased in this setting”. If the expert demonstrations are too deterministic, the transitions corresponding to some latent actions may never appear in the data, which harms the coverage of the dynamics model.