Learning What You Can Do Before Doing Anything

01 动机Motivation

智能体在真实世界中行动时需要通过感知来判断自己可以做什么、以及这些动作会带来什么后果。现有的强化学习方法需要主动交互才能获取动作空间信息，而这在很多场景下代价高昂甚至不可行（例如使用互联网视频进行模仿学习）。An agent acting in the real world must use perception to judge what it can do and what consequences those actions will have. Existing reinforcement learning methods require active interaction to obtain information about the action space, which in many settings is expensive or even infeasible (for example, imitation learning from internet video).

"Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences."

作者受婴儿学步的启发：婴儿在能够自主行走之前，已通过大量被动感知成年人行走积累了动作空间的先验知识。类比地，本文提出：纯从被动视频观察中学习动作表征，可以让后续学习更加高效——无论是动作条件视频预测还是规划任务，都只需极少量标注数据。The authors are inspired by how infants learn to walk: before being able to walk on their own, infants have already built up prior knowledge of the action space through extensive passive perception of adults walking. By analogy, this paper proposes learning action representations purely from passive video observation, which makes subsequent learning far more efficient — for both action-conditioned video prediction and planning tasks, only a tiny amount of labelled data is then needed.

Learned action space visualization — **图1：CLASP 学到的动作表征空间可视化（reacher 数据集）。**每个点代表一帧对的隐变量 z，颜色对应真实旋转角度（radians）。
*左（baseline Denton & Fergus 2018）：*隐空间与真实动作无对应关系，学习失败。
*右（CLASP）：*第一主成分解释了 99% 方差，学到的 z 与真实动作高度一致。**Figure 1: visualization of the action representation space learned by CLASP (reacher dataset).** Each point is the latent variable z of a frame pair, coloured by the ground-truth rotation angle (radians).
*Left (baseline Denton & Fergus 2018):* the latent space has no correspondence with the true action; learning fails.
*Right (CLASP):* the first principal component explains 99% of the variance, and the learned z agrees closely with the true action.

100Kreacher 无标注训练序列unlabelled reacher training sequences

~orders of magnitude所需标注数据更少less labelled data required

1.6°CLASP 视觉伺服距离误差（10K 标注）CLASP visual servoing distance error (10K labels)

30%BAIR 数据集上监督基线差距缩短比例share of the supervised-baseline gap closed on BAIR

02 方法Method

CLASP 在随机视频预测模型的隐变量上施加两个关键约束：minimality（最小性）与 composability（可组合性），从而迫使隐变量聚焦于动作相关的动态信息，同时与静态场景内容解耦。CLASP imposes two key constraints on the latent variables of a stochastic video prediction model: minimality and composability, forcing the latents to focus on action-related dynamics while being disentangled from static scene content.

**图2：CLASP 架构。**
左：随机视频预测模型（一个时间步）。训练时，推断网络（MLPinfer, CNNe）从当前帧与前一帧估计隐变量 z；测试时 z 从先验 N(0,I) 采样。生成网络（LSTM, CNNd）将 z 与 xt−1 解码为下一帧预测。
右：可组合性训练。隐变量 z 两两拼接后经过 MLPcomp 得到轨迹表征 ν，再通过同一生成网络解码并与实际图像对比。**Figure 2: the CLASP architecture.**
*Left:* the stochastic video prediction model (one time step). During training the inference networks (MLPinfer, CNNe) estimate the latent variable z from the current and previous frames; at test time z is sampled from the prior N(0,I). The generative networks (LSTM, CNNd) decode z together with xt−1 into the next-frame prediction.
*Right:* composability training. Latent variables z are concatenated pairwise and passed through MLPcomp to obtain the trajectory representation ν, which is then decoded by the same generative network and compared with the actual image.

1. 最小性（Minimality）via Information Bottleneck1. Minimality via Information Bottleneck

为使 z 仅捕捉动态信息而尽量丢弃静态内容，作者采用 variational Information Bottleneck (VIB) 目标函数：To make z capture only dynamic information and discard static content as far as possible, the authors adopt the variational Information Bottleneck (VIB) objective:

max I((z_t, x_t−1), x_t) − β_z I(z_t, x_t−1:t)

视频预测目标为标准 VAE 损失加权版本（β-VAE 风格）：最大化重建质量的同时，最小化 z 与输入帧之间的互信息。β_z 越大，z 的信息量越少，越趋于"最小化"。The video prediction objective is a weighted version of the standard VAE loss (β-VAE style): it maximizes reconstruction quality while minimizing the mutual information between z and the input frames. The larger β_z is, the less information z carries and the more "minimal" it becomes.

2. 可组合性（Composability）via 轨迹表征2. Composability via a Trajectory Representation

为学到可组合的动作表征，CLASP 引入轨迹随机变量 ν，它由若干连续的 z 组合而得（通过 MLPcomp）。同样用 IB 目标训练：ν 应足以预测对应的末帧图像，同时对单个 z 的信息量最小化。总目标为两个 loss 之和：To learn a composable action representation, CLASP introduces a trajectory random variable ν, obtained by composing several consecutive z (through MLPcomp). It is trained with the same IB objective: ν should be sufficient to predict the corresponding final frame while carrying minimal information about each individual z. The total objective is the sum of the two losses:

L^total = L^comp + L^pred

可组合性隐式地促进了解耦：因为 ν 无法看到中间帧的静态内容，仅当每个 z 已与静态内容解耦时，组合才得以高效进行。Composability implicitly encourages disentanglement: because ν cannot see the static content of the intermediate frames, composition is only efficient when each z is already disentangled from static content.

3. 控制映射接地（Grounding the Control Mapping）3. Grounding the Control Mapping

训练完视频预测模型后，只需少量带动作标注序列（如 100~10000 条）训练两个轻量 MLP（MLPlat 和 MLPact），即可在 z 与真实控制指令 u 之间建立可逆的双射映射。这一步不反向传播至视频预测模型，数据量需求极低。Once the video prediction model is trained, only a small number of action-labelled sequences (e.g. 100~10000) are needed to train two lightweight MLPs (MLPlat and MLPact), which establish an invertible bijective mapping between z and the true control command u. This step does not backpropagate into the video prediction model and its data requirement is extremely low.

03 实验Experiments

实验在两个数据集上进行：（1）reacher 数据集——模拟单自由度旋转机械臂（OpenAI Roboschool），包含 100,000 训练序列、4,000 测试序列；（2）BAIR robot pushing 数据集（Ebert et al. 2017）——44,374 训练序列，真实世界机械臂推物。基线包括 Denton & Fergus (2018)（无 composability 目标），以及完全监督方法 Oh et al. (2015)（reacher）和 Finn & Levine (2017)（BAIR）。Experiments are run on two datasets: (1) the reacher dataset — a simulated rotating arm with a single degree of freedom (OpenAI Roboschool), containing 100,000 training and 4,000 test sequences; (2) the BAIR robot pushing dataset (Ebert et al. 2017) — 44,374 training sequences of a real-world arm pushing objects. Baselines include Denton & Fergus (2018) (without the composability objective) as well as the fully supervised methods Oh et al. (2015) (reacher) and Finn & Levine (2017) (BAIR).

动作条件视频预测（Action-conditioned Video Prediction）Action-conditioned Video Prediction

方法Method	Reacher 绝对角度误差 [deg]Reacher absolute angle error [deg]	BAIR 末端位置误差 [px]BAIR end-effector position error [px]
随机猜测 (Start State)Random guess (Start State)	90.1 ± 51.8	26.6 ± 21.5
Denton & Fergus (2018)	22.6 ± 17.7	3.6 ± 4.0
CLASP（本文）CLASP (ours)	2.9 ± 2.1	3.0 ± 2.1
Supervised（全监督上界）Supervised (fully supervised upper bound)	2.6 ± 1.8	2.0 ± 1.3

CLASP 在 reacher 上达到与全监督方法几乎相同的性能，而 baseline 性能接近随机。在 BAIR 上，CLASP 较 baseline 将与监督方法的差距缩小约 30%。On reacher, CLASP reaches almost the same performance as the fully supervised method, while the baseline is close to random. On BAIR, CLASP narrows the gap to the supervised method by about 30% compared with the baseline.

视觉伺服规划（Visual Servoing）Visual Servoing

方法Method	Reacher 终点距离 [deg]Reacher final distance [deg]
Start Position	97.8 ± 23.7
Random	27.0 ± 26.8
Denton & Fergus (2018)	14.1 ± 10.7
CLASP（本文）CLASP (ours)	1.6 ± 1.0
Agrawal et al. (2016)（全监督）Agrawal et al. (2016) (fully supervised)	2.0 ± 1.5
Oh et al. (2015)（全监督）Oh et al. (2015) (fully supervised)	1.8 ± 1.5
CLASP（变化背景）CLASP (varying background)	3.0 ± 2.2
CLASP（变化机器人外形）CLASP (varying robot shape)	2.8 ± 2.9

规划算法采用 Model Predictive Control (MPC) + Cross Entropy Method (CEM)，在隐空间 z 中规划轨迹，用 VGG16 余弦距离衡量与目标帧的差异。CLASP 在视觉伺服上超越全监督方法，达到最优性能。The planner uses Model Predictive Control (MPC) + the Cross Entropy Method (CEM) to plan trajectories in the latent space z, measuring the difference from the goal frame with the VGG16 cosine distance. On visual servoing CLASP surpasses the fully supervised methods and attains the best performance.

Visual servoing results — **图3：视觉伺服结果。**
左：每一行为一个时间步重新规划的轨迹；首帧为当前状态，后续帧为模型以最低代价预测的路径，目标状态（机械臂指向左上）叠加显示在每张图上。
右：数据效率——以不同数量的带标注训练序列衡量最终伺服误差；CLASP 在极少标注下即超过监督基线。**Figure 3: visual servoing results.**
*Left:* each row is a trajectory replanned at one time step; the first frame is the current state and the following frames are the lowest-cost path predicted by the model, with the goal state (arm pointing to the upper left) overlaid on every image.
*Right:* data efficiency — the final servoing error measured against different numbers of labelled training sequences; with very few labels CLASP already surpasses the supervised baselines.

轨迹迁移（Trajectory Transplantation）Trajectory Transplantation

数据效率与鲁棒性Data Efficiency and Robustness

CLASP 用无标注数据预训练后，仅需 100~1000 条带动作标注序列即可达到与 10,000 条全监督数据相当的规划精度。在变化背景（CIFAR-10 随机图像）和变化机器人外形（72 种宽度×长度组合）条件下，CLASP 性能基本保持不变，说明学到的表征对视觉特征变化具有鲁棒性。After pre-training on unlabelled data, CLASP needs only 100~1000 action-labelled sequences to reach planning accuracy comparable to 10,000 fully supervised sequences. Under varying backgrounds (random CIFAR-10 images) and varying robot shapes (72 combinations of width and length), CLASP performance stays essentially unchanged, showing that the learned representation is robust to changes in visual appearance.

04 局限性Limitations

说明：论文未设独立局限性章节；以下各点结合论文实验范围与方法设计推断，已标注来源。Note: the paper has no standalone limitations section; the points below are inferred from the scope of its experiments and the design of the method, with the source of each indicated.

实验局限于低维度 / 低分辨率设置（推断自设计）Experiments are confined to low-dimensional / low-resolution settings (inferred from the design)

所有实验使用 64×64 像素图像，隐变量维度 dim(z) = dim(ν) = 10，模拟的 reacher 仅有一个自由度。方法在更复杂的操作场景（多自由度、高分辨率、遮挡）下的泛化性尚未验证。All experiments use 64×64 pixel images with latent dimensions dim(z) = dim(ν) = 10, and the simulated reacher has only a single degree of freedom. Generalization of the method to more complex manipulation settings (more degrees of freedom, higher resolution, occlusion) has not been verified.

BAIR 数据集上与监督方法仍存在差距（论文明确指出）A gap to supervised methods remains on the BAIR dataset (stated explicitly in the paper)

论文在 BAIR 末端位置误差上：CLASP 为 3.0 ± 2.1 px，全监督方法为 2.0 ± 1.3 px。作者承认："our model performs better than the baseline … reducing the difference … by 30 %"，但并未完全闭合差距，并指出 baseline 仍产生模糊与鬼影伪影。On the BAIR end-effector position error the paper reports 3.0 ± 2.1 px for CLASP versus 2.0 ± 1.3 px for the fully supervised method. The authors acknowledge: "our model performs better than the baseline … reducing the difference … by 30 %", yet the gap is not fully closed, and they note that the baseline still produces blur and ghosting artifacts.

需要场景动态主要由被观察智能体主导（推断自方法设计）Requires the scene dynamics to be dominated by the observed agent (inferred from the method design)

CLASP 依赖"视频中变化最大来源是智能体自身动作"的假设。若场景中有大量无关运动（行人、风吹树叶等），隐变量可能无法有效分离动作信息，使 composability 损失失效。CLASP relies on the assumption that the largest source of change in the video is the agent's own action. If the scene contains a lot of irrelevant motion (pedestrians, leaves blowing in the wind, etc.), the latent variables may fail to isolate action information, rendering the composability loss ineffective.

β 超参数选取需要人工调参（论文明确指出）Choosing the β hyperparameters requires manual tuning (stated explicitly in the paper)

论文指出："the problem of determining β is not unique to this work and occurs in all stochastic video prediction methods, as well as VIB and β-VAE"。β_z 和 β_ν 需设为"样本仍能生成高质量图像的最大值"，缺乏自动化准则。The paper notes: "the problem of determining β is not unique to this work and occurs in all stochastic video prediction methods, as well as VIB and β-VAE". β_z and β_ν must be set to "the largest value for which samples still generate high-quality images", and there is no automated criterion.