HiLAM：从无标签视频中分层提取可变长度技能，实现高效长时域操作HiLAM: hierarchically extracting variable-length skills from actionless videos for efficient long-horizon manipulation
Hanjung Kim · Lerrel Pinto · Seon Joo Kim · arXiv 2603.05815 · cs.RO
现有 Latent Action Models (LAMs) 擅长从视频中提取低层动作，但受限于短时域建模，无法捕捉更高层次的"技能"结构。HiLAM 引入动态分块机制 (Dynamic Chunking)，将低层 latent actions 自动聚合为可变长度的 latent skills，并通过层次化策略实现显著的数据效率提升。Existing Latent Action Models (LAMs) excel at extracting low-level actions from video, but are confined to short-horizon modeling and cannot capture higher-level "skill" structure. HiLAM introduces a Dynamic Chunking mechanism that automatically aggregates low-level latent actions into variable-length latent skills, and delivers substantial gains in data efficiency through a hierarchical policy.
Latent Action Models (LAMs) 通过 Inverse Dynamics Model 从观测视频中推断帧间 latent action，无需人工标注动作标签。然而，现有方法几乎全部聚焦于短时域帧间运动，对视频中本已存在的高层技能结构视而不见。Latent Action Models (LAMs) infer frame-to-frame latent actions from observation videos via an Inverse Dynamics Model, requiring no manually annotated action labels. Existing methods, however, focus almost entirely on short-horizon frame-to-frame motion and overlook the high-level skill structure already present in the video.
"existing latent action models are largely limited to short-term motion. As a result, they can capture low-level dynamics from observation-only data but often miss higher-level structure, such as temporally extended skills. This exposes a key gap where actionless videos contain not only primitive motions but also high-level skills that remain underutilized."
此前方法要么预设固定数量的 skill vectors（如 BUDS、SkillDiffuser），要么将固定长度的低层动作序列编码为 skill（如 SPiRL），均无法适应现实世界中技能时长自然变化的特性。HiLAM 的目标是：从无标签视频中自动提取可变长度、无需预先定义 skill set 的层次化 latent skills。Prior methods either presuppose a fixed number of skill vectors (e.g. BUDS, SkillDiffuser) or encode fixed-length low-level action sequences into a skill (e.g. SPiRL); neither accommodates the naturally varying durations of real-world skills. The goal of HiLAM is: to automatically extract variable-length hierarchical latent skills from actionless videos, with no predefined skill set.
Figure 1 — HiLAM 整体架构。 (a) latent skill 学习流水线：从观测视频中提取 latent actions，再经动态分块得到 latent skills；(b) 训练目标：next-latent prediction + 视觉重建 + H-Net 分块正则；(c) 利用预训练 IDM 逐帧对提取 latent actions。Figure 1 — Overall architecture of HiLAM. (a) The latent skill learning pipeline: latent actions are extracted from observation videos, then dynamically chunked into latent skills; (b) training objectives: next-latent prediction + visual reconstruction + H-Net chunking regularization; (c) latent actions are extracted frame-pair by frame-pair with a pretrained IDM.
45%HiLAM 仅用 10% 数据在 LIBERO-Long 的成功率（BAKU 仅 23%）HiLAM success rate on LIBERO-Long using only 10% of the data (BAKU only 23%)
84%HiLAM 用 50% 数据达到的成功率（≈ BAKU 100% 数据水平）Success rate HiLAM attains with 50% of the data (≈ BAKU at 100% data)
94%HiLAM 用 100% 数据在 LIBERO-Long 的成功率HiLAM success rate on LIBERO-Long with 100% of the data
4 suites全部 LIBERO 子测试均超越 BAKU baselineSurpasses the BAKU baseline on every LIBERO suite
02 Method
HiLAM 由两个阶段组成：首先在大规模无标签视频上预训练层次化 latent skill 模型；然后在目标任务中微调低层策略。核心创新在于 Dynamic Chunking Mechanism，将低层 latent action 序列自适应地分段，得到可变长度的高层 latent skill 表示。HiLAM consists of two stages: a hierarchical latent skill model is first pretrained on large-scale actionless video, then the low-level policy is fine-tuned on the target task. The core innovation is the Dynamic Chunking Mechanism, which adaptively segments the low-level latent action sequence into variable-length high-level latent skill representations.
给定观测视频 $\mathcal{V}$，首先用预训练 Inverse Dynamics Model (IDM) 提取低层 latent action 序列 $\mathbf{z}^l$。随后将其输入 H-Net 架构：Given observation videos $\mathcal{V}$, a pretrained Inverse Dynamics Model (IDM) first extracts the low-level latent action sequence $\mathbf{z}^l$. The sequence is then fed into an H-Net architecture:
Dynamic Chunking Mechanism：在每个阶段 $s$，编码器 $\mathcal{E}^s$ 将输入序列映射为 feature $h^s_t$，并计算相邻 token 的余弦相似度来预测分块边界指示 $b^s_t \in \{0,1\}$。当连续 token 差异较大时（即 $p^s_t \geq 0.5$），产生新边界，将序列自动切分为可变长度的段。Dynamic Chunking Mechanism: at each stage $s$, the encoder $\mathcal{E}^s$ maps the input sequence to features $h^s_t$ and computes the cosine similarity between adjacent tokens to predict a chunk boundary indicator $b^s_t \in \{0,1\}$. When consecutive tokens differ substantially (i.e. $p^s_t \geq 0.5$), a new boundary is emitted, automatically splitting the sequence into variable-length segments.
多级压缩：边界处的 encoder 特征被选出，作为下一阶段的输入 $\mathbf{z}^{s+1}$，实现逐级压缩。高阶段的 token（如 $\mathbf{z}^2$）代表更长时域的语义段，即 latent skills $\mathbf{z}^h$。Multi-level compression: the encoder features at boundaries are selected as the input $\mathbf{z}^{s+1}$ to the next stage, yielding stage-by-stage compression. Tokens at higher stages (e.g. $\mathbf{z}^2$) represent semantic segments over longer horizons, i.e. the latent skills $\mathbf{z}^h$.
训练目标：联合优化三项损失 $\mathcal{L} = \mathcal{L}_{\mathrm{latent}} + \lambda_{\mathrm{rec}}\,\mathcal{L}_{\mathrm{rec}} + \lambda_{\mathrm{ratio}}\,\mathcal{L}_{\mathrm{ratio}}$，分别对应 next-latent prediction（$\ell_1$ loss）、Forward Dynamics Model 的视觉重建监督、以及 H-Net 分块比例正则项。Training objective: three losses are optimized jointly, $\mathcal{L} = \mathcal{L}_{\mathrm{latent}} + \lambda_{\mathrm{rec}}\,\mathcal{L}_{\mathrm{rec}} + \lambda_{\mathrm{ratio}}\,\mathcal{L}_{\mathrm{ratio}}$, corresponding respectively to next-latent prediction (an $\ell_1$ loss), the visual reconstruction supervision of the Forward Dynamics Model, and the H-Net chunking ratio regularizer.
Figure 2 — Latent Skill 提取与策略学习。 (a) 低层 latent actions $\mathbf{z}^l$ 经层次编码后得到段级表示 $\mathbf{z}^s$，再展开回原始时序对齐为 $\mathbf{z}^h$；(b) 层次化策略整体流程：高层策略预测 latent skill，低层策略在给定 skill 条件下预测具体动作。Figure 2 — Latent skill extraction and policy learning. (a) Low-level latent actions $\mathbf{z}^l$ are hierarchically encoded into segment-level representations $\mathbf{z}^s$, then unrolled back and aligned to the original time steps as $\mathbf{z}^h$; (b) the overall hierarchical policy pipeline: the high-level policy predicts a latent skill, and the low-level policy predicts the concrete action conditioned on that skill.
利用提取的 latent skills 和 latent actions 作为 pseudo-labels，同时训练高层策略 $\pi^h$ 和低层策略 $\pi^l$：Using the extracted latent skills and latent actions as pseudo-labels, the high-level policy $\pi^h$ and the low-level policy $\pi^l$ are trained jointly:
预训练：高层策略 $\pi^h$ 根据当前观测和语言指令预测 latent skill $\hat{z}^h_t$；低层策略 $\pi^l$ 在给定 skill 条件下预测 latent action $\hat{z}^l_t$。两者均以无标签视频为数据源。Pretraining: the high-level policy $\pi^h$ predicts the latent skill $\hat{z}^h_t$ from the current observation and the language instruction; the low-level policy $\pi^l$ predicts the latent action $\hat{z}^l_t$ conditioned on that skill. Both draw on actionless video as their data source.
微调：冻结高层策略 $\pi^h$，仅用目标域的带动作标注数据微调低层策略，输出可执行的真实动作。Fine-tuning: the high-level policy $\pi^h$ is frozen and only the low-level policy is fine-tuned on action-labeled data from the target domain, so that it outputs executable real actions.
两个策略均基于 BAKU 架构，语言编码器为 T5 encoder。预训练默认使用 Something-Something V2（人类手持物体操作视频），数据处理为 observation-only（丢弃原始动作标注）。Both policies build on the BAKU architecture, with a T5 encoder as the language encoder. Pretraining uses Something-Something V2 (videos of humans manipulating handheld objects) by default, processed as observation-only (the original action annotations are discarded).
03 Experiments
在 LIBERO 仿真 benchmark 上评估，共 4 个子测试套件（Spatial、Object、Goal、Long），每套 10 个任务各提供 50 条专家演示。基线为 BAKU（当前最优）。预训练数据使用 Something-Something V2（人类视频）、Droid 和 BridgeV2（机器人视频）。预训练和微调各 100k 步。Evaluation is on the LIBERO simulation benchmark, comprising 4 suites (Spatial, Object, Goal, Long), each with 10 tasks providing 50 expert demonstrations apiece. The baseline is BAKU (the current state of the art). Pretraining data comes from Something-Something V2 (human video), Droid and BridgeV2 (robot video). Pretraining and fine-tuning each run for 100k steps.
论文原文指出："With only 10% of the demonstrations, BAKU achieves a 23% success rate, whereas HiLAM achieves 45%, nearly doubling performance. With 50% of the demonstrations, HiLAM reaches 84%, comparable to BAKU trained with 100% of the data."The paper states: "With only 10% of the demonstrations, BAKU achieves a 23% success rate, whereas HiLAM achieves 45%, nearly doubling performance. With 50% of the demonstrations, HiLAM reaches 84%, comparable to BAKU trained with 100% of the data."
Figure 3 — Skill 边界预测的定性结果。 利用预测的边界指示 $b^s_t$，将每帧分配到 skill 段 $k^s_t$，并用颜色编号标示。Segment 2：夹爪向碗运动；Segment 3：夹爪拾起碗（此处产生新边界）；Segment 4：夹爪移动到目标位置并放置。在无任何标签的无监督训练条件下，HiLAM 自动将连续 latent action 聚合成语义一致的技能段。Figure 3 — Qualitative results of skill boundary prediction. Using the predicted boundary indicator $b^s_t$, each frame is assigned to a skill segment $k^s_t$, marked by a color index. Segment 2: the gripper moves toward the bowl; Segment 3: the gripper picks up the bowl (a new boundary is emitted here); Segment 4: the gripper moves to the target location and places it. Under unsupervised training without any labels, HiLAM automatically aggregates consecutive latent actions into semantically coherent skill segments.
Ablations
在 LIBERO-Long 上的消融实验（均使用 100% 微调数据）：Ablation studies on LIBERO-Long (all using 100% of the fine-tuning data):
预训练数据：人类视频优于机器人视频（Human: 0.94 vs Robot: 0.90），因此默认使用 Something-Something V2。Pretraining data: human video outperforms robot video (Human: 0.94 vs Robot: 0.90), so Something-Something V2 is used by default.
Latent skill 阶段选择：使用 stage $s=2$ 作为 latent skill、$s=0$ 作为 latent action 条件效果最佳（成功率 0.94）；组合 $s=2$+$s=1$ 下降至 0.89。Stage-2 由最深编码器生成，捕捉更长时域上下文和更具语义凝聚力的分段。Choice of latent skill stage: using stage $s=2$ as the latent skill and $s=0$ as the latent action condition works best (0.94 success rate); combining $s=2$+$s=1$ drops to 0.89. Stage-2 is produced by the deepest encoder, capturing longer-horizon context and more semantically cohesive segments.
非层次化 baseline：仅用 latent action 条件预训练 BAKU（不做层次化）最高达 0.91，但仍低于 HiLAM 的 0.94，说明层次化策略结构本身带来了额外增益。Non-hierarchical baseline: pretraining BAKU with latent action conditioning alone (no hierarchy) reaches at most 0.91, still below the 0.94 of HiLAM, showing that the hierarchical policy structure itself brings an additional gain.
无预训练：仅在目标任务上训练层次化策略，成功率仅 0.67，说明性能提升源于大规模预训练，而非架构本身。Without pretraining: training the hierarchical policy on the target task alone yields a success rate of only 0.67, showing that the improvement stems from large-scale pretraining rather than from the architecture itself.
Figure 4 — 未来帧预测定性结果。 给定当前帧 $I_t$ 和预测的 latent action $\hat{z}^l_t$，预训练 FDM 生成未来帧 $\hat{I}_{t+k}$。尽管 $\hat{z}^l_t$ 由历史序列 $z^l_{:t-1}$ 预测，仍能产生与真实帧一致的未来帧预测，验证 HiLAM 隐式地对时序动态建模。Figure 4 — Qualitative results of future frame prediction. Given the current frame $I_t$ and the predicted latent action $\hat{z}^l_t$, the pretrained FDM generates the future frame $\hat{I}_{t+k}$. Although $\hat{z}^l_t$ is predicted from the history $z^l_{:t-1}$, it still yields future-frame predictions consistent with the ground-truth frames, confirming that HiLAM implicitly models temporal dynamics.
04 Limitations
Note:以下局限性均为论文 "Conclusion and Limitations" 一节中作者明确陈述的内容。All limitations below are stated explicitly by the authors in the paper\'s "Conclusion and Limitations" section.
仅在仿真环境中验证，缺乏真实世界实验Validated only in simulation, without real-world experiments
"our experiments are primarily conducted in simulated environments such as LIBERO. Validating the framework through real-world experiments would further demonstrate the effectiveness of the proposed method."
依赖预训练 IDM，无法端到端联合训练Depends on a pretrained IDM; cannot be trained jointly end-to-end
"to ensure computational efficiency during temporal modeling, HiLAM utilizes a pretrained IDM. However, training the entire architecture end-to-end could potentially lead to a deeper joint understanding of both low-level motion patterns and high-level skills."
未结合语言信号进行技能发现Language signals are not incorporated into skill discovery
论文指出运动线索与语言指令提供的是正交而非平行的信息，两者的结合（尤其是在复杂任务如家具组装中）有望进一步提升技能发现的质量。将层次化 latent action 建模与自然语言结合是有前景的未来方向。（论文原文："incorporating language represents a promising direction for future research"）The paper notes that motion cues and language instructions provide orthogonal rather than parallel information, and that combining the two (especially on complex tasks such as furniture assembly) promises to further improve the quality of skill discovery. Combining hierarchical latent action modeling with natural language is a promising future direction. (From the paper: "incorporating language represents a promising direction for future research")
