Latent Action Pretraining from Videos (LAPA)

01 动机Motivation

现有 Vision-Language-Action (VLA) 模型的预训练依赖于机器人遥操作收集的有标注动作数据，这使得可用数据规模极为有限。互联网上存在海量的人类操作视频（如 YouTube、Something-Something 等），却因缺少机器人动作标签而无法直接用于 VLA 预训练。如何利用这些无标注视频数据，是扩展机器人基础模型的核心瓶颈。Pretraining existing Vision-Language-Action (VLA) models relies on labeled action data collected by robot teleoperation, which keeps the amount of usable data extremely limited. The internet holds vast numbers of human manipulation videos (e.g. YouTube, Something-Something), yet they cannot be used directly for VLA pretraining because they carry no robot action labels. Exploiting such unlabeled video data is the central bottleneck for scaling robot foundation models.

"We introduce LAPA, an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels."

LAPA problem formulation — **图 1 — 问题设定。**LAPA 的目标是构建通用机器人基础模型：利用海量无动作标注的人类操作视频（左上）进行预训练，再以少量有标注机器人数据微调，最终部署到真实机器人（右下）。与需要大规模遥操作数据的现有方法不同， LAPA 完全绕过了对真实动作标签的依赖。**Figure 1 — Problem setup.**LAPA aims to build a general robot foundation model: it pretrains on vast amounts of action-free human manipulation videos (top left), then finetunes on a small amount of labeled robot data, and is finally deployed on a real robot (bottom right). Unlike existing methods that require large-scale teleoperation data, LAPA bypasses the dependence on ground-truth action labels entirely.

50.1%LAPA (Open-X) 真实机器人平均成功率LAPA (Open-X) average real-robot success rate

43.9%OpenVLA (Open-X) 对比基准成功率OpenVLA (Open-X) baseline success rate

30×LAPA 相对 OpenVLA 的预训练效率提升LAPA pretraining efficiency gain over OpenVLA

272hLAPA 预训练所用 H100 GPU 时（OpenVLA 需 21,500 A100 时）H100 GPU-hours used for LAPA pretraining (OpenVLA needs 21,500 A100 hours)

02 方法Method

LAPA 将预训练分为三个阶段：首先用 VQ-VAE 从视频帧对中学习离散 latent action；接着让 VLM 从视频观测和语言指令中预测这些 latent action（行为克隆）；最后在小规模有标注机器人数据上微调，将 latent action 映射为真实机器人指令。LAPA splits pretraining into three stages: first a VQ-VAE learns discrete latent actions from pairs of video frames; next a VLM learns to predict these latent actions from video observations and language instructions (behavior cloning); finally the model is finetuned on a small amount of labeled robot data, mapping latent actions onto real robot commands.

LAPA method overview — **图 2 — LAPA 三阶段流水线。** *Stage 1（Latent Action Quantization）：*编码器输入当前帧 x_t 与未来帧 x_t+h，输出离散 latent action z_t；解码器从 z_t 重建未来帧（自监督）。 *Stage 2（Latent Pretraining）：*VLM 学习从观测序列和语言指令预测 z_t。 *Stage 3（Action Finetuning）：*替换 latent action head，在真实机器人数据上微调为端到端控制器。**Figure 2 — The three-stage LAPA pipeline.** *Stage 1 (Latent Action Quantization):* the encoder takes the current frame x_t and the future frame x_t+h and outputs a discrete latent action z_t; the decoder reconstructs the future frame from z_t (self-supervised). *Stage 2 (Latent Pretraining):* the VLM learns to predict z_t from the observation sequence and the language instruction. *Stage 3 (Action Finetuning):* the latent action head is replaced and the model is finetuned on real robot data into an end-to-end controller.

Stage 1：Latent Action Quantization（VQ-VAE）Stage 1: Latent Action Quantization (VQ-VAE)

编码器采用 C-ViViT 架构，输入当前帧与未来帧（间隔 H 步），通过 cross-attention 提取"帧间变化"，输出离散 latent action token z_t（词表大小 K^L，默认 8⁴=4096 种）。解码器在给定 z_t 和当前帧的条件下重建未来帧。为防止梯度塌缩，使用 NSVQ（Normalized Straight-Through Vector Quantization）替代标准 VQ。整个过程完全自监督，无需任何动作标注。The encoder adopts a C-ViViT architecture: it takes the current frame and a future frame (H steps apart) and extracts the "inter-frame change" through cross-attention, producing discrete latent action tokens z_t (vocabulary size K^L, 8⁴=4096 entries by default). Conditioned on z_t and the current frame, the decoder reconstructs the future frame. To prevent gradient collapse, NSVQ (Normalized Straight-Through Vector Quantization) is used instead of standard VQ. The entire process is fully self-supervised and needs no action labels at all.

Stage 2：Latent VLA 预训练Stage 2: Latent VLA Pretraining

以预训练 VLM（7B 参数）为骨干，冻结视觉编码器，对语言模型参数进行行为克隆训练：给定视频帧序列和语言任务描述，预测对应的 latent action token 序列。训练数据可以是任意无标注视频（如 Open-X Embodiment 数据集或 Something-Something V2 人类视频），无需机器人遥操作数据。A pretrained VLM (7B parameters) serves as the backbone: the vision encoder is frozen while the language model parameters are trained by behavior cloning — given a sequence of video frames and a language task description, the model predicts the corresponding sequence of latent action tokens. The training data can be any unlabeled video (such as the Open-X Embodiment dataset or Something-Something V2 human videos); no robot teleoperation data is required.

Stage 3：Action FinetuningStage 3: Action Finetuning

仅需少量有标注的机器人演示数据：移除 latent action head，替换为任务专用 action head，微调模型将学到的 latent 表征映射到真实机器人端效器动作。由于 Stage 2 已学到丰富的视觉-语言-动作表征，Stage 3 收敛快、所需数据量少。Only a small amount of labeled robot demonstration data is needed: the latent action head is removed and replaced by a task-specific action head, and the model is finetuned to map the learned latent representations onto real robot end-effector actions. Because Stage 2 has already learned rich vision-language-action representations, Stage 3 converges fast and needs little data.

03 实验Experiments

评估在三个基准上进行：Language Table 仿真、SIMPLER 仿真（Open-X 子集）、以及真实世界桌面操作（pick、cover、knock 三类任务，共 54 次 rollout）。对比基线包括 Scratch、UniPi、VPT、ActionVLA、OpenVLA 等。Evaluation is carried out on three benchmarks: the Language Table simulator, the SIMPLER simulator (a subset of Open-X), and real-world tabletop manipulation (pick, cover and knock, three task types, 54 rollouts in total). Baselines include Scratch, UniPi, VPT, ActionVLA and OpenVLA.

Language Table 仿真结果Language Table Simulation Results

模型Model	In-Domain Seen	In-Domain Unseen	Cross-Task Seen	Cross-Task Unseen	Cross-Env Seen	Cross-Env Unseen
Scratch	15.6±9.2	15.2±8.3	27.2±13.6	22.4±11.0	15.6±9.2	15.2±8.3
UniPi	22.0±12.5	13.2±7.7	20.8±12.0	16.0±9.1	13.6±8.6	12.0±7.5
VPT	44.0±7.5	32.8±4.6	72.0±6.8	60.8±6.6	18.0±7.7	18.4±9.7
LAPA	62.0±8.7	49.6±9.5	73.2±6.8	54.8±9.1	33.6±12.7	29.6±12.0
ActionVLA	77.0±3.5	58.8±6.6	77.0±3.5	58.8±6.6	64.8±5.2	54.0±7.0

LAPA 在 In-Domain Seen 上从 Scratch 的 15.6% 提升至 62.0%，Cross-Env Seen 从 15.6% 提升至 33.6%，显示出跨环境泛化能力。ActionVLA 使用有标注动作数据预训练，作为上界参考。LAPA lifts In-Domain Seen from 15.6% for Scratch to 62.0%, and Cross-Env Seen from 15.6% to 33.6%, showing cross-environment generalization. ActionVLA is pretrained with labeled action data and serves as an upper-bound reference.

真实机器人操作结果Real-Robot Manipulation Results

Real-world manipulation results — **图 3 — 真实桌面操作成功率对比。** 评估三类泛化：Seen Obj. Unseen Combo（见过物体但未见过组合）、Unseen Obj.（未见物体）、 Unseen Instr.（未见指令）。LAPA (Open-X) 在三类泛化上均优于或持平 OpenVLA (Open-X)，平均成功率 50.1% vs. 43.9%。**Figure 3 — Success rates on real-world tabletop manipulation.** Three kinds of generalization are evaluated: Seen Obj. Unseen Combo (seen objects in unseen combinations), Unseen Obj. (unseen objects), and Unseen Instr. (unseen instructions). LAPA (Open-X) matches or beats OpenVLA (Open-X) on all three, with an average success rate of 50.1% vs. 43.9%.

模型Model	Seen Obj. Unseen Combo	Unseen Obj.	Unseen Instr.	平均 AVGAverage AVG
Scratch	18.0	20.3	25.4	21.2
ActionVLA (Bridge)	38.3	31.8	27.7	32.6
OpenVLA (Bridge)	35.6	34.6	22.1	30.8
LAPA (Bridge)	43.4	31.4	35.6	36.8
OpenVLA (Open-X)	46.2	42.1	43.4	43.9
LAPA (Open-X)	57.8	43.9	48.5	50.1
LAPA (Human Videos)	36.5	37.4	28.1	34.0

人类视频预训练（跨体态迁移）Pretraining on Human Videos (Embodiment Transfer)

LAPA 使用 Something-Something V2（人类手部操作视频，无机器人数据）进行预训练后，在真实机器人任务上仍能超越 OpenVLA (Bridge) 的平均成功率（34.0% vs. 30.8%），证明 latent action 空间可跨体态迁移（embodiment transfer）。After pretraining on Something-Something V2 (human hand manipulation videos, no robot data), LAPA still surpasses the average success rate of OpenVLA (Bridge) on real robot tasks (34.0% vs. 30.8%), showing that the latent action space transfers across embodiments.

Latent Action 可解释性分析Interpretability Analysis of Latent Actions

Latent action analysis — **图 6 — Latent Action 语义映射。** 在 Open-X Embodiment 数据集上，将 latent action 嵌入二维空间可视化，显示出与真实动作（移动方向、抓取/放置等）的语义对应关系，说明 VQ-VAE 在无监督下自动发现了具有意义的动作表征。**Figure 6 — Semantic mapping of latent actions.** On the Open-X Embodiment dataset, the latent action embeddings are visualized in a two-dimensional space, revealing a semantic correspondence with real actions (movement direction, grasping/placing, and so on), which shows that the VQ-VAE discovers meaningful action representations without supervision.

消融实验Ablations

模型规模：更大的 VLM 骨干（参数量）带来持续性能提升。Model scale: a larger VLM backbone (parameter count) brings sustained performance gains.
数据规模：预训练数据量增加，下游任务成功率稳步提升。Data scale: as the amount of pretraining data grows, downstream task success rates improve steadily.
Latent action 词表大小：Language Table 任务中，增大词表比增加序列长度更有效； SIMPLER 和真实世界任务保持 8⁴=4096 种动作空间。Latent action vocabulary size: on Language Table tasks, enlarging the vocabulary is more effective than lengthening the sequence; SIMPLER and real-world tasks keep the 8⁴=4096 action space.
窗口大小 H：默认 H=3（5Hz 视频下约 0.6 秒）效果稳健；极端值下性能下降。Window size H: the default H=3 (about 0.6 s of 5Hz video) is robust; extreme values degrade performance.
微调数据量：即便微调数据极少，LAPA 仍持续优于 Scratch。Finetuning data volume: even with very little finetuning data, LAPA still consistently outperforms Scratch.
训练轮次：1 个 epoch 即达到最优性能，无需多轮训练。Training epochs: 1 epoch already reaches the best performance; multiple epochs are unnecessary.

04 局限性Limitations

说明：以下局限性均由论文作者在原文中明确陈述（stated）。Note: all limitations below are explicitly stated by the authors in the paper.

精细抓取动作能力不足Limited fine-grained grasping ability

论文明确指出："LAPA underperforms compared to action pretraining when it comes to fine-grained motion generation tasks like grasping." 在需要精确末端执行器控制的抓取任务中，LAPA 与使用有标注动作数据预训练的 ActionVLA 相比仍有差距，例如在 pick-and-place 任务中 OpenVLA 在部分精细抓取场景优于 LAPA。The paper states explicitly: "LAPA underperforms compared to action pretraining when it comes to fine-grained motion generation tasks like grasping." On grasping tasks that require precise end-effector control, LAPA still lags behind ActionVLA, which is pretrained with labeled action data; for instance, OpenVLA outperforms LAPA in some fine-grained grasping settings of pick-and-place tasks.

实时推理延迟Real-time inference latency

"Similar to prior VLAs, LAPA also encounters latency challenges during real-time inference." 作为基于大型语言模型的 VLA，LAPA 继承了 VLA 家族共有的推理速度瓶颈，尚未针对在线控制进行专门优化。"Similar to prior VLAs, LAPA also encounters latency challenges during real-time inference." As a VLA built on a large language model, LAPA inherits the inference-speed bottleneck shared by the VLA family, and has not yet been specifically optimized for online control.

评估场景局限于操作任务Evaluation limited to manipulation tasks

当前实验主要集中于桌面操作场景。论文指出尚未探索 LAPA 在导航、自动驾驶等其他机器人应用领域（"beyond manipulation videos, such as those from self-driving cars, navigation"）的适用性，这些场景的泛化能力有待验证。The current experiments focus mainly on tabletop manipulation. The paper notes that the applicability of LAPA to other robotic domains such as navigation and autonomous driving ("beyond manipulation videos, such as those from self-driving cars, navigation") remains unexplored, and generalization in those settings has yet to be verified.