LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer
将机器人动作表示为自然语言，让 VLM 预训练知识直接迁移至未见过的机械臂Representing robot actions as natural language, so that the pre-training knowledge of a VLM transfers directly to unseen manipulators
Lihan Zha, Asher J. Hancock, Mingtong Zhang, Tenny Yin, Yixuan Huang, Dhruv Shah, Allen Z. Ren, Anirudha Majumdar · Princeton University & Physical Intelligence
LAP 提出"语言动作（language-action）"表示：将末端执行器的运动直接编码为自然语言短语（如 "move forward 5 cm"），从而使动作监督信号与 VLM 的预训练分布对齐。以此训练的 LAP-3B 在 3 个从未见过的机械臂上实现平均超 50% 的零样本成功率，比最强现有 VLA 基线提升约 2 倍；且仅需约 2.5× 更少的演示样本即可达到相当的微调性能。LAP proposes a "language-action" representation: end-effector motion is encoded directly as natural-language phrases (e.g. "move forward 5 cm"), which aligns the action supervision signal with the pre-training distribution of the VLM. LAP-3B, trained this way, attains an average zero-shot success rate above 50% on 3 manipulators never seen during training, roughly two times higher than the strongest existing VLA baseline; it also reaches comparable fine-tuning performance with about 2.5× fewer demonstrations.
language-action pre-trainingVLAzero-shot cross-embodiment具身智能embodied intelligencerobot manipulationflow matchingvision-language model跨机器人迁移cross-embodiment transfer
01动机Motivation
尽管 VLA 模型已在多机器人混合数据集上大规模预训练，"state-of-the-art VLAs still rarely function zero-shot on new robots"——哪怕只是换了一个夹爪或调整了摄像头位置，模型就会失效。问题根源在于：现有方法将 VLM 微调为直接预测连续动作或离散控制 token，这造成了"distributional mismatch"，因为 VLM 的预训练从未接触过电机级高频控制信号，也无法从这些信号中提取任何跨机器人通用的语义结构。Although VLA models have been pre-trained at scale on mixed multi-robot datasets, "state-of-the-art VLAs still rarely function zero-shot on new robots"—the model breaks down even when only the gripper is swapped or the camera position is adjusted. The root cause is that existing methods fine-tune the VLM to predict continuous actions or discrete control tokens directly, which creates a "distributional mismatch": VLM pre-training has never encountered motor-level high-frequency control signals, nor can it extract from such signals any semantic structure shared across robots.
"Zero-shot cross-embodiment transfer depends critically on how we adapt a pre-trained VLM for motor control."
图 1：LAP 框架概览与零样本迁移效果。LAP 将底层动作直接表示为自然语言，以此监督 VLM 骨干网络的预训练。LAP-3B 在训练时仅见过 DROID 机械臂，却能零样本控制 Custom Franka、YAM、Kinova 三种此前从未接触的机器人，完成多类真实操作任务。Figure 1: Overview of the LAP framework and its zero-shot transfer results.LAP represents low-level actions directly as natural language and uses them to supervise pre-training of the VLM backbone. LAP-3B sees only the DROID manipulator during training, yet controls three previously unencountered robots—Custom Franka, YAM and Kinova—zero-shot, completing a range of real-world manipulation tasks.
>50%3 个新机器人的平均零样本成功率Average zero-shot success rate on 3 new robots
~2×超越最强 VLA 基线的提升幅度Margin of improvement over the strongest VLA baseline
2.5×少于基线所需演示数量的微调效率提升Fine-tuning efficiency gain: fewer demonstrations than the baseline requires
0%所有现有开源 VLA 在未见机器人上的零样本成功率Zero-shot success rate of every existing open-source VLA on unseen robots
现有的五个开源 VLA（π0.5-DROID、π0.5-Base、X-VLA、MolmoAct、OpenVLA）在未见机器人上均完全失效，成功率均为 0%。这表明简单地增大数据规模并不能解决跨机器人泛化问题，关键在于动作表示的选择。The five existing open-source VLAs (π0.5-DROID, π0.5-Base, X-VLA, MolmoAct, OpenVLA) all fail completely on unseen robots, with a success rate of 0% in every case. This indicates that simply scaling up data does not solve cross-embodiment generalization; what matters is the choice of action representation.
02方法Method
LAP 的核心思路是：用结构化的自然语言短语描述末端执行器的运动（"language-action"），将该语言动作作为 VLM 的监督目标，使动作预测落回 VLM 擅长的语言生成任务上。同时配置一个轻量级 diffusion 动作专家将语言动作解码为连续控制信号，并通过"knowledge insulation"阻断梯度回传，保护 VLM 骨干的表征质量。The core idea of LAP is to describe end-effector motion with structured natural-language phrases ("language-action") and to use that language action as the supervision target of the VLM, so that action prediction falls back onto the language-generation task the VLM is good at. A lightweight diffusion action expert is attached to decode language actions into continuous control signals, while "knowledge insulation" blocks gradients from flowing back, protecting the representation quality of the VLM backbone.
图 2：LAP-3B 架构与语言动作表示。(a) VLM 骨干（PaliGemma-3B）以 cross-entropy 目标预测离散语言动作 token；轻量级动作专家以 flow-matching 目标将语言动作解码为连续动作；梯度从动作专家到 VLM 骨干被截断（knowledge insulation）。推理时仅运行动作专家，实现 25 Hz 实时控制。(b) DROID 数据集中语言动作样本示例，如 "move forward 5 cm"、"tilt left 20 degrees"。Figure 2: LAP-3B architecture and the language-action representation.(a) The VLM backbone (PaliGemma-3B) predicts discrete language-action tokens with a cross-entropy objective; the lightweight action expert decodes language actions into continuous actions with a flow-matching objective; gradients from the action expert to the VLM backbone are cut off (knowledge insulation). At inference only the action expert runs, achieving 25 Hz real-time control. (b) Examples of language-action samples in the DROID dataset, such as "move forward 5 cm" and "tilt left 20 degrees".
语言动作采用固定模板 "<verb> <direction> <magnitude> <unit>"，例如 "move forward 5 cm" 或 "tilt left 20 degrees"，确定性地描述末端执行器的运动。该表示无需学习 tokenizer，直接由原始动作数据按坐标约定解析生成。其设计优势在于：语义结构与自然语言空间对齐，使 VLM 的预训练知识（方向感、数量感）可直接复用于跨机器人动作预测。Language actions follow a fixed template "<verb> <direction> <magnitude> <unit>", for example "move forward 5 cm" or "tilt left 20 degrees", describing end-effector motion deterministically. The representation requires no learned tokenizer and is generated by parsing raw action data under a coordinate convention. Its design advantage is that the semantic structure is aligned with natural-language space, so the pre-training knowledge of the VLM (a sense of direction and of magnitude) can be reused directly for cross-embodiment action prediction.
双模型推理与知识隔离Two-Model Inference and Knowledge Insulation
LAP-3B 由两个模块组成：① VLM 骨干（PaliGemma-3B）：以 cross-entropy 损失在语言动作 token 序列上训练，输出离散语言动作；② 轻量级 diffusion 动作专家：以 flow-matching 目标将语言动作解码为连续 7-DoF 控制信号，支持 25 Hz 实时执行。两模块之间通过 knowledge insulation 截断反向传播，防止动作专家的连续动作损失污染 VLM 骨干已习得的通用视觉-语言表征。LAP-3B consists of two modules: ① the VLM backbone (PaliGemma-3B), trained with a cross-entropy loss over language-action token sequences and emitting discrete language actions; ② a lightweight diffusion action expert, which decodes language actions into continuous 7-DoF control signals with a flow-matching objective and supports 25 Hz real-time execution. Between the two modules, knowledge insulation truncates back-propagation, preventing the continuous-action loss of the action expert from contaminating the general vision-language representation already learned by the VLM backbone.
运动预测 VQA 协同训练Motion-Prediction VQA Co-Training
额外引入一个运动预测辅助任务（motion-prediction VQA）：给定两帧图像，模型预测描述其位移的语言动作。该目标作为逆向动力学自监督，进一步增强 VLM 骨干的动作感知能力。协同训练后 LAP-3B+VQA 在 LIBERO 上达到 97.2%，高于无 VQA 版本的 96.8%。An additional motion-prediction auxiliary task (motion-prediction VQA) is introduced: given two image frames, the model predicts the language action describing the displacement between them. As inverse-dynamics self-supervision, this objective further strengthens the action awareness of the VLM backbone. After co-training, LAP-3B+VQA reaches 97.2% on LIBERO, above the 96.8% of the variant without VQA.
训练规模Training Scale
训练数据混合：Open X-Embodiment（85.26%）+ MolmoAct（1.73%）+ 其他。在 64 块 TPU v6e 上训练约 10 小时，遍历完整数据集约 0.65 个 epoch，学习率 1×10⁻⁴（线性 warmup），批大小 2048。Training data mixture: Open X-Embodiment (85.26%) + MolmoAct (1.73%) + others. Training runs for about 10 hours on 64 TPU v6e chips, covering roughly 0.65 epochs of the full dataset, with a learning rate of 1×10⁻⁴ (linear warmup) and a batch size of 2048.
03实验Experiments
实验在 4 种机械臂（1 个训练时见过 + 3 个全新）上进行，共设计 6 类真实操作任务，累计超过 1300 次真机试验。基线分为两类：① 现有开源 VLA（π0.5-DROID/Base、X-VLA、MolmoAct、OpenVLA）；② 使用相同架构与数据重新训练的 replicated 基线（π0.5-replicated、π0-replicated、VLA-0-replicated）。Experiments are run on 4 manipulators (1 seen during training + 3 entirely new), with 6 categories of real-world manipulation tasks and more than 1300 real-robot trials in total. Baselines fall into two groups: ① existing open-source VLAs (π0.5-DROID/Base, X-VLA, MolmoAct, OpenVLA); ② replicated baselines retrained with the same architecture and data (π0.5-replicated, π0-replicated, VLA-0-replicated).
零样本跨机器人迁移（Section 4.1）Zero-Shot Cross-Embodiment Transfer (Section 4.1)
图 3：零样本跨机器人泛化性能对比。LAP-3B 在训练时见过的 DROID 机械臂上达到与 π0.5-DROID 相当的性能，而在三个全新机械臂（Custom Franka、YAM、Kinova）的六类任务上均超过 50% 平均成功率。所有现有开源 VLA 在新机器人上的成功率均为 0%。Figure 3: Comparison of zero-shot cross-embodiment generalization.On the DROID manipulator seen during training, LAP-3B matches the performance of π0.5-DROID, while on the six task categories across three entirely new manipulators (Custom Franka, YAM, Kinova) it exceeds 50% average success rate throughout. Every existing open-source VLA scores 0% on the new robots.
在 LIBERO 仿真基准上，LAP-3B 仅需 1 个 epoch 就达到 78% 成功率，6 个 epoch 达到 96.8%，显著快于基线。在真机任务（YAM 上的 "Hang Tape on Rack"）上，LAP-3B 使用约 20 个演示样本即可达到 50% 任务进度，而基线需要约 50 个演示。整体来看，LAP-3B "achieves comparable task performance using approximately 2.5× fewer demonstrations."On the LIBERO simulation benchmark, LAP-3B reaches 78% success after only 1 epoch and 96.8% after 6 epochs, markedly faster than the baselines. On a real-robot task ("Hang Tape on Rack" on YAM), LAP-3B attains 50% task progress with about 20 demonstrations, whereas the baselines need about 50. Overall, LAP-3B "achieves comparable task performance using approximately 2.5× fewer demonstrations."
图 5(a)：T-SNE 可视化。LAP-3B 的表征中训练机器人与未见机器人的特征高度重叠，而 π0.5-replicated 的表征呈现明显的机器人特定聚类。动作预测误差：LAP-3B 为 0.151，π0.5-replicated 为 0.168，π0-replicated 为 0.189。Figure 5(a): T-SNE visualization.In the representation of LAP-3B, features of training robots and unseen robots overlap heavily, whereas the representation of π0.5-replicated exhibits clear robot-specific clusters. Action prediction error: 0.151 for LAP-3B, 0.168 for π0.5-replicated and 0.189 for π0-replicated.
图 7：模型规模扩展对比（LAP vs. π0.5-replicated）。左图为 token 验证损失（相对 4B 模型的百分比下降），右图为 diffusion 动作专家的连续动作验证损失。LAP 随参数量（4B→12B→27B，基于 Gemma3）单调改善，而 π0.5-replicated 基线在大规模时出现 "early saturation and even degradation"。Figure 7: Model-scale comparison (LAP vs. π0.5-replicated).The left panel shows token validation loss (percentage drop relative to the 4B model), the right panel the continuous-action validation loss of the diffusion action expert. LAP improves monotonically with parameter count (4B→12B→27B, based on Gemma3), while the π0.5-replicated baseline exhibits "early saturation and even degradation" at large scale.
04局限性Limitations
Note:以下局限性均为作者在论文中明确陈述（stated by authors）。All limitations below are explicitly stated by the authors in the paper.
仅针对单臂操作，暂未验证双臂系统Single-arm manipulation only; bimanual systems not yet validated
论文 "focuses only on zero-shot transfer across single-arm manipulators"。作者指出 LAP 原则上可扩展至双臂系统以及缺乏精确控制信号的数据源（如人体姿态、UMI 数据、纯视频），但这些方向尚待系统探索。The paper "focuses only on zero-shot transfer across single-arm manipulators". The authors note that LAP could in principle extend to bimanual systems and to data sources lacking precise control signals (such as human poses, UMI data or pure video), but these directions await systematic exploration.
未评估高控制频率或极高精度任务Tasks with high control frequency or extreme precision not evaluated
LAP-3B 目前尚未在"requiring substantially higher control frequency or extreme precision"的任务上测试，例如快速反应控制或精细柔性物体操作。当前以 25 Hz 运行，是否满足此类场景的需求尚不明确。LAP-3B has so far not been tested on tasks "requiring substantially higher control frequency or extreme precision", such as fast reactive control or fine-grained deformable-object manipulation. It currently runs at 25 Hz, and whether this meets the demands of such scenarios remains unclear.
语言动作标注的精度上限（设计层面推断）Precision ceiling of language-action labels (inferred at the design level)
（inferred from design）将连续动作离散化为自然语言模板会引入量化误差。论文指出语言动作"naturally tolerate lower-quality labels"，这在利用噪声数据时是优势，但在需要亚毫米级精度的任务中可能构成瓶颈。对此影响的系统性研究仍属开放问题。(inferred from design) Discretizing continuous actions into natural-language templates introduces quantization error. The paper notes that language actions "naturally tolerate lower-quality labels", which is an advantage when exploiting noisy data but may become a bottleneck in tasks that require sub-millimeter precision. A systematic study of this effect remains an open question.
