Emergence of Human to Robot Transfer in Vision-Language-Action Models
人类视频数据迁移机器人技能的能力是 VLA 多样化预训练的涌现属性Transferring robot skills from human video data is an emergent property of diverse VLA pretraining
Simar Kareer, Karl Pertsch, James Darpinian, Judy Hoffman, Danfei Xu, Sergey Levine, Chelsea Finn, Suraj Nair · Georgia Tech / Stanford / Physical Intelligence
本文探究 Vision-Language-Action (VLA) 模型能否利用人类示范视频数据来提升机器人技能迁移。研究发现，human-to-robot transfer 是一种涌现能力：只有当模型在足够多样化的场景、任务和机器人形态上完成预训练后，co-training 人类视频数据才能带来显著收益。最终在多个泛化任务上成功率最高翻倍，验证了跨形态表征统一的关键作用。This paper investigates whether Vision-Language-Action (VLA) models can exploit human demonstration videos to improve the transfer of robot skills. The authors find that human-to-robot transfer is an emergent capability: only once the model has been pretrained on sufficiently diverse scenes, tasks and robot embodiments does co-training on human video data yield substantial gains. Success rates on several generalization tasks are at best doubled, confirming the key role of a unified cross-embodiment representation.
arXiv 2025-12-274 个操作任务4 manipulation tasks14 小时人类示范数据14 hours of human demonstration data📄 arXiv:2512.22414PDF
训练机器人需要大量专门的遥操作数据，成本高且难以规模化。互联网上拥有海量人类操作视频，理论上可以极大丰富训练数据多样性——但人与机器人在形态、视角、运动方式上存在巨大差异，直接利用并非易事。Training robots requires large amounts of dedicated teleoperation data, which is expensive and hard to scale. The internet holds a vast supply of human manipulation videos that could in principle greatly enrich the diversity of training data, yet humans and robots differ enormously in embodiment, viewpoint and mode of motion, so exploiting such videos directly is far from easy.
过去的研究尝试用显式的跨形态对齐（如视频预测、形态嵌入对齐）来桥接这一鸿沟，但效果不稳定。本文提出一个不同视角：这种迁移能力是否是大规模 VLA 预训练多样性带来的涌现属性，而非需要专门设计的桥接机制？Prior work has tried to bridge this gap with explicit cross-embodiment alignment (such as video prediction or embodiment-embedding alignment), but the results have been unstable. This paper takes a different view: could this transfer ability be an emergent property brought about by large-scale VLA pretraining diversity, rather than something that requires a purpose-built bridging mechanism?
"Human-to-robot transfer is an emergent property of diverse VLA pretraining."
Figure 2（原文）：Figure 2 (from the paper):"Per-task improvement from human data: We plot the difference in performance between policies fine-tuned with robot + human data versus robot-only data, isolating the lift from human supervision. Gains are largest when pre-training spans diverse tasks, scenes, and embodiments, suggesting that broad pre-training improves transfer from human videos."
— 当预训练多样性从 0% 提升至 100%，人类数据带来的增益从接近 0 跃升至显著正值，佐证了涌现假说。— As pretraining diversity rises from 0% to 100%, the gain contributed by human data jumps from close to 0 to a clearly positive value, supporting the emergence hypothesis.
+39%Spice 任务成功率提升（32%→71%）Spice task success-rate gain (32%→71%)
+25%Dresser 任务成功率提升（25%→50%）Dresser task success-rate gain (25%→50%)
+21%Sort Eggs 准确率提升（57%→78%）Sort Eggs accuracy gain (57%→78%)
14h人类示范视频总时长（4 个任务）total length of human demonstration video (4 tasks)
02方法Method
本文在 π0.5（pi-0.5）VLA 基础上，以 50-50 的比例混合人类示范与机器人数据进行 fine-tuning，同时预测 high-level 子任务和 low-level 连续动作，无需任何显式的跨形态对齐机制。Building on the π0.5 (pi-0.5) VLA, the paper fine-tunes on a 50-50 mixture of human demonstrations and robot data, jointly predicting high-level subtasks and low-level continuous actions, without any explicit cross-embodiment alignment mechanism.
系统架构：人类操作者佩戴头戴摄像头（及可选腕部摄像头），所采集的 ego-centric 视频经过手部 3D 关键点提取，转换为相对 6-DoF end-effector 轨迹，与机器人数据共同 fine-tune π0.5 VLA 模型（π0.5+ego）。System architecture: the human operator wears a head-mounted camera (plus optional wrist cameras); the collected ego-centric video goes through 3D hand keypoint extraction and is converted into relative 6-DoF end-effector trajectories, which together with robot data fine-tune the π0.5 VLA model (π0.5+ego).
数据采集：人类示范Data collection: human demonstrations
实验人员佩戴头戴摄像头（模拟机器人主相机视角）与左右腕部摄像头，在四个任务上共采集约 14 小时人类示范数据：Wearing a head-mounted camera (mimicking the robot's primary camera view) together with left and right wrist cameras, the operators collected about 14 hours of human demonstration data across four tasks:
Bussing（餐具收拾）：3 小时，移动操作任务Bussing (clearing tableware): 3 hours, a mobile manipulation task
Figure 3（原文）：Figure 3 (from the paper):"Training mixture and benchmark. Our fine-tuning mix is evenly split between human data for generalization tasks and robot data for the nearest neighbor task."
— 左侧展示人类佩戴设备采集场景，右侧为四个任务的机器人执行场景，fine-tuning 数据各占 50%。— The left side shows humans collecting data with the wearable rig, the right side shows the robot executing the four tasks; the fine-tuning data is split 50% each.
动作空间对齐Action space alignment
人类手部 3D 关键点被转换为相对 6-DoF end-effector 轨迹，维度与机器人动作空间（18 维：双臂各 6 维 + 底盘移动 6 维）相同，但不估计人类夹爪状态（夹爪动作仅从机器人数据学习）。3D human hand keypoints are converted into relative 6-DoF end-effector trajectories, whose dimensionality matches the robot action space (18 dimensions: 6 per arm plus 6 for base motion), but the human gripper state is not estimated (gripper actions are learned from robot data only).
Co-training 策略Co-training strategy
在 π0.5 基础上同时预测 high-level subtask（子任务语言描述）与 low-level continuous action（连续动作），混合比例固定为 robot:human = 50:50。研究特别考察了不同 pretraining diversity（0%～100%）对迁移效果的影响，以验证涌现假说。On top of π0.5 the model simultaneously predicts a high-level subtask (a language description of the subtask) and a low-level continuous action (the continuous action), with the mixing ratio fixed at robot:human = 50:50. The study pays particular attention to how different levels of pretraining diversity (0%-100%) affect transfer, in order to test the emergence hypothesis.
03实验Experiments
在 4 个操作任务上评估 co-training（Robot + Human）与纯机器人数据 fine-tuning（Robot only）的差异，并系统分析 pretraining diversity、wrist camera、high-level/low-level 分工等关键因素。Across 4 manipulation tasks the paper evaluates the difference between co-training (Robot + Human) and fine-tuning on robot data alone (Robot only), and systematically analyses key factors such as pretraining diversity, wrist cameras, and the division of labour between high-level and low-level prediction.
主要性能对比Main performance comparison
任务Task
Robot Only 基线Robot Only baseline
Robot + Human（本文）Robot + Human (this paper)
提升Gain
Spice Rack
32%
71%
+39%
Dresser
25%
50%
+25%
Bussing
53%
63%
+10%
Sort Eggs
57%
78%
+21%
所有数字均来自原文，verbatim。All numbers are taken verbatim from the paper.
Figure 7（原文）：Figure 7 (from the paper):"Human to robot transfer finetuning π0.5. We evaluate performance across a suite of static and mobile tasks, each testing either scene, object, or task level generalization present only in the human data. We see clear human to robot transfer resulting in nearly double the score on the target tasks."
Pretraining Diversity 的涌现效应The emergence effect of pretraining diversity
实验系统地改变预训练数据集的任务/场景/形态多样性（0%～100%），发现：The experiments systematically vary the task / scene / embodiment diversity of the pretraining dataset (0%-100%) and find:
0%～25% 多样性时，co-training 人类数据几乎无收益甚至略有下降At 0%-25% diversity, co-training on human data brings almost no gain and can even degrade performance slightly
75%～100% 多样性时，人类数据带来的提升显著跃升At 75%-100% diversity, the improvement contributed by human data jumps markedly
加入跨形态机器人数据（cross-embodiment pretraining）后效果最强Adding cross-embodiment robot data (cross-embodiment pretraining) gives the strongest effect
"Performance of Robot Finetuning on Sort Eggs plateaus, even as the pretraining diversity improves. In contrast, Human+Robot Finetuning performance scales sharply with pretraining, suggesting that broader pretraining enables more effective transfer from human data."
Figure 5（原文）：Figure 5 (from the paper):"VLA representation of human and robot data. We plot the latent embeddings of our VLA by performing a TSNE analysis on mean-pooled tokens from the final layer of the VLM backbone. With no pre-training, it is clear that the model has disjoint representations between human and robot data. But as pretraining becomes more diverse, latent overlap increases, which correlates with performance on our generalization tasks."
— 预训练多样性越高，人类与机器人的 latent embedding 越趋于融合，直接解释了涌现迁移能力的来源。— The more diverse the pretraining, the more the human and robot latent embeddings merge, which directly explains where the emergent transfer ability comes from.
消融实验Ablation studies
消融：人类数据 vs. 目标任务机器人数据对比。Sort Eggs 和 Dresser 任务中，人类数据效果接近甚至媲美目标机器人数据；Bussing 任务中目标机器人数据（65%）显著优于人类数据（25%）。Ablation: human data vs. target-task robot data. On Sort Eggs and Dresser, human data comes close to and even matches target robot data; on Bussing, target robot data (65%) is clearly better than human data (25%).
High-level vs. Low-level 分工High-level vs. low-level division of labour
对于移动操作任务（Bussing、Dresser），high-level 子任务预测和 low-level 动作预测均有必要；两者共同 co-training 才能充分利用人类示范数据。For mobile manipulation tasks (Bussing, Dresser), both high-level subtask prediction and low-level action prediction are necessary; only co-training the two together fully exploits the human demonstration data.
腕部摄像头的作用The role of wrist cameras
腕部摄像头对精细操作任务（Bussing、Dresser）有明显帮助，对其他任务（Spice、Eggs）影响有限，符合直觉："some (but not all) tasks will benefit from the added observability of wrist cameras."Wrist cameras clearly help on fine manipulation tasks (Bussing, Dresser) but have limited effect on the other tasks (Spice, Eggs), which matches the intuition: "some (but not all) tasks will benefit from the added observability of wrist cameras."
04局限性Limitations
Note：以下局限性部分来自作者在原文中的明确陈述（stated），部分为从方法设计推断（inferred）。Note: Some of the limitations below are explicitly stated by the authors in the paper, while others are inferred from the design of the method.
依赖大规模、多样化的机器人预训练数据（stated）Depends on large-scale, diverse robot pretraining data (stated)
Human-to-robot transfer 的涌现需要"vast datasets of robot teleoperation data in pretraining"作为前提。对于没有大规模预训练基础的场景，本方法的收益将大幅缩水，限制了其通用性。The emergence of human-to-robot transfer presupposes "vast datasets of robot teleoperation data in pretraining". In settings without a large-scale pretraining foundation, the benefit of this method shrinks sharply, which limits its generality.
人类夹爪状态未估计（stated）Human gripper state is not estimated (stated)
当前实现无法从人类手部关键点直接估计夹爪开合状态，夹爪动作完全依赖机器人数据学习。这是一个明确的信息缺失，可能影响需要精确抓握的任务迁移效果。The current implementation cannot estimate the gripper opening state directly from human hand keypoints, so gripper actions are learned entirely from robot data. This is a clear loss of information and may affect transfer on tasks that require precise grasping.
人类数据规模仍有限（stated）The scale of the human data is still limited (stated)
当前实验使用约 14 小时的情景式人类示范数据。作者展望未来可以扩展至更大规模的人类视频，包括日常活动的被动录制，认为模型规模持续扩大将解锁更多涌现能力。The current experiments use about 14 hours of episodic human demonstration data. The authors envisage scaling to much larger collections of human video in the future, including passive recordings of everyday activity, and argue that continued scaling of model size will unlock further emergent capabilities.
某些任务上人类数据效果有限（inferred from results）On some tasks the benefit of human data is limited (inferred from results)
Bussing 任务中，人类数据（25% success）远不及目标机器人数据（65%），说明对于运动复杂度高、形态差异大的任务，跨形态迁移存在固有瓶颈。On Bussing, human data (25% success) falls far short of target robot data (65%), showing that for tasks with high motion complexity and large embodiment gaps there is an inherent bottleneck to cross-embodiment transfer.
