METIS 提出了一套面向灵巧手机器人操作的 Vision-Language-Action (VLA) 训练框架。其核心是 EgoAtlas——一个融合 8 个来源、343K 条轨迹、89.72M 图像-动作对的大规模多源自视角数据集；以及 motion-aware dynamics——一种将视觉动态与手部运动动态联合离散化的紧凑监督信号，有效弥合人类与机器人动作之间的分布差距。METIS presents a Vision-Language-Action (VLA) training framework for dexterous-hand robotic manipulation. At its core are EgoAtlas — a large-scale multi-source egocentric dataset that unifies 8 sources, 343K trajectories and 89.72M image-action pairs — and motion-aware dynamics, a compact supervisory signal that jointly discretizes visual dynamics and hand motion dynamics, effectively bridging the distribution gap between human and robot actions.
灵巧手操作是迈向通用机器人的关键一步，但大规模带动作标注的灵巧操作数据极为稀缺。人类手部动作数据虽然规模庞大、场景多样，却存在视觉外观与机器人差距大、数据格式不统一等挑战，导致现有 VLA 模型难以直接利用。Dexterous-hand manipulation is a key step toward general-purpose robots, yet large-scale action-annotated dexterous manipulation data is extremely scarce. Human hand motion data is vast in scale and diverse in scenario, but it exhibits a large visual appearance gap with robots and inconsistent data formats, which makes it hard for existing VLA models to exploit directly.
"The bottleneck of scarce large-scale action-annotated dexterous manipulation data… Human data offers vast scale and diverse manipulation behaviors, [but] prior work faces limited scenarios and large visual gap between human and robots."
图 2：可穿戴手部运动采集系统。使用 Manus Quantum Metagloves（每手 25 个关键点）配合 VIVE 追踪器捕捉 6-DoF 腕部位姿，头戴式自视角相机同步录制第一人称视角图像，构成人机统一的动作-图像对。Figure 2: Wearable hand motion capture system. Manus Quantum Metagloves (25 keypoints per hand) are paired with VIVE trackers to capture 6-DoF wrist poses, while a head-mounted egocentric camera records first-person images in sync, yielding action-image pairs unified across humans and robots.
343KEgoAtlas 轨迹总数Total EgoAtlas trajectories
89.72M图像-动作对总量Total image-action pairs
8数据来源（人类 + 机器人）Data sources (human + robot)
6真实世界灵巧任务Real-world dexterous tasks
与仅依赖单一机器人数据的方法相比，METIS 的核心洞察是：将人类手部操作数据与机器人遥操作数据在统一动作空间下对齐，能显著提升模型的泛化能力。这一思路类比于大语言模型的预训练范式——先在海量异构数据上学习通用表征，再在下游任务上微调。Compared with methods that rely on robot data alone, the core insight of METIS is that aligning human hand manipulation data with robot teleoperation data in a unified action space markedly improves the model's generalization. This mirrors the pretraining paradigm of large language models — first learn general representations from massive heterogeneous data, then fine-tune on downstream tasks.
02方法Method
METIS 由三个核心组件构成：(a) 紧凑型灵巧操作动态表征（motion-aware dynamics），(b) 基于 EgoAtlas 的统一动作空间预训练，(c) 推理-执行一体化（reasoning-acting integration）的下游部署框架。METIS consists of three core components: (a) a compact dynamics representation for dexterous manipulation (motion-aware dynamics), (b) unified-action-space pretraining on EgoAtlas, and (c) a downstream deployment framework with reasoning-acting integration.
图 3：METIS 整体框架概览。(a) 将视觉动态（4 个 token，码本大小 16）与运动动态（40 个 token，码本大小 512）分别通过 VQ-VAE 和 RQ-VAE 离散化，构成紧凑的 motion-aware dynamics 表征；(b) 在 EgoAtlas 数据集上以统一动作空间进行大规模预训练；(c) 下游部署时整合 chain-of-thought 推理进行子任务分解，再调用动作执行分支。Figure 3: Overview of the METIS framework. (a) Visual dynamics (4 tokens, codebook size 16) and motion dynamics (40 tokens, codebook size 512) are discretized by a VQ-VAE and an RQ-VAE respectively, forming the compact motion-aware dynamics representation; (b) large-scale pretraining in a unified action space on the EgoAtlas dataset; (c) at downstream deployment, chain-of-thought reasoning is integrated for sub-task decomposition before the action execution branch is invoked.
EgoAtlas：多源统一自视角数据集EgoAtlas: A Unified Multi-Source Egocentric Dataset
EgoAtlas 整合了来自 8 个来源的数据，包括视觉捕捉系统、VR 数据集、遥操作机器人以及作者自采集的可穿戴传感器数据，共 343K 条轨迹、89.72M 图像-动作对。所有来源均映射到统一动作空间（每手 25 个关节关键点 + 6-DoF 腕部位姿），消除了跨源数据格式不一致的障碍。数据以自视角（egocentric）第一人称视角录制，与机器人实际部署时的观测视角保持一致。EgoAtlas integrates data from 8 sources, including vision-based capture systems, VR datasets, teleoperated robots and the authors' own wearable-sensor recordings, totaling 343K trajectories and 89.72M image-action pairs. All sources are mapped into a unified action space (25 joint keypoints per hand plus a 6-DoF wrist pose), removing the obstacle of inconsistent data formats across sources. The data is recorded from an egocentric first-person viewpoint, consistent with the observation viewpoint of the robot at deployment.
Motion-Aware Dynamics：紧凑的运动感知表征Motion-Aware Dynamics: A Compact Motion-Aware Representation
为了在有限的序列长度内编码丰富的操作信息，作者提出将操作动态分为两个互补组件：To encode rich manipulation information within a limited sequence length, the authors propose splitting manipulation dynamics into two complementary components:
视觉动态（Visual Dynamics）：通过 VQ-VAE 将帧间视觉变化编码为 4 个 token，码本大小为 16，捕捉场景层面的状态变化。Visual Dynamics: frame-to-frame visual changes are encoded by a VQ-VAE into 4 tokens with a codebook size of 16, capturing scene-level state changes.
运动动态（Motion Dynamics）：通过 RQ-VAE 将手部关键点轨迹编码为 40 个 token，码本大小为 512，捕捉精细的手指运动模式。Motion Dynamics: hand keypoint trajectories are encoded by an RQ-VAE into 40 tokens with a codebook size of 512, capturing fine-grained finger motion patterns.
这种离散化表征既保持了信息密度，又与语言模型的 token 序列建模范式天然兼容，无需额外的连续动作解码头。This discrete representation preserves information density while remaining naturally compatible with the token-sequence modeling paradigm of language models, requiring no extra continuous action decoding head.
模型架构：METIS VLAModel Architecture: METIS VLA
METIS 基于 Prismatic-7B VLM 构建，采用混合视觉编码器（SigLIP + DINOv2），在 LLaMA tokenizer 中扩展了特殊 dynamics token。推理阶段通过 chain-of-thought 进行子任务分解，将复杂长时序任务分解为可执行的原子动作序列，再由动作解码分支生成具体的关节控制指令。METIS is built on Prismatic-7B VLM with a hybrid vision encoder (SigLIP + DINOv2), extending the LLaMA tokenizer with special dynamics tokens. At inference, chain-of-thought reasoning performs sub-task decomposition, breaking complex long-horizon tasks into executable atomic action sequences, from which the action decoding branch generates concrete joint control commands.
图 4：六个真实世界灵巧操作任务。短时序任务（3 个）：Pick and Place、Close Laptop、Open Drawer；长时序任务（3 个）：Grasp Two Drinks into Basket、Put Cola into Basket、Open Drawer and Put Bread。所有任务均在 22-DoF 灵巧手上执行。Figure 4: Six real-world dexterous manipulation tasks. Short-horizon tasks (3): Pick and Place, Close Laptop, Open Drawer; long-horizon tasks (3): Grasp Two Drinks into Basket, Put Cola into Basket, Open Drawer and Put Bread. All tasks are executed on a 22-DoF dexterous hand.
03实验Experiments
实验在 6 个真实世界灵巧操作任务上评估 METIS，涵盖短时序与长时序任务，并测试分布外泛化（unseen backgrounds/lighting/objects）和跨机体迁移能力（22-DoF SharpaWave hands）。每个任务执行 20 次，报告成功率（SR）和阶段成功率（PSR）。The experiments evaluate METIS on 6 real-world dexterous manipulation tasks covering short- and long-horizon settings, and test out-of-distribution generalization (unseen backgrounds/lighting/objects) and cross-embodiment transfer (22-DoF SharpaWave hands). Each task is run 20 times, reporting success rate (SR) and stage success rate (PSR).
图 6：样本效率曲线。METIS 在仅使用 10% 训练数据时即可在 Pick and Place 任务上达到 50% 成功率，展示了多源预训练带来的高效迁移能力。Figure 6: Sample efficiency curve. Using only 10% of the training data, METIS already reaches a 50% success rate on the Pick and Place task, demonstrating the efficient transfer brought by multi-source pretraining.
跨机体迁移（Cross-Embodiment）Cross-Embodiment Transfer
METIS 在未经专门训练的情况下迁移至 22-DoF SharpaWave 灵巧手，在 Grasp Apple 任务上达到 85% 成功率，在 Tool Use 任务上达到 70% 成功率，验证了统一动作空间的跨机体泛化能力。Without dedicated training, METIS transfers to a 22-DoF SharpaWave dexterous hand, reaching 85% success on the Grasp Apple task and 70% on the Tool Use task, validating the cross-embodiment generalization of the unified action space.
图 7：跨机体泛化实验。METIS 迁移到 22-DoF SharpaWave 灵巧手，无需重新训练即可执行 Grasp Apple（85% SR）和 Tool Use（70% SR）任务。Figure 7: Cross-embodiment generalization. METIS transfers to a 22-DoF SharpaWave dexterous hand and executes Grasp Apple (85% SR) and Tool Use (70% SR) without retraining.
消融实验（Tables 4–5）Ablation Studies (Tables 4–5)
消融实验系统验证了各组件的贡献：The ablations systematically verify the contribution of each component:
其中最关键的发现是：移除 motion-aware dynamics 后长时序任务成功率骤降至 0%，说明运动动态表征是支撑复杂灵巧操作的核心组件，而非可选附件。The most critical finding: after removing motion-aware dynamics, the success rate on long-horizon tasks collapses to 0%, showing that the motion dynamics representation is a core component supporting complex dexterous manipulation rather than an optional add-on.
图 5：语言指令跟随能力。METIS 能够根据颜色描述（如"拿红色水果"）识别目标物体并执行对应抓取动作，展示了 VLA 的多模态理解与执行一体化能力。Figure 5: Language instruction following. METIS identifies the target object from a color description (e.g. "pick the red fruit") and executes the corresponding grasp, demonstrating the integrated multimodal understanding and execution of a VLA.
04局限性Limitations
说明：以下局限性均为论文作者在原文中明确陈述（stated by authors）。Note: All limitations below are explicitly stated by the authors in the paper.
仅依赖自视角观测，难以感知完整物体几何Relies solely on egocentric observations, making complete object geometry hard to perceive
"Model relies solely on egocentric observations, which may restrict ability to perceive complete object geometry and fine interaction details."——第一人称视角存在自遮挡问题，对需要精确感知物体形状（如透明容器、不规则形状物体）的任务构成挑战。"Model relies solely on egocentric observations, which may restrict ability to perceive complete object geometry and fine interaction details." — first-person views suffer from self-occlusion, which challenges tasks that require precise perception of object shape (such as transparent containers or irregularly shaped objects).
预训练未纳入大规模第三人称数据Pretraining does not include large-scale third-person data
"Pretraining process currently excludes large-scale third-person data available online."——互联网上存在大量人类操作的第三人称视角视频（如 YouTube cooking/crafting），这部分数据未被 EgoAtlas 利用，是未来工作的重要扩展方向。"Pretraining process currently excludes large-scale third-person data available online." — the internet holds abundant third-person videos of human manipulation (such as YouTube cooking/crafting); this data is not yet exploited by EgoAtlas and is an important direction for future work.
（推断）评估任务多为桌面场景，移动操作未涉及(Inferred) Evaluation tasks are mostly tabletop; mobile manipulation is not covered
（inferred from design）六个评估任务均为桌面固定基座灵巧操作，尚未验证 METIS 在移动机械臂或双臂协同等更复杂场景下的表现。(inferred from design) All six evaluation tasks are tabletop, fixed-base dexterous manipulation; METIS has not been validated in more complex settings such as mobile manipulators or bimanual coordination.
