EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video

01 动机Motivation

机器人灵巧操作长期受限于高质量演示数据的匮乏。现有数据集要么规模小、任务单一，要么缺乏精细的手指级追踪信息，难以支持泛化能力强的策略学习。人类每天进行大量灵巧的手部操作，如何高效捕捉这些数据并用于机器人学习，是该领域的核心挑战。Dexterous robotic manipulation has long been constrained by the scarcity of high-quality demonstration data. Existing datasets are either small in scale and narrow in task coverage, or lack fine-grained finger-level tracking, making it hard to support policy learning that generalizes well. Humans perform a great deal of dexterous hand manipulation every day, and how to capture such data efficiently and use it for robot learning is a core challenge in this field.

"EgoDex contains 829 hours of egocentric video with paired 3D hand and finger tracking data, covering 194 tabletop manipulation tasks with household objects."

与现有数据集的对比Comparison with Existing Datasets

EgoDex 在规模和标注质量上均大幅超越现有的机器人操作与人类手部交互数据集：EgoDex substantially surpasses existing robot manipulation and human hand interaction datasets in both scale and annotation quality:

数据集Dataset	演示数量Demonstrations	任务数Tasks	帧数Frames	灵巧手标注Dexterous annotations
DROID	76k	86	19M	否No
Ego4D (HOI)	89k	n/a	21M	否No
EgoDex	338k	194	90M	是Yes

在动词多样性上，"most verbs in EgoDex are above the 10³ mark"，而 DROID 中"many verbs are below the 10¹ mark"。In terms of verb diversity, "most verbs in EgoDex are above the 10³ mark", whereas in DROID "many verbs are below the 10¹ mark".

02 方法Method

EgoDex 的核心贡献分为两部分：一是大规模数据采集流程，二是基于数据集建立的两个 benchmark 及相应的模仿学习基线模型。采集使用 Apple Vision Pro 配合 visionOS 2 ARKit，无需额外硬件，直接采集裸手演示。EgoDex makes two core contributions: a large-scale data collection pipeline, and two benchmarks built on the dataset together with the corresponding imitation learning baselines. Collection uses Apple Vision Pro with visionOS 2 ARKit, requires no extra hardware, and captures bare-hand demonstrations directly.

数据采集流程Data Collection Pipeline

• 采集设备：Apple Vision Pro + visionOS 2，利用 ARKit 生产级姿态追踪，无需标记或额外传感器。Capture hardware: Apple Vision Pro + visionOS 2, using ARKit production-grade pose tracking with no markers or extra sensors.
• 动作表示：48 维向量，涵盖 2 只手 × (3D 腕部位置 + 6D 腕部朝向 + 每手 5 个指尖的 3D 位置)。Action representation: a 48-dimensional vector covering 2 hands × (3D wrist position + 6D wrist orientation + 3D positions of 5 fingertips per hand).
• 语言标注：使用自然语言描述任务，并冻结 CLIP encoder 进行语言条件化。Language annotation: tasks are described in natural language, and a frozen CLIP encoder is used for language conditioning.
• 任务构成：76 对可逆任务（如充电/断电）、28 个无需重置任务（如抛球接球）、14 个需要重置的任务。Task composition: 76 reversible task pairs (e.g. plugging in / unplugging a charger), 28 tasks that need no reset (e.g. tossing and catching a ball), and 14 tasks that require a reset.

Benchmark 设计Benchmark Design

模型架构Model Architecture

实验评估了两类架构与三种预测头的组合：The experiments evaluate combinations of two architectures and three prediction heads:

• Decoder-only (Dec)：自回归 Transformer 解码器，直接预测动作序列。Decoder-only (Dec): an autoregressive Transformer decoder that directly predicts the action sequence.
• Encoder-Decoder (EncDec)：双向 Transformer encoder 提取上下文表征，再经解码器生成轨迹。Encoder-Decoder (EncDec): a bidirectional Transformer encoder extracts contextual representations, and a decoder then generates the trajectory.
• 预测头：BC（Behavior Cloning，均方误差损失）、DDPM（去噪扩散概率模型）、FM（Flow Matching）。Prediction heads: BC (Behavior Cloning, mean squared error loss), DDPM (denoising diffusion probabilistic model), and FM (Flow Matching).

视觉编码器使用预训练 ResNet（冻结），输入分辨率 224×224。训练配置：50,000 步，batch size 2,048（8 块 A100 各 256），学习率 1e-4（Adam），每个模型约训练 72 小时。The visual encoder is a pretrained ResNet (frozen) with input resolution 224×224. Training configuration: 50,000 steps, batch size 2,048 (8 A100 GPUs at 256 each), learning rate 1e-4 (Adam), about 72 hours of training per model.

03 实验Experiments

在两个 benchmark 上系统评估了架构选择、预测时域、视觉目标条件化以及数据规模的影响。指标为手腕和指尖位置的平均距离（Avg Distance）和终点距离（Final Distance），单位为米，越低越好。The effects of architecture choice, prediction horizon, visual goal conditioning and data scale are systematically evaluated on the two benchmarks. The metrics are the average distance (Avg Distance) and the final distance (Final Distance) of wrist and fingertip positions, in meters, where lower is better.

架构对比（2 秒预测时域）Architecture Comparison (2-Second Prediction Horizon)

模型Model	Avg Distance K=1	Avg Distance K=10	Final Distance K=1	Final Distance K=10
Dec + BC	0.045m	0.045m	0.062m	0.062m
Dec + DDPM	0.053m	0.041m	0.071m	0.044m
Dec + FM	0.052m	0.040m	0.071m	0.043m
EncDec + BC	0.044m	0.044m	0.060m	0.060m
EncDec + DDPM	0.052m	0.039m	0.071m	0.043m
EncDec + FM	0.051m	0.038m	0.070m	0.041m

预测时域的影响（Decoder + BC）Effect of Prediction Horizon (Decoder + BC)

预测时域Prediction horizon	Avg Distance	Final Distance
H=30 (1 秒)H=30 (1 s)	0.031m	0.049m
H=60 (2 秒)H=60 (2 s)	0.045m	0.062m
H=90 (3 秒)H=90 (3 s)	0.053m	0.069m

视觉目标条件化（Inverse Dynamics Benchmark）Visual Goal Conditioning (Inverse Dynamics Benchmark)

加入目标图像后，平均距离从 0.045m 降至 0.035m（降低 22%），终点距离从 0.062m 降至 0.029m（降低 53%），目标图像对性能提升效果显著。Adding the goal image lowers the average distance from 0.045m to 0.035m (a 22% reduction) and the final distance from 0.062m to 0.029m (a 53% reduction), so the goal image improves performance markedly.

数据规模的影响Effect of Data Scale

模型大小Model Size

500M 参数模型与 200M 参数基线性能相同（Avg Distance 均为 0.045m，Final Distance 均为 0.062m），说明在当前数据规模下中等大小模型已足够。The 500M-parameter model performs the same as the 200M-parameter baseline (Avg Distance 0.045m and Final Distance 0.062m for both), showing that a medium-sized model already suffices at the current data scale.

分布外（OOD）任务表现Out-of-Distribution (OOD) Task Performance

在 6 个 OOD 任务上，模型表现出一定的泛化能力，但不同任务差异较大：On 6 OOD tasks the model shows some generalization ability, but performance varies considerably across tasks:

OOD 任务OOD task	Avg Distance
Jigsaw Puzzle	0.047m
Knit Scarf	0.064m
Blowdry Hair	0.083m
Stamp Paper	0.099m

消融实验结论Ablation Findings

生成式预测头（DDPM、FM）在多次采样（K=10）时优于确定性 BC，表明在多模态操作轨迹的建模上，概率模型更有优势。视觉目标图像对 inverse dynamics 任务的提升最为显著。The generative prediction heads (DDPM, FM) outperform the deterministic BC under multiple sampling (K=10), showing that probabilistic models have an advantage in modeling multimodal manipulation trajectories. The visual goal image gives the largest improvement on the inverse dynamics task.

04 局限性Limitations

场景多样性不足Insufficient Scene Diversity

论文原文指出："EgoDex has significant diversity across tasks and manipulation behaviors, it is limited in background and scene diversity."数据采集在有限的室内场景中进行，背景视觉多样性受限，可能影响策略在真实多样环境中的泛化能力。作者提出未来将通过程序化背景随机化和在更多样化环境中采集数据来解决此问题。The paper states: "EgoDex has significant diversity across tasks and manipulation behaviors, it is limited in background and scene diversity." The data were collected in a limited range of indoor scenes, so background visual diversity is constrained, which may affect how well policies generalize to diverse real-world environments. The authors propose to address this in future work through procedural background randomization and collection in more varied environments.

手部追踪精度受限于遮挡与高速运动Hand Tracking Accuracy Limited by Occlusion and Fast Motion

论文指出："dexterous annotations can also be imperfect, especially during heavy occlusion or very high speed motions, as they are themselves model predictions."手部姿态标注本身是模型预测结果（非光学标记捕获），在重度遮挡或快速运动场景下可能存在误差，进而影响策略学习质量。The paper notes: "dexterous annotations can also be imperfect, especially during heavy occlusion or very high speed motions, as they are themselves model predictions." The hand pose annotations are themselves model predictions rather than optical marker capture, so they may contain errors under heavy occlusion or fast motion, which in turn affects the quality of policy learning.

尚未完成人-机器人迁移验证Human-to-Robot Transfer Not Yet Validated

论文主要评估手部轨迹预测 benchmark，尚未直接展示在真实机器人上的端到端操作验证。人-机器人的形态差异（embodiment gap）、视角差异等问题仍需后续工作解决。The paper mainly evaluates the hand trajectory prediction benchmark and does not directly demonstrate end-to-end manipulation on a real robot. The human-robot embodiment gap, viewpoint differences and similar issues still need to be resolved by follow-up work.

模型规模与数据规模未达瓶颈Model Scale and Data Scale Have Not Saturated

实验显示 500M 参数模型与 200M 参数基线性能相同，表明当前数据规模下更大模型无法带来收益。但这也意味着随着数据规模进一步增大，更大模型或许才能释放潜力，数据-模型协同扩展的最优策略仍需探索。Experiments show that the 500M-parameter model performs the same as the 200M-parameter baseline, indicating that a larger model brings no gain at the current data scale. But this also means that as data scale grows further, a larger model may be needed to unlock the potential, so the optimal strategy for joint data-model scaling remains to be explored.